Sample Size Determination for Image Classification Accuracy Assessment and Comparison

Giles M. Foody +
School of Geography, University of Nottingham, NG7 2RD, UK

Abstract. The classification accuracy statement is the basis of the evaluation of a classification’s fitness for purpose. Accuracy statements are also used for applications such as the evaluation of classifiers, with attention focused especially on differences in the accuracy with which data are classified. Many factors influence the value of a classification accuracy assessment and evaluation programme. This paper focuses on the size of the testing set(s), and its impacts on accuracy assessment and comparison. Testing set size is important as an inappropriately large or small sample could lead to limited and sometimes erroneous assessments of accuracy and of differences in accuracy. In this paper the basic statistical principles of sample size determination are outlined. Some of the basic  issues of sample size determination for accuracy assessment and accuracy comparison are discussed. With the latter, the researcher should specify the effect size (minimum meaningful difference), significance level and power used in an analysis and ideally also fit
confidence limits to estimates. This will help design a study as well as aid interpretation. In particular, it will help avoid problems such as under-powered analyses and provide a richer information base for classification evaluation. Central to the argument is a discussion of Type II errors and their control. The paper includes equations that could be used to determine sample sizes for common applications in remote sensing, using both independent and related samples.

Keywords: remote sensing classification, sample size, Type I and II error, power, confidence interval.

In: Wan, Y. et al. (eds) Proceeding of the 8th international symposium on spatial accuracy assessment in natural resources and environmental sciences, World Academic Union (Press).

Foody2008accuracy.pdf264.06 KB