I usually generate a stratified random sample and save half of the points for validation. What is your total sample size? 

I have 335 points labeled as rice field, for example. I have converted these point to shapefile to load into ENVI. But I cannot use the function "Generate Random Sample" to generate random sample as you said. Could you guide me step by step? Do we use these random samples to select ROIs for supervised classification? etc.  

Jack Knife is method similar to what you are describing to access the error. You could simply select 10 percent of your points which are very different, leaving these data points out during construction of your classification matrix.  Then compute the error.   Another similar method is conducted leaving out 1 point, constructing the classification system, then predicting for that point.   This is done over and over again for all the data points in your study.  In your case you would have 335 residuals.   This would let you know how well your classification system in general works.

It is unfortunately too late for random points. The idea behind the random point generation is that you should generate a set of points randomly, THEN you validate them (on the field or by photo-interpretation).

If you've collected your points along a road or for the fields where you did some experiment, your sample will not be random. You can still use them to build a confusion matrix, but you need to be aware that this introduces some bias. Furthermore, you need at least two classes in order to identify the two types of errors in your map (false positive and false negative). If you only have points from the rice field, then you  can only measure the false negative.  

Are you doing a supervised classification? A general approach under supervised classification known as "Maximum Likelihood classification". You should divided your sample into 2 part: (1) ROI and (2) validation. Please make sure your samples are well distributed over your study area. I think you can get a some video tutorials on youtube. All the best..

Thank you for all reply.

Mou: I have divided the samples into 2 parts with the portion of 70% for ROI and 30% for validation. My question is we will create ROI by drawing around the sample point or neighbor. Because some samples do not belong pixels we want to label due to position error of GPS receiver. The samples for validation, do we keep in point format?

Regards,

if you think there are some errors in GPS receiver, you must make a proper buffer around each point in GIS, then , you can convert them to ROI. if the points are not in border of changes this will have low error.

It is interesting to read various expert comments on the thread. Indeed accuracy of a hand held GPS can't be compared with DGPS. I agree with @Subhajit Bandopadhyay .

But what to do now, the data has already been collected? Moreover, what is the size of the area? 335 points may or may not be good enough. Therefore, can you please mention the area of interest.

You have to consider the following points- Level of map accuracy looking, Scale of map, resolution of satellite data being used, total geographical area,  error in handheld GPS,  then use some(30-40%) of unused GCPs for validation of map. Normally achievable map accuracy is 0.5-1 mm of map scale. Add

I am classifying one scene of Landsat 8 with spatial resolution of 30 x 30 m. The scale of land use map is 1:250,000. When I plotted all collected points on the RGB image, a number of them was on the road since I could not go inside the rice fields.

http://ismir2006.ismir.net/PAPERS/ISMIR06105_Paper.pdf

http://ba1.geog.umd.edu/PapersBaraldi/TGARS_2005_1_Quality-Map-Assessment_%5B19%5D.pdf

Position accuracy in navigation can be defined as a degree of conformance between the estimated or measured position and its true position. Position accuracy can be determined as a different types of statistic. They can be calculated related to the true values of coordinates (if any) or if the actual position is not known, the average position is often used as an approximation to the actual position. Both the position solution must be based upon the same geodetic datum (WGS-84). There are 3 types of solutions:

  Predictable accuracy: The accuracy of a PNT system’s position solution with respect to the charted solution (true coordinates values).

Repeatable accuracy: The accuracy with which a user can return to a position whose coordinates has been measured at a previous time with the same PNT system. (mean coordinates values).

Relative accuracy: The accuracy with which a user can measure position relative to that of another user of the same PNT system at the same time. Applicable only for two receivers.NOT FOR YOUR SOLUTION

The values describe accuracy statistics, common used in  navigation or geodesy are: CEP (Circular Error Probable; 2D, p=0.5), SEP (Spherical Error Probable; 3D, p=0.5) RMS (Root Mean Square; 1D, p=0.68), dRMS (Distance Root Mean Square; 2D or 3D, p=0.68) or 2dRMS (Two Distance Root Mean Square 2D or 3D, p=0.95) or 3dRMS (Three Distance Root Mean Square; 2D or 3D, p=0.997).

I always realise such statictics based o own worksheet realised in Mathcad.

If you collected fixes in geodetic coordinates (degrees, minutes, seconds) You schould (before statistica analyses) transform this ccordinates according to Gauss Kruger projection equations then the coordinates  values will be in meters. Then calculate the statistics and errors.

You can go with GCPs collected after that also; GCPs means gound real condition and accuracy assessment deals, how much your classification is corrected

cross checking is must for accuracy of gps data & flied data

You are right gps: data and field data ! Dear Dnyaneshwar.

A 3 meter wide road will sometimes show up in 30 meter Landsat.   I actually did a short study to see if I could pick out roads using LandSAT with some limited success.  You might see if the system works better when you leave those ground truth points out or if under your supervised classification system you class the road sections differently.

ROADWAY IDENTIFICATION ON LANDSAT IMAGERY USING NEURAL NETWORKS

GEORGE L MASON, MARK J EMBRECHTS, DIMITRI A GRIVAS.

Publication date

1993.

Journal name

Intelligent Engineering Systems Through Artificial Neural Networks: Proceedings of the Artificial Neural Networks in Engineering (ANNIE'93) Conference, Held November 14-17, 1993, in St. Louis, Missouri, USA.

Volume 3.

Pages 423.

Publisher

Amer Society of Mechanical

In your case, I think you can use "Confusion Matrix using Ground truth ROIs" available in ENVI to compute confusion matrix for accuracy assessment. To do that your GPS points must have coordinate in the coordinate system of your image, you import these points in to Arc GIS to get a vector file. Then you can convert this file to ROI and get the confusion matrix using ENVI

You can do what Hien La mentioned. It is  because the confusion or error matrix represents the differences between the reference data and the classifier taking into account the classes that are the target of the classification. In binary accuracy assessment, the error matrix associated with a pair (i, j) represents the estimation of the number of pixels that have been classified into the class i, but in the reference data are assigned into class j. Obviously, if i = j, the expert and classifier coincide in their opinion and the rest of the matrix represents errors in the classification. Other measures, such as the Kappa coefficient of agreement in the assessment of classification accuracy have been in the use now a days, however, its real utility is being debated in the remote sensing community.

Fielding, Alan H., and John F. Bell. "A review of methods for the assessment of prediction errors in conservation presence/absence models." Environmental conservation 24.01 (1997): 38-49.

Hay, A. M. 1988. The derivation of global estimates from a confusion matrix. International Journal of Remote Sensing. Vol. 9, pp. 1395–1398.

It is very important to understand how you choose these points in terms of land cover homogenity related to the spectral data that you used for the classification; the accuracy of your observation is related to the spatial resolution of your data.

Dear you need to use your ground truth reference to do your accuracy assessment

regards