Dear Mousa, 

I think that understanding a CMC curve is much easier by using an example.

Shortly speaking, imagine that you have 5 classes. For simplicity, imagine you have one test per class. Each test produces a score when compared to each class. Let's start from test1 which belongs to class1:

As your similarity measure, here, is Euclidian distance, the more distance a test has compared to a class, the less similarity is obtained. Without loss of generality, let's suppose that the distance measures are calibrated and normalized in terms of similarity scores.  

If the score between test1 and class1 is larger than the other 4 classes (or the Euclidian distance between test1 and class1 is less than the other 4 classes), class 1 is recognized in the first rank. As an example, let's suppose the following similarity scores (not Euclidian distances):

Test1 VS class 1= .95

Test1 VS class 2= .7

Test1 VS class 3= .9

Test1 VS class 4= .72

Test1 VS class 5= .3

What do these scores say? These similarity scores say that test 1 is more similar to class 1 than the other classes. So, test 1 is correctly recognized in the first rank.

Let's suppose that test2, test 3, test 4 and test 5 are also recognized in the first rank.

So, we can conclude that we have a perfect CMC curve (y-x). because all of the 5 test are correctly recognized. A perfect CMC curve is as following:

Rank 1: 100%

Rank2: 100%

...

Rank 5:100%

Now, let's suppose the following situation:

Test1 VS class 1= .9

Test1 VS class 2= .95

Test1 VS class 3= .4

Test1 VS class 4= .72

Test1 VS class 5= .3

What do these scores say? They say that test 1 is more similar to class 2 than class 1 (This is a mismatch and it is not correct!). So, the correct match (.9) is not the top match here (the top match is .95 which is a mismatch). In other words, test 1 is not recognized as the top match, but it is recognized among top "two matches" (the top two matches are .95 and .9). This is what rank 2 recognition means!

Now, if test 1 is recognized among top two matches but the other tests (i.e., test 2, test 3,... test 5) are recognized in the first rank (Rank 1), the first rank recognition rate is 80% (test 1 is not among the top matches, i,e, 1 out of 5 is incorrect in first rank recognition which yields 80%) but the second rank recognition rate is 100%, because all of the tests are correctly recognized among the top "two matches". Some of them are recognized in the first rank (test 2, test 3, test 4, and test 5) while some of them are recognized in the second rank (test 1). So, the second rank recognition rate is 100%.

So, the CMC curve will be as following:

Rank1  80% (tests 2,3,4,5 are among the N=1 top matches )

Rank2 100% (tests 1,2,3,4,5 are among the N=2 top matches )

Rank3 100%  (tests 1,2,3,4,5 are among the N=3 top matches )

...

Rank 5 100%  (tests 1,2,3,4,5 are among the N=5 top matches )

Now, consider that test1, test 2, test 3 are recognized in the first rank, test 4 is recognized in the second rank and test 5 is recognized in the fourth rank. what is the CMC curve>? Here is the answer:

Rank 1:  60% (tests 1,2,3 are among the N=1 top matches )

Rank 2: 80% (test 1,2 ,3 ,4 are among N=2 top matches)

Rank 3: 80% (test 1,2 ,3 ,4 are among N=3 top matches)

Rank 4: 100% (test 1,2 ,3 ,4 and 5 are among N=4 top matches) 

Rank 5: 100% (test 1,2 ,3 ,4 and 5 are among N=5 top matches)

Reference:

Grother, Patrick, Ross J. Micheals, and P. Jonathon Phillips. "Face recognition vendor test 2002 performance metrics." Audio-and Video-Based Biometric Person Authentication. Springer Berlin Heidelberg, 2003.

Good Luck,

Reza.

Thanks Dear Reza for this good example, I have some trivial question, I am not expert in this domain, 

1- When you see five classes, you mean you have five different groups of images (like group 1 is lena image, group 2 is barby image and so on?)

2- The test image (Test1)  is the image under the test, When we want to produce the CMC curve in that time It is given that mage (Test1) belong to group1 or it is unkown??

3- I understand that you compair Test1 to the images in group1 and group2 until group5 inspite knowing that Test1 should be in group1??

Thanks

Dear Mohammed,

The only thing that a "scoring" classifier does is assigning a score between a test and a class (some classifiers produce just a label in their output instead of a score). Next, a human expert should evaluate the classifier performance. The human expert knows which test belongs to which class. Normally, ROC curves (sometimes EER) and CMC curves are used for this purpose by human expert.

Shortly speaking, if we have 5 classes, and 5 tests, we can arrange the scores in a table of 5 by 5, where the rows stand for the classes and the columns stand for the tests. Each entry is a similarity score between a test and a "gallery image" in the class (e.g., Lena has a gallery image which represent her class). Please see that we can generally have multiple tests and multiple gallery images (rows) to represent a class. For simplicity, I have supposed that there is one sample to represent each class. These samples are called gallery images and they are compared with the test images to generate the scores.

Best,

Reza.

P.S.

Sometimes, gallery images are also called training images as the classifier has seen them, but please keep in my mind that a training image is not necessarily a gallery image. As an example, suppose that you train a PCA with 100 males and 100 females in Germany to generate a face model which can reconstruct unknown people faces (e.g., see the video of 3D Morphable models in youtube where the model reconstructs Tom Hanks face!).  Next, some body in France uses this PCA to design a face recognition system of 30 french people. These 30 people are 30 classes. They may have multiple images in the gallery to be compared with their tests in France in future. But, the feature space (PCA) is constructed in Germany with 200 other training images. So, the test and gallery images of french people are projected to this space constructed by German people to be recognized through a criteria like minimum distance in space!

Good luck!   

Thanks dear Reza Shoja Ghiass

I have more specific question

if you can help me

It is my pleasure to answer.

I am doing encryption for video of 240 frames, the encryption is focused on the face, I would like to know the recognition rate for both original frames (240 unencrypted frames) and encrypted frames per Rank, What is the MAtlab code that I can use it to produce the recognition rate per Rank for each frame (the original frame and the encrypted one), what are the training image of this example, the face in all frames is Kimono if you know this face?

Sorry, but I am not expert in face recognition rate

Thanks in advance Reza

hi dear Dr. Reza

Thanks a lot for your good example and explanation for CMC. 

Dear Mousa,

Sorry for delay. I had not noticed your question. I do not understand your question very well. Any sample which is shown to the system before testing is called a training sample. In your case, you should clearly define your 'classes' first. Next, you should see how many samples per class are shown to the system before testing. You should not use your training samples for testing. The unseen samples are the testing ones. Calculating CMC curve is quite simple. You want to know what is the percentage of the testing samples for which the intra-class (within class) matching score stands over the other matching scores (between class matching scores). This is called Rank1 recognition rate. The example clarify this very well.

Good explanation Dr. Reza..

Dear Reza Shoja Ghiass!

Thanks a lot for your good example.

I have a question:

If the probe belongs to two or more classes, Then how can we compute the CMC?

for example, The scores of the probe for test1 for different classes are as follows:

class1: 0.95

class2: 0.6

class3: 0.2

class4: 0.15

class5: 0.55

class6: 0.99

and we know that the probe belongs to class1 & class6;

Other tests are like test1.

now how can we compute cmc curve?

@Bahare Moradi: Thanks for your question. I do not think that a test sample can belong to two different classes in the same time. Can my test facial image belong to myself and my brother in the same time? But my test facial image may show similarity to my different facial images which participated in a training scenario such as an Eigenface (PCA). 

 So, each class may contain several training images. However, FINALLY your scoring classifier will classify the test image using a single score which is obtained from your classification RULE. So, the important thing is to find this single score (and more importantly the rank associated with this score rather than the score itself), which depends on your classification rule. As a simple example, consider the RULE of K-NN,  and suppose k=3 in your example. 

To put short, a "scoring" classifier can assign a single score between a probe image and each of the classes, while you may have several training images per class. 

@Reza Shoja Ghiass : Thank you so much. 

 hello mr REZA

can you help me in my project?

my project is fingerprint recognition in compressed domain, and i work on FVC2000 to 2006 db_A, each db contaion 800 pic from 100 person with 8 image per pesron in format 1_1,1_2,..1_8,  100_1,100_2,..100_8,,

for simplisity, Gallery contain first image of each person (1_1,2_1,.....,100_1) and Probe set contain 2nd image ( 1_2,2_2,...,100_2)

i use PCA for decreasing feature space and euclidean,sperman,cosine,correlation and cityblock for classification

my quastions is:

if gallery have more than one image for example 5 image, how rank is calculate?

if probe have more than one image for ex. 3 image, how rank is calculate?

is my explain correct?  for  over 1 image in gallery or probe set, can i use average mechanism for calc recognition and rank?

please help me my Dear REZA

best regards

Dear Majid,

I do not work in the fingerprint domain. Calculating the CMC curve is just a protocol. The best strategy is to see which protocol the researchers in your domain use, i.e., if there is a paper which is well cited for this purpose. Having multiple test images is not a problem and you just need to calculate the recognition rate using any averaging protocol you like. If you have multiple images in the gallery for each class, based on your "RULE" the classifier will assign one score (just one score) never mind how many images exist in your gallery for each class. As an example, your rule can be the maximum score among all scores, or it can be the minimum distance, or the average of distances ....

So, you need to see what is the rule by which your scoring classifier yields the score for each "CLASS" . In other words, you can have multiple gallery images (each assigned with a score using let's say PCA and Euclidean distance), but finally your classifier needs to generate only one score for your probe images based on the RULE which is applied to these gallery scores of the same class. Otherwise, classification will not give any sense. The averaging about which you spoke can be simply be interpreted as one of those RULES, but it is associated with your classification task and not the CMC protocol. Please see that this is my personal opinion.

Best,

Reza.

Hello Dear Reza

thanks alot for your answer,

in the future i would asking more quastion from YOU, 

and thanks again, majid.

hello

Hello Mr. Reza.

How can evaluated CMC like a AUC for ROC

I'm waiting for your advice

thank you

Dear Fatemeh, 

I will appreciate if you could provide me with more details about your question.

Thanks.

Hello Mr.Reza.

I know that AUC gives a number for ROC curve assessment. But I need to evaluate CMC curve with weighted Rank-M  accuracy or other metric?

 this is  input to the next function.

Thank you

Dear Fatemeh,

I understand the question. I have not seen a term like AUC (Area Under Curve) for CMC in face recognition papers.

Reza.

Hello Mr.Reza.

Do use Rank-1 could be suitable criteria for CMC evaluating?

Thank you

Dear Fatemeh,

No. Rank-1 accuracy is (x1,y1) of the entire CMC curve. So, my answer is No. However, many papers only show Rank-1, Rank-3 , Rank-5 etc in a table instead of a complete CMC curve.

thank you Mr.Reza for answering

Hello

I must created a training set of FERET database contains 675 image (3 images for individual from 255 individuals) but I do not know? Please help me

That was a very brief and easily understandable explanation. Thank you "Reza Shoja Ghiass".

thank you mr reza

can you give as a generalized definition ofthe rank-1 in the CMC curves

Isn't Rank-k accuracy on the CMC curve the same as Recall@k ?

Thank you Reza Shoja Ghiass for your clarification.

It was helpful Reza, Thanx.

Then what is the different between the ROC and CMC

How to calculate rank? I m using SVM, And I have 80 classes. For CMC curve I need help.