I would suggest two things:

a. Take RGB as 3 different planes and do the way you computed MSE for Gray. Each value for a plane gives you the variations you want to check (for R , G, B)

b. Take the RGB/YCbCr values and transform it to represent as a single value for each pixel and next compute the MSE.

Digital Image processing by Anil K Jain or Gonzalez and Woods (have these formula)

@Aparna Thank you ma'am for response, I did the followed the first Method but i'm facing some problem in code: here is the code,

accRC = sum( (RC(:) - recRC(:)).^2 ); 

accGC = sum( (GC(:)- recGC(:)).^2 );

accBC = sum( (BC(:)- recBC(:)).^2 );

mse = accRC+accGC+accBC / numel(combineColorChannels);

can you check it please.

Thank you Atif, I am sure you would have done this:

1. Check for values in the GC, BC, RC, recRC ... vectors/matrices. 

2. If values are there then check for accRC... (this will be done if those matrices are having the required #s)

If the values in these vectors are missing then not being read from the source.

Double -click on the variable name (matlab) should open the matrices. 

http://mathworld.wolfram.com/Root-Mean-Square.html

I missed ! not important but square root is mathematical right I guess ...

We can calculate the MSE as in the attach file

Thank you All