a. Take RGB as 3 different planes and do the way you computed MSE for Gray. Each value for a plane gives you the variations you want to check (for R , G, B)
b. Take the RGB/YCbCr values and transform it to represent as a single value for each pixel and next compute the MSE.
Digital Image processing by Anil K Jain or Gonzalez and Woods (have these formula)