"Difference in colour" is a slightly vague description, if you could provide a little more information I may find it easier to help.
You could use tristimulus values, [https://en.wikipedia.org/wiki/CIE_1931_color_space], but depending on your application RGB may not be the best format, so you may wish to convert to a different colour model such as HSI, or CMY? I attach some notes on colour image processing from one of my previous lecturers, Dr. Adrian Evans. Below are a few references you may wish to look at if you're after edge detection within colour images (quite a bit more complicated than in B&W images).
Evans, A. N., 2006. Nonlinear edge detection in colour images. In: Marshall, S. and Sicuranza, G. L., eds. Advances in Nonlinear Signal and Image Processing. Vol. 6. New York: Hindawi, pp. 329-355. (EURASIP Book Series on Signal Processing and Communications)
Evans, A. N., 2004. Morphological Gradient Operators for Colour Images. In: IEEE International Conference on Image Processing (ICIP '04), 2004-10-24 - 2004-10-27, Singapore.