Have a look at https://au.mathworks.com/help/images/ref/corr2.html

https://au.mathworks.com/matlabcentral/answers/161039-how-to-calculate-the-correlation-coefficient-in-an-image

According to your query, this may be helpful

Article Image Encryption Based On Diffusion And Multiple Chaotic Maps

If the response is helpful then click on Recommend to help others. Thank you.

If you're trying to calculate the correlation between various images in a statistical sense then you need to take the mean of your images in the high-dimensional space. and then use the concept of correlation to identify how closely related two images are.

But if its the correlation matrix you're trying to consider the following code would be useful:

from scipy import signal

cor = signal.correlate2d (im1, im2)

where im1 and im2 are the two images

But if you're trying to look at the similarity or difference between two images I would suggest that you should implement something like the triplet loss

https://en.wikipedia.org/wiki/Triplet_loss

You can also try out clustering algorithms like:

Hierarchical clustering: https://www.displayr.com/what-is-hierarchical-clustering/

Vanilla K-means: https://en.wikipedia.org/wiki/K-means_clustering

There are implementations for the same in sklearn https://scikit-learn.org/stable/modules/clustering.html