Can anyone help me how can I write the MATLAB code for calculation of the Mutual information (MI) between two array of double numbers.
Someone said that you can use the histogram based probability estimation, calculate the independent variable histogram, and joint histogram, then calculate the independent entropy and joint entropy.
However, I dont know how to do it.
You can find some estimators in the ITE (Information Theoretical Estimators) toolbox: https://bitbucket.org/szzoli/ite/
See these links
http://www.cs.man.ac.uk/~pococka4/MIToolbox.html
http://www.mathworks.com/matlabcentral/fileexchange/14888-mutual-information-computation
For a reliable measure of the mutual information you should look at the technique proposed by A. Kraskov, H. Stögbauer, and P. Grassberger, in Phys. Rev. E 69, 066138, 2004. The histogram method has a disadvantage of determining the bin width in order to avoid the data dependency. In addition, it is important to avoid any correlation that may exist between any data points in the time series by estimating the so-called embedded parameters.
The above mentioned method also comes with a free software that can be downloaded from the following link:
http://bsp.teithe.gr/members/downloads/Milca.html
Thanks for your respond Dear @Gonzalo González-Nuevo González and @Hiqmet Kamberaj. However, I already tested the code which is not work for double type.
For example you have x=[0.2 0.5 0.6] and y=[0.5 0.7 0.6]
I need to compute this type of MI ?
I will really appreciate, if you give me MI MATLAB code for double type.
Thanks.
you will find Matlab code examples for MI calculation for continuous variables following this link:
http://xaphire.de/recipes/?p=376
The method mentioned in the following link should work for floating numbers as well.
http://bsp.teithe.gr/members/downloads/Milca.html
http://xaphire.de/recipes/?p=376
this site given by Reichel is good. But the code here is giving error in joint probability function.
There is a very small error which iam not able to recognise. Can anyone help me to calculate joint entropy, in the code given here. May be the arguments passed from calling function are not correct. What is the binsize here.
these code changes might help:
#################################
1) JointEntropy()
allow for default values for the variable binSize (the number of bins), e.g. as suggested by the author:
if nargin < 3
binSize=round([range(x)/std(x)*5 range(y)/std(y)*5]);
end
Remarks: round() is needed, since an integer is required later for hist3()
Without default values, you would need to adjust the call of JointEntropy() in
MutualInformation(). But the call needs to be adjusted anyway, see below.
#################################
2) same for Entropy()
if nargin < 2
binSize=round(range(y)/std(y)*5);
end
#################################
3) MutualInformation()
the calls of JointEntropy() needs to be adjusted (not the correct number of input arguments given). Should be something like this:
% works for 1 or 2 predictors:
if (size(X,2) > 1) % More than one predictor?
% Axiom of information theory
I = JointEntropy(X(:,1),X(:,2)) + Entropy(Y) - JointEntropy(X, Y);
else
% Axiom of information theory
I = Entropy(X) + Entropy(Y) - JointEntropy(X,Y);
end
... and Hiqmet Kamberaj is right, the determination of the bin width for MI estimation is somewhat arbitrary. The methods his link points to are better suited for this purpose.
The modified code given for mutual information is correct. These changes are needed.
For entropy, this also works:
binSize=min(y):max(y);
Can anyone help me with the binsize in joint entropy.
The code changes given above in joint entropy and entropy works, code is running but it is not giving the correct solution.
@Arpita. The solution depends on the number of bins, thus there is not just one single correct solution.
Your proposed initialization of binSize as a vector of bin centers 'binSize=min(y):max(y)' does not normalize for the range of y: the larger the range the more (probably empty) bins you get.
You can find some estimators in the ITE (Information Theoretical Estimators) toolbox: https://bitbucket.org/szzoli/ite/
You can use Mutual Information Toolbox provided by Manchester University available at:
http://www.cs.man.ac.uk/~pococka4/MIToolbox.html
You can also try Matlab library of Rudy Moddemeijer:
http://www.cs.rug.nl/~rudy/matlab/
Hi, have you found a reliable method for MI calculation? Could you share them? Thank you.
In python you can use library directly:
bins=1000;
from sklearn.metrics import mutual_info_score
c_xy = np.histogram2d(X_norm, Y_norm, bins)[0]
mi= mutual_info_score(None, None, contingency=c_xy)
*NOTE: your MI will depend on bin choice , so you can vary number of bin to observe output. Or you can Use normalized mutual information (NMI) , which is independent of bins and its value lies between 0 to 1 so can be used as metric . NMI is obtained by dividing MI by sqr root of product of individual entropy . ie NMI = MI/ sqrt(H(X).H(Y))
- Badges
- Science topic