Do you mean that you want to compute the conditional mutual information I(X1;Y|X2)? If so, since X1 independent of X2, H(X1|X2) = H(X1). Thus

I(X1;Y|X2) = H(X1|X2) - H(X1|X2,Y) = H(X1) + H(X2,Y) - H(X1,X2,Y)

So long as you know how to compute entropy of a p.m.f., you are able to compute I(X1;Y|X2) now.

Thank you Dr Geng for your reply. I am in doubt whether

I(X1;Y|X2) = H(X1|X2) - H(X1|X2,Y) or I(X1;Y|X2) = H(X1|X2) - H(Y|X2,X1) . Please correct me if I am wrong.

One more thing I am wandering that suppose X1={U1} and X2={U2,...Um} then how to find out H(X2,Y) and H(X1,X2,Y) in discrete variables case? Because for large m number of possible combinations in X2 (suppose each of the individual variables have 2 states each) will increase astronomically and we may not have the value of Y for each possible combinations of X2 and X1,X2. How to proceed in this case? I have one more question...Anyway can copula help me in this situation? Thanks again for your thoughtful reply.

While, the second equality should be I(X1;Y|X2) = H(Y|X2) - H(Y|X2,X1). You may verify this by searching 'conditional mutual information' on wikipedia, or from textbook.

The formula of H(X) is sum -p(x) log p(x) over x. For the zero probability p(x), since t log t goes to zero as t goes to zero, so those zeros don't contribute to H(X). Another view of point, H(X) is the expectation of -log p(x), so those zero p(x)'s don't come into expectation.

If you have some nice structure about p(x1,x2,y), you may use proper decomposition to get nice manipulation.

Thank you Dr Geng for your reply and clarifying my doubt.

You are welcome!

A simple probability question: Can we write the following:

P(Y,S1,S2,S3)=P(Y,S1)*P(Y,S2)*P(Y,S3) when S1,S2,S3 are independent but Y depends jointly on S1S2S3. Thanks in advance.

Not really. You may write p(y,s1,s2,s3) = p(s1)p(s2)p(s3)p(y|s1,s2,s3).

Thanks Dr Geng...

Dear Dr Geng , Sandipan,,

Thank you for this question and answers.

I have a question about the conditional mutual information (CMI)

I(X;Y|Z)=H(X|Z)-H(X|Y,Z)

my question is how to compute H(X|Y,Z)  ?

Does it called conditional joint entropy ?

Thank you.