Let's use an example. We have a function y = f(x), in which x is the input (the probability) and y is the output (the entropy). If we change y in y', can we find an x' such that f(x') = y'?
In other words, I know that when p changes, H changes; is it possible the opposite, such that if H changes, p changes?
Considering the Gibbs-Shannon entropy S, we know that
dS=-∑(lnpi)dpi
We will consider i=1,2,...,n with n>2. Then there are n-1 independent variations dpi.
You cannot uniquely determine n-1>1 independent quantities dpi from a single quantity dS.
I am not sure if this is what you were asking?
When you are trying to estimate maximum entropy you find the corresponding probability distribution. Changing the probabilities usually occurs when you have prior information, hence the formulkation of your problem becomes one of minimizing cross-entropy.
Under which circumstances would you have prior information on the level of Entropy which would influence the probabilities?
In Rényi's entropy. There are some circumstances in which a change in the Rényi's alpha exponent may occur. I wonder if, knowing the new values of entropy, I can calculate the correspondent new probabilities.
Take as a starting point a binary case with probability distribution Pr('1') = p and Pr('0') = 1-p. In this case, plotting the entropy H(p) gives you a concave function with maximum for p=0.5. For the other values of H you have two solutions for the probability distribution...
Dear Torodan, I know that, but I may choose between one of the your two solutions via Kullback-Leibler divergency. The problem is: am I allowed to start from a known E to get an unknown p, instead of the common bottom-up Shannon's procedure?
Dear Arturo,
I guess in order to answer your question one needs to do the following calculation:
1) Let p_i, for i=1..N, be such that p_i \in [0,1] and \sum_i p_i = 1 and H({p_i}) is its entropy.
2) Define the following density:
P(H)=\prod_i \int _0^1 d p_i \delta( H - H({p_i}) ) \delta(1 - \sum_i p_i)/ V, where
V= \prod_i \int _0^1 d p_i \delta(1 - \sum_i p_i).
For a given value of the entropy H the P(H) gives the fraction of the "volume" V which is "occupied" by the "configurations" p_1, ...,p_N with this entropy.
Best,
Alexander
I am not expert enough in this field to add further useful information.
Dear Arturo,
Suppose we are given the probability vector p=(p_1, ...,p_N) with the entropy H_0 (p) and we want to change p in a such a way that its entropy becomes H_1(p). If H_1 > H_0 then this can be done by the gradient ascent on H(p), starting from H_0, to attain H_1 and if H_1 < H_0 then one can use the gradient descent.
Best,
Alexander
Dear Alexander, this is exactly what I tried to do here:
https://www.researchgate.net/publication/271204916_Multifractal_exponents_and_Rnji_entropy_a_clue_for_brain_function
But somebody said to me that "if the output doesn't have coupling, then you will not observe a change in the input by changing the output".
Someone else said that "the input and the output must stay into the same domain".
Preprint Multifractal exponents and Rénji entropy: a clue for brain function?
Generally, it is not true that changes in p lead to changes in H.
let me use an example.
let we have probability pas follow:
p(x=1)=1/16, p(x=2)=1/16, p(x=3)=1/8, p(x=4)=1/2
H=-sum(p_i*log2(p_i)=9/8
now we change p to p':
p'(x=1)=1/2, p'(x=2)=1/8, p'(x=3)=1/16, p'(x=4)=1/16,
H'=9/8, in other words, p~=p' while H=H' and H does not change.
now, if we change H for this example.
assume that H (was 8/3) changes to be 2, here we can easily
say, p(x=1)=1/4, p(x=2)=1/4,p(x=3)=1/4,p(x=4)=1/4,
while if we change H to 0,
all of following p's are possible:
(1) p(x=1)=1,p(x=2)=0, p(x=3)=0,p(x=4)=0
(2) p(x=1)=0,p(x=2)=1, p(x=3)=0,p(x=4)=0
(3) p(x=1)=0,p(x=2)=0, p(x=3)=1,p(x=4)=0
(4) p(x=1)=0,p(x=2)=0, p(x=3)=0,p(x=4)=1
So, in general it is not possible to find the exact changes.
in conclusion, if H changes, it shows that p has changed , but you may not find the exact change.
First of all, as already pointed out by Hoseein Soleimani, a change in p does not imply an automatic change in the entropy S, because many different probability distributions have the same entropy value. Another thing is that, in "equilibrium" (the meaning of equilibrium may vary depending on the field you are working on), there is a certain equilibrium distribution probability p_eq that maximizes S and gives its equilibrium value S_eq, which depends on some parameters (for instance the temperature). If you change the parameter, then the equilibrium distribution changes and also its corresponding equilibrium value of the entropy S_eq. It is crucial that you should not confuse S (which is a functional of a generic probability distribution p) and S_eq=S(p_eq) (which is a function of the parameters on which the equilibrium distribution depends).
Taking it literally, the answer to your question is yes, if you change S, the corresponding distribution function must change (if not, S would remain the same). This also holds for S_eq, if you change S_eq by modifying quasistatically the parameters on whicn it depends (temperature, volume, ...), the underlying p_eq is also modified.
Considering the Gibbs-Shannon entropy S, we know that
dS=-∑(lnpi)dpi
We will consider i=1,2,...,n with n>2. Then there are n-1 independent variations dpi.
You cannot uniquely determine n-1>1 independent quantities dpi from a single quantity dS.
I am not sure if this is what you were asking?
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Probability