Dear Marti,

I know that the entropy allow for negative values (Shannon or Rényi entropy).

However, the relative entropy or Kullback-Leibler divergence, it must be allways

positive.

Javier Contreras-Reyes

https://sites.google.com/site/jcrwebpage

I would suggest you to check Shannon´s paper "A Mathematical Theory on communication" available online (http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf).

There you will find conceptual and analytical reasons for the non-negativity of the entropy.

More precisely, Shannon´s measure of entropy (denoted by H) results to be null if and only if all the probabilities but one are zero, this one having the value unity. Otherwise H is positive.

Best regards

No, by definition:

H=-sum(pn*log(pn)) ,pn

There H>=0

Dear Marti,

it is possible.

Theoretically it can be related to Gaussian approximation. Numerically, you can have many instabilities. Think about some simple dependency measures in your time series, e.g. correlation. Have a good luck, Milan

Milan, how do you define entropy ?

Due to increase of 1/(N-m+1) with m it is possible that C_i^m(r) will be also increased (notations from wikipedia). Then approximate entropy will be negative. But with m

Check your domain. If you plot the entropy function you will see that is positive in (0,1) and negative on (0,inf). So, check your domain.

Yaron, your definition is in order if you are talking about entropy of a probability mass function. However, I have seen entropy applied in other setups were the variables do not require to sum to one, or be less than one. Then, it depends on the application. Of course we can discuss how valid is such procedure, non the less, some fields use it as standard.

i am chemist

in chemical reaction G is important

when S is negative and H is negative we have equilibrium

Maybe the problem is with self-matching, and the ratio C(m+1)/C(m) is negative...you should try with SampEn if the number of data is small....

Dear Dr. Martí Rosas-Casals.

If you want, I can see your program and try to see what is going to get negative values ​​of approximate entropy.

I'm pretty sure my (MATLAB) code is correct, because I could reproduce the results of Pincus' 1992 paper.

Best regards,

Lucas.

Dear Marti,

Entropy shows the regularity of the system as yourself said. When entropy is positive, it shows disordering is increasing in the system. By contrast, when entropy is negative, it shows ordering in the system. It's rare but quite likely. For example "superlattice" structure in alloys like Au-Cu (gold-copper). The structure gets a regular array in long-range which can result negative entropy.

Marti:

In terms of the theoretical basis, no, is it not possible - what are being approximated are probabilities. It takes a string (e.g., two consecutive values, and counts the pairs that are close. In terms of implementation, computationally, the problem could arise if one does not allow the pair to be compared to itself.

Did you write your own program or get something off the internet?

Marti

How you define the entropy? In what physical (probabilistic) framework ?. Your requirement is vague.

Aldo Tagliani

Dear Martí,

If you follow Pincus, S. M. (1991) Proc. Natl. Acad. Sci. USA 88 , 2297–2301, ApEnt assigns a nonnnegative value to a time series. Probably you should check your script or see RHRV package in R.

Check the precision of the numbers and calculations. I had a problem like this when I used Excel to calculate the Entropy.

In answering this type of question one should analyse the operation which produces the result in question. From an economist's point of view I can't see of what use this concept might be but I am willing to be educated.

At any rate, looking at Wik's descrition of the algorithm (and assuming it is correct) we see that the quantities computed in step 3 are positive and an increasing function of m; therefore their logarithms are an increasing function of m. If we now go to step 5, for any given m we are summing (averaging) N-m+1 quantities; if we compare what we compute in Step 5, for m and m+1, one can rearrange the difference so that it consists of two terms, one of which is unambiguously positive ( the term for i=N-m+1, and m), and one that is of ambiguous sign, i.e (N-m/N-m+1)times the average of the first N-m terms for m and the average given in step 5 for m+1. The latter average is definitely bigger that the first avererage; the question is whether the fraction multiplying the first average is such as to allow a nonnegatve result when added to the positive term corresponding to the item for i=N-m+1 of step 4 (for m) divided by N-m+1.

An answer to that could be found perhaps probabilistically if you assume some distribution for the underlying process (but you are trying to find the series properties so you can't do that) or purely mathematically, which is certainly doab le.

Dear Martí,

Following Picus' paper, it looks like that APEn cannot be negative (page 2299, section K-S Entropy and ApEn), That is, I support the comment from Aurelio Fernandez.

Sample entropy is a more robust metric. For a solution that solves the problem of how to pick the tolerance r, see Lake and Moorman, American Journal of Physiology, 2011. Let me know if you have difficulty finding the paper. Good luck with your work.

Hi, I just developed a .m file to calculate multiple ApEn index in the same dataset. It seems to work pretty well. Should you need it, just tell me and I can send it to you!

Dear Marti,

Approximate or otherwise, entropy has to be positive. Probably, in your case, the value should be very nearly zero, which is why due to approximations in transforming your decimal data into binary mode, the value has become negative.

Yes. It all depends on the approximation algorithm.

Approximate entropy should not be negative because the pdf and the conditional distribution should normally integrate to 1. The Lagrangian constants are also non negative. Negative entropy is meaningless.

I am not sure about entropy, but calibration weights could be negative? Some one could try replacing log value with a displacement function as proposed by me in my following paper: On the calibration of design weights using a displacement function. Metrika (2012). Please see the attached file for details:

Entropy should not be less than 0. But I also have some ApEnt values for time series which are below zero. I haven't checked for the reason, but my initial thought was that this might be because of the choice of the estimator, correcting for the small sample bias problem, analogous to the Grassberger estimators for entropy (see Grassberger, 2003, Entropy Estimates from Insufficient Samplings).

Negative entropy can occur when the pdf for example has zero values

In sets of non zero measure.

For exanple the Dirac distribution would yield negatively infinite entropy.

In the finite case this can not happen since we can always assign a positive probability measure.

In the countable case the entropy is a series ( counting measure).

If the series converges only finitely many terms are non zero. Hench entropy should be positive in this case too.

At temperatures near 0 K, nearly all molecular motion ceases and ΔS = 0 for any adiabatic process, where S is the entropy. In such a circumstance, pure substances can (ideally) form perfect crystals as T → 0. Max Planck's strong form of the third law of thermodynamics states the entropy of a perfect crystal vanishes at absolute zero. The original Nernst heat theorem makes the weaker and less controversial claim that the entropy change for any isothermal process approaches zero as T → 0:

\lim_{T \to 0} \Delta S = 0

The implication is that the entropy of a perfect crystal simply approaches a constant value.

The Nernst postulate identifies the isotherm T = 0 as coincident with the adiabat S = 0, although other isotherms and adiabats are distinct. As no two adiabats intersect, no other adiabat can intersect the T = 0 isotherm. Consequently no adiabatic process initiated at nonzero temperature can lead to zero temperature. (≈ Callen, pp. 189–190)

An even stronger assertion is that It is impossible by any procedure to reduce the temperature of a system to zero in a finite number of operations. (≈ Guggenheim, p. 157)

A perfect crystal is one in which the internal lattice structure extends uninterrupted in all directions. The perfect order can be represented by translational symmetry along three (not usually orthogonal) axes. Every lattice element of the structure is in its proper place, whether it is a single atom or a molecular grouping. For substances which have two (or more) stable crystalline forms, such as diamond and graphite for carbon, there is a kind of "chemical degeneracy". The question remains whether both can have zero entropy at T = 0 even though each is perfectly ordered.

Perfect crystals never occur in practice; imperfections, and even entire amorphous materials, simply get "frozen in" at low temperatures, so transitions to more stable states do not occur.

Using the Debye model, the specific heat and entropy of a pure crystal are proportional to T 3, while the enthalpy and chemical potential are proportional to T 4. (Guggenheim, p. 111) These quantities drop toward their T = 0 limiting values and approach with zero slopes. For the specific heats at least, the limiting value itself is definitely zero, as borne out by experiments to below 10 K. Even the less detailed Einstein model shows this curious drop in specific heats. In fact, all specific heats vanish at absolute zero, not just those of crystals. Likewise for the coefficient of thermal expansion. Maxwell's relations show that various other quantities also vanish. These phenomena were unanticipated.

Since the relation between changes in Gibbs free energy (G), the enthalpy (H) and the entropy is

\Delta G = \Delta H - T \Delta S \,

thus, as T decreases, ΔG and ΔH approach each other (so long as ΔS is bounded). Experimentally, it is found that all spontaneous processes (including chemical reactions) result in a decrease in G as they proceed toward equilibrium. If ΔS and/or T are small, the condition ΔG < 0 may imply that ΔH < 0, which would indicate an exothermic reaction. However, this is not required; endothermic reactions can proceed spontaneously if the TΔS term is large enough.

Moreover, the slopes of the derivatives of ΔG and ΔH converge and are equal to zero at T = 0. This ensures that ΔG and ΔH are nearly the same over a considerable range of temperatures and justifies the approximate empirical Principle of Thomsen and Berthelot, which states that the equilibrium state to which a system proceeds is the one which evolves the greatest amount of heat, i.e. an actual process is the most exothermic one. (Callen, pp. 186–187)

One model that estimates the properties of an electron gas at absolute zero in metals is the Fermi gas. The electrons, being Fermions, have to be in different quantum states, which leads the electrons to get very high typical velocities, even at absolute zero. The maximum energy that electrons can have at absolute zero is called the Fermi energy. The Fermi temperature is defined as this maximum energy divided by Boltzmann's constant, and is of the order of 80,000 K for typical electron densities found in metals. For temperatures significantly below the Fermi temperature, the electrons behave in almost the same way as at absolute zero. This explains the failure of the classical equipartition theorem for metals that eluded classical physicists in the late 19th century.

You can set the parameters such that you get negative values. The point really is if that make any sense! A deeper understanding of the meaning of the algorithm will tell you that something is off. It is not necessarily in the algorithm but in the parameters that you specify that violate the actual meaning of what you are trying to say.

Entropy: A concept that is not a physical quantity

https://www.researchgate.net/publication/230554936_Entropy_A_concept_that_is_not_a_physical_quantity

You can read this paper, and it may encourage you to give more wider definition of Entropy:

On the calibration of design weights using a displacement function

Article On the calibration of design weights using a displacement function

Calculus is not "take for granted".

ΔQ/T can not be turned into dQ/T. That is, the so-called "entropy " doesn't exist at all.

It is well known that calculus has a definition.

Any theory should follow the same principle of calculus; thermodynamics, of course, is no exception, for there's no other calculus at all, this is common sense.

Based on the definition of calculus, we know:

to the definite integral ∫T f(T)dQ, only when Q=F(T), ∫T f(T)dQ=∫T f(T)dF(T) is meaningful.

As long as Q is not a single-valued function of T, namely, Q=F( T, X, …), then,

∫T f(T)dQ=∫T f(T)dF(T, X, …) is meaningless.

1) Now, on the one hand, we all know that Q is not a single-valued function of T, this alone is enough to determine that the definite integral ∫T f(T)dQ=∫T 1/TdQ is meaningless.

2) On the other hand, In fact, Q=f(P, V, T), then

∫T 1/TdQ = ∫T 1/Tdf(T, V, P)= ∫T dF(T, V, P) is certainly meaningless. ( in ∫T , T is subscript ).

We know that dQ/T is used for the definite integral ∫T 1/TdQ, while ∫T 1/TdQ is meaningless, so, ΔQ/T can not be turned into dQ/T at all.

that is, the so-called "entropy " doesn't exist at all.

Entropy will have positive values only.

Hello,

Entropy may be negative in some photo reaction for decolorization and degradation of dyes, hence, the small species that produced from degradation of dye, are regarded less random compared with big dye molecule. This behave will lead to negative value of entropy .

Best regards