Mehmet, I would start with the R package at the link below. Best, David

https://cran.r-project.org/web/packages/entropy/entropy.pdf

David thanks, actually I have seen some R packages like the package entropy. I need some insight about their superiorities and drawbacks. Also they seem to use discretization before calculating entropy.

I am curious if any method/package/software available to calculate those quantities without discretization or the recommended dicretization technique in case no such software is available. (I know the question title reads different, but my aim was to see alternative software, so any suggestion will be appreciated).

You can use the code of the following reference (http://bsp.teithe.gr/members/downloads/Milca.html)

Reference:

A. Kraskov, H. Stögbauer, and P. Grassberger, Phys. Rev. E 69 (6) 066138, 2004.

Hiqmet thanks for the reference. I understand that there is a deep literature for the calculation of entropy and mutual information of continuous variables. Do you have any insights about performances of different methods or any other codes?

You can have a look at the following paper, it may be helpful:

Deniz Gencaga, Kevin H. Knuth and William B. Rossow, Entropy 2015, 17, 438-470.

Thank you very much!

Mehmet, If I understand your comments you would like to to do the computations without using a discrete approximation to continuous variables? If so have you tried a symbolic math package or do I have this all wrong? Best, David PS There are packages for both R and Python as well as the commercial ones.

David, yes that's correct. I suppose that it would be preferred without discretization. Can you name those packages and their usage on entropy calculation?

Mehmet, See the links below. The first few entries from the link are the most important. There are some articles that compare packages here also. Remember though that a good approximation(need ways to check this) can be almost as good as the exact answer esp. if the exact answer is hard to impossible to compute(e.g. not all anti-derivatives exist in closed form). This is the point of numerical analysis. I would also look at a good reference in this area. The next to last link points to a favorite of mine. I hope this helps. Good luck and Happy New Year. David BTW I would suppose their use in entropy calculations would be to do things like evaluating integrals etc. The last link is to my favorite entropy book.

https://www.google.com/#q=Symbolic+Mathematical+package

https://www.google.com/#q=Symbolic+Mathematical+package+R

https://www.google.com/#q=Symbolic+Mathematical+package+Python

http://www.bing.com/search?pc=COSP&ptag=D071616-A0A14E5293D1943C986F&form=CONBDF&conlogo=CT3330954&q=Numerical+Analysis+in+Python

http://www.bing.com/search?q=Numerical+Analysis+in+R&qs=n&form=QBRE&sp=-1&pq=numerical+analysis+in+r&sc=8-23&sk=&cvid=09230B2A85284B8380B31D236F7228C0

https://www.amazon.com/Numerical-Analysis-Science-Engineering-Chapman/dp/143988448X/ref=sr_1_1?ie=UTF8&qid=1483453668&sr=8-1&keywords=using+r+for+numerical+analysis+in+science+and+engineering

https://www.amazon.com/Information-Theory-Statistics-Dover-Mathematics/dp/0486696847/ref=sr_1_1?ie=UTF8&qid=1483454185&sr=8-1&keywords=Kullback

Thanks David. I guess I need to use numerical integration to find entropies etc.

But you know the more important thing is to know the underlying distribution of present data. And generally one does not know the distribution of the data beforehand.  Therefore it seems that we need some ways to identify the underlying distribution and then calculate information theoretical quantities. There are some kernel density estimation techniques for such a goal. I am curious

i) If this is the right way of calculating those quantities (without discretization, but finding the distribution and then calculating the numerical integral)

ii) if there are different ways for finding the distribution of that sample data.

Best

Mehmet, You might look at this article at the below link. It might be of some help to you. Wish you the best, David

Practical Kullback-Leibler (KL) Divergence: Discrete Case

January 7, 2017

By msuzen

https://www.r-bloggers.com/practical-kullback-leibler-kl-divergence-discrete-case-2/

Entropy: A concept that is not a physical quantity

https://www.researchgate.net/publication/230554936_Entropy_A_concept_that_is_not_a_physical_quantity