The process of research and study are actions to determine law or seek truth from natural phenomenon or big data by mankinds self or artificial intelligence. The sort of action will making uncertainness decrease and certainness increase, and then is the process a negative Shannon entropy factor?
The process of research should be viewed as a communication channel. The information going into the channel is empirical data. The information coming out of the channel is a message M in answer to some research question Q.
Let the set of answers (messages) M speculatively entertained by a researcher in connection with a given research question Q be the set M_i, for i = 1,2.... The researcher has prior probability of expectation p_i of finding M_i through the process of research (through the channel). p_i is presumably a subjective prior probability. p_i capture the researcher's uncertainty about the answer to be delivered by the empirical data in answer question Q. Different researchers will generally have different p_i.
Now the process is applied to the data. It operates on the empirical data and yields a message M_j in answer to question Q. Then the researcher's uncertainty has been reduced by an amount -log(p_j).
If the researcher was certain of getting message M_j, then p_j = 1, and the information conveyed by the process of research is -log(p_j) = -log(1) = 0. If the researcher had no expectation of of getting message M_j, then p_j = 0, and the information conveyed by the research is infinite.
But the research processes and the data are imperfect. So the channel is imperfect, and erroneous messages are generally delivered. The channel has an unknown quality, which raises interesting questions.
When we apply Information (Shannon) entropy to the research process of finding a solution of the scientific problem itself, we can enter a relatively novel, yet still little known, area of AI, ML, and Big Data.
Shannon entropy is usually used to solve problems that are unsolvable by any other mathematical methods:
- equations,
-differential equations,
-statistics & probability.
Shannon entropy enables us to distill information from the system that can dramatically decrease its complexity. In this way, entropy serves as the input of other methods: AI, ML, and BD.
This all is done using some predefined model.
Your question is about whether we can use Shannon entropy to search for the best model itself? The answer is yes but it requires highly sophisticated approaches and knowing perfectly what you are doing.
Roughly said, you are searching through the space of all possible models of a given problem and trying to find the best suitable one. Usually, it is done manually by researchers in the way of trial and error. Nevertheless, this search can be automated.
Any step that automates such a research process is harmful to human kinds. It is doable (easy), but the consequences are irreversible, as ever seen in movies such as the Terminator series.
There are Shannon entropy, fractional entropy, Tsallis entropy, fuzzy entropy etc.
When we consider some non-equilibrium systems the entropy can decrease. In the case when we consider the cummulative entropy, such entropy always increases.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Analysis
- Data Analysis
- Dataset
- Big Data