Information theory, formulated by Claude Shannon, says that information reduces uncertainty. Diffusion of innovation theory, formulated by Everett Rogers, say that information increases uncertainty. How do we resolve this conflict in the relationship of information to uncertainty?
In his information theory, Claude Shannon defined "entropy" as a measure of uncertainty with respect to some variable or event. The greater the uncertainty, the greater the "Shannon entropy." Shannon proposed that information reduces uncertainty and therefore reduces entropy. A typical example of this principle is flipping a two-sided coin. When tossing a coin, we are uncertain about the two possible outcomes. By observing the outcome (say, heads), we gain information that reduces the uncertainty about this event.
The theory of diffusion of innovation, developed largely by Everett Rogers, is a study of how new ideas and technologies spread (or are adopted) by a population. Through his research, Rogers showed that innovation creates uncertainty, which becomes a barrier to the adoption of the innovation. For example, when teaching people who live in rural areas to boil their water before drinking it, this "innovation" is resisted because it may conflict with established cultural beliefs that boiled water makes people sick. The demand to boil water conflicts with cherished beliefs and is therefore resisted.
We can see that Shannon demonstrated that information reduces uncertainty, but Rogers demonstrated that information increases uncertainty. How do we resolve this apparent conflict? I will present a few thoughts about this and would appreciate your comments.
I would propose that Shannon and Rogers are considering two different aspects of information. Shannon's information is a signal (or indicator) with minimal or no meaning (one "bit" of information = low meaning-value); whereas Roger's information refers to multidimensional concepts with a great deal of meaning (multidimensional information = high meaning-value) . Assuming that a given unit of information can be placed on a continuum of meaning-value, then the greater the meaning embedded in a message, the greater the likelihood that the implied-meaning of the message-sender (boiled water = prevention of illness) will conflict with the perceived-meaning of the message-receiver (boiled water = cause of illness).
Gregory Bateson defines information as "the difference that makes a difference." Loet Leydesdorff defines knowledge as "the meaning that makes a difference." Perhaps, then, information with low-meaning value reduces uncertainty in a low-dimensional state space (such as binary signaling system); whereas information with high-meaning value increases uncertainty in a multi-dimensional state space (such as a cultural belief system). Ultimately, this begs the difference between information and knowledge, and how information from a sender is evaluated and assimilated into a receiver's knowledge-base.
Thoughts?
Ok, let me just say that Shannon, and others have probably mistaken the concept of entropy, when they say that increasing entropy increases uncertainty.
This was because, previous to the experiments done by Waltham, we didn't realize that order and chaos have an oscillatory relationship, where adding to the system results in order submerging into chaos, and later emerging again possibly in a more complex state.
In fact what Shannon was doing, is attempting to submerge the order of the signal in the communications medium and extract the order of the signal at the other end from the same medium. Shannon therefore decided that if he increased the information content, (Probably by using a more complex coding representation that was partly protocol) he could extract more information on the other end.
Now let us look at the Diffusion of Innovation Theory.
Diffusion of Innovation, it turns out is less about information than about semantics. In essence it is not the signal that is sent, but how that signal is interpreted that matters. It is possible to use a single bit to send a complex message if the only thing that is needed is a binary choice whether or not to make an already in situ file available to the reader. On the other hand, in the internet we often find ourselves sending the electronic equivalent to the book, even though we know that most users will only read so far before rejecting it.
Essentially, Diffusion of Innovation, is dependent on the interpretation of the information, but that interpretation is despite known redundancies in language and expression, that often make the information difficult to understand. In essence, Garbage in = Garbage out, even when the Garbage in, would translate if the reader was culturally and linguistically familiar with the topic.
Another aspect that makes diffusion of Innovation difficult is the paradigm change model, essentially too much paradigmatic change results in a sort of refusal to keep up with the change. Especially if the paradigmatic change is seen as capriciously motivated. For instance I have become resistant over MicroSofts failure to limit the paradigmatic change in their operating system, because I perceive it as arbitrary changes in name and location of useful tools as a marketing move to sell more courseware, instead of for valid coding reasons.
Well, thanks I think Roger.
I don't think boycotting Microsoft's stuff will solve all the paradigmatic problems, but it solved a whole bunch of problems I was having with the Internet, it's funny how Microsofts products do so well off the network, and so poorly on the network, but Microsoft wants everyone to hook up to the network anyway. It's almost like they planned their software to break down, just in time to get the full price for their newest and least tested version. Way too marketing oriented if you ask me.
Shannon's work is incredibly important to transmission of signals, and starts us looking at what is needed to recover a compressed form of storage. What I think is missing from information theory, is an understanding of the conversion that happens in the brain to turn relatively cut and dried data about the environment into the incredibly rich semantic content of our awareness, even when we take into account that the great masses of the data gathered and semi-analyzed by the implicit memory can't fit through the bottleneck between implicit and explicit memory.
In the GreySmith Memory Model, I try to capture this vast narrowing at the Short term memory, Within the Chunk Formation as a step involving the reconciliation of each neural group, against the partition projected onto the Short Term Memory. Since this happens at each rehearsal step, for each element stored in the Short term working memory, The bottleneck distinctly limits the number of elements that do not habituate, to those that can be rehearsed before the 3 seconds is up. One reason it works this way, is what Dr. Edelman called the "Recategorical" nature of implicit memory. Because the implicit memory is partly informed by volatile "Synaptic Memories" the brain assumes that the content of the memory being rehearsed might subtly change, and so it needs to recover the addresses of the newly significant data as well as that formerly significant.
One place where I think information theory falls down, is in the assumption of discrete memory locations, while the explicit memory is based on just such a fiction, the fiction doesn't hold as true as we might think, if only because the nature of the data-elements being retrieved at this interface with the implicit memory is NOT discrete units, but a cloud of data, much of it redundant and ambiguous. Thus the CHUNK as a list of Neural Group Addresses is needed to capture the partition of the cloud projected at the working memory (Short Term Memory) level.
Although there is some rote processing in Fodors Cerebral Modules, evidence now suggests that much more of it is done between associative areas of the brain than originally thought. Between these two stages is an analysis step, that looks to take significantly more of the so called executive function than any step before or after it, It is my contention that this step is a sub-chunk expansion, where each subchunk in the partition is randomly (With biases towards especially salient types of information) checked for salience, before it is gated on to the associative step.
The much more involved Weak (as associated to consciousness) Attention system seems to gate at every one of the 7-9 stages in processing, and thus, manage content flow out to the other parts of the brain. By philosophically describing awareness and consciousness, and physiologically describing the first few steps of the processing, we have mistakenly assumed that the other 5 steps that the brain uses to understand the semantics of its sensory input, are somehow less important.
If we really want a computer to understand semantics like humans do, we have some catch up to do. If we want to overcome human resistance to paradigm change, we have even a harder thing to do.
Good and interesting comments, Graeme and Roger. I have no ax to grind on this topic. In studying both information theory and diffusion theory, I noticed the contradiction and wondered whether the two different perspectives on information (Shannon and Rogers) could converge at a certain conceptual point, or whether they were talking about two non-convergent aspects of "information." Clearly, both Shannon and Rogers were addressing different problems.
I have had problems equating uncertainty with randomness, although I can force myself to see parallels. It seems that meaning is the factor that separates the two perspectives. Shannon-information can be transmitted as symbols that are "read" by a receiver without being "understood" by the receiver. A coin toss transmits an outcome that can be read as a signal of certainty within a system without some meta-recognition by the system (stimulus-response vs. stimulus-appreciation-response). Rogers-information (innovation) is loaded with meaning, which destabilizes the receiver and creates uncertainty.
I have had much experience with uncertainty created by the prospect of adopting an innovation. The effect does beg a theory of change. I have run training programs for mental health clinicians/counselors to adopt "evidence-based practices" in their treatment of people with mental disorders. The uptake is slow and painful. When clinicians have been practicing their "craft" for 15 years with no feedback on their effectiveness, they all assume they are above average in effectiveness. When someone comes along and tells them that research shows there is a better way to treat people, the effect is telling them that they have been wrong (or at least they are not above average) in what they have been doing for the past 15 years. My experience (and that of others) is that the clinicians will adopt the new practice while they are being "watched" (trained, supervised), and then revert back to their old, established methods once the training is finished and the supervisors have gone home.
In my search for some model of rigid system stability, I have found the concept of attractor states to explain a lot. Routine behaviors become established patterns of neural activity (years of Hebbian learning, possibly in the striatum, which has some association with obsessive-compulsive behaviors). These habits are manifestations of limit cycle attractors with deep basins of attraction (that is, well-established patterns of neural networks that cycle within a fixed range of upper and lower control limits). Given the strength of these attractor neural networks, new information that requires the adoption of new habits and the abandonment of old habits results in deep-attractors-uber-alles - i.e., absolute resistance to change. And that puts me in full agreement with Graeme's summary statement: "If we want to overcome human resistance to paradigm change, we have even a harder thing to do."
In the GreySmith Memory Model, which by the way is based on science but has not been experimentally confirmed at all, The role of the Basal Striatum is more or less to Prioritize according to experience. I liken this to the "Jelly Mold Model" Where you take a bundt cake or jelly mold, and fill it with jello, then turn it upside down, and drop warm water drops on it. At first it just speckles a bit each drop sort of sitting in a small cup it made in the jello, but sooner or later, two drops coincide and form a longer drop, and then it builds up until it rolls down the side of the jelly mold, forming a sort of chasm. As more and more drops fall, the chasms get deeper and become the preferred routes for erosion, which deepens them further.
This is reactive conditioning, given rote experience that something works one way, you will tend to continue thinking that it will continue to work that way, long after someone tells you different. It is not that the doctors didn't listen to you, it is just that they have a life experience that the other way works, and virtually no experience that this way works better. In short they are going to use the system that is well supported in books such as DSM IV or V or whatever version it is, rather than using what you suggest, because they know how to succeed they think at the old game. (At some point it becomes a chess game against the bureaucrats, to help their patients when the system doesn't care if they are sick or not as long as it comes out of someone else's budget).
The place you need to train to evidence based treatment is not the doctors, but their bureaucratic financiers, and you know how conservative THEY are, there are still somethings on the books that were based on horse carts. In fact it is the forced paperwork that is likely the culprit. Think of how you are going to explain to a bureaucrat that you don't have a diagnosis, but evidence supports this treatment over that one, so they should fund a $1000.00 blood test that no other doctor is crying for. (yet) I think the answer will be NYET! (not yet?) it doesn't matter that the evidence support the fact that they will spend literally thousands of dollars treating the person with anti-depressants that don't work, simply because their metabolism can't use a particular enzyme properly, but you don't know which one until you treat them for a year, or take the test. It's just easier to get them to pay something by having a diagnosis, and filling out the little box to meet their requirements, you still won't get the blood test, but then, no one else is either.
You could describe the uncertainty be probability theory and statistics if the uncertainties have objective characteristics (even in biotechnology). In the case of human uncertainty in the brain thinking you could use the subjective utility theory, the subjective probability theory and the theory of choice functions (the more common mathematical aspects).
Well, you could, and in fact Bayesian Theory to some extent does. Beyond all that talk of beliefs, that is based on Rev. Bayes religion, lies a statistical inference engine. But Marr tried that approach back in 1970, assuming that if he got the functions of the brain figured out with probability theory, he could use some statistical arrangement of connections to allow for the differences in individual connectivity. It didn't work that way, there is too much uncertainty in the connectivity for him to have achieved his goal. It's not just a matter of statistics, it's a matter of having to overcome uncertainty, not detect the lack of it.
Thank you Graeme,
"Resolving Uncertainty About Uncertainty ?"------>May be my opinion is something provocative. The big problem, in my opinion, is the determination of the structure in any concrete investigation. This is the so called “art of the decision”. Every time we are in a concrete situation and in a concrete investigation. The human logical thinking is restrictive and considers every investigated problem from the position of a human purpose. The world is more complex, God’s made. That is why “uncertainty” exists always. We can measure in some way and not every time our “lack of knowledge” or “knowledge”. The scientific mathematical description is in a restrictive sense, only a way to measure, a language of measuring. The big problem is in the structure, the interrelations in the sense of the system analyses.
The practical way now a day, is Probability theory, Subjective probability theory, Stochastic programming and construction (building) of new and new more profound scientific notions (philosophical, mathematical, sociological...).
Example-We can measure entropy, but this is a measure of a exact human notion, exact clear notion. We can model with this entropy our restrictive perception, our “lack of knowledge”.
Someting provokative:
De Finetti’s treatise on the theory of probability begins with the
provocative statement PROBABILITY DOES NOT EXIST, meaning that probability
does not exist in an objective sense. Rather, probability exists only subjectively
within the minds of individuals.
Uncertainty, means simply that there is a lack of certainty, caused by some information being hidden. The expansion of the Universe, suggests an origin, but we cannot see beyond the big bang, It has been suggested that this is because light, couldn't travel through the state of the Universe just before the Big Bang.
We can measure the speed of an electron, or where it is, with respect to the atom, but we cannot measure both the speed and the location at the same time.
We can clearly see that the quantum level is very dynamic, making many measurements that our equipment is accurate enough to predict, impossible to do more than approximate. However, the approximation is not the thing.
As the structures that we are trying to describe get larger, we see the uncertainty fade, until we can be relatively sure, that a table will stay a table and not become a chair without at least some energy being expended to reform it.
Probability is not like that. when the roots of a dynamic system are unknown, you can't predict the outcomes for any particular state, if only because you can't determine the roots of that state accurately enough not to fail to calculate the next state. What you can do, is develop some statistical assurance that the next step will be one of a known number of states. Which state is unknown, but the range of possible states can be analyzed for likelihood of one being chosen over another. Stochiometry or the study of random systems has taught us that if the probability of any one state, is almost equal with the probability of any other state, we cannot predict the results. However if there is any favoritism between one state and another, by doing lots of experiments we can detect that favorism even though no one experiment could have shown it. Probability is just the expression of that favoritism expressed as a number.
Favoritism of one value over another is a hint to unseen structure, yes, but it cannot detect uncertain dynamic equilibrium, a state that changes in a pattern, that responds to uncertainty but without favoring any particular stable state.
The Waltham experiments suggest that some dynamic structure survives the dissolution of certainty, even when probability and statistical means prove to be less than effective at detecting it. We need a different toolkit if we want to deal with such a subtle structural dynamic in seemingly chaotic systems.
Thanks Kees, but I should warn you that I am looking beyond pattern matching as a model for how the "Meaning gets incorporated into the definitions". I am currently writing an article on what I think the "Information Theory" for the human brain should be, and have if I remember correctly, 12 hierarchical layers of processing between the sensory input, and awareness. By eschewing pattern matching for the more uncertainty tolerant "Similarity Selection" model, a sort of Lazy pattern matching is possible that does not break with the redundancy and ambiguity, which often are required to overcome uncertainty, nor require optimal coding, or structurally static architectures.
I believe the two different approaches of treating new information would also be dependent on the stage of a change or innovation. When an innovation is being used as an experiment, information about failures can create a sense of uncertainty about the innovation. However, as the innovation becomes a tried and tested solution, information about the innovation reduces uncertainty.
The simplest answer from a mathematical viewpoint would be to add a negative sign in the second definition to bring the two in line. Unfortunately this likely isn't going to be nearly enough. First you'll need to look at the physical definitions of the two versions of information and align them precisely to be able to do anything with them down the line.
One of the most difficult problems in this or any other alignment of semantic theories with that of Claude Shannon's mathematical theory of information is in reading Shannon carefully. First and foremost, almost everyone misses the second paragraph of his seminal paper on the topic in which he states: "Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem." I think this point is something that you're missing at the ground level in your question. Too many have confused the mathematical conceptualization of information with the everyday semantic one. Again, you'll get a lot further if you can, in a mathematical sense, build up concrete and well-defined equivalent definitions in the two fields and then move on from there. Without this first step, you're doomed to fail.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence