In the example you describe there is no sequence. There is is just a single letter that can take one out of the four values A, B, C or D. If an A is observed in your second example one has obtained log_2 (1/0.28) bits. In your second example 1.85 bits is the mean value of the number of bits where the mean is taken over the four possible letters. 

In my understanding of the term Information you made some interesting steps of thoughts:

a) you build a medium value of something (here Shannon-Entropy) and then you build the difference of every involved value to this medium value. In mathematical sense you must have some values which are lower then this medium value. Therefore you get a negative value (too).

To keep your value physically real you have to name it  Entropy-Difference to Medium-Difference -  that may be negative per mathematically definition.

b) the source of your thought sequence is the comment of SHANNON besides his mathematically definition of Information and Entropy: He excluded consciously all "psychological considerations".

c) So SHANNON an his friends (HARTLEY,...) lost the today much more important  meaning of the term Information "with psychological considerations". Today the term Information got a much wider importance than SHANNON could imagine and so his small mathematically definition is not so important in Information Science. 

d) Your sequence of thoughts are a proof for SHANNON's narrow thinking in only mathematically way - you would create a negative value.

e) Look at my research over more than ten years about this area in http://www.plbg.at

@Peter : Thank you for the answer. I apologize for the ambiguity in the question. I actually wanted to mean a sequence of 4 letters (A, B, C, D)  that has above mentioned probability distributions of of two cases. I wanted to calculate both total information of the sequence and information that each letters hold in the sequence when the probability distribution changes from first to second example. Clearly entropy of first example is highest. I think any deviation from that probability distribution contributes to information. For example in the first example at third position letter C has probability of 0.25 and in the second 0.12. So the gain  in information is log2(1/0.25)-log2(1/0.12) = -1.05 bits. It was this negative information that I was referring. My question was based on paper by Gatlin :http://www.elastyc.unimore.it/fonda/ELBIOM/DNA/information_content_DNA.pdf

where information is presented as change in Shannon entropy. I would be really happy to know what it signifies or my mistakes in calculating information.

@Aakash - Just a thought on this :-" I actually wanted to mean a sequence of 4 letters (A, B, C, D) that has above mentioned probability distributions of of two cases.",  be careful with this. I don't think your problem is so straight forward. Your inferring that the possible states of A,B,C,D are limited by another set of non integer values. Unless I am misunderstanding you (highly probable ;) in this situation if B=0.42 then the solution to ACD will be combinatorial expression that sums to 0.58 in any number of ways according to the precision you have chose (in your example 2 significant digits). 

It seems that only the DIFFERENCE in information is negative for A and B in the A = 0.28, B = 0.42, C = 0.12, D = 0.18 distribution with respect to the uniform A = B = C = D = 0.25 distribution. The information ITSELF for A and B in the A = 0.28, B = 0.42, C = 0,12, D = 0.18 distribution is positive. For C and D in the A = 0.28, B = 0.42, C = 0.12, D = 0.18 distribution the difference in information with respect to the uniform distribution is positive and the information itself more strongly positive. The difference in information for A and B is negative because you would be less surprised if A or B turned up given the A = 0.28, B = 0.42, C = 0.12, D = 0.18 distribution than given the uniform distribution. The difference in information for C and D is positive because you would be more surprised if C or D turned up given the A = 0.28, B = 0.42, C = 0.12, D = 0.18 distribution than given the uniform distribution.

If we’re using the Shannon’s definition of the information: i(A) = -log2(p(A)), as 0 < p(A) 0. Therefore, the information is always positive. For a memoryless source, ve have i(ABCD) = -log2(p(ABCD)) = -log2(p(A)p(B)p(C)p(D)) = -log2(p(A)) -log2(p(B)) -log2(p(C)) -log2(p(D)) = i(A) + i(B) + i(C) + i(D). This equality holds for any values of the probabilities p(A), …, p(D).

In my view the concepts of probability theory need to be treated with proper attention and rigor. (These concepts differ form standard algebra.) My gospel of probability is Papoulis, A., 1991. Probability, random variables and stochastic processes. 3rd ed. New York: McGraw-Hill. Its chapter 15 is a nice Introduction to Entropy.

Physically, of course, information may be negative. For instance, in geology and geophysics information indicators may show that some desired objects cannot be discovred in the area under study. Some introduction of information parameter application in the Earth Sciences are given in RG DataBase:

Eppelbaum, L., Eppelbaum, V. and Ben-Avraham, Z., 2003. Formalization and estimation of integrated geological investigations: Informational Approach. Geoinformatics, 14, No.3, 233-240.

Eppelbaum, L.V., 2014. Geophysical observations at archaeological sites: Estimating informational content. Archaeological Prospection, 21, No. 2, 25-38.

I am glad to see this topic and above discussions!

Yes！  A lie or wrong prediction will  provide negative information because the codewords wil be longer if listener believe it.

Accoding to this formula I(xi;yj)=log[P(xi|yj)/(xi)], if P(xi|yj

 According to Shannon theory, the average information also may be nagative.

Assume actual source is P(X) and  you know it before. Now some one misleads you, saying the source is Q(X). So, optimized average codeword lenth will change from 

Shannon entropy H(X)=- sum i P(xi)logP(xi) to general entropy H'(X)=-sum i P(xi)logQ(xi)

Information from misleading is I(X;yj)= H'(X)-H(X)=sum i P(xi)log[Q(xi)|P(xi)]= nagative Kullback information

A difference in information can obviously be negative, but that is not the same as that information is negative.

You are making mental confusion, Peter! "Difference of Information" is based on SHANNONS mathematical formulas - so you take numbers - OK, ,they can be negative !

But "Information is negative" seems to use the common new form of Information - am I right?

Shannons Information Theory was useful in times of Telegraphy and transmitting channels. Today we need a wider definition of the term Information we need mainly the new term Data. I suppose you mix these different terms - please explain.

I agree with the first answer, not with the second one. Indeed, it is not clear for me why the average length of the codewords (denoted hereafter by Lm) is replaced with the entropy H(x). Normally, Lm = H(X)/H(S), where S = {0, 1} is the binary source obtained by coding the messages of the source X. It is not clear why we must consider H(S) = 1 Shannon in order to have Lm = H(X). Moreover, since H(X) and H(S) are positive values, Lm > 0. If someone misleads you, saying the source is Q(X), the optimized average codeword lenth will be computed with the same relation Lm = H(X)/H(S), where H(X)=- sum i P(xi)logP(xi) will be replaced by H'(X)=-sum i Q(xi)logQ(xi) ≥ 0, not with H'(X)=-sum i P(xi)logQ(xi). However, Lm > 0, no matter which is Q(X). For me, the average information is non-negative.

The amount of information and therefore the information content of the system cannot be negative. However the change in amount of information can.

Explanation: Information is carried by material carriers of information and it is proportional to their number N. The number of material carriers of information aligned in a information string cannot be negative. Consequently the amount of information contained in a string cannot be negative. Please see : Comparative study of entropy and information change in closed and open thermodynamic systems

Marko Popovic

Thanks Aakash,

if you want some more information about this you can find it at: 

http://sciforum.net/conference/ecea-2/paper/3177

There you can find both the fool text and the presentation. Its free 

Yes!　

My reasons are different.

My reasons:

1. Accoding to relative information formula I=log[P(ei|hj)/P(ei)] in classical information theory, 　I

Shannon information must be positive; yet, semantic information may be negative.

Shannon information and Kullback-leibler information are objective information, semantic information is subjective information. When predicted probability or likelihood function P(E|hj is true)=P(E|hj)=P(E|hj  is selected) , semantic information reaches its upper limit Kullback-leibler information.

However, there is a small problem. Indeed, according to theory, I = -log2(p), where p is the probability of the considered event. Hence, 0 < p < 1, so log2(p) < 0. We conclude I = -log2(p) > 0. Can you find any mistake here?

in my life I never found negative information gain, am not sure if there is any, however you said :" average entropy per symbol is 0.5", you know the average is kind of tricky it might be lower than the expected value, higher than the expected or equal to the expected value, if you assume that the probabilities are all (0.25) equal the average is definitely  0.5, but in your example the probabilities are not the same, and the entropy of some of them is slightly greater than the average, because their probabilities are greater than 0.25 (your first assumption) their entropy would be greater than the average (0,5) and therefore subtracting those values from the average (0.5) should definitely give some negative numbers, however adding them together will give the final expected results.

the negative values does not indicate information gain at that points, the total of them indicate the total information gain.

I hope I explained it well.

Ahmad Hassanat ：

Information should be I= log(P2/P1), P2 is posterior and P1 is prior. Only whe P2=1, I=-logP1. If P2

Ahmad Hassanat:

If some one tells you a stock will go up. You believe it and buy some shares. Actually, the stock goes down. Can you still say you received possitive information?

Now, not only me, others also bring likelihood into information formula to get cross entropy.

https://en.wikipedia.org/wiki/Cross_entropy

Semantic information should be measured by mutual cross entropy, which may be negative.

what if p is a pdf of a continuous random variable?

We should consider information about xi after yj is provided.

I(xi;yj)=log[P(xi|yj)/P(xi)]

which may be negateive.