At the open discussion part in my recent write-up "A brief introduction on Shannon's information theory" (full-text can be found in my contributions), there is a newly defined quantity, similar to Shannon's channel capacity. I wonder the relation between the quantity and Shannon channel capacity? To be specific:
1. is the quantity well defined?
2. If yes, will the quantity be strictly larger than the Shannon channel capacity for some channels?
3. If 2 is yes, does that mean Shannon limit can be broken?
Furthermore, it can be viewed as a pure mathematical problem without referring to information theory.
thanks
Research A Brief Introduction on Shannon's Information Theory (old version)
I am not sure that I understand your setup, but it appears that you allow codewords of different lengths. The normal setup is block coding where all input codewords have the same length and all output codewords have the same length. One may use variable lenght codewords on the input side and one may consider channels where letters may be deleted or inderted. According to the sorce channel separation theorem this does not change the capacity in the sense that for long sequence it asymptotically optimal to compress the source as much as possible and then apply a block code with long blocks to obtain the desired error correction. According to the source channel separation theorem one does not need to take the channel into account when the source is compressed and does not have to take the source into account when the error correcting code is designed. The source channel separation result holds when there are only one sender and one receiver. In MIMO systems it does not hold.
Hi Peter, thanks for your comment. It is not about considering codewords of different lengths, it is about if all codewords share the same statistic property. In my understanding, the traditional definition of channel capacity (as in Shannon's original paper), all codewords share the same statistic property, that is, each symbol used in the codewords appear with the same probability which is determined by the probability distribution maximizing the mutual information. However, it seems not necessary to put such a restriction on codewords. That is why we defined the new quantity to compare.
For a product of channels the optimal input distribution is a product of input distributions for the individual channels. If the channels are also identical the optimal inputs will also be identical. To see this it might be better first to find the optimal output distribution characterized by the Gallager-Ryabko theorem that states that the optimal output distribution achieves minimax redundancy. Note that there is a unique optimal output distributions while the there might be several optimal input distributions.
Thanks, Dr. Harremoeos.
Problem remains unsolved, more comments are welcome.
Furthermore, it can be viewed as a pure mathematical problem without referring to information theory.
Update: this question is partially answered in the paper "A brief introduction on Shannon's information theory", arXiv:1612.09316 [cs.IT].
https://arxiv.org/abs/1612.09316
- Badges
- Science topic
- Similar topics
- Mathematics
- Applied Mathematics