Let the amplitude of a continuous-time (CT) signal x(t) be distributed according to a random variable (RV) X. It's entropy (info. content) is h(X). If x(t) is sampled without quantizing, the distribution of the amplitude of a sampled signal x(n) is still the same as x(t), no matter what the sampling frequency is, implying an apparent violation of sampling theorem. How to resolve this difficulty?
Dear Manoj, interesting point
lets take a counter argument, how you say it be violating sampling theorem?
I'm not sure how that violates the sampling theorem, maybe you can clarify? Here is some food for thought... The distributions are not necessarily the same. Say your random variable is x(t) = cos(t) + v(t) where v(t) is a uniform distribution between -0.1 and 0.1. Now sample the signal at t = n*pi*T so that x[n] = x(n*pi*T) = cos(n*pi*T) + v(n*pi*T). This has a uniform distribution centered around +1 for even samples and a uniform distribution centered around -1 for odd samples. Combining this we have a uniform distribution over the disjoint interval [-1.1, -0.9] and [0.9, 1.1] whereas your original distribution has a continuous support over the interval [-1.1, 1.1]. To me, this illustrates that even without quantization, the distributions can be very different. Any thoughts?
@Imran, the information content of a signal depends only on its amplitude distribution (entropy, h(X) = -integral of -log(f(x))). when the signal is under sampled,but not quantized, its amplitude distribution and hence its information content are same as that of the original signal. but when you under-sample a signal, according to sampling theorem, some information is lost (you cannot reproduce the original signal). in the above scenario, there is apparent violation of sampling theorem
IMHO, h(X) is entropy (info. content) of a single sample, not of all signal. The sampling theorem guarantees (if the sampling frequency is appropriately set) conservation of information content of the total signal. No difficulty!
If the signal is adequately described by Gaussian model appropriate sampling yields a set of noncorrelated (and consequently independent) values. The set entropy therefore equals sum of all individual entropies. Otherwise, you must take into account mutual entropies of dependent variates.
I'm not sure how that violates the sampling theorem, maybe you can clarify? Here is some food for thought... The distributions are not necessarily the same. Say your random variable is x(t) = cos(t) + v(t) where v(t) is a uniform distribution between -0.1 and 0.1. Now sample the signal at t = n*pi*T so that x[n] = x(n*pi*T) = cos(n*pi*T) + v(n*pi*T). This has a uniform distribution centered around +1 for even samples and a uniform distribution centered around -1 for odd samples. Combining this we have a uniform distribution over the disjoint interval [-1.1, -0.9] and [0.9, 1.1] whereas your original distribution has a continuous support over the interval [-1.1, 1.1]. To me, this illustrates that even without quantization, the distributions can be very different.
h(X) is entropy (info. content) of a single sample, not of all signal. The sampling theorem guarantees (if the sampling frequency is appropriately set) conservation of information content of the total signal. No difficulty!..
If the signal is adequately described by Gaussian model appropriate sampling yields a set of noncorrelated (and consequently independent) values. The set entropy therefore equals sum of all individual entropies. Otherwise, you must take into account mutual entropies of dependent variates.
It does not look like this violates sampling theorem. Even though the distribution of the signal does not depend on the sampling frequency, it should be noted that the signal itself does. Two very different sampled signals might have the same distribution. Sampling theorem only talks about the signal and not the distribution.
I agree with Mahmood that the sampling theorem is not violated. The sampling theorem that I am familiar with is about (deterministic) reconstruction of the original signal from its samples. That does not necessarily relate to the entropy of a signal's samples. Are you referring to a different sampling theorem? Can you state it explicitly, so we can all better understand your question/argument?
Manoj,
The sampling theorem says that you may reconstitute entirely a signal if you upper sample the signal and if the signal is sufficiently periodic (not available for distribution).
Entropie is an informations amount evaluation the signal carries.
For example, if you under sample the signal then you lost information and the signal entropy decrease.
h(i)=pi.log(1/pi) decrease because :
pi statistically decrease or increase away from the maximum of h(i) / pi curve.
But you can continue to use your signal. It is like you have noise on the signal.
Sampling theorem is just a limit as :
-------------------------------------------------
Low Bit rate but low efficienty
- Badges
- Science topic