Standard deviation has become the most widely used measure of dispersion of data.
A measure of dispersion can, in the true sense, be regarded as the proper measure of dispersion if the measure is based on the deviations between all pairs of data.
Standard deviation is, however, not based on the deviations between all pairs of data.
Accordingly, the perfectness of standard deviation, as measure of dispersion, is questionable.
Therefore, the question is
" Is standard deviation a proper measure of dispersion of data ? "
Dear Dhritikesh Chakrabarty,
Yes standard deviation called sigma is a proper measure of dispersion of data. For example, in mechanics, a dimension is given with its mean value and the tolerance interval from -3*sigma to +3*sigma.
The probability area from -sigma to +sigma around the mean value is equal to 68 % more than this value the density of probability curve has a change of curvature with an inflexion point at -sigma and at +sigma. With 6*sigma the probability area becomes nearly 100 %.
You say : " Standard deviation is, however, not based on the deviations between all pairs of data. "
However, each x_i is subtracted from the mean (and then things done to it as with the standard deviation), this is the mean of the difference for each pair. Since R is free I'll use it. I have made x just a few random numbers, but you can choose other vectors.
> x sd(x)
[1] 0.9547663
> sqrt(sum((mean(x)-x)^2)/(length(x)-1))
[1] 0.9547663
>
> diffs1 diffs2 round(max(diffs1 - diffs2),10)
[1] 0
>
I believe that the dispersion measure Standard Deviation (S.D) is more appealing because of a solid geometrical interpretation, it is the square root of variance, and in linear algebraic terms, the Euclidean norm of the deviation vector.
the Question arises what is the intuition behind partitioning the observation data vector into the two mutually orthogonal components, the mean vector , which is the multiple of unit vector in the appropriate Euclidean space and is the Projection of the Data vector in this 1 dim. space and the deviation vector, which is the Projection of the Data vector in the Orthogonal Complement Subspace of the unit vector space, which therefore has dimension= n-1, n is the dimension of the data vector space.
the observed response (Data), has functional dependence on the process variables, ideally, if there are no uncontrolled factors playing role, once the process is set at a particular combination of these process variables setting, the output is expected to be constant throughout and conditional on the precise input settings, changing the process variables to a different settings combination will in general result in a different, but constant value again, this is the reason we estimate the mean of the process output through the component of the data vector in the unit vector subspace , if the process is unstable or following a more complex output behavior , this estimate of the mean would no longer be appropriate, in such situation we try to postulate a form of dependence of the output variable on process variables and test for validity of the model through Data based mathematical analysis e.g Regression modeling, here the role of the deviation vector is taken up by the Residual space, of dim =n-k-1, k=no. of Process variables in the postulated Model.
@ Mohammed Lamine Moussaoui,
The tolerance interval and also the confidence interval have been derived in terms of standard deviation.
However, this can not be ta logical justification for standard deviation to be a proper measure of dispersion of data.
Had there been other measure of dispersion, it might have been possible to derive the tolerance interval and the confidence interval in terms of that measure of dispersion.
@ Depopam Ghosh ,
I am willing to be certain (100% confident) on whether standard deviation a proper measure of dispersion of data.
@ Daniel Wright ,
I have not been able to understand how is standard deviation, which is not based on the deviations between all pairs of data, a proper measure of dispersion ?
The explanation is not as clear as to convince that standard deviation, is a proper measure of dispersion.
Dhritikesh, the mean is the sum of the values divided by the number of values. What equation are you using to calculate sd? Do you agree that mean(x)-x is in that equation. That is what the code above demonstrates compares with value with each other. Is it something in the code above that you need explaining. Consider a simple example with x just 1,2,3.
> x mean(x) - x
[1] 1 0 -1
# This is comparing each value with each other, and then takes the mean
# deviation.
> c((1-1 + 2-1 + 3-1)/3,(1-2+2-2+3-2)/3,(1-3+2-3+3-3)/3)
[1] 1 0 -1
> # the first of each pair is 1+2+3 is the sum of all values (6)
> # and the second is just 3 times the value
> c((6 - 3)/3,(6 - 6)/3,(6-9)/3)
[1] 1 0 -1
> # which is
> c(2-1,2-2,2-3)
[1] 1 0 -1
> # or the mean minus each value
>
On another issue, I am not sure what you mean by proper, but there are lots of other measures of dispersion that are more robust. The inter-quartile range is common, but there are lots of others.
@ Daniel Wright ,
It is true that there are lots of measures of dispersion of data. However, each of them has merits and demerits. Standard deviation is one which has been found to be the most meritorious among the existing measures of dispersion. Though standard deviation is the most meritorious, it may not be proper measure of dispersion. A measure can be recognized as proper measure if and only if it can actually measure what is wanted to measure by it.
@ Daniel Wright ,
Standard deviation, which is a measure of dispersion of data, is not based on the all possible deviations between the pairs of values in the data set. In other words, in computing standard deviation we do not take in to account the deviation between each pair of values in the data set. Therefore, it is questionable whether standard deviation can actually measure the dispersion of data i.e. whether whether standard deviation is a proper measure of dispersion of data.
Standard deviation (SD) is the most commonly used measure of dispersion. It is a measure of spread of data about the mean. SD is the square root of sum of squared deviation from the mean divided by the number of observations.
The reason why SD is a very useful measure of dispersion is that, if the observations are from a normal distribution, then68% of observations lie between mean ± 1 SD 95% of observations lie between mean ± 2 SD and 99.7% of observations lie between mean ± 3 SD
The other advantage of SD is that along with mean it can be used to detect skewness.
The disadvantage of SD is that it is an inappropriate measure of dispersion for skewed data.
Yes, Standard deviation (SD) is the most common measure for dispersion of data. Standard deviation is a widely used concept in statistics and it tells how much variation (spread or dispersion) is in the data set.
@ Farangis Shahidzade
The two points
(1) Standard deviation is a measure of spread of data about the mean.
& (2) The disadvantage of SD is that it is an inappropriate measure of
dispersion for skewed data
in your answer justify the significance of the question
" Is standard deviation a proper measure of dispersion of data ? "
@ Abdullah Noori ,
The two points
(1) Standard deviation (SD) is the most common measure for dispersion
of data
& (2) Standard deviation is a widely used concept in statistics and it tells
how much variation (spread or dispersion) is in the data set ,
as mentioned , do not imply that standard deviation a proper measure of dispersion of data .
If standard deviation a proper measure of dispersion of data, as indicated in the above answers, then how can the definition of /principle behind the proper measure namely
"A measure of dispersion can, in the true sense, be regarded as the proper measure of dispersion if the measure is based on the deviations between all pairs of data."
be explained / justified to be valid in the case of standard deviation ?
We have learnt that Mean, Median, Mode are the measures of central tendency. We also know that mean is considered as the best measures of central tendency as it depends on all the observations while median and mode do not depend on all the data set. So, assuming the mean as the best measure of central tendency, it is logical to know how the observations are varying with respect to mean. There is no point in knowing the variability in each and every observation. Practically, it is not possible. Consider that you have 10 observations. This means you can have 10C2 = 45 pairs of observations. Next consider you have 30 observations, then there should be 30C2 = 435 pairs of observations. So, as you increase the number of observations, the pairs of observations will also increase. So to make so many comparisons is not possible. While SD will give you the deviations from the mean which is often of our interest of observation. Why SD is best, many others have answered, so, I will not go again about them but definitely stress you to consider this two points:
1) The mean is the most representative observation of the data set; 2) So, it is logical to know how the other points are varying with respect to mean.
If any body does the research and suggests that the inter-pair observations are better reflection of variation then SD for all the selected sets of data then we all have to agree with it. Until, then we can agree with SD as the best indicator of variation of the data. Further, you know that range can also give you better idea of variability.
I agree with the view of Ramnath Takiar that the arithmetic mean is the best measure of central tendency of data among the five classical measures of central tendency namely arithmetic mean (AM), geometric mean (GM), harmonic mean (HM), median and mode.
However, the best among some may not be a perfect one.
Accordingly, AM may not be a perfect measure of central tendency of data.
AM may not be able to yield the value of the central tendency of data correctly (i.e. with zero error).
Recently, some more measures of central tendency of data have been developed. These are:
(1) Arithmetic-Geometric Mean (AGM),
(2) Arithmetic-Harmonic Mean (AHM),
(1) Geometric-Harmonic Mean (GHM),
& (1) Arithmetic-Geometric-Harmonic Mean (AGHM).
These can be more accurate measures of central tendency of data.
Thus standard deviation, which is based on AM, carries an weakness of being a proper measure of dispersion of data.
In statistics, the standard deviation is a measure of the amount of variation or dispersion of a set of values. A low standard deviation indicates that the values tend to be close to the mean of the set, while a high standard deviation indicates that the values are spread out over a wider range.
Therefore it is a proper measure of dispersion of data
I have understood, from the answer given by Festus Daisi Famule , the logic behind the supposition that standard deviation is a proper measure of dispersion of data.
The logic, as I have understood from that stated by Festus Daisi Famule , is that as the standard deviation of data decreases, the numerical values of the data tend to be close to their mean and vice versa. This implies that when the standard deviation is zero, the numerical values of the data become identical with their mean (which means that there is no dispersion among the data). Conversely, when there is no dispersion among the data then the numerical values of the data become identical with their mean and the standard deviation will be zero.
Now, one point arises at this stage which is that the same logic also holds good if in the formula of standard deviation, mean is replaced by median.
Accordingly, the formula obtained from the formula of standard deviation replacing mean by median can be treated as another measure of dispersion of data.
- Badges
- Science topic