It is bit confusing with the notations. Can you illustrate using a sample dataset?

Let's say you got three sample data like: 42,93, 99

simple mean by normal method would be: (42+93+99)/3 = 78

but, when you use the method that I have described:

w1=/(78-42)^2   w2=1/(78-93)^2    w3=1/(78-99)^2

or, w1=0.00077160493,  w2=0.00444444444,   w3=0.00226757369

hence, average weighted mean= (0.00077160493 x 42+ 0.00444444444 x 93 + 0.00226757369 x 99)/(0.00077160493 + 0.00444444444+ 0.00226757369)

=89.559

As you can see, our result is more weighted towards the cluster point and decreasing the effect of outliers.

I hope,, this gives the idea of what I am asking ??

There is a fundamental issue with your formulation. Let the measurements be of length, so units are meters. Your weights W have  units of m^-2.  And your "new" mean has units of m^-1, that is, inverse of length.

Clearly, you are not calculating "mean" of length! Hope that clarifies you.

Hello Vikash,

It's dimensionally sound.

Let's say we are calculating mean of length. then dimension of w will be m^-2.

then, average weighted mean dimension will be: (m^-2 x m)/m^-2 = m

μ(wx)=wi ∗xi/(w1+ w2+w3)  as you can see from this formula. dimension of wi is m^-2 and dimension of xi is m. 

Oh Sorry Rajkamal, I had missed the denominator part. There is possible limitation though, what if the normal mean matches with any of the data, that is if μx = xi, then wi blows to infinity. 

As mentioned above, your calculated mean is not a mean in ideal sense but it is somewhat biased in the form of weights given as inverse square difference.

in case of... any normal mean matching with any of the data, we will have to take the limit and our value which is matching with mean value will be the mean value....so it won't blows to infinity. (try to see the final expression, it will have infinity both in numerator and denominator part.)

It's the whole purpose of taking weightage average. We want mean to represent the sample and mean should be much closer to cluster points and take less consideration of any outliers. (outliers can be due to some errors in experiment, so their's effect can be reduced by this expression.)

yes but then how do you exactly deal with infinity in both numerator and denominator. Can you show it in the example of 1,1,1,1,1.?

our normal mean is 1.

now weighted mean= limit a approaches 1...  [1x1/(a-1)^2+1 x 1/(a-1)^2+1 x 1/(a-1)^2+1 x 1/(a-1)^2+1 x 1/(a-1)^2)/  [1/(a-1)^2+1/(a-1)^2+1/(a-1)^2+1/(a-1)^2+1/(a-1)^2)

=limit a approaches 1. [5/(a-1)^2]/[5/[a-1]^2]=1

Why do you apply this condition " limit a approaches 1". You are biasing your result then! And I still do not get how you are dealing with infinity in numerator and denominator? Are you bluntly cancelling them?

Please see the attached expression of final weighted mean calculation. Where, mu is (x1+x2+x3)/3.

as you can clearly see, we can calculate the limit of that expression to find the weighted mean, which will be equal to any value if that value matches with normal average or mean.

What do you say about this example?

1, 101

u = 51

w1 = 1/(1-51)^2 = 0.0004

w2 = 1/(1-101)^2 = 0.0001

w1+w2 = 0.0005

mu = 0.0004*1/0.0005 + 0.0001*101/0.0005 = 0.8 + 20.2 = 21

I hope I have done the calculations correctly.

there is one problem with your calculations:

1,101

u=51

w1= 1/(51-1)^2= 1/2500=0.0004

w2=1/(51-101)^2=1/2500=0.0004

thus, weight mean= [0.0004 x 1+ 0.0004 x 101]/[0.0004+0.0004]=51 which will be equal to normal average/mean.

wi=1/(u-xi)^2 (please this is the expression, not what you have used.)

hmm.. thanks for correcting. Apart from the limiting issue, it seems to work, but have you tried it in a combination of positive and negative dataset? I will get back to you.

Hi Rajkamal, 

Good afternoon. I created a quick simulation attached, using 200,000 random values. The set was normally distributed with mean = 0 and SD = 3. We can see a convergence between your weighted average and mean. Then, I did the same with just 50 random values. The mean was closer to 0 than the weighted average. I believe with bigger data sets we will have a convergence for both estimators, but on smaller ones, it is not certain that weighted average will present  the best convergence to the highest frequency. Your thoughts, please?

Good job Ricardo. I really want to play with this formula, but busy with my thesis for next few days.

Let me give you two examples:

Let someone did two experiments. In first set, his readings was 11,87,97.

Then, his normal mean would be (11+87+97)/3 = 65

       Let his second experiment’s readings were 76,77,72. Then, his normal mean would be (76+77+72)/3 = 75

Now, try using weightage mean calculation.

for first set, weightage mean= 82.18

for second set, weightage mean = 75.86

As you can see from the above results, due to some error if there is some mistake in one of the experiment, then overall result is drastically affected.

     if we use normal mean to find the conclusion, then it would be first set will result in less value than the second set.

     However, if we use weightage mean to find the conclusion, then it would be second set will result in less value than the first set. (Which would be the correct assumption.)

If we do a lot of experiments in each set, then it will show same result as weighted mean or normal mean.

Hi Ricardo,

Thanks for showing the simulation.From your results, it is clearly that when we use large number of data points, then scattering of points will not play any role in determining the mean value. (normal mean and weighted mean will play same role.)

    But, if we have very small number of data points, then scattering will matter. And for the sample properties, we have to take more considerations of cluster points (those points which are closer to each other) and reduce the effect of those points which can drastically change your calculation. (As shown in my previous statement.)

It seems that the underlying idea of what you are trying to do is close to that covered in an established topic that goes under the names of "robust estimation" and "robust statistics". This is now a fairly established topic, going back to about 1964: see for example https://projecteuclid.org/euclid.aoms/1177703732  . A search using these terms should lead to more recent stuff that is relevant to you.

JEFF Report 18 by Fritz Froehner, "Evaluation and Analysis of Nuclear Resonance Data" discusses the subject of means and variances from a group-theoretical basis. I don't understand everything he says but it's interesting to read.  

Different methods are discussed in the literature for estimating the mean and variance of a set of measurements of some physical quantity. In a perfect world the one to choose would be the one with the smallest variance. In the real world  the fuzziness of all the relevant data, most importantly the unknown sources of error which are not accounted for in the quoted variances, makes all the calculated means and variances subject to additional variation that is unknown. So users of such uncertainties should be prepared for 5 or 6 sigma if anything really important depends on their accuracy.  If they can't do that they should go back and redesign their procedures to increase their accuracy. That's still my opinion. 

Another point: If someone made two measurements of the same quantity that were 1 and 101 I would simply tell them that they don't have a clue as to what the value is and I would make sure that the quantity being measured was not of any significance to our design procedures.

Dear Rajkamal Kumar,

I like your idea of diminishing the role of outstanding data, which is an especially important feature for measurements/observations exposed to extraordinary random perturbations. However, there are some places, where we cannot aply the term of the 'true' mean value, unless a particular model of the perturbation is assumed and then analysed. Seemingly, a simle and directly correponding to your idea example can be given as follows:

The statistical procedure measures a rv $X$ with pd being a convex combination of the PROPER pd (with pr. p) and the FALSE pd Q (with pr. q = 1- p), in symbols:

$P_X = p \cdot P + q \cdot Q$

Now, for being more specific, let both be normal

${\cal N}(m,v)$ and ${\cal N}(M,V)$, respectively,

where the expectations are different ( $m \ne M$ ) and with variances satisfying $0 < v

This is a correction of an obvious error of the last aswer. The proper claim of PS:

"The following is a sketch of a reason to ASSUME, that Kumar's average $K(x)$ of $d$ entries  $ x = ( x_1, . . . x_d)$  takes the arithmetic average value  (NOT zero), at points, where at least one coordinate equals the arithmetic average, denoted further by $A(x)$:"

I would like to give the following observations:

• The function w(x,µ)=1/(x-µ)2 leads to the weights Wi for the sample xi (with i=1,...,n) in the following way: Wi=wi /(w1+w2+...+wn) for i=1,...,n with wi=w(xi,µ). Obviously all Wi are nonnegativ and W1+W2+...+Wn=1. So the average obtained via these weights is well defined.
• The function w(x,µ) has a singularity at x=µ, so that for xi close to µ the weights Wi are very large. If, let us say m samples xj are equal to µ, then their weights are Wj=1/m  and the weights of all other samples are 0. The new mean µnew calculated from these new weights is µ again. Thus, when the mean equals a sample, then µ is self-consistent, i.e. when it is used for the weights for the samples, then the new average is µ again.
• If the number of samples xi in the "cloud" is large and the distance between them is small, and, therefore, also the distance between µ and the nearest neighbouring sample, then the new mean will move towards this nearest neighbour since its weight dominates. This may be an improvement with respect to the old average or not, it is a matter of chance. In any case, the update of the mean will be small (as has been already pointed out in an answer before). If the distance between samples xi is large, then the situation is less obvious.
• The updated mean depends on the old mean: As an example we consider the samples of the former example: x1=1, x2=19, x3=21, and x4=23. The cloud consists of 19, 21, and 23, the point 1 is considered as a measurement with error. Now we determine a sequence of means in the way, that we start with a µ0 for calculating the initial weights and determine µ1, then we use µ1 to determine µ2, etc. We find with µ0=1/4 (x1+x2+x3+x4)=16:                                            µ0=16, µ1=19.46, µ2=19.21, µ3=19.03, µ4=19.00, ... µ∞=x2.                                        Starting with µ0=20 we get:                                                                                                      µ0=20, µ1=20.13, µ2=20.38, µ3=20.76, µ4=20.98, µ5=21.00, µ∞=x3.                     For µ0=5:                                                                                                              µ0=5, µ1=4.19, µ2=2.97, µ3=1.70, µ4=1.08, µ5=1.00, ... µ∞=x1.                             So we find that the updated new mean strongly depends on the former mean and that all samples xi are self-consistent means.

It will be difficult to draw general conclusions about the quality of a mean updated by this method using w(x,µ) with its singularity without putting further constraints onto the distribution of the samples xi.

However, we may improve the situation by modifying the "sharp" function w(x,µ) into a "soft" function v(x,µ), e.g. to a bell-shaped function like a Gaussian function, or v(x,µ)=1/(1+(x-µ)2/∆2 ) . It represents a generalisation of the function w(x,µ) in the sense, that v(x,µ) leads with ∆->0 to the same weights like w(x,µ). But for finite ∆'s samples xi around µ are treated by almost equal weights whereas only samples xi further away from µ have smaller weights.

For comparison, for our former example we may choose for a first ∆ half the diameter of the cloud:  ∆=(x4-x1)/2=11. Next we perform the calculation for the means with the µ0's before:

µ0=16.00, µ1=18.36, µ2=19.04, µ3=19.19, µ4=19.22,   ...   µ∞=19.231,                                

and

µ0=20,  µ1=19.37, µ2=19.26,  µ3=19.24,  µ4=19.23,   ...   µ∞=19.231,

and

µ0=1, µ1=9.18, µ2=14.56, µ3=17.81, µ4=18.90, µ5=19.16, µ6=19.22, µ7=19.23,   ... µ∞=19.231.

It is easy to show, that the mean µ=19.231 (with ∆=11) is the only and unique self-consistent solution.

Next we may optimise the ∆: There may be several possibilities and criteria, e.g. the one to choose the smallest ∆ for which µ has a unique self-consistent solution: We then find for our samples xi that ∆=2.96 and the mean µ=20.71.

So we may overcome with v(x,µ) problems which we have to face by using w(x,µ).

Dear Werner,

your nice idea deserves further investigation. Morever a question appears:

Is your method applicable to statistical algorithms for problems like the one formulated in my last but one answer above?

More precisely, how to apply the W[erner]-mean for determining the mean of the RIGHT p.d. P, i.e. for neglecting as much as possible the FALSE data coming from  Q. Obviously, this requires suitable formulation of what is given about the parameters  p, P, q  and Q of  P_X etc.

Regards

Dear Joachim,

I was very busy with work recently, therefore my answer is a little bit late.

I think that the method is also applicable to statistical algorithms for problems like the one formulated in your last but one answer above, because of the following observations:

For large values of ∆, in particular for ∆ -> ∞, the weights Wi for the samples xi (with i=1,...,n) are all the same, 1/n, and the corresponding mean µ0 will be the arithmetical mean of the samples xi. According to our assumptions this will be µ0 = p m + q M. If p>q, µ0 will be closer to m than to M and the subsequent µ1, µ2, µ3, ...µ∞ converge towards m. The optimised ∆ will approximate the v (times a constant). If p~q then we will get two self-consistent µ's (one starting the sequence e.g. with µ1=(x1+µ0)/2 and one for µ1=(µ0+xn)/2 etc.).

Regards