Quite likely, but it depends on what you want to do. I assume you have a dataset of different samples that you apply the above procedure to?

What is the distribution of results that you expect? Data reduction methods can be used to capture the most common recurring patterns. Is there one threshold that optimises across the dataset? This is the simplest solution.

Or if you need a more adaptive threshold look for a rule that gives an optimal result.

If the absolute values between samples do not matter and it is just the variability with each then perhaps first max-min scaling then threshold at a certain fraction of the range?

But if the values mean something between samples then you need a rule that is defined based on your whole dataset.

Have you tried setting different percentiles rather than the plain mean?

Quite likely, but it depends on what you want to do. I assume you have a dataset of different samples that you apply the above procedure to?

What is the distribution of results that you expect? Data reduction methods can be used to capture the most common recurring patterns. Is there one threshold that optimises across the dataset? This is the simplest solution.

Or if you need a more adaptive threshold look for a rule that gives an optimal result.

If the absolute values between samples do not matter and it is just the variability with each then perhaps first max-min scaling then threshold at a certain fraction of the range?

But if the values mean something between samples then you need a rule that is defined based on your whole dataset.

Have you tried setting different percentiles rather than the plain mean?

Thank you Dr. James Renwick Beattie for your comments.

Yes you are right, we are working on a dataset with different samples. Your suggestion regarding adaptive threshold is quite correct and we are working on it. The distribution varies between each 3x3 block inside the same sample, and also varies from sample to another. Arithmetic mean and max-min scaling were all applied in the literature.

Dear James Renwick Beattie;

I hope that my colleague Wasseem Al-Obaydy answered your questions. do you have any idea about a statistical measure better than the arithmetic mean?

Best regards

Sarmad

Thank you Aparna Sathya Murthy;

A statistical measure other than the mean is required, all help appreciated?

Best regards

Hi dear,

I agree with James. In chemical analysis, the average is considered as a quick method not precise and also inexpensive to know the order of magnitude. But, the statistical method is more precise and more expensive.

With my best regards!

Adel

Medians or modes might also be useful, as might other descriptors: standard deviation, skew, kurtosis. Do you get a different answer if you use the median of the 8 differences rather than the arithmetic mean? What happens if you look at the differences in variance (or standard deviation) rather then means?

Graphical: plot the data, though with a 3x3 table I am not sure this will help.

Simulation based on the existing data. Of course the simulation is only as good as the original data. If sample sizes are small in the original data sets, then this is a bad idea.

Is it appropriate to think of this as a circle, with data in the center, and a value at 0, 45, 90, ... 315 degrees? Directional (or circular) statistics might be useful (https://en.wikipedia.org/wiki/Directional_statistics)?

Thank you @Timothy A Ebert;

Both the Median and slandered deviation gave a lower accuracy in the result than that of the mean, on the other hand the variance accuracy gave a bad results. so, we are seeking a better statistical measure.

Best regards.

3rd Quartile is appropriate as a threshold.

Is it possible to combine several statistical measures together, rather than using them individually?

@Tom. Yes it is possible, but I would suggest that it is unwise if I understand correctly what you propose. So I try the mean, no luck. I then try median, standard deviation, skew, and kurtosis. Still no luck. I then try (mean/median)^(standard deviation/(skew+kurtosis)) and various other combinations of varying complexity. This is a fishing expedition, unless there is some very good theory to explain why "this" combination is the reasonable path. Finding a significant combination and working backwards to a reasonable theory is not a good idea unless you have independent data to confirm that the new theory is reasonable.

At some point one needs to decide that it is time to refocus the research. Accept a failed project, or look at the project very cynically and ask if it is useful. Given that another researcher in some other country is considering this same experiment, what would you advise them to do? If you were that researcher, would you believe that advice? Knowing that something doesn't work is sometimes as useful as knowing that it does work.

@Timothy. I was thinking more along the lines of computing the statistical distribution of the 8 elements and expressing it in terms of probability density function. Then you could use a probability value (or inner-quartile range) as the variance threshold.

@Tom. That is much better than where I was going. Thank you for clarifying.

@Sarmad. Is there any outlier in your data set? If there is any outlier then simply mean is not appropriate here. In that case you may use median however it does not consider all observations. In that case you may use weighted mean. Please have a look at http://naturalspublishing.com/Article.asp?ArtcID=11225

@Md. Moyazzem Hossain. from where could I acquire the full text of "Proposed Mean (Robust) in the Presence of Outlier" please. Many thanks for your help.

@Waldemar Koczkodaj. Could you please elaborate more on your suggestion or direct me towards a reference that helps. Many thanks.

For outliers you could do a sensitivity analysis. This is some form of leave one (or many) out, and recalculate the model. This is usually followed by trying to predict the observations that were left out, and quantifying the successful placement of these value. If your results are entirely dependent on the presence/absence of one value then you can focus on that one value.

This approach could lead you to maximum accuracy methods (see: ODAJournal.com). It might be possible to do something similar in R. The RCOR package seems like a good place to start. However, you should know that I have not used R for this purpose. This was the first thing that seemed close in a quick search of the internet.

Dear colleagues.

the question can not be A STATISTICAL MEASURE BETTER THAN THE MEAN ?.

but what measure corresponds to show XXXXX ... for this type of data. ??? looking to respond to this goal!

There is NO better measure, but the appropriate measure for what I need to answer.

From what has been read, the problem is not the arithmetic mean, but the type of data obtained, its representativeness and the presence of atypical values.

But I could use a weighted measure as a function of weights given by the sample spaces

beware of atypical values, they should investigate why it is considered atypical.

3 reasons

1. Measurement error given by the instrument.

2. error due to human error,

3. incidental data.

the decisions of what to do with the outlier depends on one of those 3 reasons and on the sample size

Sarmad,

Test your data sets for skewness . As one said already, the best measures of central tendency and variability about that measure is the one that is most appropriate for the data being analysed. if the data is normally distributed, then mean and standard deviation are appropriate. If the data set is skewed, then median and quantiles are to be preferred. Once you have results using your most appropriate measure of central tendency and dispersion around it, then it comes down to interpretation of the results.

George

Proposed Mean (Robust) in the Presence of Outlier

@Md. Moyazzem Hossain, thanks for suggesting (again) to us your paper - that's a nice choice for weights in the weighted mean. However, this paper seems somewhat incomplete, since it only compares the proposed choice of weights for the weighted mean to the arithmetic mean. What about all those other means: https://en.wikipedia.org/wiki/Mean ?

Md. Moyazzem Hossain has added another approach to the classical problem of "an apparent outlier (or outliers)" -- I personally applaud the effort, as it both 1) draws attention to the issue, and 2) offers an alternative that might be appropriate under some circumstances. There are a multitude that have been offered over the decades as noted by Tom Arjannikov with wikipedia reference. One might added to alternatives use of truncated values (e.g., median or mean of middle quartile), or use of selected transformations (e.g., 1/x which has a long history of use with reaction time data). My major point is

to consider if one might obtain some advantage in the choice of method for handling outliers. For example, I have frequently found a transformation of the form Ln(x+a) with "a" chosen to eliminate neg or 0 values (say 1 in the example that stated this thread) will tend to linearize data that represent a hiost of kinetic, chemical and/or physics related studies (a point well made re: choice of transformations by Scheffe, Analysis of Variance, 1959 Wiley & Sons). Box also made this point repeatedly in a time-series class -- based on his expereince in the chemical industry, but also relevant to a range of problems -- air quality, aero-engineering, econnmics as well as a varienty of behavioral Likewise, 1/x or 1/(x+a) sometimes also does the same (but it should be noted that these transformations oft have similiar power-strenght (cf. Tukey, 1957 Comparative Anatomy of Transformations)

The choice of dual purpose transformation to handle outliers often may be quite apparent a-priori just from theory based considerations (e.g., Log-linear response models for toxic exposures and drugs.

Sarmad; I am appalled at many of the recommendations you have been offered. Lets examine a little bit of history of statistics.

First: Assuming that your data is a random sample of some population (and no one has made an error in entering the scores of that sample), then the mean of any sample you take from that population gives you an unbiased estimate of the population mean. We should realize that it really doesn’t matter what the shape of the population is, the mean of a random sample of that population will be an unbiased estimate of the population’s mean. This is one very important reason for using the arithmetic mean. Also we know that the mean gives the “best” estimate of the sample’s scores from the standpoint of minimizing the average error and the average error squared. That is, if one subtracts the mean from each score (in order to get a measure of the “error” one would make if one assigned the mean score to each score in the sample) the average deviation will be zero.

Second: If one now gets the mean of the squares of these deviation scores, one obtains the variance of the sample (i.e., a measure of how much the scores differs from the Raw score mean). While this measure doesn’t provide an unbiased estimate of the population of scores, if one divides by N-1 instead of N, you do obtain an unbiased estimate of the variance of the population. That is, if you took an infinite number of samples of size N and computed the average of those unbiased sample variances, you would get the variance of the population. Also the square root of the sample’s variance provides the “standard deviation” (and the square root of the sample’s unbiased variance provides the unbiased standard deviation of the population).

Third: If one divides each of the sample’s deviation scores by the sample’s standard deviation, one obtains the samples “standardized scores.” If one now cubes (takes to the third power) the sample’s standard scores and gets their mean, one obtains the measure of the skewness of the sample’s scores.

Fourth: If one takes each of the sample’s standard scores to the fourth power and obtains their mean, one gets the measure of kurtosis of the sample.

Fifth: Assume one has a second set of standard scores which, say, measure a different attribute of the same individual members of the sample. If one multiplies each sample’s standard score for the two attrributes measured, the mean of those products will be the sample’s correlation between the two attributes measured. And the sample’s correlation will be an unbiased estimate of the population’s correlation between those two attributes.

Note that each measure we have discussed (i.e., the mean, the variance, the skewness, the kurtosis, and the correlation) are all means of either raw scores deviation scores or standard scores. The mean, variance and correlation involve only linear transformations of the raw scores while the measures of skewness and kurtosis involve non-linear transformations of standard scores. The strength of these measures come, not because they are all linear transformations of the raw score, but because they are all means and their strength is why these measures are the choice of most statisticians.

If I believe there is a non-linear relationship between two variables (X and Y), I would not hesitate to get the linear relationship between the two variables, but I probably would use X, x2, x3, x4, ... as potential predictors of Y using a Multiple Correlation program to see if there was a significant non-linear relationship. And I see nothing wrong with using other non-linear transformations like Ln(x+a) (such as Bittner suggested) as a possible predictor of Y. By the way, one can use a multiple correlation program to analyze the data from an ANOVA experiment and you will obtain the exact same results.

Finally, let’s talk about “outliers.” You have received some good advice about looking for “outliers,“ but we must talk about what we mean by an outlier. In most of the statistical programs I wrote, I always had the program look for the highest and lowest scores of each variable. In some programs, I would have the program keep track of which cases had scores lower or higher than some expected “boundary scores.” That is, as the cases were being processed by the program, it would print out what I thought should be flagged as outliers. In this way, I could go back to the raw data to see if someone had entered an erroneous value. One certainly needs to remove any human data entry errors. If the outlier data was caused by the subject not properly following directions, then one might decide to say that one is interested only in the behavior of subjects that will correctly follow the test procedures and thus eliminate that subject.

Using statistics, one should be interested in the relationships between various things and whether the relationships found are real or if they happened by chance alone. Here we get into tests of significance (z-test, t-tests, F-tests, Chi-square, etc.). Most of these are taught as being based on assuming underlying populations of scores that are normally distributed, but that is not actually necessary. If one samples sufficient data, the shape of the means of those samples will approach normality based on the Central Limit Theorem. And most of these tests are simply based on the difference between the means of various subgroups of data and the overall unbiased variance of the data.

In summary, I would be extremely cautious to use some term other than the mean as the “most representative score” for any sample, but don’t include any true outliers that were caused by someone’s error.

SARMAD has indicated that the mean, in his initial scale does not meet his needs, so our suggestions have been toward alternatives -- and apparently he has yet to find what he needs (see above his responses to query suggestions). One of the problems in our responding adequately is the lack of context behind his query, ii.e. what is the purpose for the operations behind his query??? After looking at his described operations -- and his relatively recent research publications -- one might guess that his query is related to his research regarding his " finite difference method (FDM) for solving TPFBVP (with two point boundary requirements if my quick reading is correct). Hence, based on that, my past experience in emprically fitting differential conditions -- with empirical boundary conditions -- I suggested some care in selecting the scale in which to work. AT the same time, it should be recognized that "3x3 dithering" about a local value -- can provide a local linear model which may provide for 2-D control applications which might be his purpose as require some (but not all) of the operations described (and it also would profit from scale choice). But this is just guessing.

In any case, I would suggest SARMAD outline the goal behind the operations, so as to be best advised.

AB

Many Thanks to everyone for your answers, this question came from a discussion with a colleague with a different research focus, and I with Wasseem Al-Obaydy described about the problem as much as It is allowed to be shared.

Best regards

Sarmad.

https://www.bis.org/ifc/publ/ifcb32k.pdf

http://www.lps.uci.edu/~johnsonk/CLASSES/MeasurementTheory/Hand1996.StatisticsAndTheTheoryOfMeasurement.pdf

http://www.ilo.org/wcmsp5/groups/public/---dgreports/---stat/documents/presentation/wcms_195747.pdf

Dera Sarmad A. Jameel Altaie,

Why have you taken the arithmetic mean of the 8 elements to obtain the threshold value? Is it based on some principle? First, be clear on this point.

I have not understood about what you are going to do i.e., what is your objective.

If you want to test something then test it on the basis of 8 values. It will yield more accurate result.

This threshold is applied on the differences to produce binary stream. Is there a statistical measure that would be better than the mean to calculate the threshold and give a better description to the texture of the original data?

Thanks to Sarmad A. Jameel Altaie for recommending my answer.

Kindly describe your problem point-wise clearly.

If your data is plagues by outliers oftentimes using the median is a better way to set the threshold.

Moreover if you have enough data to calculate the standard deviation it is usually better to cast the data into the form z(t) = (x(t) - mean(x(t)))/sd(x(t)) and define your thresholds on the z statistic. That has at least some meaning, versus just some set threshold which value might change every sample.

You don't describe your problem very well, but also it is better to use autoregressive estimates of the mean and standard deviation, which you update each sample. That will smooth out random fluctuations in your data sets over time.