I recently came across an experiment designed to investigate the relationship between two variables: X and Y.
The experiment was conducted in four steps, as follows:
o First, the X scale was divided into three sections: Low, Medium and High.
o Second, a single experimental unit was selected from each section.
o Third, values of X and Y were observed for each of the 3 experimental units. This yielded a total of three (X,Y) pairs.
o Fourth, the three (X,Y) pairs were plotted. The plot showed that ONE of the three pairs was not consistent with the hypothesized relationship.
Based on this evidence, the authors concluded that variation in Y is not “well predicted” by variation in X.
Am I alone in questioning the validity of this conclusion?
Not really. They got 3 times lucky! They have not performed a 'statistically significant' analysis (square root of N + 1).
What is the "hypothesized relationship?" There should be some model quantitatively linking the values of X to those of Y.
The problem I have is that you say X is one of "low", "medium" and "high", and in this case we cannot define any quantitative relationship, what in turn makes it impossible to to judge how well the observed data would fit (to what, precisely?).
I see two possibilities to define a relationship. One is to map X to a set of three numeric values X->X' and then a hypothesized relationship could be formalized as a function of X', line a simple linear relation Y = mX' + b, a quadratic relation Y = a(X'-b)²+c, an exponential relation like Y = a*eX' and so on. The other possibility is to code X as two indicator variables: Xmed = 1 if X=medium and 0 otherwise; Xhigh=1 if X = high and 0 otherwise, and to use a linear model like Y = a + b*Xmed + c*Xhigh. All these models represent the relation in the coefficients (a, b, c), so one can specify what these coefficients should be and then see how well the observed data corresponds.
Maybe there are other ways - but these are the possibilities that come into my mind.
It depends on what the expectation for the relationship is.
For example, imagine you have an analytical machine set up to determine the amount of a pollutant in water samples. The machine is calibrated. It is expected that the raw machine response can determine the concentration of the pollutant in an unknown sample to a high degree of accuracy. During the course of analyzing unknown samples, you run a check standard of known concentration. If that check standard if off by some threshold, say 10%, then you know something is wrong. So one observation outside of expectations can indicate that the relationship doesn't hold.
In a similar vein, in this types of analytical machines, the relationship between the machine response and the concentration of the pollutant is linear, and that the r-squared will be, say, at least 0.975. It would take running only 3 standards to see that this relationship is not holding.
One consideration is that I am talking about physics and chemistry systems in which it is possible to decrease the effects of confounding variables to a negligible effect, and that relationship is known through many many attempts at perfecting the analytical method.
Even so, the relationship will hold only over a given range of either the x or y variable. So, if you are outside this range, you would wrongly conclude that the relationship doesn't hold in a practical way.
Also, once you get outside the realm of chemistry and physics, relationships are more complex, confounding factors much more recalcitrant, and r-square values are typically much lower. So you need many data points to understand the relationship of the x and y variables.
A sample size of 3 works if there is no variability, and the relationship is linear within the span of the data. Outside of those conditions, I don't see three data points as being useful. It is not even a useful pilot study. However, you will find other things that slip through the review process and editors. Just because it is published does not make it true, correct, or even relevant. If you read enough scientific papers carefully enough you will get a collection of "how did this get published?" There is also no justice/fairness: "this got published" so why are you complaining about "this" in my manuscript? If you submit your manuscript enough times you will probably find a set of 2 or 3 reviewers that are sufficiently disinterested or too tired to catch problems. Recognizing this problem, some journals ask reviewers if their review can follow the manuscript. Thus when I resubmit my manuscript to a different journal, the review will follow and (presumably) someone will check to see if the negative comments were addressed.
Without a better understanding of the question asked and the system studied it is not possible to suggest a more reasonable experimental design.
Since I am familiar with the original case about which the question was asked, I'd like to provide more context. I will not reveal the original study, but provide a close analogy.
Suppose we investigate how fast people can carry various stones. We use stone-weight as predictor. We then pick three stones, by weight: a light stone, a medium stone and a heavy stone. Now we ask people (say N=29) to carry each stone once (in random order) for some short distance, and measure the speed. We would predict that the average carrying speed is highest for the light stone, and slowest for the heavy stone. Indeed we find that the average carrying speed for medium stone is higher than that for the heavy stone. However, the average speed with the light stone is actually lower than that with the medium stone. Surprise! So, the researchers said that stone-weight is not a good predictor of stone-carrying speed.
To validate that last statement, we would need to try with many more stones (not just three). I think that is agreed.
But the fact that average speeds did not align as predicted tells us something.
Maybe there are additional factors that influence the carry-speed. Maybe the shape of a stone is also important, etc.
In such a setting, misalignment of three data points is not negligible.
It does not PROVE that a given predictor is bad (not even statistically),
but it raises questions for further research.
These are great responses.
Both Mangiofiaco's example (analyzing samples with known levels of a pollutant) and Flor's example (predicting carry distance for stones of known weight) illustrate an important special case: when the X variable can be measured with little or no error.
In the referenced experiment, however, the X variable is not measured precisely. It is this more general case that I am wondering about.
- Badges
- Science topic