I agree with George, but would go further: one shouldn't start using least squares statistics (or even non parametric methods?) before making sure that each data point is a candidate member of the statistical population in every relevant aspect.

To illustrate, examples of known local influences George refers to are the type and processing of the tobacco, the habitual length (tax-dependent?) of the unburned tab end (which retains tar), and reluctance in some cultures to put 'cancer' on a death certificate (in some countries only the ultimate cause such as pneumonia was given, whatever the illness that led to that).

If the objective is to demonstrate cause-effect, it would be better to group data from more homogeneous populations (Scandanavian here); some studies are confined to a single profession.

Two well-separated clusters of data give better confidence limits than the same number of points evenly spread, but only if you can be confident that it's a straight line. For a carcinogen that acts by reacting chemically with DNA, you may well get a straight line. Cancer due to more general toxic mechanisms including plain irritation may show the classical sigmoid dose-response curve. The subject is very complicated, but I guess the overall function could be a combination; clearly there aren't enough points here to do a curve fit, given the scatter.

I didn't find the source of the graph, but it may be that the data points were carefully selected from a larger study in order to provide teaching material full of warnings for a course in epidemiology or toxicology.

Suppose a more complete set of data still looked like that graph. Someone whose job is to forecast research and health care investment requirements may well have to use the data as it stands: they'd call in statistical experts like Jim. An epidemiologist might be more interested in explaining the likely regional differences as reported, in the hope of identifying particular risk factors. As mentioned, someone wanting to study exposure-response (including baseline incidence), and to study toxic mechanisms, may use the information to locate relatively homogeneous populations suitable for that kind of study.

This in an interesting graph.  For a 'simple example' there is a lot going on!

As well as the more statistical issues you have mentioned, I would also want an epidemiologist to be thinking about the relative weight of the data points (Iceland population at the time was only 100,000), the competing risks of different causes of death, relative ascertainment of exposure and outcome in different countries, potential differences in the nature of the exposure, and more generally issues related to ecological comparisons.

I agree with George, but would go further: one shouldn't start using least squares statistics (or even non parametric methods?) before making sure that each data point is a candidate member of the statistical population in every relevant aspect.

To illustrate, examples of known local influences George refers to are the type and processing of the tobacco, the habitual length (tax-dependent?) of the unburned tab end (which retains tar), and reluctance in some cultures to put 'cancer' on a death certificate (in some countries only the ultimate cause such as pneumonia was given, whatever the illness that led to that).

If the objective is to demonstrate cause-effect, it would be better to group data from more homogeneous populations (Scandanavian here); some studies are confined to a single profession.

Two well-separated clusters of data give better confidence limits than the same number of points evenly spread, but only if you can be confident that it's a straight line. For a carcinogen that acts by reacting chemically with DNA, you may well get a straight line. Cancer due to more general toxic mechanisms including plain irritation may show the classical sigmoid dose-response curve. The subject is very complicated, but I guess the overall function could be a combination; clearly there aren't enough points here to do a curve fit, given the scatter.

I didn't find the source of the graph, but it may be that the data points were carefully selected from a larger study in order to provide teaching material full of warnings for a course in epidemiology or toxicology.

Suppose a more complete set of data still looked like that graph. Someone whose job is to forecast research and health care investment requirements may well have to use the data as it stands: they'd call in statistical experts like Jim. An epidemiologist might be more interested in explaining the likely regional differences as reported, in the hope of identifying particular risk factors. As mentioned, someone wanting to study exposure-response (including baseline incidence), and to study toxic mechanisms, may use the information to locate relatively homogeneous populations suitable for that kind of study.

I'm not discuss about the epidemiological considerations of the problem, I think that George and Christopher explain this point very well, but regression is a thrilling field of statistics because you can use it in many many ways. I think that if the main purpose of the regression is to find a model to predict the dependent variable with the maximum efficiency, you must not to remove the “outlier” in this example. If you do the regression with a log-log model with all the data, this results in a significative regression (p=0.012) and the R2 is 0.7081. If you use a second order equation, the regression is better (p=0.0026, R2 = 0.7745). All the resultant equations explains the relationship between deaths by cigarette consumption, and you can choose the better model in order of a significant regression with better R2, with reasonable confident limits, because the model will be the best that fits the data.

Dear Dr. James,

Considering the responses above, I my opinion the use of regression or even correlation is not the best way to describe these data. Based on the flow chart from Scheiner and Gurevitch, 1993, p. 12, fig. 1.2, data collected from the experiment about hypothesis of the study must be first submitted to an EDA (exploratory data analysis) and then to additional analysis, if any. In my opinion, statistics is just one method to help the researcher to organize his idea based on his hypothesis. The decision has to be the result of his understanding and experience about the case under study. The main subject in the study is the researcher who has to make appropriate decisions. The R-sq of 89% is suggesting that some data was still out control during this survey. In the current study, data from States or provinces, for example, should complement the data base. As these data arouse my interest, I felt free to attach the EDA to this response, looking for a feedback. I clustered these data about eleven countries two times though the first one had a group with only four countries (sorry and the minimum recommended is 5. However, it can illustrate the idea I have applied when the variance is high than expected. In 2014, the Nature methods published several papers about the use of exploratory data analysis, (box-plot-and-whiskers, in particular) and how they can improve our understanding about the study. In summary, I disagree in eliminating outliers from the analysis, because the outlier is like a meteorite: it is just one within several other meteorites but in contrast it is large, heavy and travelling right into the Pacific Ocean.

Best regards,

Walter

Thanks to James for this very interesting question. I agree with George and Christopher. It should not be disregarded that this case study has a clearly linkage with epidemiology. In that field of investigation it is well known that, in several circumstances, using aggregated data may drive to the so called "ecological fallacy", a concept assigned to Robison (1950) and, over any statistical consideration, it may be evoked here. For this reason, data like those used here, should be studied using a different approach than the classical regressive one: e.g., by using the meta-regression analysis (see for example Higgins and Thompson; Stat. Med.2004; 23:1663-1682).

Another way of looking at this question (as posed by James) is to separate the chart from the issue.  The chart does what charts do: sets out a two dimensional scatter plot.  Without applying any numerical tests at all, it is clear that the US is an outlier in this two dimensional chart.  But of course the issue - smoking and putative smoking-related illnesses - is actually a multidimensional concept.  'Countries' are proxies for the behaviors of their populations, of which cigarette sales per capita are only one measure of tobacco consumption, and 'deaths by lung cancer' are the product of a complex path from death to death certification to death registration to national statistical analysis, all subject to diagnostic fashions, ascertainment biases and so forth.  So rather than get tied up in alternative statistical approaches, a more fruitful way forward would be to explore which aspects of the pathways by which the data are gathered vary between countries.  Understanding variation, and the reasons for it, is not entirely a statistical question.

Interesting question and interesting answers for the range of reaction - what immediately came to my mind  was " means on means analysis is meaningless"! as the mean does not reflect the within group (country) relationship

Aitkin, M., Longford, N. (1986), "Statistical modelling issues in school effectiveness studies", Journal of the Royal Statistical Society, Vol. 149 No.1, pp.1-43. 

I am talking about the ecological fallacy and its counterpart  - the atomistic fallacy - see

https://www.researchgate.net/publication/23959447_Revisiting_Robinson_the_perils_of_individualistic_and_ecologic_fallacy

It would be interesting to use the Seven countries study to look at this as they collected individual data across countries 

http://en.wikipedia.org/wiki/Seven_Countries_Study

Article Revisiting Robinson: The Perils of Individualistic and Ecolo...

I think that the above scatterplot is an excellent example of how over-over-estimated has become the regression analysis. Dear All, why should there exist a common functional relation for all those extremely different countries? What is the legitimation of such a (hidden) hypothesis? Nothing. We just assume that we can explain the whole universe by doing an orthogonal projection and then we are trying to find what was gone wrong...

On my opinion, for a regression model for application, firstly the number of samples should be big enough; secondly, the influence of heteroscedasticity should be considered. For evaluation of a regression model, we presented 6 statistics, you can see details in the attached file. Sorry, only English abstract was provided, the paper are written with Chinese.

Making a brief documentary research about this example, I found that it was used (and is still useful) to teach how the outliers affects the regression line (without any epidemiological inferences). D. Freedman in many documents discussed about if regression models are helpful, successful and if the statistical assumptions are correct or not in each particular situation. The original data (Doll and Hill, 1950) were not analyzed in function of regression, it was established an association relationship between the variables, and then epidemiological use of regression in this case is unuseful, there wasn't any underlying or occult hypothesis.

I want to remember the original question of James above: what are the most important considerations for a regression evaluation?

For that reasons, I maintain my point of view that if a regression procedure is the response that the researcher wants to establish a predictive model between the variables and the scatterplot shows some atypical points, maybe the linear regression is not the answer and is better prove another non-linear solutions. You must to prove the assumptions before and if the data are non-continuous like this example (both are counts in order of number of deaths and number of cigarettes), you can try a non-parametric regression.

Interesting topic and nice discussion:

It is well known that the linear regression has 5 assumptions. Fitting a regression model is complex. I agree the interesting discussions which have been posted. I also agree this is a teaching example. Except for considering high leverage and high influence, the heteroscedasticity, which should be also considered. It is clear that the variance of deaths is a function of cigarettes. In such a case the transformation on death should be considered. From the scatterplot, it can be seen:

Larger death has larger variance

Larger cigarette  has larger variance

Hence the following transformations could be considered:

Deatthsq=sqrt(deaths)

Deathlog=log(deaths)  

Deathcig=deaths/deathsq,

Using 1) the R-square is 0.557, which is for sqrt(deaths) with cigarette

Using 2), the r-square is 0.540, which is for log(deaths) with cigarette

Using 3), the R-square is 0.167, which is for Deaths/sqrt(cigarette)

 However we should not compare the R-square directly, because they are for different outcomes, we should transform them back to deaths, then re-calculate the r-square. After the transformation, the variance  heteroscedasticity is improved a lot.

 See attachment:

In addition, in practice, the deaths and cigarettes are from different size population, The estimation from different countries have different variance of itself. The estimation from larger populations is more accurate. The weighted analyses should be considered. Simple delete USA will loose important information, even it has high leverage and high influence. Definitely, use cigarette only as regressor is also not good enough. It is ok for teaching and discussion.

This is a very interesting discussion. Yuanzhang shows a very interesting transformation if we can assume the data is sound and can reasonably explain the research question.  The problem is, as I see it, this data should not be used for this question to begin with.  The issue is the time span and locations from which the data was collected. The data looks at a time span of 1930-1950. WWII would have removed many of your Western European 1930 smokers prior to 1950. 

I agree with everyone that this is an excellent teaching tool addressing regression diagnostics. Martin's critique sums up the substantive message that should accompany accompany the message about the limitation of the method... perhaps with the additional point that, since the observational units are represented using summary measures, that there are additional concerns regarding the variability...

This looks like a problem for which linear regression is inappropriate. I would first try a non linear function. Then I would ask some cultural and stratification (demographic) questions about who (proportions) in the population smokes.  

The first point I would make about influence, the second about confounding. There are all sorts of confounds here. I imagine the death rate from lung cancer per capita might could be higher in the UK because cigarette consumption increased and the number of healthy males decreased (both due to WW2). This might be wrong but there all sorts of country level confounds ...

I would like to add a littele more about heteroscedasticity, if we group the counties by cigarette into five groups (2,2,2,2,3 ) respectively, the scatter plot of cigarette and the standard deviation and the cigarette is shown almost perfect linear relationship (see attachment).

It means V(deaths|cigarette)=cigarate^2*σ2, then V(deaths/cigarette)= σ2, which is constant.

Hence we use Y=deaths/cigarette, ciginv=1/cigarette, as transformation

Deaths= α1+β1*cigarette +ε1 (1) classic

Model y=α2+β2*ciginv +ε2 (2) to fit the data,

When transform it back, we have

Deaths= α2*cigarette+β2+cigarette*ε (3)

The interception of α2 in Model (2) is the actual slope, the cigarette effect β2, the slope in model (2) is actual the interception, in Model (3), it show the variance is proportional to the cigarette.

Using transformation model (2) to fit the data with and without USA, the difference of the two regression lines are much smaller. It implies, if controlling the heteroscedasticity, the effect of USA as outlier is reduced .

α2=0.352, stderr=0.071, comaring with model 1, β1=0.228, stderr=0.70. for with USA.

I have a different take on this scatterplot: the data support that "A countries high cigarette consumption is necessary but not sufficient for a countries high death rate by lunch cancer". The data-analysis method Necessary Condition Analysis (NCA) is now available to analyse such triangular scatterplots. NCA draws a ceiling line on the data instead of a central line through the data. I used the NCA calculator on the website www.erim.nl/nca to get attached scatterplot with ceiling line. The calculation shows that the necessity effect size is 0.5, which is considered a very large necessity effect. I am curious to hear your comments after exploring the website about this approach.

http://www.erim.eur.nl/centres/necessary-condition-analysis/

Perhaps "necessary but not sufficient" may not always be true if you get into a situation where you have Simpson's Paradox. But that ("necessary but not sufficient") might also be of practical consideration. 

Jan -

I see you have a paper, "Comparing NCA with OLS."  What about WLS?  I find OLS drastically overused, even when it is clearly not true.  You might want to consider that sometime, if you haven't already.  

Best wishes  - Jim

PS -  It looks like the "ceiling line" is very much sample size dependent, but I suppose that you account for that somewhere. 

James,

Regression lines with OLS and WLS (which addresses heteroskedasticity) both decribe central tendencies, whereas the NCA ceiling line describes a maximum tendency (maximum value of Y for a given X). For NCA heteroskedasticity is not an issue: it only tells that the necessary condition is not sufficient (for a given X there are also values below the ceiling).

The paper on comparing NCA with OLS is not about considering a central trend line (like OLS or WLS regression line) as an alternative for the ceiling line of NCA. This working paper (to be presented next week at the European Academy of Managment conference in Paris, see attached link for the Abstract) compares the fundamentally different logics and results of NCA and OLS. OLS is based on additive logic: each X can change/produce Y, and X's can compensate for each other. This means that each X is sufficient but not necessary for changing Y. However, NCA uses necessity logic: a (level of) X is necessary but not sufficient for (a level of) Y. NCA understands cause-effect relations in terms of "necessary but not sufficient". It means that without the right level of the condition a certain effect cannot occur. This is independent of other causes, thus the necessary condition can be a single bottleneck, critical factor, constraint, disqualifier, etc. In practice the right level of necessary condition must be put and kept in place to avoid guaranteed failure to achieve a certain level of Y. Other causes cannot compensate for this. 

The paper on comparing NCA with OLS makes the point that NCA does not suffer from "omitted variable bias" (OVB). OVB is one of the main potential endogeneity problems of OLS and similar models:  when an incomplete model is used to predict the outcome the predicted regression coefficients may be flawed. NCA is immune for leaving out other causes because the necessary condition operates in isolation from the rest of the causal structure (that is the reason why it is necessary). 

Best wishes,

Jan

PS. Indeed just like in other data analysis methods, the results depend on having  a proper sample (see the section about "NCA limitations" on the NCA website www.erim.nl/nca under FAQ). We are working on formulating confidence lines to address sampling variation.

https://www.erim.eur.nl/centres/necessary-condition-analysis/personal-pages/james-lebreton/