My view is that statistical analysis is about decision making (say about size of effects) under conditions of uncertainty. Consequently, I would often using statistical proeceures when using a complete census, but the numbers are quite small in the 'table' being analysed.

To take a specific example - I am interested in whether a particular area has a high risk of mortality and I have all the records of death in a particular year to calculate the relative risk of (observed death/ expected death) * 100; so that 100 is an average area, or equivalently the same as a national average. Say I calculate the estimated risk to be 223 (based on all the deaths that occur in that year) – that is quite elevated and suggests that I might want to transfer monies away from other areas to help alleviate the problem. My problem with this (in spite of it being all the data) is that this is a descriptive statistic and needs to take account of natural variation if I am going to get at the parameter of the underlying risk. I would argue that I need to examine the simple statistic in relation to the standard error of the relative risk which is given by (Observed^0.5/Expected) as there are small numbers involved and there will be natural variation in the observed counts from year to year. Indeed, when I examined the data for MSOAs in England and Wales (the country is divided into c7000 of these); I found this sequence

E02006843 (Sheffield): SMRs, 2002 - 2012 All Cause mortality , Males

44, 223, 231, 140, 168, 69, 79, 88, 76, 77, 99.

that is fivefold differences in risk from year to year that are occurring naturally- some years this is a great place to live, in others it is quite a high risk, but nothing is really changing; it is just very uncertain. My one off year of 223 despite being a census does not contain sufficient information to make a good judgement. Calculating the level of uncertainty – the standard error and the associated confidence intervals, would have alerted me to the problem. We generally underestimate this sort of variability but need to take it into account in any analysis. As Kahenman, D( 2011, 5) in Thinking fast, thinking slow -has written “Even statisticians are not good intuitive statisticians- they consistently underestimate uncertainty”.

When R. A. Fisher introduced the term 'parameter' [ in "On the Mathematical Foundations of Theoretical Statistics", Philosophical Transactions of the Royal Society of London, Ser. A. 222, (1922) 309-368] he saw the need for two terms (the descriptive statistic and the parameter]):

"it has happened that in statistics a purely verbal confusion has hindered the distinct formulation of statistical problems; for it is customary to apply the same name, mean, standard deviation, correlation coefficient, etc. both to the true value which we should like to know but can only estimate, and to the particular value at which we arrive by our method of estimation ..." p. 311

He would recall, "I was quite deliberate in choosing unlike words for these ideas which it was important to distinguish as clearly as possible." (letter (p. 81) in J. H. Bennett Statistical Inference and Analysis: Selected Correspondence of R. A. Fisher (1990)).

In the mortality example I am interesed in the parameter of the relative risk for a place and the descriptive statistic for a particular year is not telling me all I need to know; I have to consider it in relation to the underlying uncertainty.

You may thing that these small number issues would be obvious, but consider this. In other work using the census, the data being analyse is a set of counts for a very large table for usual residents aged 16 and over in England and Wales by highest level of qualification, by ethnic group and by age as of census day, 27 March 2011. There are 4 ethnic groups, 4 age categories, and 5 levels of educational qualification thereby giving 80 ‘cells’ or groups of people with common characteristics repeated for 7201 Middle Super Output Areas. It may be thought that there is little need from modelling and an inferential framework when we are dealing with a population of millions and we have the complete population of 7,201 MSOA's and every adult in England and Wales.

Indeed, Gorard (2013))for example, strongly argues that a modelling and inferential approach is not needed in these circumstances. For him inference should be confined to inferring from imprecise samples to true, but unknown population values. However, for me the imprecision arises not as a result of sampling variation but because of natural or chance variation. In the total 576,080 (80 by 7201) cells while the mean count is 54, the median is only 5, and the lower quartile is just 1. If we were to calculate a Standardised Rate by dividing the observed by expected and following convention multiplying by 100) we would typically get a rate with a great deal of uncertainty – with such small numbers very slight changes could result in large changes to the rate Thus the observed count should be considered to be the outcome of a stochastic process which could produce different results under the same circumstances. It is this underlying process that is of interest and the actual observed values give only an imprecise estimate of this. [ Gorard (2013, 54) “all traditional statistical analysis, including all tests of significance, and the use of standard errors and confidence intervals… are, of course irrelevant when the full population of cases is used since then there is no sampling variation”. Research Design: Robust approaches for the social sciences. London: SAGE}

Here are two examples (both on Research Gate) where quite spohisticated modelling and inferential procedures are used with 'census' data -we are not trying to generalise to the observed population but about the parameter of interest when there is quite a lot of stochastic variation in the data

Multilevel Modeling of Social Segregation; George Leckie, Rebecca Pillinger, Kelvyn Jones, Harvey Goldstein https://www.researchgate.net/publication/225303320_Multilevel_Modeling_of_Social_Segregation

and Multilevel perspectives on modeling census data; S V Subramanian, Craig Duncan, Kelvyn Jones,Environment and Planning A

https://www.researchgate.net/publication/23539033_Multilevel_perspectives_on_modeling_census_data?ev=prf_pub

FInally one could take a Bayesian view on this.

The Bayesian approach, has a very different perceptive on inference and does not concern itself with repeated samples from hypothetical super-populations. Rather, it asks the question what is the degree of support for a parameter, combining the researcher’s beliefs about the parameter and the evidence from the data? It is also worth stressing the difference between (standard frequentist) confidence and (Bayesian) credible intervals. The former as a frequentist concept and describe the anticipated performance of the interval across repeated samples. If you had 100 repeated samples from the population and you constructed 95% confidence intervals for each sample, you would expect 95 of the intervals to include the population parameter. The probability statement therefore applies to the data not to the parameter. This is also the case for a null-hypothesis significance test. It is the proportion of repeated samples that would give a test statistic equal to or greater than that of the data. In contrast, the Bayesian credible interval is more natural and applies to the parameter and not the data, giving a 95% probability that the parameters falls between the lower and upper values. That is we are interested in the weight of the evidence supporting a certain size of effect. In some cases even with a census there is so much natural variation around that we have to be careful about our judgements.

A long answer - but I think it is naive to think that if you have observed what you think is everybody then automatially only a descriptive account is needed.

Article Multilevel Modeling of Social Segregation

Article Multilevel perspectives on modeling census data

Dear Mahmoud,

I believe the answer to your question lies within the explanation you put with your question. If the population based data were collected accurately then the p values for the significance should have a sense that could be translated in a meaningful way.

Best-

Dear Amritlal, Thank you for your response. But let me explain my question. We know the true values of the parameters (means, proprtion,...) only if we have census data from the entire population. So, we usually draw a random sample from the study population and make inference on the unknown population parameters. The meaning of the p-value and significance tests in this situation is well-known. Now, in some studies the study population can be small (like university professors in a specific field in a university or top-level managers in a company,.. ) and it is possible to collect data from the entire population. Then no randomness in the data and no (or negligible) error in the results but some still use confidence intervals and significance tests and deal with their data as if it is a random sample from unlimited population. What is the interpretation of confidence intervals for example in such a case?

Thank you Ahed but if the study population and study sample are the same then there are no random (statistical) errors . What is the meaning of statistical significance then?

Mahmoud:

In general, the larger the sample size the smaller the standard error. The smaller the standard error, the smaller the p-value (as p value is related to standard error). The smaller the p-value, the more significant the test result. That said, if you have sampled the entire population, or nearly so, there is no need for "drawing inferences" from "part to whole" or "sample to population" (i.e., no need for "inductive reasoning") .... you have actual, verifiable conclusions and not not rely on hypothesized conclusions. I hope this answers your original question. You seem to have arrived at the same conclusion per your final remark above in response to Ahed.

And as for those who sample an entire population, or nearly so, and then go about building confidence intervals, this is a meaningless exercise unless one takes the view, in your example context, that the population is not merely the professors at the particular university, but all professors at all similar universities or some such, and this population is significantly larger than that at the university in question.

Dear Mahmoud,

There is no need for testing in a census studies as each single observation is a reflection for a fact in the population and you can draw a decision directly from the descriptive statistics. The testing used only when you have a random sample from the entire population. Therefore there is no meaning for the p value in a census surveys.

Thank you Ram.. I agree with all what you said in your both responses ..

You're welcome, Mahmoud. All of this business of statistical hypothesis testing and building confidence intervals, is murky, quite controversial, and heavily misused. Perhaps the increasing availability of Big Data in many contexts, coupled with more recent emphasis on Big Data Analytics, will help alleviate such problems.

Best, Ram

Dear Dr. Mahmoud: Thanks for sharing the question and subsequent discussions.

when we know the universe we know everything. So no errors, no statistical inference.

Thank you Subrata.

My view is that statistical analysis is about decision making (say about size of effects) under conditions of uncertainty. Consequently, I would often using statistical proeceures when using a complete census, but the numbers are quite small in the 'table' being analysed.

To take a specific example - I am interested in whether a particular area has a high risk of mortality and I have all the records of death in a particular year to calculate the relative risk of (observed death/ expected death) * 100; so that 100 is an average area, or equivalently the same as a national average. Say I calculate the estimated risk to be 223 (based on all the deaths that occur in that year) – that is quite elevated and suggests that I might want to transfer monies away from other areas to help alleviate the problem. My problem with this (in spite of it being all the data) is that this is a descriptive statistic and needs to take account of natural variation if I am going to get at the parameter of the underlying risk. I would argue that I need to examine the simple statistic in relation to the standard error of the relative risk which is given by (Observed^0.5/Expected) as there are small numbers involved and there will be natural variation in the observed counts from year to year. Indeed, when I examined the data for MSOAs in England and Wales (the country is divided into c7000 of these); I found this sequence

E02006843 (Sheffield): SMRs, 2002 - 2012 All Cause mortality , Males

44, 223, 231, 140, 168, 69, 79, 88, 76, 77, 99.

that is fivefold differences in risk from year to year that are occurring naturally- some years this is a great place to live, in others it is quite a high risk, but nothing is really changing; it is just very uncertain. My one off year of 223 despite being a census does not contain sufficient information to make a good judgement. Calculating the level of uncertainty – the standard error and the associated confidence intervals, would have alerted me to the problem. We generally underestimate this sort of variability but need to take it into account in any analysis. As Kahenman, D( 2011, 5) in Thinking fast, thinking slow -has written “Even statisticians are not good intuitive statisticians- they consistently underestimate uncertainty”.

When R. A. Fisher introduced the term 'parameter' [ in "On the Mathematical Foundations of Theoretical Statistics", Philosophical Transactions of the Royal Society of London, Ser. A. 222, (1922) 309-368] he saw the need for two terms (the descriptive statistic and the parameter]):

"it has happened that in statistics a purely verbal confusion has hindered the distinct formulation of statistical problems; for it is customary to apply the same name, mean, standard deviation, correlation coefficient, etc. both to the true value which we should like to know but can only estimate, and to the particular value at which we arrive by our method of estimation ..." p. 311

He would recall, "I was quite deliberate in choosing unlike words for these ideas which it was important to distinguish as clearly as possible." (letter (p. 81) in J. H. Bennett Statistical Inference and Analysis: Selected Correspondence of R. A. Fisher (1990)).

In the mortality example I am interesed in the parameter of the relative risk for a place and the descriptive statistic for a particular year is not telling me all I need to know; I have to consider it in relation to the underlying uncertainty.

You may thing that these small number issues would be obvious, but consider this. In other work using the census, the data being analyse is a set of counts for a very large table for usual residents aged 16 and over in England and Wales by highest level of qualification, by ethnic group and by age as of census day, 27 March 2011. There are 4 ethnic groups, 4 age categories, and 5 levels of educational qualification thereby giving 80 ‘cells’ or groups of people with common characteristics repeated for 7201 Middle Super Output Areas. It may be thought that there is little need from modelling and an inferential framework when we are dealing with a population of millions and we have the complete population of 7,201 MSOA's and every adult in England and Wales.

Indeed, Gorard (2013))for example, strongly argues that a modelling and inferential approach is not needed in these circumstances. For him inference should be confined to inferring from imprecise samples to true, but unknown population values. However, for me the imprecision arises not as a result of sampling variation but because of natural or chance variation. In the total 576,080 (80 by 7201) cells while the mean count is 54, the median is only 5, and the lower quartile is just 1. If we were to calculate a Standardised Rate by dividing the observed by expected and following convention multiplying by 100) we would typically get a rate with a great deal of uncertainty – with such small numbers very slight changes could result in large changes to the rate Thus the observed count should be considered to be the outcome of a stochastic process which could produce different results under the same circumstances. It is this underlying process that is of interest and the actual observed values give only an imprecise estimate of this. [ Gorard (2013, 54) “all traditional statistical analysis, including all tests of significance, and the use of standard errors and confidence intervals… are, of course irrelevant when the full population of cases is used since then there is no sampling variation”. Research Design: Robust approaches for the social sciences. London: SAGE}

Here are two examples (both on Research Gate) where quite spohisticated modelling and inferential procedures are used with 'census' data -we are not trying to generalise to the observed population but about the parameter of interest when there is quite a lot of stochastic variation in the data

Multilevel Modeling of Social Segregation; George Leckie, Rebecca Pillinger, Kelvyn Jones, Harvey Goldstein https://www.researchgate.net/publication/225303320_Multilevel_Modeling_of_Social_Segregation

and Multilevel perspectives on modeling census data; S V Subramanian, Craig Duncan, Kelvyn Jones,Environment and Planning A

https://www.researchgate.net/publication/23539033_Multilevel_perspectives_on_modeling_census_data?ev=prf_pub

FInally one could take a Bayesian view on this.

The Bayesian approach, has a very different perceptive on inference and does not concern itself with repeated samples from hypothetical super-populations. Rather, it asks the question what is the degree of support for a parameter, combining the researcher’s beliefs about the parameter and the evidence from the data? It is also worth stressing the difference between (standard frequentist) confidence and (Bayesian) credible intervals. The former as a frequentist concept and describe the anticipated performance of the interval across repeated samples. If you had 100 repeated samples from the population and you constructed 95% confidence intervals for each sample, you would expect 95 of the intervals to include the population parameter. The probability statement therefore applies to the data not to the parameter. This is also the case for a null-hypothesis significance test. It is the proportion of repeated samples that would give a test statistic equal to or greater than that of the data. In contrast, the Bayesian credible interval is more natural and applies to the parameter and not the data, giving a 95% probability that the parameters falls between the lower and upper values. That is we are interested in the weight of the evidence supporting a certain size of effect. In some cases even with a census there is so much natural variation around that we have to be careful about our judgements.

A long answer - but I think it is naive to think that if you have observed what you think is everybody then automatially only a descriptive account is needed.

Article Multilevel Modeling of Social Segregation

Article Multilevel perspectives on modeling census data

In the case of a census (or in thought experiments) the p-level may still be used as a guide for deciding whether or not an observed effect (e.g., difference in means or proportions) is worth interpreting. The comparison involves the distribution that would have occured if randomization were present and I do not wish to interpret a result that is no better than chance (even though chance has not operated). The use of such theoretical universes is quite common in stat practice and I think is appropriate in your situation.

Thanks for the explanation by Dayton. Is this a case of so called super population or a underlying probability model of that which generates the population? Can we have an example where such an idea is implemented.

I think this notion is applied routinely, but perhaps unwittingly, in many studies. Case-control studies have no randomization but p-values are always reported. Similarly, for prospective studies that are not experiments (cohort studies). And, all kinds of fancy stats are reported for time-series analyses but there is no random selection or assignment. Clinical trials with randomization would seem to have a basis for interpreting significance levels but Ss are virtually never sampled originally - available samples (convenience samples) are randomly assigned among conditions but there is no "real" population to which to generalize. In any case, generalization to a broader population depends upon logical arguments (my Ss are like the general population because etc. etc.), not on sampling considerations.

Many thanks my colleagues for all your useful responses.

Very meaningful

p-values in data analysis refer to the information to give sicnificance to a experimental results