Hi Steve,

The essence of sampling is to save cost and time. If you have the resources and time to include every eligible participant (census) in you study, why not? Go for it. Even better for your study inferential power!!

Regarding P value, you could go for a lower p value, say only P

Confidence intervals are not applicable - these are estimates of the range within which you are likely to get the answer if your repeat a random subsampling in the same manner as the original protocol. But you are using all the sample, so there is no randomisation possible.

Frequentist statistics will guide on how likely the result will be against the null hypothesis (what you observe can be explained purely by random chance) so it is still useful but should not be over-interpreted. You cannot repeat the sampling again so there is no room for a reiteration of random chance. If you have multiple year data then there is more can be done as there will be some variation year to year., but different years are not independent samples! They can help understand the underlying volatility of the data though. While there is no sampling error, there may be some magnitude of measurement error present depending on the rigor and compatibility of the data collection processes.

However, assessing your results in terms of practical significance is more useful. You have exact answers for the population, not estimates, so the question is if the difference you observe big enough to make a practical difference.

Rather then p values you should cite the size of difference observed and the impact that difference had and try to indicate how meaningful that difference is. Bayesian methods may be more useful to explore.

A critical thing to remember for whole population analysis is that the results are not generalisable at all. They only apply to that set of data. Sub sampling is designed to extrapolate results from a sub population onto the whole population, but that is not possible here.

What matters is getting a good, representative sample of your population. In fact, trying to collect results from a larger sample size can add costs – without significantly improving your results

You have an entire population but an N of only 23 institutions, which is a small number. Start by describing the characteristics of the institutions that you find interesting without weighting. Then I suggest you weight your analyses using number of students or teachers depending on the question, to focus on the population impact of different policies. In other words, how many students or teachers were there, what policies were in place, and what is the outcome in terms of completion etc. And I would also suggest as did someone else that you try a Bayesian model.

If you use the whole lot of students that there are, you are taking a census and what you get are parameters; then, all you need is to describe their performances since there will be no statistics. Watch out for your unit of data collection (e.g., individual students) and unit of analysis (e.g., the institution) , to avoid ecological fallacy., when writing the report.

Confidence intervals are designed to account for sampling error, and as you note you have no sampling error. Whatever you find correctly describes the 23 institutions at the point of time that you measured (assuming no measurement error; nonsampling error is often greater than sampling error). Of course, if you wish to generalize to more than the 23 institutions, or to more than the time of measurement, then there is a sense in which you are drawing a sample. When dealing with models, it is still appropriate to measure goodness of fit; even if your data are without error (an unlikely prospect), that doesn't mean the model is without error.

Steve,

The responses thus far are pretty much on point, with John Sinclair's being quite helpful. Statistical inference is all about trying to estimate parameters for the full population. But, you have the parameters! Hence, no need for assumptions, asymptotically correct methods, and prayers for robustness. The numbers

I like the suggestion of James Beattie to focus on the magnitude of various effects--trying to link those to thresholds that policy-makers would perceive as 'noteworthy.'

Good luck with your work!

If you are using whole population then it will census not sampling. Sample surveys are used when it is not possible or practical to conduct a census to count each individual of an entire population. Census provides a true measure of the population. While taking whole population, the problem of generalizability will become irrelevant. If unit of analysis is organization level then N =23 is not sufficient for any kind of statistical tests. It will be turned into a practitioner’s project rather than an academic research.

You may be interested in conducting analysis within your census—such as comparing curriculum designs across terminal AA degree programs versus AA degree programs designed to facilitate transfer to BA/BS programs. in such cases, all the assumptions of any statistical analysis, such as t-tests, would apply; sample versus census issues are irrelevant to assumptions of interval measure, homoscatasticity, etc.

In addition to all said above my simple question is on resources; Do you have enough resources - money, manpower to take the whole population?

If that really is the population to which your conclusions will apply, go for it. With the availability of (1) big data and (2) higher CPU and RAM processing power, I expect we will see more population testing rather than sample testing which may in turn force new thinking in statistics along the lines of the above responses.

Population testing puts the researcher on the front foot in looking for unanticipated deviations that are invisible when using sample testing. This is a wonderful development.

Because sample sizes are often not large enough I have been doing population testing for some years in my own field of audit testing with surprising discoveries precisely because of the limitations of sampling.

Thanks to all for the informative discussions. Indeed, what I have is the population and that is truly all I need for the interests of this project. The goal for me to is to determine whether their is a relationship within the specific environment of Ohio's funding formula between business curriculum internationalization and the variety of variables that constitute the funding formula's metrics.

As each state effectively has its own funding formula for higher education, there's not an urgency to try to extrapolate the results to other states or to community colleges, at large (indeed, this is already an identified limitation of the research). However, from a practical level, if a significantly strong correlation does exist, it might be logical to conclude that (at the very least) testing this phenomenon elsewhere would be a worthwhile endeavor.

I suppose my main question at this juncture is simply, "What is the best method going forward?" There are still obviously covariate factors that will need to be filtered (as much as possible) to attempt to isolate the impact that business curriculum internationalization has on the dependent variables. Does MANCOVA still apply for such a scenario or is there a better alternative for accomplishing this task when working with an entire population rather than just a sample?

I suppose I'm having some difficulty wrapping my head around the process here as in my statistics training (which is admittedly a bit rusty), we only dealt with situations where we were using samples, not entire populations.

Thanks again for any and all help!

Steve,

The fact that you have a census rather than a sample is largely irrelevant. The key factors are to develop appropriate hypotheses, models to test them, and statistical measures. One thing you need is variation, and this may be where focusing on a single state may hurt you (even with a census). If all of the colleges are the same (i.e., following a common state policy), then you have no way of measuring the impact of the policy. What you need is variation over time (e.g., the state changed its policy, or some colleges changed their programs at different times than others), or variation across colleges (perhaps in implementation; this often occurs, and in that case you need measures of the implementation). Alternatively, you might consider a multi-state study, looking for differences in outcomes that are related to differences in state policies.

Variation could be found in the N=23 population; e.g. some schools may be performing better than others or have higher or lower staff/student ethnic diversity. My concern would be that with only 23 observations there may be difficult to come up with insights that policy makers can rely upon with some confidence (and as conventional stats are not applicable you have no way of gauging confidence of the results). Even a broken watch shows the correct time twice a day; so you need more than one observation of it (with more than a minute apart) to figure out that the watch is broken :-) So if I were you, I would do the descriptive stats for my population in the areas of interest and then use these stats to pick a subset of schools for further/deeper qualitative/case-based study. The schools I would pick for further study would be those that are exemplary in some way of the identified variety; e.g. the best/worst performers, rural/urban, max/min diversity, and so on (whatever the census data identify as better reflecting the identified variation). Maybe focus on three schools in each dimension of interest min-middle-max. Maybe order the schools across the set of relevant dimensions and pick those to focus in terms of a particular combination of dimensions of interest. As you have small number (N=23) a population-based analysis cannot be of great policy confidence; but if you deepen in some cases and are able to identify for example the mechanisms involved in translating the policy, budget or whatever; this could inspire more confidence to formulate policy on the basis of those insights than now cover the mechanisms involved in your population instead of the mere description of its characteristics given the policy stimuli of the previous year or the previous X amount of years.

Assuming an expected effect size of 0.5 and for p=0.05, N=23 has a statistical power of only 0.2 which is only one-quarter of the conventional power of 0.8. In other words, if there is in fact an effect, the chances of detecting it is very low and this may lead to a wrong conclusion of no effect. This obviously is a disadvantage to the researcher as the effort might be wasted just because of small N.

If your research question is purely descriptive or retrospective then you don't have a sample. If your aim is to predict, then you do have a sample. Organizations and people change over time and certain interventions or conditions may have a different effect from year to year. Be reminded that you are not just drawing a sample from institutions, but samples from all possible observations to be made on these institutions in past, present or future. Also be reminded, that some of your measures are likely to be contaminated by measurement errors. That is a source of uncertainty and therefore inferential statistics may well be needed.