Hi,

I notice that you are almost taking the whole population (9 out of 10 schools) which is more than enough. Then you are aiming at 40% of each school students, which is not less than 120 per school. This is adequate number to have reliability. 

The other important issue is the validity and generalizability, which i think it is met with your sampling technique. 

The above answers which considered the sample as the 9 schools themselves, I dont think that is correct and this is not your target.

In general the % is not the issue - they key is the extent to which the pupils who have answered are a representative sample.

.I am sorry to say that if the focus of interest is the school climate then you have a sample of 9 which will not be a large sample to say something about a variable at the school level and you will be really struggling if you have alternative hypotheses at the school level that you want to evaluate ( eg is it school climate or school resources that is driving attainment?). Putting it simply if you have a hypothesis about schools you need lots of schools - it is more powerful to have 10  pupils from a 100 schools than than 100 pupils from 10 schools although I appreciate the great deal of difficulty in increasing the data collection and doing so across multiple schools.

Some of the issues in what is a multilevel research design are considered here

http://www.bristol.ac.uk/cmm/software/mlwin/mlwin-resources.html#datadesign

see power

Thank you for your answer.

We only have 10 schools in the city of Limassol and so 9 is 90% of the senior high schools (lyceum) in Limassol. I think based on that that I will be able to draw general conclusions about the school climate in Limassol schools, won't I? (Cyprus is an island with less that 40 senior high schools in total).

The purpose of the study is to (1)  assess the school climate in each of the schools so that the school unit can strategically plan changes to improve the school climate (2) to assess the general picture for Limassol high schools.

Based on this, in a school of 400 students, how many students would you suggest would give a reliable picture of the school climate?

I would be very grateful for your thoughts on this.

The issue is not 9 out of 10 but that 9 is a small number -sampling theory is in general about  n - the absolute  number-  not the ratio of n/N.

That being said I would have though that a 100 pupils will give you a reliable estimate of a school level attribute provided that they are a representative sample, and not say all the 'compliant' children. 

My issue is that you will then have only 9 observation if you want to use this to account for say pupil level outcomes.

The basic advice in the multilevel literature is a minimum of 25 pupils in 25 schools  with a preference for 25 pupils in 100 schools.

"One important issue which arises is the sample size necessary to carry out a satisfactory analysis. It is difficult to provide general guidelines here, but a key factor is the number of higher-level units, for example schools. Experience suggests that studies with fewer than 25 schools are unable to provide very precise model estimates. Studies with more than 100 schools, on the other hand, have been able to provide useful insights. Likewise, for the estimation of the intercept or slope parameters for a single school, at least about 25 students are required."

http://www.bris.ac.uk/media-library/sites/cmm/migrated/documents/introduction-to-multilevel-models1.pdf

Thank you very much for going to the trouble of explaining this to me. I very much appreciate it.

Kind regards, Miranda

This is a complicated design. The most conservative approach would involve alpha-testing the sampling methodology and survey at one school and evaluating the findings prior to system-wide implementation.

As Dr. Jones comments, the sampling methodology is crucial--the same method must be used at every school if the results from different schools are to be meaningfully compared (e.g., by examining confidence interval overlap separately for each point estimate across the schools). Stratified random sampling is the best alternative, and the same number of students at each grade level (freshman, sophomore, etc.) should be sampled. With 100 students per school, this would mean 25 randomly selected students at each grade level. However, I would think that it is important to assess data for males and females individually, which would reduce this to 12 students for each combination of grade level and gender. This may be too few (given greater than marginal response heterogeneity) to support inter-school or inter-category comparisons (e.g., via examination of CI overlap).

Combining data from multiple samples can induce a form of Simpson's Paradox. This includes not only combining data from different schools, but also combining data from different grade levels, genders, ethnic categories, socioeconomic categories, and so forth. For example, imagine a school in which the seniors (or the boys, the minority ethnic group, or the poor) are unhappy, but the freshmen (or the girls, the majority ethnic group, or the wealthy) are happy. Combining the data would produce a result suggesting the combined sample was ambivalent--a finding that may be unrepresentative of any of the constituent groups.

Hi,

I notice that you are almost taking the whole population (9 out of 10 schools) which is more than enough. Then you are aiming at 40% of each school students, which is not less than 120 per school. This is adequate number to have reliability. 

The other important issue is the validity and generalizability, which i think it is met with your sampling technique. 

The above answers which considered the sample as the 9 schools themselves, I dont think that is correct and this is not your target.

Thank you to you all.

Miranda

These are all interesting conversations but probably not helping you!

I should stop pontificating and ask you what you are planning to do once the data are collected - the key is what is your target of inference; to whom or what are you trying to infer; what are you going to do with school climate once data have been collected?

At the risk of causing more confusion I see that you skill set includes Rasch modeling. From a multilevel modeling perspective, this is a one factor multilevel multivariate model with a set of discrete outcomes. In that context you could regard your research as a complex structure with multiple items nested within individual pupils nested with classes within schools. This does not have to be a strict hierarchy as  pupils could be nested within one or more classes ( a multiple membership model).

On Rasch and multilevel see

http://scholar.harvard.edu/files/sampson/files/2003_socmeth_multivariate.pdf

https://core.ac.uk/download/files/153/6303142.pdf

https://ppw.kuleuven.be/okp/_pdf/DeBoeck2011TEOIR.pdf

on different types of structure and classifications see

http://www.bristol.ac.uk/cmm/software/mlwin/mlwin-resources.html#mlmbkgrd  

This is a thesis I supervised that goes onto some detail behind the approach

http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.556725

Kelvyn

Kelvyn

My skills set should not include Rasch modelling as I have not claimed to have expertise in Rasch. I co-authored 3 articles which included Rasch methods - but my colleague was the Rasch expert. My interests are in Educational research, foreign language anxiety, motivation and school climate. But I have a basic understanding of Rasch and shall read the resources you kindly added here.

I am an insider researcher (as an upper secondary school teacher) and my aim is to produce an empirical evaluation of School climate. I believe that this is essential for evidence-based school improvement. Since students’ perceptions influence their emotional and psychological engagement in the learning process, it is vital that they be consulted in such an evaluation. This research aims to provide  data to be used in assessing the school climate and areas for improvement so that the school body has an opportunity to reflect on the students' perceptions of the school climate and to consider ways to bring about improvement for the following school year.

Upper senior high school students (which is 3 classes here) will be administered a questionnaire, which is the result of an extensive review of the research literature and in careful consultation with experienced school councillors.

The questionnaires consist of 20 items (statements) for which students (1) indicate the extent to which they agree with the statements (2) indicate the need and urgency for improvement in aspects of the school climate.

The research has two levels:

1) to provide descriptive statistics for each school, in the form of a case study.

2) to provide descriptive statistics on the students' perceptions of school climate in Limassol Senior high schools (in other words I plan to combine the data from the schools)

Details:

There are 10 senior high schools in Limassol. I have negotiated access to 9

In each I have aimed to collect data from randomly selected students (4c. 40% of the 1st graders, 40% of the 2nd graders and 40% of the 3rd graders).

In one school I am having trouble getting 40% of the population, hence wondering what the threshold would be for reliability of results.

Thank you very much for your time.

Miranda

That is much clearer- so your target of inference is each of the  9 schools and how each one may change over time. So what matters is the number of pupils in the sample in each one;  and you appear to be doing fine on that. It is the absolute number that counts not the percentage and that they are a representative not a biased sample of pupils.

Thank you again

There is no straight forward answer to your question. You can follow suggestion by prominent researchers as to how many sample to obtain for a particular study. You can take a look at Krajie-Morgan (1970) and also Hair et al.(2010) regarding the sample size for the strudy.

The percentage of the population is not usually the issue, unless the sampling rate is very high. Normally it is the sample size which determines the standard error. Sometimes very low sampling rates (

Beware of pooling groups from different schools (or within different schools) without testing for homogeneity of variance and balance by external factors (such as demographics).  Also, in many cases, such as Hispanic populations, there is an upward or downward response scale bias in questionnaire items.  If you end up with groups that are too small, you can use bootstrapping to see whether small sample size is undermining your results.

Thank you very much for your advice. With a homogeneous population, as is the case in Greek Cypriot schools (perhaps as having to speak Greek is a prerequisite on enrolling), do you believe it is 'safe' ?

I will learn how to use bootstrapping as a precaution.

Miranda

Are there differences in social class or household income?

I would say no (or at least certainly not a significant one).

More specifically, I would say that in 7 of the 9 schools I am investigating (on an individual basis and then collectively as 'State run senior high schools in Cyprus) no. In the other two, within a student population of 600 in one school and 800 in the other, there may be up to 10 students who come from a particularly wealthy background. And some of these may or may not be in the sample population.