I am analyzing data from a field study. The main goal of this experiment is to understand what intervention can help increase students verbal reasoning.
There are 4 kinds of interventions (say Factor A,B,C,D)in total, the combination of which result in 6 experiment groups.
group1: no treatment; group2: A; group3: A +B; group4 A+B+C, group5: A+B+D, group6: A+B+C+D.
We randomly chose some schools and assigned them into 6 different groups. Students' verbal reasoning skills were tested before and after the intervention for each school.
I suggested simply compare group1 v.s. group2 to determine effect of A, group2 v.s group3 to determine effect of B; and use a 2*2 ANOVA on group 3~6 to determine main and interaction effect of C and D.
My co-author suggested that we should check out analysis for nested model. I am confused as whether or not this is a nested design and how should i change my analysis if so?
Any suggestion is highly appreciated!!
Yes, a multi-level model with students nested under schools maybe a good option for your study. Using such a model you maybe able to fit a single model to your entire dataset with the consequence of increased statistical power. You will also be able to evaluate differences across groups. The following tutorial explains this well using data from students: https://rpubs.com/rslbliss/r_mlm_ws
Hi, I am suggesting that in your oral examination you will be asked what is the difference in randomly choosing schools and randomly assigning students to a treatment.? which would R. A. Fisher recommend and why? What effect does your choice have on the results of the study?.For sure, If I were the reviewer of the paper I would want to hear your answer. Best wishes, David Booth
Hello Bowen,
To the good advice posted earlier (by David Eugene Booth and Usman Rashid ), I would add this-
Yes, it's a multi-level (hierarchical/nested) model. Schools were the experimental unit (you assigned schools, not students, to condition). What that means is that your net sample size of some number of schools (one or more per treatment combination under consideration) will likely not be very sensitive, unless treatment effects are strong or unless you have a lot of schools involved. (This despite the fact that schools vary far less than do individual students on any single variable.)
Given the treatment conditions and how they were combined, I think I would code results as 0/1 for each of the four treatments, using a regression framework: Group 1: 0,0,0,0
Group 2: 1,0,0,0
Group 3: 1,1,0,0
Group 4: 1,1,1,0
Group 5: 1,1,0,1
Group 6: 1,1,1,1
Good luck with your work.
This is indeed a multilevel problem and this has been the standard analysis for 25 years plus. In this context the main problem of not using it is type 1 errors of the school level variables. The analysis does not “know” about schools and students.
being harsh, no reputable journal in the field would publish this. To understand more have a look at this site with its extensive resources
http://www.bristol.ac.uk/cmm/software/mlwin/mlwin-resources.html
Hi All,
Thank you all for the useful information.
Usman Rashid David Morse Thanks for suggesting using a regression model. I tried using a regression model with 4 dummy variables to represent the treatments. But as David Morse said, the problem of using schools as our experiment units is that unless the effects are strong, we are unlikely to see significant result.
Do you think it is valid that I do analysis on student-level and use a regression model that specifies fixed effects of treatment and a random effect of schools.
Thanks Kelvyn Jones for pointing me to some useful resources!
Cheers,
Bowen
- Badges
- Science topic