I use PISA data of 28 countries. I’m interested in a cross-level interaction effect using Mplus. The question is: Does a country level variable (DI = measure of school segregation) moderate the effect of SES on achievement?
First I use a 2level model (I = students, c = countries):
Level 1
Yic = boc + b1c(SESic – SEScountry_mean) + eic
Level 2
boc = gamma00 + gamma01(DI) + r0c
b1c = gamma10 + gamma11(DI) + r1c
gamma11 is significant and positive, indicating that in countries with a high school segregation the association between SES and achievement is stronger than in other countries. à This is also what I expected.
Second I switch to a 3level model (I = stundets, j = schools, c = countries,
Level 1
Yijc = b0jc + b1jc(SESijc – SESschool_mean) + eijc
Level 2
b0jc = gamma00c + r0jc
b1jc = gamma10c + r1jc
Level 3
gamma00c = gamma000 + gamma001(DI) + u00c
gamma10c = gamma100 + gamma101(DI) + u10c
gamma101 is significant and negative, indicating that in countries with a high school segregation the association between SES and achievement is less pronounced than in other countries.
These different results somewhat confuse me. I tried different estimation methods (Bayes, ML, …), centering methods (country mean vs. school mean for the 3level model) and weighting approaches (unweighted data, only final student weights, final student weights and final school weights, all provided within the PISA data set), the results are stable.
Intuitively I thought that the L2- and L3-model should yield similar results and that the sign switch has to be an artefact. But now I think the results might be meaningful.
Now to my question: Is the following line of argumentation correct, or am I wrong?
When I estimate the 2level model (Students, countries) the random slope contains both individual and compositional effects, because school level effects are not controlled for.
When I estimate the 3level model (Students, schools, countries) the L2-slope variance captures the slope differences which are due to school differences (e.g. differences in SES-composition) and L3-slope variance is due to country differences. Thus, the 2level-slope “contains compositional effects”, the 3level-slope does not.
The interpretation of the L3-cross level interaction thus is: the more segregated a country is, the smaller is the effect of SES on achievement within schools. In segregated countries there is less within school SES-variation, what promotes small within school SES-effects.
Now it follows that the difference between the 2level and 3level results is the different consideration of the SES-school-compositional effects.
Consequently, I run a model, where I estimate a cross level –interaction with DI and the SES-school composition. This effect is positive: The higher the segregation in a country, the stronger are the compositional effects.
Thus I conclude: High segregation in an educational system leads
(A) to a smaller SES effect on achievement within schools – what is due to the smaller within school SES variation.
(B) to stronger compositional effects, what finally induces more inequality in segregated systems
Is this line of argumentation tenable?
Any comments are welcome!
Christoph
What is the school the segregation of ,,,, attainment from a previous year?
It is difficult appreciate exactly what you are doing - eg there is no subscript on the DI - I suggest that you write out the full combined model ( by substituting the macro equation in to the micro equation) to see what is exactly in the fixed and random part of both specifications . I would also normally include the variable you are using for centering in the fixed part of the model ie is the SES means.
Thanks for the quick response
DIc is the dissimilarity index (on Country Level) based on the PISA-SES-measure (split at the Country median, see Jenkins et al 2008 for this approach). I used the Leckie et al 2012 procedure for simulating DI based on a multilevel binomial response model. Additionally I also used the ICC(SES) as a kind of segregation measure for continious variables (this leads to the similar results).
I will look at the combined model and will come back to the thread tomorrow.
christoph
dear Kelvyn, you can find the combined model equations in the attached word doc (because of better possibilities for writing the equations).
thanks
Christoph
It is good to see you using our modeled segregations index.
I have thought a bit more about this and I am concerned that the cross level-interaction effect of segregation and de-meaned SES is an artifact of not modeling all the relevant means.
I do not know if this is possible but I would consider running another multilevel model with SESijk as the outcome and then obtaining the precision weighted v0k for the mean for each country and u0jk mean for each school and eoijk as the pupil de-meaned ( by country and school) SES. Then taking these estimates and including them in the fixed part of the model before doing the cross-level interactions. I would then do cross level interactions with these school and country averages (the first moment) before doing segregation (the second moment of the SES distribution).
This (unfinished) paper makes the case for doing this to minimize measurement error of the derived school and country variables
https://www.researchgate.net/publication/252146040_Do_multilevel_models_ever_give_different_results
I think you really need the means in when you are doing a within - between analysis - see attached which are a few slides from a course
Article Do multilevel models ever give different results?.
I like the model based DI, especially the possibilities for explaining differences in segregation. But I wonder why the “segregation index literature (e.g. some recent paper on bias correction for DI)” apparently does not recognize this approach…
Back to my problem:
What you suggest is similar to the multilevel latent covariate approach (e.g. Lüdkte et al., 2008) which is implemented in Mplus. This approach deals also with the low reliability of L2-measures (formed via manifest aggregation of L1-measures) due to sampling error at L1. I already entered L2 and L3 latent (and also manifest) SES means as predictors. The interaction effect remains negative and significant.
For me the “miracle” starts with the simple random slope model (see doc; --> This way, missing predictors shouldn’t be a problem) I saved the factor scores of the country level slopes from the 2level and 3level model. They are only correlated with r = 0.09 (see also scatterplot in the doc). If I simply correlate these factor scores with DI, I get the same results: the country level slope based on the 3level model and DI correlate with r = -0.434; p
I found some futher evidence that the negative interaction effect isn’t an artefact.
I ran a multigroup model for L1- and L2(compositional) SES-effects and used these estimates for some correlational analysis on country level. The country level slope (factor score from the 3level model) correlates with the L1-(within school) effect with r = 0.96 .--> Thus, the country level slope from the threelevel model captures country differences in the within school effect. The within school effect is further correlated with DI (r = -0.42) what supports the negative interaction effect. Finally the school level random slope (from the 3level model) correlates with the compositional effect with r = 0.57 à Thus, the school level slope contains compositional effect variation between countries.
What do you think about this? Do you have some further suggestions?
Fine
That looks sensible,
Personally I would include the school and country means as I see it a bit like not including main effects when looking at interactions in standard regression and I would also be interested in their substantive effects.
If you have not seen this - it may be of interest,
Endogeneity Problems in Multilevel Estimation of Education Production Functions: an Analysis Using PISA Data
http://www.llakes.org/wp-content/uploads/2010/11/HanchaneMostafa-14-final-online.pdf
On the modeling of segregation; I too have seen the recent papers on bias correction to D - I suspect that if you are not familiar with multilevel modeling it will take a lot to make you switch. I have been using the approach a lot recently with a Poisson and not just a binomial
https://www.researchgate.net/publication/270579850_Ethnic_Residential_Segregation_a_Multi-Level_Multi-Group_Multi-Scale_Approach_-_Exemplified_by_London_in_2011
https://www.researchgate.net/publication/290433969_Spatial_Polarisation_of_Presidential_Voting_in_the_United_States_1992-2012_the_%27Big_Sort%27_Revisited
https://www.researchgate.net/publication/287796771_The_melting-pot_and_the_economic_integration_of_immigrant_families_ancestral_and_generational_variations_in_Australia
Kelvyn
Article Ethnic Residential Segregation: a Multi-Level, Multi-Group, ...
Article Spatial Polarization of Presidential Voting in the United St...
Article The melting-pot and the economic integration of immigrant fa...
Dear Kelvin, thanks a lot for all your valueable comments and suggestions, these helped a lot! Further thanks for the papers!
ps. I also inlcuded in my final models the cluster means of SES on L2 and L3. The results are stable.
Christoph
- Badges
- Science topic
- Similar topics
- Education