Hi,

I choose to run a level 1 version of the regression model with and without them and compare the residual plots, and also look at R-square measures. Then I do the same for the level 2 models and see how it works out.

For a limited number of observations belonging to one 'nest', it may make sense to just not take it into account. There is a lower limit of observations needed for the most basic (e.g. OLS) regression, and that should be taken into account when doing multilevel regression, that is having a minimum of observations per nest.

Here techniques such as bootstrap and jacknife can be useful in raising the number of observations and allowing you to run a regression with enough variables. These techniques are used when there are very few cases, e.g in life sciences.

It is hard to imagine a multi-level regression. Errors, how they are distributed by levels

Let the levels will be flat. Then how the observations are made?

With the deepest respect, Gennady

Hi Adrian,

thank you for your response. I very much agree with your approach of carefully examining the data and models: indeed this should be basic stuff, but with complex statistics easily available it is easy to forget. Also, bootstrapping and jacknifing make for more robust estimates indeed.

The analysis of residuals, on the other hand, may not be very suitable for detecting influential cases. The problem is that when a case is very influential, it 'draws' the regression slope towards itself, so that this case ends up with a very small residual.

Dear Gennady,

thank you for your reply. I'm not entirely sure I undestand what you mean. Multilevel regression models (also referred to mixed effects models) are quite common in various disciplines, but of course not without their problems. Commonly, the group-level variation (errors) are assumed to be distribution Gaussian, but some software allows for the specification of other distributions.

Agree, Rense, but you can still see its influence in many cases, e.g. as a lone dot on a scatter plot Another good way to isolate outliers is to do a cluster analysis if you can. It certainly helped me in some cases.

I agree that a good-old scatterplot is still a very good way to detect problems with the data, including some influential cases.

This just to say that the MLwiN programme has (and has had for quite a long time) a set of diagnostic graphing facilities for looking at the multilevel residuals at any level. post estimation That is if countries at level 2 – you can evaluate informally the Normality distribution via a normal probability plot as well as Deletion residuals; Leverage values and Influence values using the methods of Langford, I. and Lewis, T. (1998). Outliers in multilevel data. Journal of the Royal Statistical Society, A. 161: 121-160.

As the between country variances depend on not having outliers, these procedures should be used. Here are two examples where it was definitely worth looking

Article: Explaining Fixed Effects: Random Effects modelling of Time-Series Cross-Sectional and Panel Data Andrew Bell, Kelvyn Jones

And

Article: Stylised fact or situated messiness? The diverse effects of increasing debt on national economic growth Andrew Bell, Ron Johnston, Kelvyn Jones

Both papers are on Research gate

I accept that if the analysis is something like an individual growth model where there are thousands of individuals at level 2, these procedures are probably not so valuable. But they are quick and easy to do and well worth at least a quick look.

A good discussion. Probably one could also take a look at group level errors with their confidence intervals to see how they matter. I am taking a liberty of asking another question related to Multilevel models. Can we use average or fraction calculated from individual level data as group level variables in multilevel model just to control for group level observables provided that those variables already appear as independent variable at individual level in the model.

In response to Md Siddiqui

Two aspects

1) If you compare the robust standard errors and ordinary standard errors of an individual variable and they are different, this is a good indication that there should be random effects for that variable – ie random slopes. This is a useful and simple diagnostic.

2) In relation to the question : ” Can we use average or fraction calculated from individual level data as group level variables in multilevel model just to control for group level observables provided that those variables already appear as independent variable at individual level in the model” . The answer is yes and these are known as contextual effects based on aggregated variables. It is also worth doing a cross-level interaction between the individual and group- level counterpart: eg the able child makes even greater progress when the average ability around them is high – peer group effects. That is they are often of substantive value . A word of caution – if there are not many observations on which to base the group variables you are introducing measurement error. A simple way around this is outlined in this paper and its accompanying data set. I must get back to finishing this off as it is a useful procedure!

Do multilevel models ever give different results?. Kelvyn Jones, Caroline Wright, Andrew Bell

https://www.researchgate.net/publication/252146040_Do_multilevel_models_ever_give_different_results?ev=prf_pub

Do multilevel models ever give different results: the data that goes with the paper

https://www.researchgate.net/publication/242350144_Do_multilevel_models_ever_give_different_results_the_data_that_goes_with_the_paper?ev=prf_pub

Article Do multilevel models ever give different results?.

Data Do multilevel models ever give different results: the data t...