Dear Marco,

on a practical level, you will probably find that a multilevel model will be difficult to estimate with your data, particularly if you have a binary dependent variables and/or multiple random effects. 

It might be OK to select a single observation per level 2 unit, in one way or the other. However, the real question is why you actually have multiple observations for a limited subset of your data. Is that just random, or the result of a proces that is relevant to what you are studying? It would be helpful to have more information on your data collection and substantive area of research before deciding how to deal with your data.

Dear Rense,,

many thanks for your reply (!).

Indeed the estimates were troublesome. Apparently, within and between variance changing weirdly.

The dataset is a survey among universities (level 2); the respondents (level 1)  are one or two people for each university. Having more than one respondent is just random.

The dependent variables are the responses to a question with multiple option to be weighted on a ordinal scale 0-3. I  would employ the results of a factor analysis, hence continuous normally distributed, and run models separately for each factor.

The predicting variables are mostly university features (level 2) and a few other resulting from responses to other questions (level 1).

Dear Marco,

Please find below some thoughts that might not be a complete or definitive answer, but hopefully are of some use (some answer require that you’d have a substantial sample of universities).

1. Overall, it seems like the way forward is to ‘reduce’ the analyses to a single-level analysis of universities. You simply do not have the lvl-1 variation required to estimate the multilevel model. This implies that you need to use your individual-level observations to represent characteristics of universities.

2. It could be that you have surveyed individuals (within universities) to get some information characteristics of the university. For instance: name of the institute, number of faculties, or number of students. In that case you would hope that each individual gives the same answer, although in practice some variation would likely occur (in increasing likelihood given the examples in the previous sentence). You could then use the universities in which you have multiple individuals to assess to what extent the data you get from those individuals is reliably representative to the (level of your) universities.

3. As you suggest in your question, you could take the average between the individual-level responses within universities. Of course, you can only do that if an average between two values make substantive sense in your application.

4. Alternatively, for those universities where you have two individual-level observations, you could randomly select 1. You could repeat that procedure multiple times to see if the random selection of individuals (to represent universities) affects your findings. If not, you have an argument in favour of the quality of your sampling design.

5. If your sample is large enough, you could do the above (point 4) just on the subsample of universities for which you have two individual-level observations. That would provide an even stronger test.

Some recent papers by Daniel McNeish address this and he describes how GEE estimation can be used. He is on researchgate. Several years ago I ran some simulations on this situation for binary outcomes, I'll attach that paper.

Article Extra-binomial variation in multilevel logistic models with ...

Article The Effect of Small Sample Size on Two Level Model Estimates...

Article Modeling Sparsely Clustered Data: Design-Based, Model-Based,...

Article Analyzing Clustered Data with OLS Regression: The Effect of ...

Thanks a lot Rense and Daniel. Very helpful!

I had 552 respondents for 520 universities. Data on university features are given from  statistical sources. The respondents had questions on issues like "what drives university decisions..".

Hence, I will consider the University values as my level 1, and take the average value for the few cases where I have double responses.  

I have a somewhat different take on this - in that the problem is not the analysis but the data - you have predictors at level 2 about universities but the people you have sampled may not be representative of that institution - you need to collect more data.

This is an example where sampled units may not be representative of the higher-level unit (eg 3 sampled counties not representing a State); see Lucas , SR (2013) An inconvenient dataset: bias and inappropriate inference with the multilevel model, Quality and Quantity

On the modeling side we often find it helps (with MLwiN at least) in ordinal outcome models to try out choosing different reference base to omit - that is try dropping the highest category if omitting the  lowest does not work - the estimation engine may have difficulty in estimating the 'threshold' values on the logit/probit scale if there are not many observations in a category and you are also trying to estimate a random intercept on top of this.

But as I say the real issue is the data and its sparseness and un-representativeness - you mainly have a sample of size 1.

Dear Kelvyn, many thanks for your reply.

Unfortunately I cannot collect more data, as I have received the dataset. So, I am trying to do the best that I can with what I have.. 

On the other hand, there is a conceptual argument to support that responses are somehow representative in that the questions are about universities'  Internationalization and the respondents holds key positions on this regards, namely the Rectors either the responsible for international relations. On the empirical side, responses from the same institutions display high agreement (although inter rater agreement indexes can't be computed of course).

On the modelling side, I have run the models both considering universities as level 1 (nested into countries at level 2, by the way) as well as respondents at level 1 (again, nested into countries). The results are pretty similar.

By adding these clarifications, do you think is acceptable to run a 2level with country at level 2 and respondents or universities at level 1?

Marco

Yes I would do that but think about using MCMC - have a look at this

How Many Countries for Multilevel Modeling? A Comparison of Frequentist and Bayesian Approaches http://onlinelibrary.wiley.com/doi/10.1111/ajps.12001/abstract

It shows that you can recover parameter estimates with even quite low number of units at the higher level.

Kelvyn,

Indeed I am using MCMC in MLwiN. The countries at level 2 are 34, the number of universities at level 1 ranges between 1 and 82 (median of 9). The difference with maximum likelihood estimates is visible..