If the number of parameters to be estimated is larger than the number of observations then there is no one optimal point in the parameter space (that yields the best fit to the data), but instead you have an optimal set (usually subspace) in this space (with all its points producing equivalently good fit). That's the rationale behind the word "unidentifiable".

And also this means that no method (bayesian or not) can prefer one point in the optimal set to another one. The effect that in practice some methods seem to work "but some parameters are very slow to converge" I would attribute to the inevitable numerical errors (and those slowly "converging" parameters must act as the basis in that optimal subspace).

Besides that, if your model is unidentifiable or near to it then its fitting to data will likely produce lots of overfitting effects that will make the obtained estimates of the parameters meaningless again. It is the rule of thumb that the number of parameters to be identified should be at least several times less than the number of data points. Try to simplify your model by introducing some reasonable parameter constraints (especially some coupling between the parameters).

If the number of parameters to be estimated is larger than the number of observations then there is no one optimal point in the parameter space (that yields the best fit to the data), but instead you have an optimal set (usually subspace) in this space (with all its points producing equivalently good fit). That's the rationale behind the word "unidentifiable".

And also this means that no method (bayesian or not) can prefer one point in the optimal set to another one. The effect that in practice some methods seem to work "but some parameters are very slow to converge" I would attribute to the inevitable numerical errors (and those slowly "converging" parameters must act as the basis in that optimal subspace).

Besides that, if your model is unidentifiable or near to it then its fitting to data will likely produce lots of overfitting effects that will make the obtained estimates of the parameters meaningless again. It is the rule of thumb that the number of parameters to be identified should be at least several times less than the number of data points. Try to simplify your model by introducing some reasonable parameter constraints (especially some coupling between the parameters).

It is possible that you can introduce a few less than odious constraints that would allow your model to be suffiently identified so that you can estmate the remaining parameters? In your case I would recommend setting uninteresting, across group paramters equal. This corresponds to presuming a null hypothesis of no difference. If this does not suit the actual situation, you would learn that quickly enough as your overall model fit will be poor. If the overall fit is decent, then you can go on to examine the significance of the parameters that were sufficiently meaningful to estimate.

Thank you both.

For the specific problem, I ended up with many random parameters because I am specifying varying coefficients for some variables and the interactions between them.

However the varying coefficients I am actually interested in are the coefficient of the interactions.

Is it meaningful constrain the two interacting variables using fixed coefficients but still estimate varying coefficients for their interaction?

A simplified version of the problem in lmer notation would look something like this

lmer(y~x+m+x:m+(x+m+x:m|teams), data=df)

Is it ok to just use

lmer(y~x+m+x:m+(x:m|teams), data=df)

or would this be a problem?

Thanks for posting this, George. This is very helpful. I think I had a similar problem and wasn't sure what to make of it.

George

This is going to be quite a long answer but I will give it a go! The bottom line is that you do not necessarily have an unidentifiable model because the number of parameters may actually be smaller than the number of observations. Whether there are any problems depends on the specification of the model, the method of estimation, the target of inference, and the nature of the problem being estimated, and finally the information content of the data you are analysing.

To take a specific example at the outset – if I had 500 individuals repeatedly measured on 5 occasions in terms of their annual (log) income, I do not think I would have any difficulty at all in estimating the

5 fixed effects giving mean the average income across all individuals for each year ;

5 variances giving me the between-individual inequality in each year;

And 10 covariances (once standardised ) which would give me the correlation between any pair of years across individuals.

I could estimate this with maximum likelihood or MCMC and I would anticipate the results would not differ a great deal because the response would be continuous and not discrete; the structure is simple and not complex (two –level hierarchical) ; the target of interest is the means and variance- covariances for which I have one hundred observations and not the specific individuals for whom I typically only have 5 pieces of information. Also I would expect in our unequal world that the variances will be substantially different from zero; that is far away from the boundary space of zero. Such a multivariate model with is unstructured covariance level 2 matrix is the most complex that can be fitted with these data (see Chapter in Volume 2 below). I could reduce the complexity to a Toeplitz model in which the covariances the same lag apart are constrained to be the same (the covariance between occasion 1 and 3 is forced to be the same as 2 and 4, and 3 and 6 etc), and I could simplify it further to a compound symmetry model with only one higher level (random intercept variance); the latter specification implies there is equal correlation between any pair of occasions.

So I would see no difficulty here. In your team example I presume you are interested in between-team differences (the level 2 variance function which depends on the variances and covariances) and not in inferring to the specific team – the latter would be difficult (impossible?) as you have so little information (and therefore there is a great deal of uncertainty about the estimate/ prediction). This difference in target of inference is important and the paradigmic example is individuals in twin pairs – you can say very little about specific pairs (their estimates would have very wide uncertainty), but that does not stop you drawing conclusions about between-twin variation if you have enough pairs.

This is of course all related to statistical power and in the educational literature of pupils in schools where typically some 10% of the variation lies at the school level, Bryk and Raudenbush (1992) Linear Hierarchical Models suggest that ; with 60 students in each of 160 schools you can estimate 4 random variables at school level.

Now to some more technical comments.

If you are estimating these models by procedures that maximize the likelihood such as Goldstein’s (1986) Iterative Generalised Least Squares, estimation is a two stage procedure (see the chapter in Volume 1 below). You estimate the fixed means and the variances and covariances at each level. So if you have a homoscedastic model at level 1 there will be 5 fixed terms; one level-1 variance and 15 variance-covariances at level 2, and you have 100 teams to estimate level 2 terms and 500 observations to estimate the level 1 variance and the fixed means. You are not estimating the random effects and then summarising their variance- covariances, but the reverse – having estimated the variance-covariance you can try and estimate (predict) the specific random effects (residuals) for the teams – but you would not probably not bother as your target of inference cannot really be the specific teams with only 5 observations each . Even estimating only the overall parameters (and not the specific random terms) you might run into problems – this will show up as failure to converge, impossible empirically negative variance terms, and covariances with implied impossible correlations outside the +/- 1. The solution is collect more teams, simplify the model or switch estimation. I will same a bit more about the latter.

The maximum likelihood estimation estimates the fixed part treating the variances as unknown, and estimates the variances as if the fixed part is known. In contrast restricted maximum likelihood (REML or RIGLS) estimates the variances on the basis that the fixed part has been estimated – this is conceptually equivalent to using n-1 in the estimation of the variances as 1 degree of freedom has been estimated in calculating the mean. This is more realistic and in practice it often helps with convergence when a small number of higher-level units are being analysed.

Finally to you question about MCMC. This is of course Full Bayesian error propagation (see the chapter in Volume 1 below), and the uncertainty in every parameter is fed into all the other parameters – the fixed part is estimated given that the random part is an estimate and vice –versa. In this approach the random effects for each team are estimated alongside their variance- covariances. Hence your concern about identifiability. But the Bayesian setup does not treat each random effect for a team as a separate estimable parameter but as coming from a distribution; and the prior for this typically has a mean of zero and a Normal distribution so that the differential random effects are shrunk back to this zero value. This shows up quite nicely in the Deviance Information Criterion that is often used to evaluate models where the Pd term gives you an estimate of how many degrees of freedom have been consumed in the fit; this might be a lot smaller than the nominal parameters obtained by counting the random effects. If you do lack information to estimate the model it will show up in a very slow convergence to a distribution and highly auto-correlated traces for the MCMC chains. But the key parameters would remain for me the averages and variance-covariances and not the specific teams; and you may have enough information to do this. You have to inspect the traces and judge if convergence to a distribution has been achieved. To improve things you can collect more data, simplify the specification, and run the chains for a very long time, increasing the thrown-away burnin and use techniques such as hierarchical centering (in the MCMC estimation; see Chapter in Volume 1 below).to reduce the dependency in the chains

These items which are on ResearchGate maybe of help – I am definitely not a statistician so they are written to try and explain the underlying concepts as simply as possible hopefully without doing an injustice to the material. If I am misinformed I would really like to know. Apologies for the shameless publicity!

Developing multilevel models for analysing contextuality, heterogeneity and change using MLwiN, Volume 1 (updated June 2014) Kelvyn Jones, V. S. Subramanian

https://www.researchgate.net/publication/260771330_Developing_multilevel_models_for_analysing_contextuality_heterogeneity_and_change_using_MLwiN_Volume_1_%28updated_June_2014%29?ev=prf_pub

Developing multilevel models for analysing contextuality, heterogeneity and change using MLwiN, Volume 2Kelvyn Jones, VS Subramanian

https://www.researchgate.net/publication/260772180_Developing_multilevel_models_for_analysing_contextuality_heterogeneity_and_change_using_MLwiN_Volume_2?ev=prf_pub

This covers variance functions at each level

Modelling complexity: analysing between-individual and between-place variation -- a multilevel tutorial N Bullen, K Jones, C Duncan Environment and Planning A (Impact Factor: 1.89). 01/1997; 29(4):585-609.

https://www.researchgate.net/publication/23538539_Modelling_complexity_analysing_between-individual_and_between-place_variation_--_a_multilevel_tutorial?ev=prf_pub

And this covers the precision weighted estimation of the random effects in the two step procedure

CONTEXTUAL MODELS OF URBAN HOUSE PRICES - A COMPARISON OF FIXED-COEFFICIENT AND RANDOM-COEFFICIENT MODELS DEVELOPED BY EXPANSION

K JONES, N BULLEN

Economic Geography (Impact Factor: 3.97). 09/1994; 70:252-272. DOI:10.2307/143993

https://www.researchgate.net/publication/225303298_CONTEXTUAL_MODELS_OF_URBAN_HOUSE_PRICES_-_A_COMPARISON_OF_FIXED-COEFFICIENT_AND_RANDOM-COEFFICIENT_MODELS_DEVELOPED_BY_EXPANSION?ev=prf_pub

Finally I and a colleague - Andy Bell – are in the middle of some large scale simulations of seeing what is possible using likelihood and Bayesian procedures – we are especially comverned with how the nature of the data structure makes a difference (eg many higher level units compared to few lower level units (as in bio-statistical panels) as compared to few high level units and many lower level units (as in political analysis of countries) – we will post it on RGate when it is finished.

I have just seen your recent post about the interactions - given that, my advice is to speed up estimation (less correlated chains) by using Orthogonal Parameterisation. This option is always at least efficient as not using and normally an improvement, especially if the predictor variables are correlated. I also suggested using Hierarchical Centring, particularly if the cluster-level variance is substantial. This is generally more efficient but sometimes worse if the higher level variance is relatively small. Deploying a relatively short trial run is useful to see what happens in practice.

Kelvyn

Article Modelling complexity: Analysing between-individual and betwe...

Article Contextual Models of Urban House Prices: A Comparison of Fix...

Book Developing multilevel models for analysing contextuality, he...

Book Developing multilevel models for analysing contextuality, he...

Try using Mplus 7.2 software (statmodel.com) to get Bayesian SEM models. I used JAGS, OpenBugs and others R packages, but I found Mplus more easy and accurate to get model fit.

A big picture issue I see here with your approach is you are doing no preliminary model building. Personally, I think it is a mistake to just hop into a complex model without first building and examining simpler models. The place I would start is a main effects only -one level model ignoring groups. Then I would add groups, then I would add interactions. Actually, I would probably add across group constraints before interactions. At each step I would look at model fit and model issues. Rome wasn't built in a day. This sort of stuff is a component of my professional business, so, really, I am speaking from some experience and a modicum of success. Best of luck!

What about using an Exploratory Structural Equation Modeling? http://www.statmodel.com/esem.shtml

@kelvyn. Thanks so much for such a detail response. This certainly helps. I am very new to Bayesian estimation so I am still trying to get my head around these issues. I will look into DIC and Pd to see if indeed there is an issue.

@Karen. Yes. That is actually the approach I am taking (but sometimes I get too excited and just want to test the final model)

Do you have any views regarding my follow-up question? If I want to test the effect of two predictors (e.g. x and w) and their interaction (xw), is it a problem to model their interaction as a varying slope whilst specifying each of these as a fixed slope?

Another question that would help me simplify the model: what happens If i want to test a crosslevel interaction but allowing for a varying slope does not improve my model. Is testing a crosslevel interaction between two fixed effects meaningful at all?

As to fixing the main effects and estimating the interaction, yes, you can do this. But think about the example of a linear regression with an interaction term. Generally if you have two predictors in a main effects model and there is an unmodeled interaction effect, the parameter extimates of the main effects can be quite off. If you use the main effects values you obtained in your main effects model to fix these values in order to estimate your interaction, it may not be an optimal model- of couse you can check the diagnostic statistics that would tell you if freeing either of the main effects would significantlt improvement model fit.

It seems that your primary interest in this model is at the group level. I would recommend estimating group mean values for all your variables and model that as a one level model. If you are not seeing interactions there, I doubt they would propogate at a two level model.

Also, did you check the intraclass correlations for your outcome variables with respect to group? If they are less than .20 (my personal rule of thumb- without citation) then you can treat the data as iid (independently identically distributed). This would make it simple to test group by individual level interactions.

So, first, I would check ICCs, if substantial, I would first do group mean level analysisdescribed in second paragraph, if not substantial, I would treat data as iid.

Best of luck with your modeling.

This reminds me of Dale Barr's recent papers where he argues that you need to include all the random slopes to avoid inflating error rates. I think in his appendix he suggests that fitting the model without random intercepts but with random slopes is an OK fallback if you have convergence problems (which is often if you have lots of predictors). Alternatively you can try and fit models with fewer covariances or different parameterizations.

Hi Thom ,

Thanks for this. Can you provide the citation or a link to the paper. This is really useful as my model seems to work better when all the slopes are random!!!  

I still believe you can! May I see your model and data? Thanks [email protected]

Here is a link to Dale Barr's blog with links to one of the papers:

http://talklab.psy.gla.ac.uk

Thanks!