As the model is saturated, any sort of fit index is helpful. I would think the best answer to your question is to look at the estimates. Are they statistically significant? Are they in the right direction?

This question touches on the conceptual meaning of df and what a model is. Ideally, our models impose a restriction upon the data (i.e., they are parsimonious and falsifiable). A model with zero df imposes no restrictions on the data (e.g., the observed covariances) and thus does not test anything more than was already there.

The 3-variable example above, however, could be tested against restricted or nested versions in which certain parameters are forced to zero. However, doing all possible paths would exhaust all df, circling back to the orginal problem and also Type-I problems.

The solution, at least in my opinion, is theoretical rather than statistical. At least two rival interesting theories are needed which can be tested. One theory posits a certain set of relations (hopefully not zero df) and the other theory posits a different set of relations. A statistical search of all possible paths would merely be fishing (but may suggest a confirmatory model for the future).

As the model is saturated, any sort of fit index is helpful. I would think the best answer to your question is to look at the estimates. Are they statistically significant? Are they in the right direction?

Sometimes, not often, a saturated model with zero df, does not have perfect fit. A case in point: A single-factor model with three indicators where two of the correlations are positive and one is negative. So there are the rare cases that some saturated models can be falsified.

Thanks Prof. Kenny for your helpful comments. It is a pleasant surprise for me to see your comments. In fact, I have started learning the basics of doing mediation and moderation analysis from your website and using your methods and spss/excell macros. Thanks once again to hear from you whom I consider my first teacher for such statistical analyses. Though, latter I added to my knowledge of this area from several others either from their web pages or books and still in the process of learning.

I tested one such model and found that all the direct paths as well as the indirect effect are significant (using the conventional test of significance as well as using bootstrap re-sampling method). All the variance explained based indices of fit (e.g., GFI, CFI) were 1, the Chi Square with 0 dif was 0 . The RMSEA was not reported for the default model but for independent model it was in non-acceptable range (.53). What I have learnt from you, I would infer that the relationship is partially mediated as the direct effect is also significant. But how should I report the validity of the overall model in terms of goodness of fit indices provided by the AMOS as all suggest a perfect fit.

I also tested one factor model with three indicators, one having positive, another negative and the third one zero correlation. And as suggested by you this model with zero df yielded a good fit but not a perfect fit.

Thanks a lot. I would highly appreciate if you could throw some light on why in some cases a perfect fit is obtained but not in others despite that fact that all models are saturated i.e., having a df zero.

As with any saturated model, models with 3 variables have an infinite number of equivalent models. You mention one rarely estimated but useful model which is all three measures are indicators of one latent variable. (This model was suggested a long time ago by Brewer, Crano, and Campbell.) I believe this model will always have a decent fit, unless one has inconsistent mediation, the direct and indirect effects are opposite signs.

One more point. All too often, we have equated a good model as one having a good fit index. A good model should have good fit, but it should have other things: plausibility, parsimony, and reasonable estimates (likely others can add to this list). Note that if in the population all the covariances are zero, most every model (assuming it runs) will have a good fit!

Dear Prof Kenny

Thanks for your valuable comments.

I do agree with you, " A good model should have good fit, but it should have other things: plausibility, parsimony, and reasonable estimates (likely others can add to this list)"

I call the "reasonable estimates" as a parameter of local fit and all the fit indices like GFI, TLI, CFI etc as global fit indices. A model must have an adequate global and local ft in addition to what you call the plausibility and parsimony.

Kind regards

Hi there. Interesting discussion.

I would like to add that when evaluating the fit of a model, it is important to consider what (i.e., which restrictions) are actually tested and if a failure to hold thse restrictions adds anything to the causal interpretation/foundation of the whole model.

The most researchers I know do not really know which restriction their model contains.

In my current view, there are three kinds of restrictions.

a) Non-essential restrictions (nice to have but "so what")

b) Unreasonable restrictions

c) Essential/helpful restrictions

a) These are effects fixed to zero just because the researchers does not have any definite idea if these should be freed (e.g., fixing paths to zero). If the models fails, these effects are non-zero - but that doesn't say anything about the specification of the other parts.

b) These are fixed-to-zero covariances between the errors of the latent endogenous variables. These imply that the researcher hold the assumption that all common causes of the two variables are contained in the model. This is very unlikely and not necessary. A model that fails because of such a restriction is a pitty. Hence: add error covariances. Econometricians do this by default.

c) These are important restrictions. Examples are local independability in latent variable models. These are vital because, hey, wie try to model unobserved things. These can only be labeled as scientific if the measurement model cleanly fits. Any deviance is a signal that the latent variable probably is wrong or doesn't even exist.

Another form of important restrictions stems from "backdoor variables" (see Pearl), for instance instrumental variables. These can help to place the causal interpretaton of the descentend paths' on more solid grounds as they help to elimininate alternatives.

A saturated path model is not really compelling - regardless of the estimates, because if the X-variable is endogenous, you cannot be sure about its effects on Z and Y. Because of its endogeneity and the estimated direct effect, you either cannot be sure about the Z-->Y-effect, even when you estimated the error covariance between Z and Y (by the way, I think that your model has less than 0 df because you estimate the indirect, and the direct effect and the covariance...?).

Any thoughts on this?

With best regards,

Holger

Well, one way to get around this problem is that you can treat observed variables as single indicator constructs. You can fix error variance of these single indicator constructs as 1 - cronbach alpha (Kline). This is especially advisable in case of exogenous variables because exogenous variables in SEM are assumed as having no measurement error which is far from truth in many cases. By converting them into single indicator constructs, we incorporate error which is more realistic, plus it also adds DF so the problem of zero DF is also solved. 

I am a bit concerned about the remark that a saturated model is of little value because it always can be reproduced by the underlying data. In doing some exploratory research, my interest, at this stage, is not necessarily to identify a theoretically well-underpinned parsimonious restricted causal model, and seek  confirmation from a plethora of goodness-of-fit stats when tested against empirical data.

For instance, I am developing some sort of general theory (exploratory research) involving exogenous independents, and endogenous mediator-, control - and dependent variables. In doing so, I may end up with a well-underpinned saturated theoretical model. Nothing wrong with this, so far, I think. Of course, I want to empirically investigate whether this general model holds if tested against the same (i.e. similar) study population in different country-settings. If country-specific information is unavailable to specify a priori country-specific versions of the general model, than I think the well-motivated general saturated model is convenient and can be tested using, first, pooled country data, and, second, using country-specific data (multi-group analysis). If country-specific configurations turn out, then a  such a configuration could be of help to detect reasons, previously unknown, for that specific configuration (for instance from comparisons with configurations of other countries).

Thus, in such kind of exploratory research, I may not be interested so much in seeking for a parsimonious and/or restricted causal model, but in the size, direction and statistical significance of direct and indirect effect (path) coefficients of all possible effects included in the saturated model.

Is this a plausible way of reasoning, providing support for the use of saturated models in particular (exploratory) research settings. Thank you in advance for sharing your views.

Hi George, different goals imply different approaches. Your idea and plan is completely sound. However, the question remains what the goal of any - even exploratory - approach is - especially what you mean when you use terms like "direct and indirect effect". Do you assume that the term "effect" refers to a causal (treatment) effect or some kind "relationship". If you have the first in mind, don't you think that it would be scientifically attractive/interesting/important to increase the evidence for the model before you move to the next level (e.g., cross-cultural comparisons)?

It may be that you core model of interest is saturated but after you have explored this model, think further: Which variables could you incorporate into this model that imply some valuable restrictions and that simply have some kind of instrumental functions. 

Take for instance, a fully saturated mediation model (consisting of three variables A, B, and C): Once the core model is established one could think if specific predictors of A, B and C that further test the causal directions implied by the model (among A, B, and C). Incorporating such variables not only allow testing but (if the test is successfully passed) increase the causal evidence of the model and its effects. Of course, no test in the world can prove that the effects are correct but testing strengthens the support. This all can be part of a larger exploratory endeavor. 

Best,

Holger

Hi Holger,

Thank you very much for sharing your views/knowledge! The first part of the example is, I think, what I am doing. In social and political debates, A, B, and C are considered as related processes. However, there is little, if any, empirical evidence for this, and certainly not from an international comparative perspective.

Based on scattered literature and inductive reasoning, I think, I manage to defend the proposed causal sequence/structure of a 'simple' saturated model 1. It is about relating three social processes A, B, and C. The model specifies that A influences  both B and C, and that B (also) affects C. More specific, variable A is known to consist of 2 dimensions and is therefore represented in model 1  as two distinct but related variables A1 and A2 (I link the two (A1 and A2) with a double arrow through their respective error terms, which is technically sound, according to the literature). For substantive reasons, this link is also of interest in the research.I tested this model using pooled country data and find support for (saturated) model 1. I then proceed by expanding model 1 by incorporating three 'underlying' factors. I motivate their position in the causal chain of model 2. In model 2, it is argued that variable A  (i.e. A1 and A2) mediates part of the effect that the underlying variables X, Y, Z (i.e. exogenous variables) have on B and C. Model 2 is also a saturated model and I think I manage to underpin all the assumed relations between model variables. I test model 2 in the same way as I tested model 1 and find support for most of the relations, except that effects of these underlying variables exclusively run through one dimension of A only (which is an interesting and somewhat unexpected finding, but is interpret able).

After these results, the question arises (of course) 'do these two 'general' models equally well apply if country-of-residence of respondents is entered into the equation. Multi-group analysis results indicate that the general model applies pretty well to the situation in 2 countries, but the results of other countries suggest that a somewhat different configuration of the two general models applies in those countries (i.e. the mediating role of A is confirmed in all cases, though mediation runs through 1 dimension of A only, which is an interesting finding) . I am currently trying to explain the deviant configurations, after carefully examining statistical non-significance of particular causal relations and try to figure out what  the substantive reason(s)  might be that particular context and the time-frame in which the data were collected.

I think I am following a sound path, but I am open for critics and would appreciate very much your view on this (and of others of course). Thank you in advance for a reaction.

Kind regards,

George

Hi George, just a view thoughts:

a) you never should test the model in a mixture of culturally diverse sub-samples as these will naturally be more heterogeneous than each of the subsample. The better approach would be to begin spefically and move towards comparisons afterwards.

b) the bidirectional arrows between exogeneous variables are covariances between the variables - not their error terms (this is the case for endogeneous variables).

c) If you incorporate X, Y, Z as predictors of A1 and A2 I guess you estimate all possible effects, thus, ending again with a saturated model. Is that necessary? Wouldn't some (at least ad-hoc) reasoning not allow to specify more specific effects?

d) Your "support" for both models is only based on the reasonableness and significance of the path coefficients. This is ok, but rather weak support as - you can arbitrarily flip the arrows and you will get some effects, see

Thoemmes, F. (2015). Reversing arrows in mediation models does not distinguish plausible models. Basic and Applied Social Psychology, 37(4), 226-234. doi:10.1080/01973533.2015.1049351

Kline, R. B. (2015). The mediation myth. Basic and Applied Social Psychology, 37(4), 202-213. doi:10.1080/01973533.2015.1049349

HTH

Holger 

Hi Holger,

I will digest your remarks more thoroughly, but I do want verify whether I understand comment (a) correctly. Does this mean that it is o.k. to specify a general model (assumed to be valid across countries), because there is simply insufficient country-specific evidence available to a priori specify and test country-specific versions of the 'general' model. Then, test the general models immediately to country-specific data, examine differences and common features, try to interpret  those differences with the limited country-specific information available, and see to what extent we may generalize across countries. Thus, leaving out the pooled analysis altogether?. Thank you in advance for replying

Thanks for the literature references!!

George

Hi,

the main point is that when you throw diverse data in one bowl, you are always unsure that any mishmash-estimate means. Having a rather homogeneous set of individuals always makes interpretations easier - not matter if you have raw data or - for instance - meta-analytical data. I do much research with meta-analytical structural equation model. The data is always a mess as you throw everything in one bowl. A subsequent model leaves always unclear what the estimates really mean. 

Best,

Holger

Hi Holger,

Before continuing, I like to say that i appreciate very much your views.

Points (b-c-d).

Regarding (b). Yes, I did not describe it clear enough apparently. A1 and A2 are specified as the two main mediator variables. However, they are 2 dimensions of the same construct (A). More specificially, they are the two dimensions of Berry's acculturation model/quadrant (et al., 1986). Depending on the situation people may choose to either act on the basis of the norms, values and customs characterizing the majority population culture (e.g. German, Dutch) or they may choose to act on the basis of those of their ethnic/immigrant  group culture (e.g. Turkish, Moroccan). I want to include both dimensions in the model and examine whether effects of particular 'underlying factors' on an outcome (dependent) variable are partly conveyed (are partly expressed) in people's acculturation preferences. The data do not permit me to specify something different than a recursive model and I somehow do need to express in the model specification that A1 and A2 are somehow 'related'. I read that I may express this by including a double arrow between the error terms of A1 and A2 (instead of between  A1 and A2 directly, which would jeopardize the recursive nature of the model).

(c and d) Thank you again. I think these observations put me on the right track (I think). Previously I did not understand that a base (restricted) model may fit very well to the data of all 6 countries, but that it is possible that a particular coefficient may turn out to be statistically non-significant in some countries and significant in others, so that (still) country-specific patterns of 'statistically significant' coefficients may still emerge (as is the case). I just wonder whether this may only be due to statistical/data reasons, such as country-specific differences in standard deviations of that particular variable, to sample size differences, and to a combination of both (standard errors). In one of the countries, where there are relatively more non-significant path-coefficients, sample size is smaller (250) than in other countries (450-500). I think, ex post,  that there may be a good reason why in that particular country, particular 'relations' may not be statistically significant. But if that indeed plays a role how can it be that the general model still fits well to the data of that country?

Hi George,

regarding the error covariance. An error covariance normally reflects the assumption that there may be other common but omitted causes of both A1 and A2.

 You could insert one effect - but yes inserting two effects representing both directions would need at least one exclusive "instrument" for A1 and A2 respectively.

With regard to the country-differences. If I did understand you correctly: The overall model will only misfit, if you leave out /fix parameters that actually are significant - not if you include non-significant parameters.

Best,

Holger

Hi Holger,

Thanks again for your reaction. It is not entirely clear to me, the third paragraph. I think I mean something different (?).

Supported by the literature review I first developed a saturated model comprising 3 underlying determinants, two (related) mediator variables and 2 outcome variables. The saturated model includes a number of relations between model variables for which I do not expect an effect at all (and I motivate this). However, my idea was, lets include this and see whether I am right across countries and to what extent, based on country-specific path-coefficient significance, what remains left of the saturated general model if only statistically significant path coefficients are included.

I think (given your comments) this is possible but a weak approach. Better to reduce the saturated model to a restricted model, and first see whether this restricted model can be replicated by the empirical data of each country, and then study why certain coefficients/path are statistically significant in one country and not in another country. This is what I did, and the restricted a priori general model, according to the goodness-of-fit statistics, fits well to all country data and also to the pooled data.I was a bit surprised though about this because goodness of fit of the general restricted model to all individual country-data seemed incompatible to me with non-significance of some paths/model coefficients in one country and their significance in other countries. If I understood it correctly, this may happen because sample sizes vary between countries and because distribution of variable scores may differ across countries. However, I was also looking for more substantive explanations.

(Just to check for my self, I, on purpose, deleted some important relations from the model and checked the fit of this 'test' model, and immediately the model fit was poor). Although I could try to find out what the best 'trimmed' model is that fits the data of all countries, I feel that such an approach is not 'fair' because the base-model should be a priori specified, based on empirical, theoretical and common-sense reasoning, and then tested against the data. Trimming it to fit the data of the current countries may result in mis-fit if applied to data sets of other countries for which data may come available in future. I hope this is clear?