Farough, what you found is called method effect. It is well-known that negatively worded items not only load onto the intended trait factor, but also (or solely) on a different factor that only consists of negatively worded items, i.e., a method factor.

Here are some articles by Podsakoff and colleagues that explain the problem of in more detail:

psych.colorado.edu/~willcutt/pdfs/Podsakoff_2003.pdf

psych.colorado.edu/~willcutt/pdfs/Podsakoff_2003.pdf

Negatively worded items often have a lower loading on the common trait factor than positively worded items and therefore they affect reliability.

If you know that your items are supposed to measure five constructs, why not use confirmatory factor analysis? The error variables of negatively worded items may correlate (as modification indices may tell you), but this could be justified by referring to method effects.

Regards, Karin

There is a very interesting case in point: there was some suggestion in the literature that the Centre for Epidemiological Studies Depression scale (CES-D) measured a continuum from depression to well-being, on the basis of two factors, one with negative and one with positive items. This was soundly debunked as a method effect. 

Wood, A.M., Taylor, P.J. & Joseph, S., 2010. Does the CES-D measure a continuum from depression to happiness? Comparing substantive and artifactual models. Psychiatry Research, 177(1-2), pp.120–123.

Ronan, thanks for pointing to this article. This is exactly what I have often found: Negatively worded items form a method factor.

Here is another article dealing with the same problem in the context of optimism and pessimism:

Rauch, W. A., Schweizer, K. & Moosbrugger, H. (2007). Method effects due to social desirability as a parsimonious explanation of the deviation from unidimensionality in LOT-R scores. Personality and Individual Differences, 42(8), 1597-1607.

Abstract:

Dispositional optimism measured by the Life Orientation Test (Revised; LOT-R, Scheier et al., 1994) was originally proposed as a unidimensional construct pertaining to individual differences in positive outcome expectancies. However, factor analytic research has suggested two separate ‘‘optimism’’ and ‘‘pessimism’’ factors associated with positively and negatively worded items respectively. Using confirmatory factor analysis with data from N = 658 students it is shown that a general optimism model with method effects of the positively worded items fits even better than the ‘‘optimism–pessimism’’ model in terms of the AIC statistic, which accounts for both model parsimony and model fit. Furthermore, an attempt is made to explain the method effects by social desirability: impression management, one of two components of socially desirable responding, correlates significantly with the method factor whereas the other component, self-deceptive enhancement, is related to the general optimism factor. Thus it is shown that the deviation from unidimensionality of observed scores does not imply deviation from unidimensionality of optimism when method effects are incorporated in the model.

Regards, Karin

Thanks, Karin and Ronan.

The original version of the scale was intended to measure pregnancy-related anxiety, with 7 positive and 7 negative-worded items. In fact, all items should be negatively worded. CFA confirmed well the five-factor structure of the scale after removing one item. Referees asked for the EFA results. The results of the EFA indicated four factors while the scale consisted of five factors. I also found that there were weak intercorrelation coefficients between factors with positive and negative-worded items. Is this finding important to be reported?

Forough, I don't understand why reviewers asked for EFA results. Did they provide a reason for this?

Results of your EFA: Which method did you use, orthogonal or oblique rotation, and according to which criteria did you judge the number of factors to retain?

Regards, Karin

I agree with Karin that it is unusual for reviewers to request an EFA, when you have supplied the superior specification of CFA. But sadly, there is very little that one can do about that.

At a minimum, make sure that you have done the kind of EFA that is most closely related to your CFA, which would be to use Maximum Likelihood as a method and oblique rotation (correlated factors).

 I used Promax rotation and the criterion was eigen value >1. 

I reported that there were low factor correlations (T values < 2.56).

Promax is an orthogonal rotation. Use Oblimin instead.

Also, if you expect 5 factors, then use that rather than the eigen value to specify your model.

As David suggested force EFA to extract 5 factors. Eigenvalue > 1 is not a good criterion  because it is based on PCA. Parallel analysis would have been a better option. What do your results look like when you

...My answer did not go through. What do your results look like when you force the program to extract 5 factors?

There is another problem I would like to point out: With 14 items and 5 factors you only have 2 to 3 items per factor. If the factors are not correlated your model will not be identified. This may be the reason why a reviewer wanted you to reort an EFA.

Regards,  Karin 

Thanks, allI ran an EFA and used Oblimin rotation and forced to extract 5 factors. 2 factors with 2 items, two factors with 3 items and one with 4 items. four

I ran an EFA and used Oblimin rotation and forced to extract 5 factors. 2 factors with 2 items, two factors with 3 items and one with 4 items. four factor correlations are weak. It looks like somewhat the proposed structure, but a new factor emerged and 3 factors lost one item. the goodness of fit test is not significant.

results of the CFA on 13 items is good. (RMSEA= 0.066 (CI [0.053, 0.078]), SRMR=0.069, Chi-square/df=2.71, P

Hi Forough 

I think, it is good to reverse the negative items before calculate reliability,because its affects the reliability coefficient if we do not reverse the negative items and find reliability. 

Hi Muhammad

I reversed the positive items before analysing. since the scale measured anxiety, the author asked us to reverse positive items.

Well, model fit is actually not good, just on the border to being satisfactory. There is some misfit in the model, because the chi-square (which you should always provide, together with df) is obviously too large, p-value should be > .01, not .97 (or at least .95), and GFI and SRMR are fit indexes you cannot trust (dependent on sample size).

What do your modification indices suggest? How large is your sample size? Which estimation method did you use? Did your treat your items as being continuous or ordinal scaled? What kind of rating scale is used for your items? How did you deal with missing data? Are the items about normally distributed or rather skewed?

If the scale is supposed to measure anxiety, then the factors should be correlated, because I guess that you need to build a sum score, is this correct? But this can only be done if the factors are correlated. This seems to be a problem.

Regards, Karin

Hi Karin,

Thank you. I must check them with the statistician of the study.

what I know is:

The sample size was 400. The modification indices suggested removing item 9.

Four correlations were low (< 0.1). items were treated as ordinal. The rating scale was 4-point likert.  yes, I built a total score.   

There are different criteria for accepting a model. The statistician of the study told me if 2 criteria out of three (Chi-square/df < 3, RMSEA 0.05) meet, the model fits our data.

Kline RB: Principles and Practice of Structural Equation Modeling 2edn. New York: The Guilford Press. ; 2005.

Tabachnick BG, Fidell LS: Using Multivariate Statistics. Boston: Allyn and Bacon; 2001

Hi Forough, your sample size is large enough if these are valid cases, and treating the variables as ordinal is also fine. I don't know how modification indices can suggest to remove a variable. These indices tell you which parameter to freely estimate that had been fixed before to zero.

Model fit: If the chi-square value is significant, the model does not fit the data. However, some misfit may be accepted as an almost well-fitting model may be reasonable and because a large sample size may enhance the chi-square value. But how large can a misfit be? There are several rules-of-thumb, but these are dependent on sample size, model complexity, distribution of variables, effect sizes, and more. I have the new edition of Rex B. Kline's book (4th edition, 2016). Here (on page 272) he cautions against using chi-square/df (normed chi-square): "...there were never any 'acceptable' clear-cut guidelines about maximum values of the normed chi-square (e.g., < 2.0? - < 3.0?). Because there is little statistical or logical foundation for the normed chi-square, it should have no role in global fit testing."

Of course you can try to justify your decision to keep the model based on some rule-of-thumb criteria, these are quite often cited:

https://www.researchgate.net/publication/251060246_Evaluating_the_Fit_of_Structural_Equation_Models_Tests_of_Significance_and_Descriptive_Goodness-of-Fit_Measures

But I wonder what exactly your results indicate. As the factors are not correlated, but you seem to need the sum score, what exactly are your items measuring? What would a sum score mean based on these items? In order to help you more, I would need an output and more information on your research question. But maybe that your statistician can help you here.

Regards, Karin

Article Evaluating the Fit of Structural Equation Models: Tests of S...

Thank you so much for your guidance Karin