I am new to statistics and currently trying to conducting data analysis for my pilot study and I am confused about the procedures. Overall, I have around 150 respondents. Attached there is a model that I am trying to analyse.
The affective variable is a latent factor that is presented in 4 dimensions based on the degree of extrinsic and intrinsic influence (28 items). It affects learning phenomenon that influences students' achievements in language learning. The phenomenon might have receptive and productive nature and therefore there were two scales (8 and 6 items respectively) to measure each. Finally, the achievements are represented by two separate lexical tests with two separate overall scores and one variable that is accounted to measure students' aspiration to practice their language (8 items). All items were assessed by respondents by using a 7-point Likert scale, except for the lexical tests which were measured by overall score.
All the instruments I have been using were validated before in a different context. However, since I have been conducting study in a new context, I have decided to conduct EFA to ensure validity and reliability of scales. For affective variable, in order to understand whether each of 4 different dimensions were represented separately, I have conducted EFA. It was successful and after deleting several factors that had cross-loadings on other dimensions, all 4 dimensions were represented separately with minimum of 3 variables per item.
Along with that, I have checked reliability and conducted EFA for the items that were used to measure productive and receptive nature of phenomenon to ensure that they do not have any cross loadings.
During the next step I used AMOS. I have inserted all items for 4 dimensions of affective variable, all items that represented receptive and productive items for measuring the learning phenomenon and one item that was used to measure one language learning outcome. For measurement model, Chi-square was 460.648, df=309 with probability level =.000., the data for model fit was the following CFI=.930, RMSEA=.058 with PCLOSE=.114, however AGFI=.784, GFI=.823 and NFI=.816 were low. All factor loadings of items were above .60. Being new to statistics, I believe that it might be due to a small sample size and a number of variables. So, now I am uncertain about my following actions.
I need to check my hypotheses by checking structural model, but I am hesitating about further procedures. The first option that I have: to get factor scores for all latent factors and then to conduct path analysis. The second option: to use Partial Least squares in PLSmart since it can deal with formative and not only reflective scales and to measure my structural model for all factors at once.
I apologise If I have described wrong procedures but I would really appreciate any suggestions and feedback. Sorry for a long text.
Hello Artem,
the most salient issue is the misfit of your model that should be addressed before going to more substantial issues.
Most striking for me is the bit unclear explanation about your independent variable. A latent variable is a most-possible specific thing--that is a singular dimension. Hence, I do not know where these 4 dimensions come into play. Perhaps you can clarify that. It is a second-order factor or rather a composite? My goal would always to conceptualize the latent variables in a model as specific/singular existing entities with a likewise specific meaning and then to identify 2-3 indicators that causally reflect these entities as directly as possible.
Did you do a CFA before setting up the structural model? It is not optimal that you work with AMOS as AMOS' diagnostic possibilities are limited. Using lavaan in R would be helpful and would cost you 1 day (read the tutorials on lavaan.org). I can offer to coach if you like. In this thread there is a nice example where this happened. Unfortunately, Monika (who initialized the thread) did stop halfways.
https://www.researchgate.net/post/Is_it_possible_to_perform_mediation_analysis_in_SEM_with_dependent_latent_variable#view=5ddd17dd0f95f1376a0388d1
Best,
Holger
Dear Professor Holger,
I very much appreciate your dedication to work and your reply. I have been reading your comments to many other threads in Researchgate before and without some of your suggestions I would not come to the point of writing my own question now.
Let me describe my study and the variables I have with more particularities.
To be specific, in my case, affective variable is represented by students' motivation that might have extrinsic or intrinsic regulation. Based on the answers of students to the items, it is possible to measure whether engagement in phenomenon happens due to more external or more internal regulation of motivation. In this case, EFA allowed me to differentiate 4 dimensions of motivation types that make impact on the engagement in particular activities. My initial goal was to test several structural models to check empirically the hypotheses of how each of these regulations influences the phenomenon and further how the engagement in phenomenon influences language learning outcomes. Therefore, after conducting EFA (due to the fact that the data were collected in the context and by using the language that has not been applied before) 4 different factors emerged. Taking into account correlation of these factors I used promax rotation and ML as data extraction method. All items loadings were above .550. KMO was .853 with chi-square of 1615.87 and df=153.
After that, I have proved theoretical underpinnings by extracting these four dimensions. Now, I am not sure whether I should conduct 4 different SEM analyses while using different dimensions for motivation regulation. In total each factor (latent variable) is represented by 3 to 6 items that "reflect the entities" as you put it in previous message. Composite reliability of all factors was fluctuating from .833 to .879 for each factor.
I have conducted CFA with AMOS as well specifically for 4 dimensions of this variable. CFI=.91, IFI=.911, TLI=.893, GFI=.837, CMIN=268.343 and df=129. RMSEA was also low =.086, and I believe that could be due to a small sample size.
But affective factor is only the beginning of my problems. Another difficult point is connected to considering all different outcomes that I have adapted for my study: two total scores for lexical tests and one factor that was measured through 8 items to understand students' aspiration to practice the language.
I have tried to use R and lavaan but since I am just starting my path in statistics, I found it difficult to use. And yes, I would appreciate any of your suggestions and help.
Regards,
Artem
Hello Artem,
so let's untangle that. You write:
" it is possible to measure whether engagement in phenomenon happens due to more external or more internal regulation of motivation"
A) This seems to suggest that you have a specific engagement and you ask about the reasons why the person is engaged. First of all, this is not a very good way to measure motivation as all you get are the attributions of people. But here we are so let's continue. When your list contains several reasons for being engaged (which seems to fall into the extrinsic and intrinsic categories), than the result is an aggregate not a singular variable--although I am a bit confused what the numerical value could represent. It would be helpful if you post the items. At present I imagine something like "I am engaged in X because of...." 1) reason A, 2) reasons B, 3) reason C with non-exclusive response formats. Hence, person could be motivated due to one reason, ore several. The overall value would just represent the number of reasons. I am unsure what this theoretically means...
B) Your EFA seems to show that there are for types/categories of reason. Again, this is unclear. Showing this with specific items would help. In any case, you have 4 latent variables (or 4 composites/aggregates depending on the structure)--not one. Hence the left side of your model is undercomplex.
C) The next sentence is confusing: " My initial goal was to test several structural models to check empirically the hypotheses of how each of these regulations influences the phenomenon"
What is now "the phenomenon"?
D) You write: "Now, I am not sure whether I should conduct 4 different SEM analyses while using different dimensions for motivation regulation"
Again: do you think these 4 regulation forms are different existing entities with unique causal effects? If yes then include all of them at once.
E) You write: "In total each factor (latent variable) is represented by 3 to 6 items". Perhaps too much. I would choose maximally 3 items that are homogeneous and crystal-clear representations of the underlying latent variable. Such a subset are still enough and implies strong testability of the implications with at least a chance to fit (whereas 6 items imply almost always an over-simplified, hence, misspecified and misfitting model)
F) The model does not fit which is not due to the sample size--to the contrary. The low N reduces power which in your case was still high enough to reveal the misspecification.
I leave the rest of the posting with the outcomes for the moment. One comment however with regard to lavaan. You don't have to have any statistical deeper understanding--just how to set up the code. Here is a workflow for a CFA that you should be able to adapt.
#########################################################
#Install libraries
install.packages("lavaan")
install.packages("tidyverse")
#load packages
library(lavaan)
library(tidyverse)
#load data (the data was saved as a csv file)
data %
print(n=100) #or any number. If you just type "data" you get the first N=10
#Building the CFA structure
cfamodel
Hello again Dr. Holger,
Ok, let me answer and provide clarifications in order.
A-B) To be precise, I am not asking about the reason of the person to be engaged. I am asking questions that could reflect intrinsic or extrinsic foundation to be involved in particular learning practices (I call it phenomenon for this thread to avoid excessive information). The statements were provided to students to assess while answering one single question: Why are you learning English? The example of statements are as follows: "In order to obtain a more prestigious job later on" or "Because I want to have "the good life" later on" (Vallerand et al., 1992). Therefore, there are several items that help to assess each regulation type. But theoretically some types of regulation are supposed to influence person to be involved in specific practices (in learning phenomenon) more than other types of regulation.
To sum up, the regulation types influence learning outcomes differently. More intrinsically regulated motivation lead to positive outcomes and vice versa. However, I hypothesise that it does not only positively predicts the outcomes but also influences students to be engaged in particular learning activities which, in their turn, influence their outcomes (indirect effect). Hope now it sounds clear.
C) Implementing several structural models, in my opinion (which might and most likely is false) was considered as an option to understand how different type of motivational regulation affects the phenomenon and the outcomes. Therefore, I thought to insert into the model each type of regulation (intrinsic, extrinsic I type and extrinsic II type, amotivation) separately but only after understanding that all factors are distinct factors after conducting EFA and reliability analyses. That is why I have been questioning whether I should include all regulation types at the same time since I predict that some of them might not have an effect on students' engagement in learning practices.
E) Well, what if after conducting stat. operations, the presence of 6 items does not deteriorate the model fit? does it still mean that it is better to delete some items that have lower loadings?
F) Thanks for your comment of the sample size.
Meanwhile, I will try to use your guidelines for R.
P.s. I decided not to provide all information about my theoretical foundations and I am trying to avoid extra terms based on the theory to rather focus on statistical operations than on the discussion of the theory. Probably, I have puzzled you because of this strategy.
Artem
Dear Dr. Holger, as I told you, I do not feel comfortable while using R and lavaan. Thanks for providing instructions. Attached please find the doc.
Dear Artem, thanks.
You see: lavaan is easy ;) You see that the results are quite the same (differences being due to using FIML and the Yuan-Bentler correction or non-normality. And again the model is far off being acceptable.
Now to your prior posting.
The first paragraph is confusing. You reject my assumption that you ask for reasons for being engaged but then provide exact those examples that I had assumed !? The phenomenon (towards which the respondents are engaged) is learning English and you ask for reasons. All that I wrote is valid, hence.
As a consequence, it COULD BE that some "clustering reasons" (i.e., those sets of items that correlate highly) indeed represent some underlying more fundamental motivation (hence conforming to the factor model) but a misfit of a corresponding structure implies that this picture is too simple. Your model clearly shows that this is the case. So there are two options: a) As I recommended, select 2-3 crystal clear items, b) adopt some kind of re-specification with the danger of fitting a model to the data set that may not be generalizable.
This will be ok IF each set of clustering reasons is indeed a mere representation of one latent factor. In this case, using a valid subset will still reveal this factor (you do not need 5 measures of blood pressure to measure it). If however, the factor structure of the clustering reasons is more complex (with the extreme that each reason is a singular and specific thing albeit being correlated with other reasons), THEN using a subset of 2-3 reasons may lead to an under-representation of the overall *category* /index/aggregate.
Before proceeding think about whether the "factor-thinking" makes sense and whether using a subset could work. Actually, the more easy way would really to conduct a PCA (instead of a EFA), regard them as aggregate constructs and form composites. If you want to continue "walking the latent-variable-road", then
a) choose a subset of 2-3 items per factor
b) repeat the CFA with these
c) present the list of question wordings of these items
Best,
Holger
Dear Dr. Holger,
I would say that I rejected your assumption based on the wording you used. I believe I did it unintentionally by considering motivational regulation as a determinant rather than a reason to act in a specific way (i.e. to be engaged in learning activities). Anyways, I apologise for confusing.
I would like to ask the question about your quote "...COULD BE that some "clustering reasons" (i.e., those sets of items that correlate highly) indeed represent some underlying more fundamental motivation (hence conforming to the factor model) but a misfit of a corresponding structure implies that this picture is too simple". Would you be so kind to explain it in a simpler way? What results made you think so?
Taking into account, that the questionnaire has been adapted to my study and previous studies demonstrated its applicability, I believe that I cannot try to adapt my model to the dataset I have. This is also true due to further inability to generalise findings.
Following your suggestion, I have tried to conduct PCA in SPSS (since I am more aware of the ways it operates rather than R). I attach the document with the output. I have deleted several factors based on the cross loadings results; however, I am afraid that it will be hard to find other justifications for reducing the items since all factors load as they are supposed to with an absolute value of .25 with no cross loadings noted. What is more, cumulative loadings is more than 71%. For you to see the wording of the statements, I have included them into the word document by factor.
Now, I have further question regarding the following procedures. If I form composite scores, what would I need to report in my methodology in regards to switching from latent variables towards aggregated constructs? Also, how will I need to structure my further analysis? Does it mean that I will need to continue with path analysis procedures as I have described in my first post to this thread?
Sorry again if I am confusing you with my questions, I appreciate your feedback and replies.
Regards,
Artem
Dear Professor Holger,
I have been exploring more literature and discussions on similar topics and my question is connected to the normality of the data I have. Can it be true that the the results of FA that I have attached earlier have non-normality of the data that influences the model fit?
Thanks in advance for your reply.
Hi Artem,
I did not see that you already posted a message.
You wrote
-----------------
I would like to ask the question about your quote "...COULD BE that some "clustering reasons" (i.e., those sets of items that correlate highly) indeed represent some underlying more fundamental motivation (hence conforming to the factor model) but a misfit of a corresponding structure implies that this picture is too simple". Would you be so kind to explain it in a simpler way? What results made you think so?
---------------
Yes that was unclear. The key issue is that you have a bunch of inter-correlated items (stated reasons for learning English). The question is: what is the cause of these inter-correlations? The "common factor model" is ONE answer. It states that these items are reflections of a latent underlying common cause--that influences all items and creates these (spurious) correlations. This model is very valuable but comes with great challenges. In order to believe in it, you have to test the implications and these are high barriers. These testable implications are "local independences" of the items: IF the model is correct than these spurious correlations should disappear once the latent cause held constant. THIS is what the chisquare test evaluates.
In contrast, the PC model ( what you did in SPSS and what is a synonym for "aggregate construct", "formative construct" or simply "index". These are somewhat "cheap" because the form composites on the basis of these inter-correlations but make no assumptions that go behind the data. It is simply an aggregate. You see: no risk (composites cannot be wrong or false) but not very valuable.
So the million dollar question in your case is: Is the common factor model reasonable (despite the misfit showing at least problems)? It could be as specific reasons may reflect some more general underlying motivations. For instance, the statements "I want to learn something new", "I want to master challenges" and "I want to personally develop" could simply be reflections of some underlying mastery-motive. The misfit however shows that this simplicitity is at least to strong. You will never now, if the factor model is completely wrong (e.g., the reasons chould be facets in complex network instead of reflectionf of the factor) or if it is *in principle* correct albeit a little bit too simple. You chould re-specify your model to keep the latent variables with some changes. But you run the risk of adapting a wrong model to the data (capitalization on chance or overfitting). This is the risk.
On the other hand, you could skip the latent factor perspective and go the PC-road. That is, use the PCA results, create composites and use them as manifest variables (perhaps with some theoretically based error correction ) as variables in your model.
And no, you would not report switching because you nobody wants to read your personal road to enlightment. Just present your reasoning.
Best,
Holger
Dear Dr. Holger,
Thanks for your fascinating answer.
Let me firstly give my comments regarding common factor model. If I understand correctly, you are referring to common latent factor (https://www.youtube.com/watch?v=Y7Le5Vb7_jg)? am I right? if so, I will try to conduct the analysis and will come back with the results for the sake of this discussion and for the usage of others who might refer to this thread later. If I am wrong, please correct whether I understood you correctly (again, I am attaching the example that has been provided in AMOS since I feel more comfortable while using it).
Indeed, every part of my model is based on the theoretical assumption and probably it will make more sense to stick to the "PC-road". Especially taking into account that the questionnaire has been already used and validated before in similar contexts. That is why, I will also continue walking PC-road to see the results. But again, here I have two questions. Firstly, referring to manifest variables, do you imply using aggregated score by using a mean score or a factor score for the variables? Secondly, could you clarify your quote "perhaps with some theoretically based error correction"?
Thanks a lot for clarifications about the methodological part.
Update about part 1. I guess I have been wrong and after reading (https://psu-psychology.github.io/psy-597-SEM/06_factor_models/factor_models.html#formal-specification-of-the-common-factor-model) I got what you meant.
Regards,
Archie
Holger Steinmetz
Dear Pr. Holger,
I have conducted several tests for my models with different outcomes (please see the attached file). I have used factor scores of items and conducted path analysis. As you can see, the results are not very satisfactory.
What would you propose, based on your experience, to address these issues?
Additional theoretical information:
Initially, I hypothesized that only intrinsic motivation is supposed to have a positive influence on IDLER and IDLEP practices while amotivation was hypothesized to have negative but significant influence on students' informal engagement. And further, external motivation was supposed to have a small effect on students' engagement.
upd: I have observed some issues with kurtosis for some of the variables after the analysis. will it make sense to exclude the variables to avoid non-normality?
Regards,
Artem
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size