I conducted a research in which we compared the effects of two training methods in the acquisition and retention of competence in a particular clinical skill. After submitting a paper to a peer-review journal, one of the reviewers said that comparing mean results was not accurate and that we should compare the amount/proportion of students who achieved the benchmark (pass).
The hypothesis was that training #1 was more effective than training #2 in improving students' acquisition and retention of the competence.
In order to measure the level of competence, we assessed: 1) psychomotor skills (percentage of certain tasks performed according pre-defined standards); 2) cognitive knowledge (percentage of correct answers obtained in a MCQ), and 3) self-efficacy levels (with a validated tool that measured self-efficay from 0-100).
This was assessed at pre-test, post-test and retention-test (3 months later) for 2 groups: control group (CG) and experimental group (EG).
Now I wonder how I can analyse:
1) The differences in proportions of students who passed the benchmark within the same group across time.
2) The differences in proportions of students who passed the benchmark between groups across time.
I know about chi-squared and McNemar test, but is there an equivalent to mixed ANOVA for this kind of analysis that can be performed with SPSS? I would need to take into consideration the correlation between results across time depending on the intervention, I think...
Any help would be very much appreciated.
I see you have listed SPSS as one of the topics, so presumably you have access to it. One option would be to use GENLIN to perform a "mixed design" logistic regression. The Exp(B) column in the table of coefficients gives you odds ratios for this type of model.
Here is syntax I used when a colleague had the same design you described (i.e., she had a treatment group vs a control group and 3 time points).
****************************************************************** .
*** GEE: Quit = const + treat + t2 + t3 + t2*treat + t3*treat' *** .
****************************************************************** .
* Generalized Estimating Equations.
GENLIN quit (REFERENCE=FIRST) WITH treat t2 t3
/MODEL treat t2 t3 treat*t2 treat*t3 INTERCEPT=YES
DISTRIBUTION=BINOMIAL LINK=LOGIT
/CRITERIA METHOD=FISHER(1) SCALE=1 MAXITERATIONS=100 MAXSTEPHALVING=5 PCONVERGE=1E-006(ABSOLUTE)
SINGULAR=1E-012 ANALYSISTYPE=3(WALD) CILEVEL=95 LIKELIHOOD=FULL
/REPEATED SUBJECT=mrnum WITHINSUBJECT=t2*t3 SORT=YES CORRTYPE=unstructured ADJUSTCORR=YES
COVB=ROBUST MAXITERATIONS=100 PCONVERGE=1e-006(ABSOLUTE) UPDATECORR=1
/MISSING CLASSMISSING=EXCLUDE
/PRINT CPS DESCRIPTIVES MODELINFO FIT SUMMARY SOLUTION (EXPONENTIATED).
Variables t2 and t3 were 1-0 indicators for time points 2 and 3. And the data were arranged such that there was one row per time point, or 3 rows per person. If you need to rearrange your data to that format, see VARSTOCASES.
Finally, in the data I analyzed, there was no hint of a treatment x time interaction, so we ended up estimating another model without the interaction term. Here's the syntax for that one:
******************************************* .
*** GEE: Quit = const + treat + t2 + t3 *** .
******************************************* .
* Generalized Estimating Equations.
GENLIN quit (REFERENCE=FIRST) WITH treat t2 t3
/MODEL treat t2 t3 INTERCEPT=YES
DISTRIBUTION=BINOMIAL LINK=LOGIT
/CRITERIA METHOD=FISHER(1) SCALE=1 MAXITERATIONS=100 MAXSTEPHALVING=5 PCONVERGE=1E-006(ABSOLUTE)
SINGULAR=1E-012 ANALYSISTYPE=3(WALD) CILEVEL=95 LIKELIHOOD=FULL
/REPEATED SUBJECT=mrnum WITHINSUBJECT=t2*t3 SORT=YES CORRTYPE=unstructured ADJUSTCORR=YES
COVB=ROBUST MAXITERATIONS=100 PCONVERGE=1e-006(ABSOLUTE) UPDATECORR=1
/MISSING CLASSMISSING=EXCLUDE
/PRINT CPS DESCRIPTIVES MODELINFO FIT SUMMARY SOLUTION (EXPONENTIATED).
I hope this helps.
I think Bruce is correct (again): Mixed logistic regression seems to be the right tool for this problem.
I'll just add that when I was developing that syntax, GENLIN was still pretty new to me, and I wanted some way to validate my approach. So here's what I did. I estimated the following 3 models via GENLIN:
I also estimated ordinary binary logistic regression models with "treat" as the only explanatory variable at each time point separately. And what I found was that the partial effects of treat in the 3 models estimated via GENLIN matched exactly the effects of treat in the corresponding logistic regressions for each time point.* I was hoping that would be the case, and when it was, I was reassured that I was estimating a sensible model via GENLIN.
* Of course, I had to filter out any cases that did not have data for all 3 time points to get this exact correspondence.
HTH.
Thanks for your answers.
I must admit I have no previous experience with GENLIN at all. Therefore, I will have to research how to do it. However, after reading your last comment I have got a concern. I have got 23 dropouts who did not complete the retention phase. I removed them from the initial analysis and the reviewers commented that this could bias my results. They strongly recommended to perform the analysis including these cases, which by the look of it cannot be done using the model you propose, Bruce. Am I correct?
Hi Jose. The only reason I excluded cases with incomplete data was for purposes of validating my GENLIN model. If I had included them, the treatment effects from the 3 separate binary logistic regressions (one per time point) would not have matched exactly the corresponding estimates from the GENLIN model. But after having performed those validation analyses, I did indeed revert to using all of the available data. This is possible because the data file is configured to have one row per observation rather than one row per person. (So you need an ID variable to indicate who each observation belongs to--e.g., "SUBJECT=mrnum" in the syntax I posted earlier.) HTH.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA