Hi. I have the following experiment
An independent-measures (between-subjects) quasi-experimental pre-post intervention (within-subjects) design was used with an intervention and control group.
Each participant provided a pre (baseline measure) and post intervention score in only one of the groups.
A convenience sample of 400 employees from four of the five functional departments were recruited to the study (100 participants from each of the four departments), with two of the departments (200 participants) allocated to the intervention group and the other two departments (200 participants) allocated to the control group.
Employee intention to violate IS policy by intending to engage in risky cyber behaviour was measured one month before the education intervention (preintervention measure) and one month after the end of the intervention (postintervention measure).
The research uses a summative Likert Scale which has been designed to allow for interval measurement / scoring
My question is what the most appropriate statistical test. I am considering that using a t-test to compare pre / post / gain score on the intervention group assuming that the data meets all of the parametric tests and if not Mann Whitney. Does that feel right? What other tests could I use for inferential analysis?
Try a 2 (experimental vs. control) by 2 (time: pre & post) Repeated-Measure ANOVA. A t-test on the change with time would give you less information and would be statistically inappropriate because it's not accounting for the degree of freedom of time. More seriously, though, is your confound of department with condition, so if you have an effect of condition, it might really be an effect of department. There's no statistical way to tell those explanations apart. To correct for this, you might try collecting 100 more participants from each department in the opposite conditions (not ideal, because of time, but better). Then add department as a third dimension in the ANOVA. Best wishes with your study, Damian. ~ Kevin
As Jochen Wilhelm and Katie Hope Grobman state, the confound is pretty detrimental, so might be best to start a new study. But, a couple of questions. Do you have other information on the employees that would allow you to match individuals across departments, and was this study conducted during COVID lockdown (so less interaction among individuals). These won't get rid of the problem, so I think I do agree with the previous responders that regardless of the answers it may be best to design a new study. If you design it without the confound, look up Lord's paradox for issues about using a 2x2 mixed anova (as Kevin mentions) or an ANCOVA.
Daniel Wright Hi. I do have demographics and details on previous exposure to cyber attacks / fraud that I could subdivide the focus groups into through matched paired t-test between groups. Why would the department be confounding?
"with two of the departments (200 participants) allocated to the intervention group and the other two departments (200 participants) allocated to the control group. "
You are not able to differentiate if a possible effect of intervention vs control truely stems from differences between departments. Maybe there is already a difference between the departments without any intervention, then this pre-existing difference will also appear in your data.
For example, your DV is "reading skills" and your "departments" are high school and elementary school. If you now assign all high school students to the intervention group and all elementary students to the control group, your effect will surely come from the "department" not your intervention. It would have been better to assign the participants randomly to control and intervention group and use "department" as another predictor.
Damian Crosby , with matching you can correct for some obvious and known differences between the depratments, but the problem is more severe, since you simply don't know in what aspects the employees of different departments differ. Without randomizing the subjects into the treatment/intervention groups I don't see how you could solve the problem effectively. I would do a full stop here and request a more sensible study design. Whatever you do with the data you have will very likely put you in trouble. If you want to continuos with these data, then "department" is your unit of randomization (not "employee"!), so you have a total n of 4, with n = 2 per treatment group. This is statistcally ok, but the sample size is way too small to come up with any confident conclusion (so it's practically worthless).
Jochen Wilhelm if I were to correct the randomisation errors in my sample and the confounding Variables would the t-test and Mann Whitney be the most appropriate para and non parametric tests? I thought about ancova to but wasn’t sure. Thanks.
I don't think that it is possible to "correct randomization errors" in the statistical analysis. Matching or adjusting for covariables can help to account for possibly relevant effects of these variables, but it does not solve the confounding issue.
A table of analysis of variance can give you all the indicators needed in the analysis experience
I think can proceed with mann whitney test as you have used the Likert Scale to measure the outcome variable. Moreover post test was done only once, one month after the the end of intervention so ANOVA wont be suitable.
- Badges
- Science topic
- Similar topics
- Correction