I'm currently conducting a pre/post-test quasi-experimental research project involving a drug free program in a high school. I want to know if program-enrolled student's opinions differ from the general population because of the program, so I've surveyed two groups of students (those who are in the program and those who aren't) in a pretest to get a baseline, and measured them in the post to see if their scores differ. The post survey included quantitative data of the program's effectiveness over a span of 10 years, which was the "treatment" factor for both groups. The main idea was to see if the program-enrolled student's scores changed significantly compared to the general population's scores. Originally, I thought this would be NEGD, but the scores from the pre and post for both groups are anonymous, so that doesn't quite fit anymore. Any suggestions on what methodology and analysis method I should use?
Just so I am sure that I understand, are you not able to match a person's score on the post test with that person's score on the pre test? And assuming this is right, do you at least know which set of pre and post scores belong to the treatment condition?
Yes, that is correct. And yes, I can differentiate between the scores of the two groups on both the pre-test and post-test.
How much overlap is there between the distributions on the pre test? Were the control group people how still took drugs, but just did not have the treatment on offer, or was it between the high school population and a group who I assume were mostly required to have the treatment? And the treatment specific to opiods?
What was the retention rate, so that was everybody at time 1 also at time 2?
With these non-matched samples it is more tricky and requires more assumptions, and therefore is more open to skepticism/criticism, but I can imagine in this case you might have been worried about having an identifier. My hope is that whatever you are interested in the groups began similar, but I am guessing probably not.
One more question, is the distributions for the control group at time 1 and time 2 pretty similar to each other?
The control group in this case were the students that had not participated in the drug-preventative program. The treatment group were the students that had participated. The goal of the study was to determine whether or not the treatment (aka, being a part of a Voluntary Randomized Student Drug Testing Program) affected the two group's overall qualitative opinions of VRSDT. In short, I want to know if whether or not VRSDT is considered effective by both groups by determining their scores on a survey. The more positive the score, the more positive an opinion a person had.
The same classrooms were surveyed in the pre and post tests, however, a lot students were missing/absent during the post (due to Christmas break).
In any case, I'm interested in the results regardless if the groups differ, or if they're similar to each other.
Aaah, okay, so: "I want to know whether or not VRSDT is considered effective by both groups." In that case, you aren't comparing the groups, you just have to decide what is effective (i.e., what value on the scale) what it means for a group to consider it effective (e.g., the 25th percentile being above that point, and probably calculate the uncertainty of that estimate). If you are worried about bias in the post-new year sample, was this do to kids leaving the school, or just the absences? I assume in this sample drug users are more likely to be chronically absent. Is that true?
Thank you for your answer! How would I go about calculating uncertainty? And what methodology would said analysis process fit under? (I have to specify the specific methodology and analysis for the project). As for the kids leaving the school before break, I would assume yes when considering a portion of those may be drug users, however, there's no sure fire way to tell. Overall, the post test sample size from both the comparison and treatment group had a similar reduction in size due to this problem, so I think I just need to qualify that when talking about the methods.
The method for calculating uncertainty depends on what you measure. The suggestion above, 25th quantile above some threshold, was just off of the top of my head and without any knowledge of the subject area, but Wilcox (in his robust statistics books) gives a method for confidence interval for quantiles. Those books are at my office, but if I remember he used a percentile bootstrap within his function. BUT, you should decide what you mean by effective and what constitutes a group being above this threshold. Your summary statement about wanting to see just if the treamenet was considered effective by both groups is different from many objectives (like seeing if the treatment was effective), so you should discuss these issues with colleagues.
- Badges
- Science topic
- Similar topics
- Computer Science
- Program