In my research, I propose a software that helps researchers to conduct a randomized controlled trial to evaluate interventions in the educational context, somewhat similar to ASSISTments / E-TRIALS, Terracotta and UpGrade.
I need to run an experiment where researchers will use this software and a manual alternative, where the activities performed in the software are done manually, such as sending emails and randomization, for example. And then use the Technology Acceptance Model to evaluate the software in comparison to the manual alternative.
The manual part is becoming a bit complex because the researcher is required to send e-mails, create a pre-test, post-test and interventions, randomize the research groups, among other activities. If the activities are too long and laborious, I am afraid that this could harm the experiment.
Is it possible to perform this experiment in another way, with another design? Is there a design that is commonly used for this kind of experiment, to evaluate software with a manual alternative?
(Sorry for mistakes in the writing, I am using the translator)
The first consideration that you will want to make it what are the outcomes that you are interested in measuring, and how you will measure these outcomes. This is the most important consideration, because without valid and reliable measures, it will not matter what methodology you are using.
The next consideration is determining your comparison group. You want to avoid creating a "strawman". In other words, is your comparison group an accurate representation of the current best practice? One can imagine creating a control condition so bad that you will find a significant effect for the software, but the difference being meaningless. It sounds like the manual you have designed may be very complex because it is not a natural comparison (i.e., no one does it this way - though I don't have a lot of details so, I could be wrong).
It sounds like the research might benefit from a more qualitative approach to refine both the software (e.g., what are the steps in the RCT process where researchers need automation), and the manual for the comparison condition.
You also might consider conducting tests of individual (or a small number) of the steps at a time as pilot tests. Perhaps just the sending emails and randomization steps in one experiment.
- Badges
- Science topic
- Similar topics
- Linguistics
- Composition Studies
- Writing