I would like to conduct a survey to assess the level of awareness students have regarding internet threats and their vulnerability to social engineering. However, I plan to conduct the survey twice, once before providing them with training and the second time after the training session is completed.
What are the statistical methods recommended for that? Note that the population is the same and the survey questions will be the same.
It sounds like a t-Test would be appropriate, and if you can compare the scores from the same individual at each time point then a paired-comparison t-Test would be even better.
Your problem is your null hypothesis. No matter how bad the teacher is, sudents are going to learn something. So the null hypothesis of the t-test is that students learn nothing at all from the class. zip.
Not a credible hypothesis.
In education interventions, the null hypothesis has to be that the students don't learn enough to make it worthwhile to take the time and resources needed to run the learning activity.
I suggest that you draw up a lesson plan that includes stated learning objectives, and the percentage of students who will meet those objectives following the class. You then set your null hypothesis values as the percentage below which the class would be deemed not to have met its objectives.
For example, you might set an outcome that 90% of students would be able to identify two cues that suggest that an email is phishing. And you set a lower level of 70% below which you would regard the class as having failed.
You can use the confidence interval around your effect size as a test. If it doesn't include 70%, you win.
Hello Hamza Kweyu Omullah. Please tell us more about the survey. Does it already exist? If yes, is it published? Or is it something you intend to develop yourself? And how is "level of awareness students have regarding internet threats and their vulnerability to social engineering"? E.g., is there a single score? Thank you for clarifying.
Re the concerns Ronán Michael Conroy expressed, another option would be to include another (control) group that also takes the survey twice, but with some kind of "placebo" training in place of your actual training session.
Bruce Weaver – I guess my point is that the null hypothesis in education isn't set by defining 'no effect' but by defining 'not enough effect to justify the time and resources required'.
And your point is a good one: what is the / are the outcome measure(s)? An educational intervention is frequently a complex intervention with multiple inputs and multiple objectives. So the evaluation has to be an overall bang-for-bucks perspective.
Ronán Michael Conroy Okay noted, thanks for your input and insights. Bruce Weaver it is a research project that I am working on at the moment. Thank you for your input.
Jos Feys, Thank you. I have planned to structure my two questionnaires in a Google form as a continuous process. Firstly, the participants will need to answer the first questionnaire and then submit it. After submitting, they will be automatically directed to watch the training video. Once the video is completed, the second questionnaire will appear, and they will complete the survey by responding to the second questionnaire. I am assuming that all participants will be able to participate in the survey from start to finish. So, I think I will be able to achieve the same number of subjects.
Ronán Michael Conroy, what you described sounds like superiority testing with a margin, as discussed by Shafiq & Malhotra (2015).
Article Superiority trials: Raising the bar of null hypothesis stati...
Hamza.
although the answers given by colleagues are generally correct, consider the following:
- is your instrument created from the answer with memory loss?
- How much time is necessary for there to be a loss of memory of the previous answer?
- Does the way of scaling the response allow me to register the change I expect?
I suppose that what you are looking for is a change in attitude or behavior, so the reagent is key.
then if it gets him to recognize some "idea" and he doesn't know it, he will learn it with some process, (what RONAN mentions),
but it is different from evaluating how he responds attitudinally to a stimulus,
Thanks for the answer Carlos Jimenez-Gallardo
My interest is after answering the first questionnaire, the subjects will take some 10 minutes to be trained on various internet threats and social engineering tactics. Once they're done with the training, which will take like 10 minutes, they will be directed to the second questionnaire which will have the same questions as the first one but framed differently. I just want to examine the impact of the training on their response.
Hamza
you must understand how you validate the hypothesis "there is change due to training".
this will not have much construct validity, the experiment is very "light".
it is very certain that what Ronan tells you will happen from the educational point of view..
you plan an OXO design,
I would recommend
1. 0X0X0X0X0....
2. 0X00000...
3.
0 X1 000...
0 X2 000...
0 corresponds to the observation or measurement
X corresponds to the intervention or experiment
for the training to work it has to move from short term memory to long term memory or become meaningful,
Design 1. answers how many trainings should I do to make the content meaningful.
Design 2, answers to how long after the training can I return to the same behavior as before.
the third one looks for what kind of training can be the most effective to provoke meanings in people.
for example
X1 can be training in ideal behaviors
X2 may be showing the results of not analyzing the messages.