We have developed a software which is a pedagogical tool that helps in teaching computer science to students with hearing impairment. We are planning to have a study on the effectiveness of the tool. What are the steps that we need to take to measure the validity of the tool?Is their any similar study conducted in this area?
You can use questionnaire such as TAM model. You may google TAM model questionnaire and modify according to you area. I think validity and reliability will be assessed once you prepared questionnaire and conduct pilot study.
I look forward to seeing the results of your study of students using your tool when compared to results of students not using your tool. To me, that would be where to begin. I know that students with hearing loss or deaf students do have particular unique struggles with computer science that hearing students don't usually have, usually language-based.
I wonder if the tool does what it is supposed to do, so I would consider a small study with a small number of people from your target group and try to see what they think of it. Perhaps even pay them for their comments and include them in your research team, probably they will think of questions that you never even considered asking.
I think the validity of a pedagogical tool is depend the feedback of Student That how Much Students gained the Knowledge efficiently and effectively.....
Hi Waldemar,
If to know means to measure I do not know. How could I measure this philosophy? ;-) Teaching, and programming, I used to change once I understood a need. Like use small numbers to keep it simple. Perhaps sometimes I changed too easily.
Still, try to understand needs before you try to measure satisfaction seems reasonable to my mind... as is to consider the n=5 based usability heuristic (from Nielsen et al) that if 5 experts can not find it, they will have found enough to keep you occupied until the next test....
Joris
Dear Raji,
It depends on whether you want it for your personal practice/use or you want to do a research and come out with a more generalizable result.
In the latter case, Catherine M. already suggested you to test it with a control group. This means that you would need to use your pedagogical tool with one group while leaving the control group without it. This stage is called intervention. Then you would need to measure the effect size (change in learning) in both groups. Your tool would be valid, if the effect size of your experimental group is significantly (statistically) higher than control group. Usually in pedagogical science, there is a big grey scale so you should not expect black and white difference.
You also need to be careful with the Pygmalion effect. Since your tool is your 'darling,' you would tend (unconsciously) to favour its validity.
The question is how authentic is the teaching through this particular tool and if the achieved result is the one the tool is designed for. First the aims of the teaching programme have to be clear (and I believe that this is already been considered). According to the aims the results have to be analysed. The necessity is to collect these results. Collection can be done by using feedback tools, using different techniques, as scales, group analysis, written reflection, and finally will be the final examination. All these then have to be analysed within the department
So depends on what you want. Research or educate, test hypotheses or document results, grow or improve, and so on....
Hello
I suggest that you follow the steps of usability tests. You can do different tests on different phases (http://www.usability.gov/how-to-and-tools/methods/usability-testing.html). You can test user satisfaction, errors, functionality and usability, based on Nielsen heristics, for exemple (https://www.nngroup.com/articles/ten-usability-heuristics/).
If you want to test the functionality and pedagogical impacto, you have the Phillip´s model, that measure: Reaction, Satisfaction, & Planned Action, learning; Application and Implementation (http://www.busgurus.ca/media/pdf/the-kirkpatrick-phillips-evaluation-model-en.pdf).
You can use questionnaire such as TAM model. You may google TAM model questionnaire and modify according to you area. I think validity and reliability will be assessed once you prepared questionnaire and conduct pilot study.
Your tool is to teach computer science and you can easily check the effectiveness as follows:-
1. Using your tool , teach any programming language first and then give exercise to your students and find out whether they are able to carry out the problems. If they are able to solve the problems, then your tool is really effective.
2. Similarly test them in Database creation , query and reports generation and then find out the effectiveness.
Pedagogy means process and implementation
So you have to find out the effectiveness of your tool from the angle of process and implementation.
To teach understanding is not easily tested when you do not involve the target group(s) and just ask what is their understanding. Therefor to me pedagogy is more than ' process and implementation'.
Thank you all for the valuable suggestions.
After review of literature we have identified deaf undergraduate students in two different institutions. One group is taught by deaf instructor and other by hearing instructor.The sample size is 25 on average in each group.We are planning to conduct three tests in each group.
1. pretest before administrating the tool
2. posttest before administrating the tool
3. posttest after administrating the tool.
The control group and experimental group are same.Since the mode of communication of instructors are different,we have decided to conduct the 3 tests in each group separately.
Kindly comment .
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Software