My goal is to use large-scale assessment data (answers from the respondents to the items and the respective Q-matrix) to verify whether it is possible to infer prerequisite relationships among the skills assessed in a multi-year application of a national large-scale assesment (like NAEP or SAT) using perfomance data.
I researched some psychometric and data mining methods like Tatsuoka's Rule-Space which could be used to determine those relations. They have been used in intelligent tutoring systems, but in a much greater granularity.
Few skills and items are applied each year, but considering multiple applications, I might have a better picture with sufficient coverage, and, as a summative assessment, is a snapshot disregarding the learning present in the process.
Is there any study in the literature that claims such relationships from the flat model used in these assessments (although it is quite)? Is this a valid path?
Dear friend
To the best of my knowledge its valid to acquire knowledge of skills and its interesting also. To make the task easy, try using well defined structures of the data
you have from questionnaire. Then use correlation and biplot concept. I think that shall be fine
I think an argument for the validity (depending on what is meant by validity) of using this approach could be made. I would recommend adding a component that also reviews the identified rules with respect to the curriculum, an educational theory, skills, or the knowledge components that are addressed by the questions that are identified as prerequisite to one another.
With respect to validity, you may want to look at how its many forms have been defined. Chapelle also wrote an interesting article on validity that's worth a read.
Chapelle, C. A., Enright, M. K., & Jamieson, J. (2010). Does an Argument-Based Approach to Validity Make a Difference? Educational Measurement: Issues and Practice, 29(1), 3–13. http://doi.org/10.1111/j.1745-3992.2009.00165.x
Thanks for all the answers.
@Carrie, what I was thinking of validity is regarding the construct. Since large-scale assessments assess a single trait (e.g. proficiency in Math and its IRT score), is it valid to build a attribute hierarchy from the skills being assessed?
I imagine a question like that: is putting the pieces together (order the skills) valid to explain the whole (Math proficiency)?
I am expecting to cross these results with the national Math curriculum and have some experts evaluate the internal validity.
Construct & Validity Related Comments
The Math construct? A smaller sub-type of math (e.g., algebra or addition)? How would you argue that defining the prerequisite skill/knowledge structure within Math shows the validity of the construct? My understanding of large scale tests is that they are created to operationalize a particular construct.
I don't work with NAEP/SAT type assessments, so I'm going to ground this in large scale language tests where each test (e.g., IELTS, TOEIC, or TOEFL) has multiple sub-tests each with their own constructs (e.g., speaking ability or reading comprehension). Each of these tests aims to measure that construct and individual questions are used to determine students' knowledge within a particular area or ability to perform certain types of tasks . Question responses then provide evidence of the test-writer's abilities/knowledge with respect to the targeted construct.
Hines, S. (2010). Evidence-Centered Design: The TOEIC Speaking and Writing Tests (Compendium Study No. TC-10-17). Educational Testing Service. Retrieved from https://www.ets.org/research/policy_research_reports/tc-10-07
Research Question Related Comments
I think it's interesting to try to pull apart potential prerequisites from text-taker data and then use that to try and explain overall student performance in an area. I expect that a modelling approach will be needed, suggesting that questions should ask the extent to which this method identifies appropriate prerequisite paths or how well those paths explain proficiency. (instead of a binary question)
It might be worth considering the relationship between question IRT scores and the identified prerequisites. I wouldn't expect an exact match but something that is pre-requisite to something else shouldn't be wildly harder.
What I have as input is the Math proficiency as a whole, being assessed by the test, and a lot of skills (~30-40) which are, as you said, students' knowledge within a particular area or ability to perform certain types of tasks - measured by individual items.
What I am trying to achieve is to find relationships with those skills, so that I can build a graph of growing complexities of skills, with different paths. The ultimate goal is to better assist instructional improvements, driven by data evidence.
For example: in Math, I might find that ''A-reading information from tables" is a prerequisite both for "B-identifying equivalent fractions" and for "C-identify triangle properties". They are all Math skills, but with different (and maybe related) specific knowledge components and, so, I should master A before moving on to B or C.
PS: the link you posted is broken.
I think that's fine, but you'll want to spend some time thinking about how to validate the resulting rules (beyond the basic machine learning approaches).
Sorry about the link. ETS appears to have reorganized their site since I entered that report into zotero. https://www.ets.org/Media/Research/pdf/TC-10-07.pdf
I am not sure if it is of direct relevance to your work, but I think the following link on educational data mining research may be useful.
http://www.aspiringminds.com/research-articles
best wishes
- Badges
- Science topic
- Similar topics
- Methodology
- Research Methodology
- Research Papers