There is an interesting Law of the Hammer (see link *), which states the following truism: "If the only tool you have is a hammer, it is tempting to treat everything as if it were a nail." This Law may be working in some areas of Psychometrics, too.
If the only cognitive tools you have, are ordinary arithmetic and statistics on the real line (i.e., all negative and positive numbers), then you may be tempted to treat ALL measures as REAL numbers and apply the usual arithmetic and statistical operations (plus, times, arithmetic mean, etc.) on REAL numbers, to get whatever you want, e.g. estimates and predictions.
The Hammer of IRT/Rasch
This seems to be the case for IRT based on a naive concept of scores. Scores, in ITS's view, are just like real numbers, X for short, which can be neatly associated with (cumulative) probabilities via an elegant transformation (conversion) rule:
- P(X) = exp(X) / (1 + exp(X) )
Once you have those probabilities, the whole machinery of probability theory and statistics can be invoked for descriptive or testing purposes. Adding one, two or three parameters (which have to be estimated) increases the explanatory power of the approach even if your responses are still binary 0 and 1's. Generalizing to polytomous and multidimensional IRT etc. is possible within the same adopted framework.
Still, all these developments rest on a highly questionable, implausible interpretation of X as the genuine measurement of an underlying (psychological) feature and P(X) as its associated probability. However, there is no empirical evidence or formal argument, that such an interpretation is possible, necessary, meaningful or even correct.
On the contrary: it is highly counter-intuitive. There are no (latent) psychological features or variables like X which are strictly unbounded to either side, i.e. could become - infinity or + infinity. If you happen to know one, please let me know.
An alternative view avoiding the concept of probability
The alternative is, to view the real number X as a transformed score f(S) from the double-bounded interval [A,B] between given A and B to the real numbers.
- For instance, A could be 1 and B could be n, with equal-spaced anchor points 1, 2, 3, ..., n, so that [A,B] is the underlying continuous scale of a discrete n-point ruler or scale for the user/respondent.
Now let S be such a score on the n-point scale from 1 to n, and define X such that :
- exp(X) := (S - A) / (B - S)
Then X is well-defined: it is the (natural) logarithm of (S - A) / (B - S). Now, after a few simple manipulations we get:
- S = (1- P(X)) * A + P(X) * B so that P(X) / (1 - P(X)) = (S - A) / (B - S).
In other words: P(X) is just a weighting factor required to get the position (location) of the score S on the chosen n-point scale from A to B. Indeed, we could replace P(X) just be W(S), i.e. the weight of S on [A,B]) to get:
- S = (1- W(S)) * A + W(S) * B so that W(S) / (1 - W(S)) = (S - A) / (B - S).
In other words, W(S) = P(X) is just a normalized version of S given that S is constrained to [A,B]. No probability interpretation is required!
What's more: we can define the quasi-addition (+) of scores S and T on any scale [A,B]. It is, in fact, relatively simple:
- S (+) T := (S - A) * (T - A) divided by (S - A)*(T - A) + (B - S)*(B - T)
This quasi-sum (+) of S and T has (almost) ALL the properties of the usual sum of two real numbers, however, it is closed on the interval between A and B. Defining quasi-multiplication (*) of a score S with a real number is also quite simple. And once we have quasi-multiplication, the quasi-additive inverse is correctly defined as (-1) (*) S. Together, Module Theory provides the right framework for working out the algebra of bounded scales and scores.
Introducing the interval scale on [A,B]
The biggest surprise of all: it turns out that the n-point scale endowed with this quasi-arithmetic addition operation is indeed an interval scale. The same holds for the standard percentage scale [0,1], so often used in educational contexts, but also in Fuzzy Set Theory and Fuzzy Logic.
CONCLUSION:
Rasch, one of the founders of IRT, was an original mathematician and statistician, but he was rather focussed on classical statistics and measurement theory from the physical sciences, and probably he didn't know about the then-recent developments from what is known now as quasi-arithmetic calculus and module theory.
Also, in his time, Stevens was one of the hero's of a rather naive (=operationalist) scale theory for psychology and other social sciences (see e.g. the critiques of J. Michell, Luce, and other measurement theorists).
I am sure, that Rasch would be delighted to see that it is indeed possible to do serious measurement (theory) in the social sciences without the counter-intuitive assumptions that he had to make.
________________________________________________________________________
*) https://en.wikipedia.org/wiki/Law_of_the_instrument
What about
Article 11 Applications of Multidimensional Scaling in Psychometrics
https://www.springer.com/gp/book/9783642318481
http://citeseerx.ist.psu.edu/showciting?cid=5016686
Maybe, it is out of 'fashion'?
Paul Hubert Vossen, I found your post interesting, but I confess to not understanding everything my any means.
However, I wonder what your thoughts are concerning classical test theory, particularly in relation to what you have posted as your question.
Robert Trevethan : Thanks for your reply. Yes, I am aware, that my approach to scoring and rating without using the common statistical mindset and framework leads to many questions.
The purpose of this discussion thread is exactly to try and build up such an alternative mindset and framework from the basics, avoiding as much as possible the mathematics on which it is based, because that would unnecessarily scare off a lot of people. Which would be a pity, because there are so many people, e.g. teachers and others who have to provide evaluative judgements, who desperately need such a non-statistical approach.
I am very lucky to work together with Dr. Suraj Ajit from the University of Nottingham on implementing a web-based tool for peer assessment based on my general approach to scoring and rating. His many questions for explanations and examples has helped me much to see the problems people may have with understanding and accepting this new approach.
If you can't wait for my further contributions to this discussion thread, I recommend you to read the joint paper with Dr. Ajit about the required paradigm change in group and peer assessment in software engineering education. The paper will be discussed in a virtual session at the upcoming conference CSEE&T, on Thursday next week.
P.S.: As a student, we were taught all the basics of Classical Test Theory (CTT) along with the early statistical theories in psychology. However, CTT isn't a viable alternative to modern psychometrics, based on a sound measurement theory which was just building up in the time that Lord wrote the "bible" for CTT.
Here are the three most recent publications explaining the basics of a non-statistical approach to measuring, scoring and rating in a non-research context. Start with the 1-slide teaser. Then go on with the video. For full details read the paper.
______________________________________
https://ase.in.tum.de/cseet2020/index.php/presenter-discussant-assignments-with-presentation-material/#feedback
What about
Article 11 Applications of Multidimensional Scaling in Psychometrics
https://www.springer.com/gp/book/9783642318481
http://citeseerx.ist.psu.edu/showciting?cid=5016686
Maybe, it is out of 'fashion'?
Congratulations Paul for raising a fundamentally important question. I've worked at it for 50 years and obtained findings that I have been able to convince a few hundred people of their value (see my Teachers, learners, modes of practice: Theory and methodology for identifying knowledge development and www.changingwisdoms.net). But that is a drop in the bucket of what is needed. Adding answers to multiple choice questions is a prime example of your Law of the Hammer. Its results are as meaningful as counting the items in a produce department (1 pea + 1 watermelon = 2 items), but most educators still do it. Instead I made developmental rubrics with multiple dimensions each having a few levels of complexity that I called modes of practice. When I plot the frequency of usage across time, I find that within a dimension the modes compete like species in an ecosystem with frequencies determined by initial frequency, reproduction rate, and competitive strength. Programs that sustain reliable use of these rubrics for student evaluation result in faster knowledge development and better student retention in the program. But getting to reliable, sustained use requires understanding of your Law of the Hammer and commitment to a reasonable and effective alternative, even if it requires more work to get it right.
Stephen I. Ternyik : Out of fashion? Psychometrics? I had to smile a bit as I happened to read just yesterday the paper "The fashionable scientific fraud: Collingwood's critique of psychometrics" by Joel Michell (University of Sydney, Austria) in History of the Human Sciences, vol. 33(2) p. 3-21, 2020.
Well, first of all, psychometrics is much more than Rasch or general IRT models. I might be that the success of and publicity around IRT in the domain of psychological and educational testing has led many people to think that IRT and psychometrics are the same.
Therefore, and for some other reasons, I prefer the more general and neutral phrase "mathematical psychology" for all theoretical and empirical research in psychology which is aimed at creating and testing formal models of psychological phenomena, including as the case may be the measurement of relevant variables.
And yes, multidimensional scaling is one of the many mathematical models and techniques which have been invented with the intention to place scientific psychology on a sound basis, which - surely - includes psychological measurement, but goes far beyond that.
Thus, no, I don't believe that this field is out of fashion. See the New Handbook of Mathematical Psychology: Volume 2, Modeling and Measurement which appeared only two years ago....
https://www.amazon.com/-/de/dp/1107029074
David Dirlam : Thanks for your nice congratulations! You made me curious to your book "Teachers, learners, modes of practice: Theory and methodology for identifying knowledge development" but curiously enough I couldn't locate it through the central academic library lending service here in Germany. Thus, for the time being, I will start with reading your recent paper "How Modes of Practice Revolutionize Learning and its Assessment", which I found here on RG. I'll contact you again as soon as I have had time to read it thoroughly (after my conference presentation next week).
With respect to your comment:
There is an interesting Law of the Hammer (see link *), which states the
following truism: "If the only tool you have is a hammer, it is tempting
to treat everything as if it were a nail." This Law may be working in
some areas of Psychometrics, too.
Comment 1: Before going further please consider the following constraint as to the consequence of the driven nail.
https://www.google.com/imgres?imgurl=https%3A%2F%2Fi.pinimg.com%2Foriginals%2Fb4%2Ff3%2Fa7%2Fb4f3a7f71d5c977df8349dfe3f604c36.jpg&imgrefurl=https%3A%2F%2Fwww.pinterest.com%2Fpin%2F527554543847745342%2F&tbnid=zsQ2V3wfWq3fIM&vet=12ahUKEwjzzpHQtPDsAhURMqwKHa5WCtMQMygBegUIARCkAQ..i&docid=oUOQVaFqULHBzM&w=1280&h=720&q=lessons%20taught%20by%20life%20nails&ved=2ahUKEwjzzpHQtPDsAhURMqwKHa5WCtMQMygBegUIARCkAQ
What is clearly evident in the preceding video is the effect of "bio-feedback" and it's effect on the social system. This system in and of itself requires clarification by the understanding of logic and emotion (these of which are being designed into the the QMH (Quantifying Mental Health) forensic simulator.
Regarding:
If the only cognitive tools you have, are ordinary arithmetic and
statistics on the real line (i.e., all negative and positive numbers), then
you may be tempted to treat ALL measures as REAL numbers and
apply the usual arithmetic and statistical operations
(plus, times, arithmetic mean, etc.) on REAL numbers, to get
whatever you want, e.g. estimates and predictions.
Comment 2: Real measures in themselves have statics and dynamics embedded within and without. Extreme caution is required in the "modeling of these notions"
in order to avoid "instabilities and statistical drift" .
All systems have: a) static, and b) dynamic characteristics and as such can be modeled using CLM theory by the 2 equations of the system:
1) Measurements (m in number) , and the
2) State (n in number)
Both the measurements and the system must incorporate their respective probabilities. Specifically the measurement uncertainties are given via it's
m x m measurement R covariance matrix. Whereas the state propagator uncertainties are defined a priori via its Q m x m covariance matrix. In addition it is assumed that R and Q are independent processes -- see Kalman Filter theory.
The forensic simulator of QMH is built on the foundation of CLM (Closed Loop Methodology). CLM is based on it's modeling theorem which can be stated as follow:
The CLM Modeling Theorem: CLM is a successful integration of estimation, identification, and control algorithms adapted from well-established methods in state-space variables, parameter/state estimation, and control theory. The resulting, demonstrated capabilities of CLM make it possible to state a general theorem about system modeling: A system model will have sufficient fidelity to predict or reproduce measurements from the actual system if and only if it contains an adaptive processor comprising algorithms for generating the functions of system Estimation, Identification, and Control-see attachments.
The estimation problem in CLM is statistically based in order to minimize the covariance if the system which satisfies Newton's Laws. As such the solution is guaranteed to converge to the proper mean provided the system is stable. Stability may be quantified by examination of the system eigenvalues. Eigenvalues are obtainable once the nonlinear system is linearized about a nominal operating point to be known as one's cognition which is obtainable through the solution to the human's SID (System IDentification ) problem. This is fundamental in order to insure system stability which will yield the proper statistics in all static and dynamic systems. For example: See the information given in the paper "Closed Loop Methodology (CLM) Applied to Signal Processing - A Revisitation "
This paper presents the vital role mathematical modeling plays in the solution to the signal processing problem. The paper will compare the traditional open loop methodology of signal processing to the modern closed loop methodology. Advantages and disadvantages will be discussed. In the post, system discriminants have been derived from knowledge of the signal bandwidth and characteristics of the receiver while actual signal propagation had been ignored. The result of such a formulation generally leads to incorrect statistics. This traditional approach has been improved by use of a closed loop methodology (CLM) technique which employs an adaptive processor to provide correct signature statistics.
--see Article Closed Loop Methodology (CLM) Applied to Signal Processing -...
Hopefully the information provided above is of value to your thought processes rather than a diversion to it.
Respectfully,
Al Fermelia
CLM Associates
email: [email protected]
Please consider the following comparisons as attached
1) probability error function and
2) Fibonacci mete physical time function
Al Fermelia : Thanks for your contributions on modelling and simulation of dynamic systems. Yes, of course, under the umbrella of mathematical psychology, you will find several approaches to constructive cognitive psychology, i.e. the idea that one may come to understand cognitive processes better by building computational models of those processes and thus simulating human sensory, memory, thinking, problem solving, decision-making etc.
- Two of the most famous pioneers of this sort of psychology are Nobel Prize Winner Herbert Simon ("The Sciences of the Artificial") & Allen Newell in their book Human Problem Solving (new edition 2019). Another famous researcher in this area: John R. Anderson in his "The Architecture of Cognition". You will find more recent research in this area in the Second Handbook of Mathematical Psychology I mentioned earlier in this discussion.
Having said this, I must quickly add that this discussion thread has another focus/intention. It is solely dedicated to the goals of psychometrics and edumetrics to come up with measurement models in the social sciences, especially in the field of human abilities and performance, clearly relevant to educational settings, but also of interest to e.g. ergonomics.
_______________________________________
Dear Paul,
Your comments have produced a level of excitement within me that supports
the current goal of QMH. That is to model the human system with respect to it's optimization of the human's unconscious interactions of Ethics, Economics and Education as stimulated from the time of birth
It is interesting to note that 3 points are required to produce a plane but only 2 to create a teeter-totter. I have also noticed the many discussions go to fast. For example: In your reply , an addition word that could be highlighted is focus. For it this word from which ethics according to the Michael Jensen the co-founder of SSRN may be quantified.
Thank you Psul for your insight and guidance.
Respectfully,
Al Fermelia
I have added some stuff in the description of this discussion thread.
Dear Paul, many thanks for raising this point. You are completely right that the WLE from an IRT modelling and simple sum scores relate to r > .95 in almost all cases, where the scales are not too skewed - I hope, I got your point correct. So, these are almost always interchangeable. IRT is a theory with specific axioms and these are testable. The huge advantage is that you can test model assumptions and select items or identify persons with deviating answering patterns. I however agree with you that there is much much more to psychometrics than just 1PL, 2PL and 3PL IRT models, like for example simultaneously modelling accuracy and time (Fox et al., 2007) or speeded tests with time cutoffs with Poisson Counts Rasch Models (or further developments; Boris Forthmann et al, 2020; https://www.researchgate.net/project/Scaling-speeded-tests-with-the-Rasch-Poisson-Counts-Model ). The draw back of the last decades is the immense focus on Rasch models and much of the development in psychometrics was not noticed or was restricted to discussions in specialized journals. In addition, there has been a narrowing of certain approaches, as aspects such as fluency are much more difficult to model with Rasch.
Regarding your remark on the normally distributed latent ability, this of course is a theoretical assumption as well, but practically, only the range of +/s 4 sd is relevant at all. And in reality, scales are always restricted. Consequently, in applied psychometrics you always need some kind of norm score to model the relation between persons. For this, you indeed do not need to do an IRT-based scaling (but of course for the item selection, model tests …), but you can model the raw and norm scores directly, either parametrically with approaches like GAMLSS or via simple Taylor polynomials, as it is done in cNORM. Maybe, the following paper is of interest for you, since it simulates, how raw scores stem from a latent trait and are subsequently modelled with Taylor polynomials to predict the latent trait in a huge, representative sample (cross validated). In the end, in reality you get regression functions functions that relate the real scores to the latent trait: https://www.researchgate.net/publication/341192767_Improvement_of_Norm_Score_Quality_via_Regression-Based_Continuous_Norming
Dear Wolfgang, many thanks for your thoughtful comments and references! Nice birthday present for me, too. After I have been able to read your comments and some of the references therein, I'll reply more fully. Regards, Paulus
Dear Wolfgang,
Yes, you are right: one of my rather pragmatic points is, indeed, the question of whether it is necessary and relevant to adopt IRT's or related statistical machinery at all if in daily practice much simpler approximate approaches (still in the form of well-defined methods and techniques!) will do as well.
The daily practice I and my co-workers/co-authors are mostly interested in is educational assessment in the classroom, mostly in the form of quantitative summative assessment, i.e., at the end of the day (so to speak) a teacher or instructor needs to be able to provide a report on what the individual students in a given class have achieved in terms of learning outcomes and/or how they have performed in terms of learning objectives, using either the standard percentage scale or some derived grading scale (mostly a step-function defined on the percentage scale). Not a typical research context.
In our case, the target population/context has mostly been software engineering classes/courses, where the exams weren't classical tests of some standard type but rather complex construction assignments to be completed under authentic, i.e., semi-realistic conditions, sometimes also in the form of group projects. Time to thoroughly prepare and evaluate the used assessment criteria, rules and procedures is mostly not available. Worse: the assignments and assessment schemes may change from semester to semester.
The assessors are highly specialized and competent in software engineering techniques, but usually lack the psychometric background and knowledge to adopt an approach as e.g. IRT and adapt it to their highly specialized assignments (indeed possible but highly sophisticated, see e.g. https://www.researchgate.net/lab/Maomi-Ueno-Lab).
Thus we have been looking for assessment approaches and methods which mimic very closely how instructors in design areas like software engineering conceive of and deal with their assessment (judgement) tasks, especially in the context of group projects cum peer assessment.
One highly successful approach turned out to be based on the theory of quasi-arithmetic scoring using n-point scales (where n is rather small) and an arbitrary list of quality criteria/items (where the list may be rather long). This approach assumes that scores can be added and multiplied by a scalar, i.e. they form a well-known algebraic structure called module. Measurement turns out to be on an interval scale. The rest follows easily.
Another approach I am currently working on (taking up previous work from around 2015) centres around the concept of the lexicographic ordering of discrete probability distributions which gets mapped (respecting the ordering) to the same scale used for the original n-point measurements (using generalized multinomial coefficients and the BETA function; see e.g. Handbook of Beta Distribution and Its Applications https://www.amazon.de/-/en/Arjun-K-Gupta/dp/0367578328). This is purely ordinal measurement which nonetheless yields quantitative scores which can be converted to percentages or grades. In that respect, it is perhaps not much different from using percentiles which you mention in your paper.
Dear Paul,
ah, interessting. I think you are right and IRT might not be the ideal framework in that case (small groups, non overlapping item sets ...). The scenario, you describe, rather ressembles a criterion based test for me, where it is not necessary to do a complete scaling: Either the testees manage to solve the task, which is a sign that they reached the learning goals, or they do not. In university tests, we partly do scaling of the data to identify problematic alternatives / questions and determine reliability, but overall, we use a Bernoulli process to identify chance pobability and set the criterion for passing the tests accodingly. We then can determine if somebody scored significantly above chance and we define thersholds like 30% above chance for success. Our exams are multiple choice and single choice tests however and this might not apply for the complex scenarios, you describe.
Best regards,
Wolfgang
P.S.: I think, there is nothing wrong with using the raw scores in your case, but it is rather a question how to set thresholds for grading.
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics