Hello Björn,

Yes, there's been discussion of methods to overcome this challenge.

One, that was mostly applied to classical test theory models, was multiple matrix sampling. David Shoemaker's expositions (see below) are a very good introduction.

The second, which arose from equating in classical test theory models, and calibration of large pools in item response theory models, is that of anchor tests or items (common item sets and unique item sets administered to subsets of the total sample).

Good luck with your work!

MMS references:

Shoemaker, D. M. (1973). Principles and procedures of multiple matrix sampling. Ballinger.

Shoemaker, D. M. (2006). Multiple matrix sampling. In Encyclopedia of Statistical Sciences. New York: Wiley. (Online DOI: 10.1002/9781118445112.stat02622)

Gonzalez, J. M., & Eltinge, J. L. (2007). Multiple matrix sampling: A review. Washington, DC: U. S. Bureau of Labor Statistics, Office of Survey Methods Research. https://www.bls.gov/osmr/pdf/st070070.pdf

Equating/IRT item linking for calibration references:

Kolen, M. J., & Brennan, R. L. (2013). Test equating, scaling, and linking: Methods and practices (2nd ed.). New York, NY: Springer.

Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago, IL: MESA Press. (Available for free download: https://research.acer.edu.au/measurement/1/) (see ch 5)

Sounds like you've created an instrument which is sound as it is based on theory - but this version should only be the pilot version, rather than the final instrument. As you see - there are too many items, which will affect the quality of the data collected. Have you conducted any sort of pilot study - factor analysis - and tested to see how those items relate? You'll probably find some items are redundant, and possibly even some scales ... use EFA to explore how the items load - then as you delete items - the number of items in each scale will decrease - you can then delete scales on the basis of insufficient items.

I find David De Vaus' work on survey design and validation very useful:

https://www.amazon.co.uk/Surveys-Social-Research-Today/dp/0415530180/ref=pd_lpo_sbs_14_t_0?_encoding=UTF8&psc=1&refRID=ER3KY0K00XZC84NB7GHW

It is also worth considering is an instrument should be trying to accurately measure 24 constructs. For many uses it users of the scale might be interested in only a subset. Are they meaningful ways to divide this instrument by construct type?

Hello Björn,

I think you've been given some excellent advice above - and thank you for posting your question because a colleague and I are facing a similar dilemma - though not on the same scale as yours. We are starting with 43 items, having scoured the relevant literature. That initial stage required us to adopt and adapt items from previous instruments where copyright didn't apply. (Might copyright be an issue in your research?)

We initially argued back and forth about which items were essentially redundant on face value, and we winnowed / culled items substantially at that time. (Might you have done enough of that kind of activity?)

On top of that, we are currently collecting data from a small group of people who are representative of our future participants, to get an idea about the relevance of each item (assessed on a 4-point Likert scale). As a result of their feedback, we might be able to remove some more items as being too "off beam".

After that we'll factor analyse the data that we get, but, if we have about 40 items, we'll need about 400 respondents in order to satisfy what is often regarded as an appropriate number of participants relative to the number of items being analysed. Might that kind of thing be relevant to the issue(s) you face - i.e., being able to access a sufficient number of participants for factor analyses given the number of items on hand?

We are concerned that 40 items might be too many for people to want to commit time to, so we are taking particular care with the design of our instrument so that it is not likely to come across to participants as burdensome or, worse, a waste of their time to the point that they cease responding mid-way through.

I hope these reflections are useful for you.

Robt.