Dear Shelly, 

It is OK to shorten an instrument but you have to check Cronbach's alpha again so that it is at least 0.70 or higher

Yes, as Beatrice says, you can shorten the instruments, but you have to check whether the new version still performs well enough -- so you would be taking a risk.

My advice would be to look further for short versions of your instruments. For example, the Pearlin Mastery scale (locus of control) is only seven items. 

Thanks Beatrice and David! that was very helpful.

I concur with the responses above, but want to add that you need to consider how you will be using the data. If the original instruments are standardized, and rely on pre- established norms for scoring and interpretation, than "borrowing" just a few items here and there may be risky in terms of lowering the validity of the instrument you are creating,and of course in such a case, you cannot rely on the previously established norms.

However, if you are not in need of standardized instruments, you should be able to shorten the original instrument. I suggest that in addition to testing for reliability again, as proposed by my colleagues,  you conduct a factor analysis to establish the validity of the revised instrument. Finally, watch for issues associated with copy-right of the original instrument. You may need o get permission from the author/publisher to revise the instrument.

Behazlacha!

To use some items from a standardized instrument does not take away the standardization from the items!

What I meant was that it was taking away from the ability to compare scores from a revised or shortened instrument   to the norm groups on which the test was standardized.  But thinking about it as per your comment Beatrice, I have to respectfully disagree.  Using only a few items very much interferes with standardization, because a standardized tool is designed to be administered  in a uniform way.  using only a few items, perhaps in a different  order than they appear in the original,  compromises the uniformity ( standardization)  of the instrument and of the administration.

Dear Orly, 

The standardized test items work as well as they worked within a very big group. To take away some items does not compromise the procedure of standardization, as long as you control that the set of items live up to the validity and reliability  of the original scale.

https://en.wikipedia.org/wiki/Standardized_test

Adapting a questionnaire- is it ok to omit items or only use parts of a questionnaire?

Yes you can - that's why we use the word "Adapt".  Adaptation sometimes might include changing certain words to match your research context e.g. targeting project managers instead of sales people, rewrite the sentences to make your respondents have better understanding on your questions asked, translation from foreign to local languages etc.  But all these need to be handled with care, seek advice from expert, walkthrough with pilot respondents for improvement etc.  Also the acid test will be to run Cronbach Alpha / Composite Reliability test on your pilot study to ensure > 0.7.

Yes it is ok

Dear Beatrice, we will have to  agree to disagree. For a detailed discussion of standardization of data collection instruments, see Grinnell's work. Having said that,  As long as Shelly  is aware of all the empirical and psychometric implications of adapting/shortening tools, she should be able to do so. 

I love that we are able to have a conversation about research that encompasses scholars from around the world! 

Hi Shelly,

Quite an interesting question that you have posed and often among the decsions that researchers have to make when undertaking empirical studies (i.e. development of a new instrument OR using existing measures). The approach you have undertaken in developing an instrument is the most economical one, namely searching the literature to find an instrument that may be suitable for the study under investigation. For example, if you had opted for the first option, namely that of developing a new instrument, though it might be desirable where a new concept is being studied, it has its own negatives. For example, some studies have shown that developing new constructs or scales of measurement is a complex task. In contrast, using preexisting instruments have the added advantage. Furthermore, it is recommended to use pre tested constructs from past empirical studies to ensure their validity and reliability.

In additional to the correct suggestions posted (i.e. OK to shorten the questionnaire), I would also suggest providing some explanations and justifications for the refinements (or adaptations as referred to by Hans) made to the existing instrument(s). In fact some of the reasons for the refinement are already included in your question as posted (i.e. items not relevant for your study). And as just suggested by Beatrice, it is still important to carry out the testing for reliability, undimensionality and construct validation for the shortened instrument. The following studies by Faragarasum and Kumar (2002), and Tata et al. (1999) are worth looking at as they provide some insights into the issues around options 1 & 2 (i.e. development of a new instrument OR using existing measures):

Faragarasum, M. and Kumar, S. (2002), Measurement instruments and data collection: a consideration of constructs and biases in ergonomics research, International Journal of Industrial Ergonomics, 30: 355-369.

Tata, J., Prasad, S., and Thorn, R. (1999), The influence of organisational structure on the effectiveness of TQM programs, Journal of Managerial Issues, 11(4), 440-453.

All the best with your research,

Nicholas

If your original articles published factor analyses or equivalent information (such as Structural Equation Models) on the scales in question, then you could pick the items that had the highest loadings. This would be your best bet for ensuring that those variables are still highly correlated in your short-form version.

Thank you all so much for taking the time to write such detailed and informative answers. That is very helpful.

I have another question. For almost every construct, there are general and context - specific instruments available (e.g technology acceptance versus car technology acceptance OR locus of control versus driving externality - internality ). It is no surprise that the general questionnaires are usually more cited and well established. However, I suspect that using context specific questionnaires would have higher external validity among other benefits. (My context is autonomous vehicles willingness to use). Do you have any thoughts on this topic? Perhaps including general instruments for some of the constructs and context specific instruments for other constructs will be a good idea?

Thank you .

I agree that context-specific instruments will have a stronger relationship with other content-specific variables, as opposed to the more generic versions of such measures. That is certainly the case for locus of control.

The main reason that I can think of for including both generic and context-specific measures would be methodological -- i.e., to examine the difference in the effect sizes etc. But if you are mostly interested in maximizing your ability to account for a dependent variable, then you should use the context-specific measures.

Thank you David, I appreciate your answers. I am currently considering shortening my questionnaire because it is too long. I was trying to include shorter instruments as you previously suggested. However, most of them are general (i.g., Pearlin Mastery scale of LOC). Considering all of your suggestions, perhaps it will be best for me to use context specific instruments (if available) and shorten them with careful consideration.

I don't know about other instruments but the Pearlin Mastery has certainly been converted to context-specific versions. So one option is to make your own context-specific version of an existing, shorter scale.

hi shelly,

the answers of David Morgan and Nicholas are very intersting. I follow your discussionin in methodology carefully ,  it is fruitful.

Hello all

I looked online but was not able to find unambiguous answer.

What sample size do I need to conduct alpha Cronbach test ? I have about six constructs (meaning six different instruments), each has 7-22 items.

The the total number of items for all constructs is 87.

As this is a pre- test, I am trying to be as efficient as possible.

Thank you

There is no size constraint on Chronbach's alpha itself, since it is simply based on a correlation matrix. You could look up power analyses on correlations to get a sense of how sample sizes affect the standard error. Alternatively, for a full scale test of a measurement model, it is common to get a recommendation of 5-10 respondents per correlation parameter in the model.

In general, I suspect you have a problem where your items were generated outside a survey setting, probably with "captive" populations who had considerable time to fill in long instruments. Unless you have further information about the factor structure of those instruments, there isn't a lot you can do about that problem.

Thank you for your answer. For a factor analysis, what is the minimum sample size? I have about six constructs (meaning six different instruments), each has 7-22 items.

The total number of items in my questionnaire is 87/

There a number of different online "calculators" that will help you estimate sample size. There are also a number of less formal recommendations for determining sample sizes for factor analysis, such as a N = 10 per loading in your model.

Just to make it more clear. The N=10 per loading means that if I have 90 items in my questionnaire, I will need a sample size of 900?

I appreciate your thoughts very much. Thank you.

If you followed that "rule of thumb," that is what it would imply, which suggest that you you a different approach, such as an online "power" calculator. 

If you do not have 900 subjects but 90 items,

all of the 90 do not work and you delete them

when you calculate your Crohnbach's alpha for a group of items in 

order to get a good internal reliability. 

88

The standard advice is to have at least 10 participants per item on your scale. However, this should be regarded as the bare minimum.

There are surprising differences of opinion in the literature, however, on how small your sample can be, and the best advice is based on simulation studies where authors have studied the power of samples of various sizes to detect a given alpha.

The answer depends on the structure of your scale. Sample sizes as small as 30 can measure alpha reliably so long as the scale items have strong inter-correlations.

First step : principal components analysis

Your analysis should begin with a principal components analysis. A principal components analysis identifies underlying ‘dimensions’ that account for the variation in a set of items. In the case of reliability, you should only examine the first principal component. There is a good reason for this: alpha has no interpretation when scales combine items that measure different constructs. The first principal component measures the degree to which the items measure the same construct.

Samuels, summarising the literature, makes these recommendations

1. Don’t run reliability analysis with less than 30 participants

2. If you have between 30 and 50 participants, remove items that have loadings of less than 0·4 on the first principal component. This means that that very little of the variation in the responses to that item are shared with the other scale items.

3. Rerun the principal components analysis and examine the first eigenvalue (the eigenvalue for the first principal component). If this is less than 6, do not attempt a reliability analysis; the items just don’t show enough homogeneity to yield a reliable estimate of alpha.

4. Ideally, scale items should have a loading of 0·8 or more on the first principal component. Items between 0·4 and 0·8 need to be considered carefully as candidates for inclusion.

5. If your sample size is between 50 and 100, then follow the same steps, but if your eigenvalue falls between 3 and 6, then only perform a reliability analysis if the sample size is at least 75. See Yurdugül for details of how these figures are arrived at.

Samuels, P., 2015. Statistical Methods – Scale reliability analysis with small samples, Birmingham City University, Centre for Academic Success. DOI: 10.13140/RG.2.1.1495.5364. https://www.researchgate.net/publication/280936182_Advice_on_Reliability_Analysis_with_Small_Samples

Yurdugül, H., 2008. Minimum sample size for Cronbach's coefficient alpha : a Monte-Carlo study. Hacettepe University Journal of Education, 35, pp.397–405. http://www.efdergi.hacettepe.edu.tr/200835HALİL%20YURDUGÜL.pdf

Technical Report Advice on Reliability Analysis with Small Samples

Ronan- Thank you so much for the informative answer- I will try to follow those steps.

Thank you Beatrice and David, I appreciate your thoughts very much.

Hi

I am a little confused over this matter: I have an instrument that is composed of 6 existing scales I found in the literature. Wherever possible, I used the original scale's factor analysis (if they published one) to eliminate items that loaded weakly. However, that wasn't enough and the questionnaire now has 87 items total (which is way too long). I wish to reduce it by half. I administered the questionnaire to 100 respondents. Now, I am not sure weather I should conduct a Factor analysis, Cronbach's alpha test, both (and if so how/ in what order). I got mixed answers about that before...

Thanks you so much again.

Shelly,  if your goal is to reduce the number of items and still wind up with a valid and reliable instrument,  you should conduct both a factor analysis, eliminating the items with the weaker loading  (which will address the validity of the adapted instrument ),as well as cronbach analysis, which will address the reliability of these remaining items in the adapted instrument. 

With data from 100 respondents,  you should be able to trust the coefficients of both these procedures and produce a new scale, as it is,  wit demonstrated validity and reliability.  

Thank you so much Orly.

Should I run a factor analysis for each scale (or sub scale to be exact) separately or for the entire questionnaire altogether? 

Unfortunately, some scales now have Cronbach's alpha of under 0.7 (one is 0.3, another is 0.5) any suggestion what can I do about that? Is there any way around omitting these scales completely? 

The one that got a alpha of 0.3  is "need for control"- I used Burger's and Cooper's scale and I couldn't find any other scale that measures exactly that, I will appreciate any suggestion. 

Hi, not knowing the nature of the scales,  it is hard to know how to structure the factor analysis.  But generally,  use all the items,  ensure that items that are supposed to measure the same construct load high, and those that measure different construct 

Load low.  

As for reliability,  .7 abd even .5 is not bad. Low reliability is a factor of error , both random and systematic.  You cannot do much about random error,  except acknowledging it. As for systematic error,  sometimes you can correct this by wording questions better etc. But I would be wary of further modifying the instruments you are using by changing the language of the questions.  

For the scale with an alpha of .3 I would suggest that you examine the correlation matrix among the items, because alpha is based on an average of the correlations. If only one or two items are "off," that can lower the average correlation for the full set.

With 87 items, a different approach from eliminating the items with low loadings is to concentrate on creating pool of items with high loadings, so reduce the size of the scales.

Thank you so much Orly and David

i am agree with Ronan Michal..... his answer is supportive and to the point.

dear Shelly Ben Zvi , maybe the following post can possibly help you to understand

http://korbedpsych.com/R09aAdopt.html#:~:text=The%20first%20is%20to%20take,instrument%2C%20which%20is%20called%20adapting.&text=However%2C%20sometimes%20an%20instrument%20is,therefore%20should%20not%20be%20adopted.

Thank you

Hi,

Yes, you can do that. But you need to consider the following points:

1. Take permission to modify the scale from the developer of the scale.

2. Perform the validity analysis

Content validation (experts in the field)

Criterion validation (using commonly used scales on similar domains)

Construct validation (factor analysis or Rasch analysis)

Face validation (by interviewing the participant from the intended population)

3. Perform reliability analysis

Internal consistency (Cronbach's alpha)- make sure your sample size is enough;

multiply the no of items with 10 or calculate using Raosoft Calculator

http://www.raosoft.com/samplesize.html

Two weeks or one-month test-retest reliability analysis

Person and item reliability by Rasch analysis- less sample size sensitive

Thank you