Yes, It is not recommended to choose arbitrarily some items because you modify the psychometric quality of the scale unless you test all items and you adapt the scale consequently...

Dear Youjae

Actually five-item or seven item scale is the general practice. I think three items may not give the actual results. So best try to avoid three-item scale.

Thank you.

Yes, It is not recommended to choose arbitrarily some items because you modify the psychometric quality of the scale unless you test all items and you adapt the scale consequently...

Dear Youjae

The final decision depend on the result of ststistical analysis of both exparatory factor analysis and then confirmatory factor analysis

Dear Youse,

Fuzzy scale is another alternative, it can tackle possible message from the subjects.

Dear Mr Youjae,

This is a question of faith of reviewer. Just present Information in a clear way that how you've really done.

Dear Youjae,

I guess the reviewer's concern is about the content validity of the multi-item scale you use in your study.  I also assume that you use a reflective measurement scale.

In my opinion, reducing from 5 to 3 items may negatively affect the content validity and reliability of the scale. Therefore, you should check the conceptualization of the concept you are measuring and explain / make sure that the three items are still able to capture all facets of the concept. Also, the reliability of the scale should be above the threshold.   

Good luck !

Hi, the answers are interesting. I suspect your choice can raise an issue of MEASUREMENT ERROR of the variable. I remember an MPhil defence in which a student clubbed three responses in to two to use Binary setting models like Probit. The serious issue of measurement error of the LHS variable makes all the quantitative exercise spurious and biased.

Few items scale violates the assumption of parametric statistical analysis of IID. Which is Independently, Identically, and Normally distributed data.

Restricted number of items scale hinder the Independence choice of your respondents meaning you are forcing them to choose from a restricted number of choice.

I suggest you to use the interval scale from 1 to 10 instead.

Let's be honest here. Your choice of which three questions to use was arbitrary. You didn't run a formal data analysis that showed that a three item scale has just as good psychometric properties as a five item scale.

So just say in the methods section that you needed to shorten the survey and the choices as to what items to remove was made based on the subjective opinion of a subject matter expert (yourself).

Then when the excessively picky reviewer (in my opinion) rejects your paper, send it to a different journal.  In general, it's a bad idea to change scales because a reviewer is likely to raise concerns. But I think that the practicalities of research sometimes require you to shorten a survey, because a shorter survey with uncertain psychometric properties might still be preferred to a longer survey that half of your volunteers refuse to complete.

Research is often a series of difficult compromises between theoretical ideals and practical realities. Shortening your scale is limitation that you do need to acknowledge, but having a limitation like this should not disqualify you from publishing your results. If we only published research with no limitations, the research journals would be a lot thinner.

The one thing that you emphasize is that the decision to shorten to three items rather than five items was made PRIOR TO THE COLLECTION OF ANY DATA. You can prove this, if the peer-reviewer doesn't trust you, by offering to share the protocol that you send to and that was approved by the IRB. It's still an arbitrary choice, but it is not one that would be biased by the data analysis. It's not like you ran ten different analyses and then chose the one with the smallest p-value.

I hope this helps. Good luck with getting your paper published.

Thanks a lot for your kind answers.

I have used three items of the five-item scale. A reviewer raises a question about this procedure. How should I respond to this reviewer?

The lower the scale e.g. 3-point scale will provide the following disadvantages that the reviewer(s) might challenge the author:

For more details you might want to refer to this article "The Measurement Imperative" by Wittink & Bayer (2003).

If your survey questionnaire is adopted / adapted from other source(s) perhaps you can refer to the original questionnaire on how many item scale being used & is there any literature reviewed to justify the use of 3-point scale? 

In order to address the reduction of time length for the survey, perhaps you can use computer online survey (which is fast & easier to administer) or other scaled survey questionnaire.  E.g. if you are Likert scale questionnaire - perhaps you can convert the Likert scale questionnaire (e.g. 1=Effective, 2=Some How Effective, 3=Moderately Effective, 4=Neutral, 5=Moderately Not Effective, 6=Some How Not Effective, 7=Not Effective) to Semantic Differential scale using Bipolar Adjectives (e.g. Effective1, 2, 3, 4, 5, 6, 7 Not Effective) so that respondents can fill up faster.  Wishing you all the best.

Dear Youjae

I have read all the answers and realized that there are two different opinions. As few people considering that you have made the scale 3-point by reducing the 5-point scale where as few including me consider that you are talking about number of questions/statements in your scale that you have reduced to get better response.

Please clarify which one is correct perspective.

I am talking about the number of items or questions.  I am not referring to the 3-point scale rathert than 5-point.  Sorry for causing misunderstanding.

Hi Yi,

Based on your clarification, my earlier answer was not addressing your question since you were reducing your question items from 5 to 3 questions for a construct / variable.  Agreed with some scholars' comments that reducing your questionnaire items can affect your reliability or / and validity of your questionnaire (in which you need to re-test to confirm) especially you were adapting / adopting from other sources & those questions are formative questions (i.e. each question item playing a contributing role to the effect of the construct).  Another alternative is to re-develop your own questionnaire with 3 items if you really understand how the construct is being operationlized.  Wishing you all the best.