I am putting together a questionnaire looking at leadership style and a handful of organizational variables. I have explained the development of the individual scales, which are established survey instruments (discussing alpha reliability, use in previous research, and justification for going with a short-form scale).

However, one of my committee members has raised the question of whether my *overall* questionnaire will have good psychometric properties, since I am “creating my own instrument by combining multiple measures.” My initial response was that I am not really creating my own survey instrument — I am merely administering multiple established instruments in a single survey. Her response was the same — that by using multiple scales, I am essentially creating my own new questionnaire.

One thing I gather is that I need to justify the ordering of measures within my overall questionnaire (e.g., not priming participants by posing demographic questions first). However, I am stumped by her suggestion that I need to justify the overall validity of the combined questionnaire...because I am measuring different constructs (and it seems that almost all social scientific survey research does this). Perhaps I am missing something, and I am not having much luck with getting direct clarification from my committee member.

Does anybody have any insights on what “validating” a questionnaire with multiple instruments entails? Or any recommended resources on the topic?