Dear Diptiman,

My advice is to perform an exploratory actor analysis, doing the same rotation method as the original authors. Try to find the same dimensions. If it is not possible, please suggest other factors, considering loading higher than .40 (minimum) and eigenvalues over 1 8mininum) - see also the scree plto. Good luck with your research!

Dear Diptiman,

If you only need to validate a single scale I would not advise a full exploratory factor analysis. Instead, you can run a Principal Component Analysis, check the size of the first eigenvalue and remove any items which have a weighting of < 0.4 on the first component - see my guide:

https://www.researchgate.net/publication/280936182_Advice_on_Reliability_Analysis_with_Small_Samples

Technical Report Advice on Reliability Analysis with Small Samples

Thank you, Lisete and Peter! Your help is sincerely appreciated! 

Lisete: Your solution is appealing for its simplicity right away! And thanks for your wishes.

Pater: Your technical paper is very helpful! Thank you so much for sharing this! You are right in your understanding that I need to validate a single scale (that has 11 items), with a sample size of just over 100. That is something that you have covered in your note.

I am now wondering what are the relative pros and cons of each approach, that is, exploratory factor analysis vs. principal component analysis, for scale validation? Will sincerely appreciate your wisdom on this! 

Dear Mr Diptiman,

In true sense, only performing EFA doesn't suffice scale validation since EFA only provides reliability measures. However, as I understand, you have only one set of sample which perhaps deters you from any further validity test. If your reviewers are "convinced" with only EFA, you may need to justify the cut off value of factor loading which should ideally be 0.5 or higher (Hair, Black, Babin, & Anderson, 2009). 

Thanks, Taufique, for your answer! That helps!

Dear Diptiman,

Satisfactory validation of instruments can involve complex and protracted processes, and among these you might not have the opportunity to engage in (m)any that are substantial beyond factor analysis.  Actually, factor analysis can be used to investigate BOTH reliability (i.e., internal consistency of an instrument) as well as factorial validity (i.e., the "theoretical" structure of the construct that you are measuring). This can seem confusing, but in reality it's not.  It depends which "hat" you are wearing (reliability or validity) at a particular time in your analyses.

The following article that I wrote about instrument development might be of help: 

Trevethan, R. (2009).  Self-assessment of foot health:  Requirements, issues, practicalities, and challenges.  Journal of the American Podiatric Medical Association, 99, 460-471. 

That article focuses on foot health, but has much wider application.  You'll find there that there are 13 kinds of validity. Under the circumstances, you might be able to use/claim only face validity, content validity, and factorial validity.

It might be necessary to acknowledge in the discussion section of your research that full validation of the instrument was not possible - and that might be partly why you didn't obtain the results you expected OR that you obtained the expected results despite not having been able to fully validate the scale but that it would appear that the scale had remained valid.

Every good wish with your research.

Dear Dr. Trevethan,

Thank you so much for your detailed response! Your guidance, including what you have suggested to include in the limitations section is sincerely appreciated!

I downloaded your article. At first glance, it appears exhaustive, and will surely be of help to me.

Thanks again!

Dear Diptiman,

Feel free to call me Robert - and thanks for your response.

You might find my article EXHAUSTING!  In it I tried to provide comprehensive information because I had found many problems and misunderstandings related to instrument development not only in podiatry but in a number of other spheres.  One person got back to me and said he'd read the article "millions of times" because it was so dense but useful.

In your research you might not need to say a lot about validity because you probably have limited prospects for dealing with it, but the reviewer is justified in requiring you to at least address the issue of validity in some way, and you can do it only as best you can.  It's important not to ignore it even if you can't do a lot about it.

Robt.

Dear Robert,

Thanks so much for taking time from your busy schedule to help!

You've added a lot of knowledge here at a conceptual level. Further, what particular method would you recommend - factor analysis or CPA? I've seen authors using both methods in different ways, and I'm a bit confused here. I'm wondering which method will be more useful for understanding reliability and validity of the scale adaptation used.! 

Thank you for your kind consideration!

Warm regards.

Dear Diptiman,

Ha! You've now hit me at a weakness.  I've used factor analysis quite a lot, but I have to confess (and in public - how embarrassing!)  that I really don't know enough about the differences between PCA, exploratory FA, and confirmatory FA to give you good advice. Maybe a statistician could help - or one of the people who made earlier posts up above (who I welcome to disagree with me if they think my advice is awry; we can all progress together in this).  Or maybe you can get good advice from trusted sites on the Internet. 

At the risk of being wrong, but to avoid you perhaps feeling abandoned, I'd be inclined to do a confirmatory factor analysis (using, if they tell you, the same kind of setup as the original test developers in terms of such things as the nature of the rotation they used and how many factors they decided to extract) and look at what your data produce. I gather that the test you used has only one dimension, so it might be best to ask the analysis to be based on only one factor. However, as Lisete recommended above, do look at the scree plot in case there are two or more factors in your (small number of) items.

Some of what you do could depend on the sophistication of the original scale development.  If something like factor analysis wasn't used back then, there is little point in your attempting to replicate what they did in terms of FA. You can't!  Nevertheless, even if they didn't use FA, it might be good for you to do it to ensure that your instrument didn't have apples and oranges in it and that you were inappropriately adding them up.  That's part of ensuring factorial validity -as you'll see in my JAPMA article.

In case it helps, I am attaching a set of PowerPoint slides that I used when I was teaching a basic research methods subject to undergraduate students of health sciences. These slides were the backbone of the more elaborated face-to-face verbal delivery so they are somewhat sketchy, but they might be helpful for you.  In particular, do ensure that you have enough participants (I think you have, with a bit over 100 participants and only 11 items) and that your inter-item correlations, KMO index, and Bartlett's test of sphericity are satisfactory.

There is some material about FA starting on page 462 of my JAPMA article that you might find useful as a way of nudging into the background issues more comfortably if you're not very familiar with FA.  There is Add your answermore information on page 467, specifically about factorial validity.

I hope all of this is helpful rather than bamboozling..

Robt.

Have you looked at the COSMIN guidelines they are very comprehensive and give you a check list for validating your scale.  If you are using it in a different context you will need to look and reliability and validity in that context.  The checklist walks you through what needs to be done to assessment the measurement properties of the scale.

Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737-45.

Terwee CB. Consensus-based Standards for the Selection of Health Measurement Instruments checklist. 2010.

Anita

Dear Dr. Slade, Thanks so much for pointing me towards the COSMIN guidelines- I was not aware of them! Your help is sincerely appreciated!

Hello again Diptiman,

I have been doing more thinking about your original question - going right back to the start.  I wonder whether the reviewer was perhaps making an inappropriate request of you.  From what you have written, you did not adapt the instrument, but simply used the original instrument but in a different context from the one in which it was originally developed. That means that the reviewer's request for "validation of scale adaptation" is barking up the wrong tree.  Reviewers are only human, and, in my experience, they quite often get things wrong.  So I think that you should not feel obliged to address the reviewer's request if that request is awry.

The issue is more whether the original scale has validity for the context in which you were using it. An instrument's validity is undoubtedly important, and other posts in response to your question should be helpful regarding that. Perhaps the reviewer really was simply requesting that you provide evidence [as much as possible] that your instrument was valid.

Picking up the pieces from some of the above posts, if the original instrument developers conducted factor analysis (they might not have), it would be good for you to try replicating that.  If they didn't, it might still be good for you to run PCA or factor analysis to ensure that you were/are scoring the instrument appropriately (e.g., not adding up apples and oranges when they should have been kept separate, or adding up the apples and oranges in their most appropriate groups).

One point that you might need (or be hopefully able) to defend is that the instrument, although being used in a different context, retained its original face and content validity.

Apart from that, it might be good to look at things like Cronbach's alpha that give you a sense of an instrument's reliability / internal consistency.  After all, an instrument can't be valid unless it is reliable.

Robt.

Thank you so much, Robert, for your detailed opinion, and for explaining some points in candid detail! In fact, I got pointers "validating" some doubts of my own! Sincerely appreciate your kind consideration!

Thanks Robert for your inputs. I too benefitted.

Regards,

Vikas.

My pleasure, Vikas, and thank you for the feedback.  Doing research isn't always easy, and the decisions that need to be  made are not always clear cut.  I'm glad that ResearchGate provides a venue for an exchange of ideas and suggestions.

All the best with you work.

Robt.

Robert Trevethan and Anita L Slade . Thank you for the detailed responses. I wanted to understand would I have to follow the same complex procedure, if I wanted to curtail/ adapt items , but only for a part of the pre-test of running a bigger experiment.

Do I neled to use reliable/ valis scales for pre-tests too say to do a manipulation chk / identify the right stimuli OR can I just frame my own questions , based upon literature ?

Surabhi Jain. Hello Surabhi: I'm sorry, but I don't understand your questions sufficiently to provide responses that might be appropriate. (Don't worry about the typos; I can overcome them.) Do you want to try to frame your concerns differently? If so, maybe I could help.

Thank you for your response Robert. I am in the midst of designing a pre test for an experiment. I understand that for measuring DVs for the experiment, we would require a reliable valid scale. However , would we need a reliable / valid scale for measuring a varaible , even for the pre test of an experiment? For instance if I want to measure the believability of a given situation, can I just not ask a simple qn and leave it at that or will I have to look up a scale to measure it. I really hope I am clearer now.

Surabhi Jain. Thanks for the extra information. At times a single question can be very useful, but often this type of question might not yield enough information for you. If you are going to measure something "properly", yes, it's best to have a scale that is reliable and valid.

But I still don't understand your situation. Are you going to conduct a before / after type of experiment in which scores on a scale might change? If so, I think that use of the same (good) scale on both occasions of measurement (before and after the intervention) would be best.

I might not be understanding your situation, however. Feel free to get back with more information in case I can be of more help.