Dear Mohammed,

I am not sure if I get the problem correctly. I think we need some more information to answer in a helpful way. What kind of study did you conduct? Does the problem refer to measurement models that need to be confirmed? Are the psychometric properties of the data bad? etc...

e.g. If you used a well established measurement tool, a pilot study is not obligatory (in my experience).

Best regards, Nicole

Thank you Nicole for your quick answer,

the thing is i did quantitative research in my Phd and i have used questionnaires to measure my study variables, i have missed doing the pilot study which is normally should happen before doing data analysis in order to do reliability and validity for the study measurement before data collection. the problem now happen during my viva presentation where the examiner asked me where is the pilot study ?? why you didnt make ?? and my answer was coz i have used previous study measurement which was already valid and reliable so why i need to do again ? bu the examiner said since you conduct the research on other environment or country so you have to do. so now anyway can give justify or to do something to recover the problem.

Thank you Much

Dear Mohammed,

thank you for the clarification. If the research context or field of application is highly different, I understand the argumentation of your examiner. But still, from my experience, a pilot study is not always necessary. If you can argue that the scales you used are well established in many research contexts over the years, and you can show for your data that the scales are highly reliable and valid (by using exploratory and comfirmatory factor analysis, including all the established psychometric criteria) everything is fine for your main analyses. In the end, ensuring the quality of the data is the reason for doing pretests; otherwise this would be a formal necessity of the examiner only.

Nonetheless, thinking about a possible solution.... Doing a pilot study post hoc, clearly, is not an option. It may be a solution to split your sample and to use a little part (e.g. n = 30) to conduct pretests using exploratory factor analysis, analysis of internal consistency, etc.. This could work if dimensionality, reliability and validity can be confirmed in the small sample; and even if you have to exclude one or two items. With the main part of your data you could go further and conduct also confirmatory analyses. If there are more problems with the variables, of course, this approach is not helpful as you cannot modify the variables used in the data collection. Of course, this is not an exemplary approach (e.g. as the point of time in pretest and main study would be the same) but it may be better than nothing. I guess it is not possible to use the data you have collected until now as a pretest and do another main survey due to time and cost effort, is it?

All the best, Nicole

Dear Nicole,

Thank YOU very much, your answer really helpful, appreciate you.

warm regards

Dear Mohammed hello. In my opinion, you would rather do a pretest since the questionnaires are administered in a sample of another country from the country they were originally used. So I agree with the examiner. It is true that their validity and reliability should be tested in a sub sample before the main survey takes place. However, as Nicole mentioned before if the scales have adequate psychometric properties in a great deal of surveys (especially recent ones), are used in many countries and different cultures and current analyses reveal they have high internal consistency and construct validity (supported by factor analysis showing that the initial factors are the same with the ones extracted in your survey) you can consider them as valid and reliable enough to use in your main survey without being cautious about the generalizability of research results. If the sample in the survey you have already conducted is relatively high better not use it as a pilot survey due to cost, time and further required effort as well, although this depends much on the availability of the sample and the convenience it could be recollected if necessary. If I were you I would not try to fix the problem by conducting another one survey.

Good luck with your research.

Kind regards,

Andronikos

Thank you MUCH Andronikos Kaliris

Since the step has already been overtaken by events, the only solution is to go ahead and analyze the data (assuming that all the questions were answered by the respondents). This will only work if the normality test on data is passed. It is however important to do factor analysis and extract those that do not meet the threshold but you need to clearly explain any action that you take on the data analysis.

Mohammed, in other words an exploratory factor analysis could offer you some important preliminary data for the factor structure of the questionnaires used in the research. In order to do confirmatory factor analysis your sample should be quite large, be cautious with this analysis. If questionnaires structure is similar to the structure of the ones used in the original survey or other surveys which include those instruments you can go ahead analyzing data. Also, I agree with Mary Ragui that you have to make sure that the distribution is normal (Kaiser Smirnoff tests could be used to support the hypothesis). On the other hand, you do not have to extract factors which do not meet the threshold. Perhaps, a differentiated factor structure could be extracted due to cultural, age or educational level differences between your population and other ones studied elsewhere. Words, phrases and habits or attitudes do not have the same meaning both for people in China and other people in US, Iran, Turkey, Italy or Greece. This is a possible and quite convincing reason which could be easily stated in your dissertation.

thank you guys, i need your opinion also in one solution has been suggested from one of my friends, he said that you can also cover the problem by get evaluation about the items of my questionnaire from expert people ( Phd Holders and Prof), by making scale about my items to be evaluated from them whether these items matching with the study objectives or not and make 5- likret scale starting from ( strongly unsatisfied about the item to strongly satisfied ) then see the percentage of the respondents about the items and give justification based on their answer whether the items are reliable and valid. because as you know guys in Phd level for solving any problem it is very important to give satisfactory justification or otherwise your work will be rejected. how do you think guys . and thank you all

Perhaps you are referring to face validity. If most Professors or PhD holders (experts) agree with the relevance of the items used with the study you can claim your scales are valid but are they reliable? Pilot testing is mainly used to check reliability coefficients (Cronbach's α, inter-item correlations, test-retest reliability etc). Yes, judges' agreement on items could be an option but is the less powerful form of validity and by no means adequate. Normality tests, Factor analysis and reliabilities should be your main tools at the moment. In my view, face validity and an extensive psychometric control could complement each other in order to overweigh the disadvantage of not doing pretest. Additionally, you have to justify the selection of the scales based on theory and relative bibliography revealing a trend to fit many cultures and populations with similar results. The above are enough..Another investigator in the future could test anew the scales to prove their validity, reliability and coherence.

There is nothing more you can do..that's science..Always something is missing from the puzzle.

Best Regards,

Andronikos

Thank you and really appreciate

The best way to compensate for the error, is taking other developed pilot research study. That way, your project will build on these previous studies and present new approaches and research to achieve new knowledge.

Best of luck!

Dear Federico Del Giorgio Solfa, thank you for your answer i think it is really powerfull helpful answer.

what i have understood from your point that i have to find previous authors who have test pilot study using the same scale in different countries or environment and build justification based on their pilot test . did u mean that??

Thank you again

I partially agree with Federico. Mohammed it holds true that taking other surveys' results, analyzing and interpreting them could justify the the use of the instruments due to adequate reliability and validity coefficients shown in these studies. However, I insist you have to be based on completed studies, and not just pilot tests, with quite big and representative samples. Those studies should have been undertaken in other countries all around the world or only in your country. At least some of the instruments you use might have been used in a related research field. The more studies you are based on the better the justification of your results. Focus on the psychometric properties. This process could justify the fact you did not do pilot testing but running one new study to test the instruments at the moment is a solution I would definitely not recommend. Well grounded theory and high quality studies in your field should guide your work from now on.

Good luck!

Best regards

Andronikos

thank you Andronikos Kaliris for your justification and good answer, i will do what all of you suggested to me. i appreciated too much for all of you

thank you

warm regards

thank you very much Regina Gyampoh-Vidogah

Exactly Mohammed, I think you can use some parameters and tested structures and from them, you can evaluate your research object, which together with other findings, you can get new insights and knowledge.

No harm doing more experiments (it just adds to the rigorosity) of your research. Or another option is to add a chapter on sensitivity analysis. What if I did this? or that?

Pilot study is undertaken to find out the validity of the questions in the questionnaire so that the questions can be added/ deleted or modified as the case may be. Since you have already done the full analysis. Still you can use the first 30 or 40 questionnaires collected as pilot study and analyze them separately and check it with the full analysis. In all probability there may not be much difference.

Also see whether your findings are in tune with earlier research.

If there is some difference you can go for further experiments on those portions to make the study more rigorous

Wish you all the best

What if it involves purposive sampling of 30-40 respondents and these are the same respondents being studied upon and who will respond to the crafted questionnsire? Can pilot study be exempted?

A pilot is intended to test relaibility and validity of your instrument before its used in the main study. However, if the item were adopted with Cronbach's Alpha above .70 and the context is not so different, then you can do away with a pilot. I guess your finding would not significantly affected.

Pilot study may not be necessary if the validated tool used in the study

has good internal consistency i.e Cronbach alpha.