I am using group interviewing technique for a research study. I conducted a pilot study on few respondents. Can I use the results of the pilot study also to write my findings and conclusion? So if I interviewed 10 respondents in the pilot study and 30 later, can I say that my sample size is 40? Thanks.
Hi Ruchi
Just to add :
Leon, Davis ,Kraemwr (2011) mentioned that the very purpose of conducting a pilot study (PS) is to examine the feasibility of an approach that is intended to be used in a larger scale study.PS can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention. But among problems of PS are:
1.Completing a pilot study (PS) successfully is not a guarantee of the success of the full-scale survey.
2.PS findings may offer some indications of the likely size of the response rate in the main survey, they cannot guarantee this because they do not have a statistical foundation and are always based on small numbers.
3. Contamination may arise in the following ways: (i)where data from the pilot study are included in the main results; (ii) where pilot participants are included in the main study, but new data are collected from these people.
Social scientists engaged in quantitative research are likely to argue that: "an essential feature of a PS is that the data are not used to test a hypothesis or included with data from the actual study when the results are reported" (Peat et al. 2002: 57). The obvious concern is that if there were problems with the research tool and modifications had to be made in the light of the findings from the pilot study, data could be flawed. But if an established and validated tool is being used and the pilot study is determining other methodological aspects such as recruitment rates, it could be argued that such data may be of value. So when to decide whether to include pilot study participants or site(s) in the main study?
Respondents who have already been exposed to an intervention may respond differently from those who have not previously experienced it. It may be positive, when participants may become more adept at using a new tool/ procedure. It may also be negative with participants showing a decline in following a protocol because it is no longer novel. Indeed both changes in behaviour have long been recognised and a 'run in' period, where an intervention is introduced prior to a study, is often used for these reasons:
(1)The concern about including participants from the pilot study in the main study arises because only those involved in the pilot, and not the whole group, will have had the experience. In some cases, it is simply not possible to exclude these pilot-study participants because to do so would result in too small a sample in the main study.
(2)When samples are clusters, for example schools, prisons / hospitals. In such cases one can conduct a sensitivity analysis (or sub-group analysis) to assess to what extent the process of piloting influences the size of the intervention effect.
Contamination is less of a concern in qualitative research, where researchers often use some or all of their pilot data as part of the main study.
Qualitative data collection and analysis is often progressive, in that a second or subsequent interview in a series should be 'better' than the previous one as the interviewer may have gained insights from previous interviews to improve interview schedules and specific questions.
Some argued that in qualitative approaches separate pilot studies are not necessary (e.g. Holloway 1997: 121). For example, a qualitative interviewer conducting 15 focus group interviews will listen to the recordings or read through the transcripts of the first three or four in order to improve the questions, the way of introducing the issues into the group interview or even to add new topics. Thus, although there is no specific pilot study, analysis of the earlier focus groups may help improve the later ones.
Frankland and Bloor (1999: 154) argue that piloting provides the qualitative researcher with a "clear definition of the focus of the study" which in turn helps the researcher to concentrate data collection on a narrow spectrum of projected analytical topics. Piloting of qualitative approaches can also be carried out if "the researcher lacks confidence or is a novice, particularly when using the interview technique" (Holloway 1997: 121).
Another problem may arise where a PS requires a significant investment of resources, making it difficult for the study team to call a halt to the research after an unsuccessful PS.
Researchers might be tempted to make considerable changes in the main study, rather than deciding that the proposed study is not possible with the available resources, time, population, etc. It follows that papers reporting methodological issues, such as those identified during the pilot phase of a study, will also be less attractive to publishers. In the past, editors have recognised the dangers of publication bias with respect to clinical trials and have offered 'an amnesty for unpublished trials' in an attempt to overcome these problems (Smith & Roberts, 1997). However, it is equally important to ensure that lessons learned with respect to the research method are shared. Thus, when pilot studies are mentioned in more detail in academic papers and reports, researchers regularly comment that they "had learned from the pilot study" and made the necessary changes, without offering the reader details about what exactly was learnt.
Some of these processes and outcomes from both successful and failed pilot studies might be very useful to others embarking on projects using similar methods and instruments. This is particularly important because pilot studies can be "time-consuming, frustrating, and fraught with unanticipated problems, but it is better to deal with them before investing a great deal of time, money, and effort in the full study" (Mason and Zuercher, 1995).It has also been argued that the current research climate demands accountability from researchers, which means that there is a need to ensure the best possible use of research results (Crosswaite and Curtice 1994). Therefore, investigators should be encouraged to report their pilot studies, and in particular to report in more detail the actual improvements made to the study design and the research process(Leon AC, Davis LL, Kraemer 2011).
References
Crosswaite, C. and Curtice, L. (1994), Disseminating research results-the challenge of bridging the gap between health research and health action. Health Promotion International 9: 289.
Frankland, J. and Bloor, M. (1999), Some issues arising in the systematic analysis of focus group material, In: Barbour, R. and Kitzinger, J. (eds) Developing Focus Group Research: Politics, Theory & Practice, London: Sage
Holloway, I. (1997). Basic Concepts for Qualitative Research, Oxford: Blackwell Science.
Leon AC, Davis LL, Kraemer HC. J Psychiatr Res. 2011 May; 45(5):626-9. doi: 10.1016/j.jpsychires. Epub 2010 Oct 28.The role and interpretation of pilot studies in clinical research.
Mason, D.J. and Zuercher, S.L. (1995). Pilot studies in clinical nursing research. Journal of the New York State Nursing Association 26: 11.
Peat, J., Mellis, C., Williams, K. and Xuan W. (2002), Health Science Research: A Handbook of Quantitative Methods, London: Sage.
Smith, R. and Roberts, I. (1997) An amnesty for unpublished trials British Medical Journal 315: 622
In business research, it is often recommended NOT to include pilot study in the main study samples. Some health science researchers also hold this view (ref: Peat, J., Mellis, C., & Williams, K. (Eds.). (2002). Health science research: a handbook of quantitative methods. Sage).
However, there are other researchers in health science that allow (and recommend) to include pilot studies with the main study upon the condition that same methodology has been followed, in order to strengthen the findings of main study. (ref: Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., … Goldsmith, C. H. (2010). A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology, 10(1), 1–10. http://doi.org/10.1186/1471-2288-10-1.).
So researchers need to make a judgement based on the concerns as addressed in those two articles. They primarily talked about the possibility of intervention and contamination of data if pilot studies are included. However, if pilot study methods and sample characteristics are very close, the authors recommended to include this to the main study. Hope it helps.
Can the sample used for pilot testing be included in the final research sample?
Generally is not recommended. Reason being 1 of the objectives of a pilot study is to test a researcher's instrument be it interview / observation etc. (qualitative research) or survey questionnaire / experiment etc. (quantitative research) - so that after the pilot study, the instrument can be improved before being used in the final data collection. If there is a change / improvement of the instrument, quality of data collected during pilot study and actual study is different - combining both the data can impact the outcome of the data analysis. Another reason from quantitative research perspective is that - convenient sampling normally is used during pilot study whereas random sampling typically is used during the final data collection (for parametric statistical testing). Hence, combining both the data collected can also impact the result of the data analysis.
However, some researchers find that after the pilot study & if there is no change / improvement in the instrument whereby the same instrument is used in the final data collection, then there is no harm combining both the pilot data and final data together for data analysis.
I will agree with my fellow researchers that it is generally not recommended.
When you say that you used "group interviews," does this mean that you are doing qualitative research using focus groups? If so, the answer would be different than if you are doing survey research.
In particular, qualitative researchers typically do include their early (pilot) interviews, as long as they are reasonably comparable to the questions that you asked in later interviews.
David, you are right about it. In qualitative design, inclusion of pilot study results is inevitable. Thanks for highlighting this perspective.
Hi Ruchi
Just to add :
Leon, Davis ,Kraemwr (2011) mentioned that the very purpose of conducting a pilot study (PS) is to examine the feasibility of an approach that is intended to be used in a larger scale study.PS can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention. But among problems of PS are:
1.Completing a pilot study (PS) successfully is not a guarantee of the success of the full-scale survey.
2.PS findings may offer some indications of the likely size of the response rate in the main survey, they cannot guarantee this because they do not have a statistical foundation and are always based on small numbers.
3. Contamination may arise in the following ways: (i)where data from the pilot study are included in the main results; (ii) where pilot participants are included in the main study, but new data are collected from these people.
Social scientists engaged in quantitative research are likely to argue that: "an essential feature of a PS is that the data are not used to test a hypothesis or included with data from the actual study when the results are reported" (Peat et al. 2002: 57). The obvious concern is that if there were problems with the research tool and modifications had to be made in the light of the findings from the pilot study, data could be flawed. But if an established and validated tool is being used and the pilot study is determining other methodological aspects such as recruitment rates, it could be argued that such data may be of value. So when to decide whether to include pilot study participants or site(s) in the main study?
Respondents who have already been exposed to an intervention may respond differently from those who have not previously experienced it. It may be positive, when participants may become more adept at using a new tool/ procedure. It may also be negative with participants showing a decline in following a protocol because it is no longer novel. Indeed both changes in behaviour have long been recognised and a 'run in' period, where an intervention is introduced prior to a study, is often used for these reasons:
(1)The concern about including participants from the pilot study in the main study arises because only those involved in the pilot, and not the whole group, will have had the experience. In some cases, it is simply not possible to exclude these pilot-study participants because to do so would result in too small a sample in the main study.
(2)When samples are clusters, for example schools, prisons / hospitals. In such cases one can conduct a sensitivity analysis (or sub-group analysis) to assess to what extent the process of piloting influences the size of the intervention effect.
Contamination is less of a concern in qualitative research, where researchers often use some or all of their pilot data as part of the main study.
Qualitative data collection and analysis is often progressive, in that a second or subsequent interview in a series should be 'better' than the previous one as the interviewer may have gained insights from previous interviews to improve interview schedules and specific questions.
Some argued that in qualitative approaches separate pilot studies are not necessary (e.g. Holloway 1997: 121). For example, a qualitative interviewer conducting 15 focus group interviews will listen to the recordings or read through the transcripts of the first three or four in order to improve the questions, the way of introducing the issues into the group interview or even to add new topics. Thus, although there is no specific pilot study, analysis of the earlier focus groups may help improve the later ones.
Frankland and Bloor (1999: 154) argue that piloting provides the qualitative researcher with a "clear definition of the focus of the study" which in turn helps the researcher to concentrate data collection on a narrow spectrum of projected analytical topics. Piloting of qualitative approaches can also be carried out if "the researcher lacks confidence or is a novice, particularly when using the interview technique" (Holloway 1997: 121).
Another problem may arise where a PS requires a significant investment of resources, making it difficult for the study team to call a halt to the research after an unsuccessful PS.
Researchers might be tempted to make considerable changes in the main study, rather than deciding that the proposed study is not possible with the available resources, time, population, etc. It follows that papers reporting methodological issues, such as those identified during the pilot phase of a study, will also be less attractive to publishers. In the past, editors have recognised the dangers of publication bias with respect to clinical trials and have offered 'an amnesty for unpublished trials' in an attempt to overcome these problems (Smith & Roberts, 1997). However, it is equally important to ensure that lessons learned with respect to the research method are shared. Thus, when pilot studies are mentioned in more detail in academic papers and reports, researchers regularly comment that they "had learned from the pilot study" and made the necessary changes, without offering the reader details about what exactly was learnt.
Some of these processes and outcomes from both successful and failed pilot studies might be very useful to others embarking on projects using similar methods and instruments. This is particularly important because pilot studies can be "time-consuming, frustrating, and fraught with unanticipated problems, but it is better to deal with them before investing a great deal of time, money, and effort in the full study" (Mason and Zuercher, 1995).It has also been argued that the current research climate demands accountability from researchers, which means that there is a need to ensure the best possible use of research results (Crosswaite and Curtice 1994). Therefore, investigators should be encouraged to report their pilot studies, and in particular to report in more detail the actual improvements made to the study design and the research process(Leon AC, Davis LL, Kraemer 2011).
References
Crosswaite, C. and Curtice, L. (1994), Disseminating research results-the challenge of bridging the gap between health research and health action. Health Promotion International 9: 289.
Frankland, J. and Bloor, M. (1999), Some issues arising in the systematic analysis of focus group material, In: Barbour, R. and Kitzinger, J. (eds) Developing Focus Group Research: Politics, Theory & Practice, London: Sage
Holloway, I. (1997). Basic Concepts for Qualitative Research, Oxford: Blackwell Science.
Leon AC, Davis LL, Kraemer HC. J Psychiatr Res. 2011 May; 45(5):626-9. doi: 10.1016/j.jpsychires. Epub 2010 Oct 28.The role and interpretation of pilot studies in clinical research.
Mason, D.J. and Zuercher, S.L. (1995). Pilot studies in clinical nursing research. Journal of the New York State Nursing Association 26: 11.
Peat, J., Mellis, C., Williams, K. and Xuan W. (2002), Health Science Research: A Handbook of Quantitative Methods, London: Sage.
Smith, R. and Roberts, I. (1997) An amnesty for unpublished trials British Medical Journal 315: 622
Thank you, Noor Saadah Zainal Abidin. Sincerely appreciate your nuanced response.
- Thank you for creating this debate; some very useful readings and suggestions. The discussion helped me to justify my reason for including the data from the pilot study. THANK YOU
Appreciate your feedback Eva and hope you benefit from the explanation.
Good luck in your study.
Very interesting answers. I had used the pilot sample in my final research sample count largely because there was no change in the scale / instrument of research and secondly, the sample of the pilot was random sampling and exact representative of the population considered for the final study. If the scaling in the instrument would have changed in the course of the pilot study, I would not have considered them in the final sample.
I've got the same question bothering my mind for weeks. Thanks for all the nice comments, here.
I have just been wondering the same thing as Ruchi, and I am not sure the answers above help me (sorry!). Ruchi wanted to know if people she had piloted her interviews in, could be included in final analysis (if I have read that correctly). I am also wondering whether people I interview to test my interview questions on could be then included in analysis (if the data they give is of a good standard/there are no real changes in interview guide). However, the answers above related to pilot studies and piloting of surveys/questionnaires which are very different..(unless I am reading things wrong).
Hi, I have included my pilot interviews as they were fitting nicely (same theoretical framework, same 'type' of participants etc. However, I have added a page long justification.
one of my students asked me the same question.
I referred him to this page and to the references mentioned in the answers.
thank you
No you can not include because of the intricacies involved in the sample size. we can not generalize the results with pilot-testing results. This does not guarantee reliable outcome to meet the objectives. However, in some of the medical research/health science the pilot testing samples was included for final research. However, logistic regression analyses carried out in the most of the studies. Research literature is available on web resources
Thank you for the discussion. It has been an eye opener. Actually, I was looking for an answer as I was wondering why the student I am supervising has included the pilot results of the qualitative section in the findings of the project. The target group and pilot have similar characteristic.
It is not the norm, when done need to justify, as well as be sure that the data had not got flawed and contaminated.
Thanks for all the answers and clarification, surely it is going to help me in my research.
This is very useful and helpful information for me. I once read that a time lapse allows pilot samples to be retested for actual study. Can anyone recommend further reading on this? Thank you so much
Mikuska, E. (2017). The importance of piloting or pre-testing semi-structured interviews and narratives. SAGE Research Methods Cases. http://dx.doi.org/10.4135/9781473977754
Pilot testing is done away from your study site so that it will not influence your data collection.Therefore by theory pilot tested sample cannot be from the sample you wanted to do the research.
Hi guys, I am want to add the pilot respondents to the final survey, please I need justification by giving examples of other studies (including reference) that add the respondents from pilot to the actual survey.
thank you
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size