I am designing a research questionnaire on the experiences of adults learning numeracy. To assess the reliability and validity of my questionnaire I woul like to see a copy of a similar instrument that has been used successfully in a similar inquiry. Or a relevant source or literature.
Hi Norman,
In qualitative research, where the questionnaires are either semi-structured or unstructured, we don't usually test tools for reliability and validity - as we do for structured quantitative questionnaires. Qualitative questionnaires are often 'unique' and not designed to be generalisable to other studies. Therefore - it is unlikely that you would want to directly 'copy' another qualitative questionnaire. Qualitative researchers may certainly employ pilot or member-checking procedures to seek objective opinion as to how questions could be made easier to understand, avoid bias or leading questions and/or avoid any potential ambiguity. This is more related to the study's overall 'trustworthiness' though - and not its validity of reliability.
Qualitative 'trustworthiness', as would your eventual questionnaire, can take on many different positions - such as the following - taken from a qualitative research chapter of mine. Perhaps the first position is most in line with your current thinking:
Position 1 — using the criteria of quantitative research - This position argues that the process of quantitative reliability and validity remain appropriate concepts for ensuring rigour in qualitative research. It emphasises procedural precision and the use of ‘verification strategies integral and self-correcting during the conduct of inquiry itself’ (Morse et al. 2002 p 1). To a certain extent, meta-synthesis suits this position.
Position 2 — parallel methodological criteria - This position argues that qualitative research requires a different set of criteria for evaluating trustworthiness. Commonly cited are those of Guba and Lincoln (1989) who developed criteria which have a parallel relationship to those used in quantitative research. These are: ‘credibility with internal validity’, ‘auditability (dependability) with reliability’, ‘fittingness (transferability) with external validity’ and ‘confirmability with objectivity’.
Position 3 — multiple criteria - This position argues one list of criteria per qualitative research approach. For example, it has been proposed that the trustworthiness of an ethnographic report can be evaluated by the application of three criteria: veracity, objectivity and perspicacity (Stewart 1998). In grounded theory various forms have been suggested. McCann and Clark (2003) highlight the differences between the criteria proposed by Glaser of ‘fit’, ‘work’, ‘relevance’ and ‘modifiability’ with other grounded theorists e.g. Strauss and Corbin (1998).
Position 4 — fresh and universal criteria - There are unique ‘general’ criteria for evaluating the rigour of qualitative research (all approaches). These criteria are yet to be identified and agreed on, although there have been numerous suggestions as to what the criteria should be. For instance, Morse and Richards (2002) suggest generic criteria grouped under the headings of: ‘asking the right question’, ‘ensuring an appropriate design’, ‘making trustworthy data’, ‘verification or completion’ and ‘solid theory- building’, if that is an aim.
Position 5 — each study develops suitable, justifiable criteria - Growing in popularity is the flexibility for researchers to develop their own list of criteria for trustworthiness of a study. The choice is justified because of the qualitative research approach used and also on other grounds, such as philosophical, ethical and political reasons. 
Position 6 — no criteria is necessary There is also a post-modern position that rejects the need for criteria to be selected or stated by a researcher in regard to trustworthiness of a qualitative research study or its product. Part of the rationale for this position is that the findings in qualitative research are ‘both a process and a product in which the researcher is deeply and unavoidably implicated’ (Sandelowski & Barroso 2002). The findings are therefore a subjective construction in which the knowledge, beliefs and activities of the researcher’s play a significant role. The findings are ‘unique social interactions’ and, for this reason, qualitative research can never be truly ‘generalisable’. The research and its reported product are accepted or rejected by the reader of the report or user of the product according to their own subjective criteria.
Hi Norman,
In qualitative research, where the questionnaires are either semi-structured or unstructured, we don't usually test tools for reliability and validity - as we do for structured quantitative questionnaires. Qualitative questionnaires are often 'unique' and not designed to be generalisable to other studies. Therefore - it is unlikely that you would want to directly 'copy' another qualitative questionnaire. Qualitative researchers may certainly employ pilot or member-checking procedures to seek objective opinion as to how questions could be made easier to understand, avoid bias or leading questions and/or avoid any potential ambiguity. This is more related to the study's overall 'trustworthiness' though - and not its validity of reliability.
Qualitative 'trustworthiness', as would your eventual questionnaire, can take on many different positions - such as the following - taken from a qualitative research chapter of mine. Perhaps the first position is most in line with your current thinking:
Position 1 — using the criteria of quantitative research - This position argues that the process of quantitative reliability and validity remain appropriate concepts for ensuring rigour in qualitative research. It emphasises procedural precision and the use of ‘verification strategies integral and self-correcting during the conduct of inquiry itself’ (Morse et al. 2002 p 1). To a certain extent, meta-synthesis suits this position.
Position 2 — parallel methodological criteria - This position argues that qualitative research requires a different set of criteria for evaluating trustworthiness. Commonly cited are those of Guba and Lincoln (1989) who developed criteria which have a parallel relationship to those used in quantitative research. These are: ‘credibility with internal validity’, ‘auditability (dependability) with reliability’, ‘fittingness (transferability) with external validity’ and ‘confirmability with objectivity’.
Position 3 — multiple criteria - This position argues one list of criteria per qualitative research approach. For example, it has been proposed that the trustworthiness of an ethnographic report can be evaluated by the application of three criteria: veracity, objectivity and perspicacity (Stewart 1998). In grounded theory various forms have been suggested. McCann and Clark (2003) highlight the differences between the criteria proposed by Glaser of ‘fit’, ‘work’, ‘relevance’ and ‘modifiability’ with other grounded theorists e.g. Strauss and Corbin (1998).
Position 4 — fresh and universal criteria - There are unique ‘general’ criteria for evaluating the rigour of qualitative research (all approaches). These criteria are yet to be identified and agreed on, although there have been numerous suggestions as to what the criteria should be. For instance, Morse and Richards (2002) suggest generic criteria grouped under the headings of: ‘asking the right question’, ‘ensuring an appropriate design’, ‘making trustworthy data’, ‘verification or completion’ and ‘solid theory- building’, if that is an aim.
Position 5 — each study develops suitable, justifiable criteria - Growing in popularity is the flexibility for researchers to develop their own list of criteria for trustworthiness of a study. The choice is justified because of the qualitative research approach used and also on other grounds, such as philosophical, ethical and political reasons. 
Position 6 — no criteria is necessary There is also a post-modern position that rejects the need for criteria to be selected or stated by a researcher in regard to trustworthiness of a qualitative research study or its product. Part of the rationale for this position is that the findings in qualitative research are ‘both a process and a product in which the researcher is deeply and unavoidably implicated’ (Sandelowski & Barroso 2002). The findings are therefore a subjective construction in which the knowledge, beliefs and activities of the researcher’s play a significant role. The findings are ‘unique social interactions’ and, for this reason, qualitative research can never be truly ‘generalisable’. The research and its reported product are accepted or rejected by the reader of the report or user of the product according to their own subjective criteria.
For qualitative reseach there are some certain critera to be used to increase the rigor and trustworthiness of the research. In qualitative research, I've read my articles, some of them indicate to using same criterias used for quantitative to be used for qualitative too, but some others disagree with the idea. but eventually I personally came to the consesus that, Qualitative has alternative terms for Reliability and Validity in which they are Rigor and Trustwothiness. for further information I recommend you to refer to:
1- Guba, E. G., 1981. Criteria for Assessing the Trustworthiness of Naturalistic inquiries. Springer, 29(2), pp.75-91
2- Guba, E. G., & Lincoln, Y.S. (1994). Competing paradaigms in qualitative research. In N.K Denizin & Y.
3- S. Lincoln, Handbook of qualiative research (pp.105-117). Thousand Oaks, CA: Sage.
the criterias I used to increase my research Reliability and Validity of my MA was the following:
1- Respondent Validation or Member Checking
2- Peer viewing Debriefing,
3- Validation of the Interview
4- Thick Descriptions
5- Audit trail
If you would like to know the process of each of those above please refer to:
Bowen, G. A., 2005. Preparing a Qualitative Research-Based Dissertation: lessons learned.
Eventually, Documenting all the process is also increaseing the trustworthiness and rigour of the research.
I hope that helps.
Best Regards.
Dean does a good job of laying out the various options. I personally prefer his position #3 which argues that qualitative and quantitative research should have different criteria. In particular, the concepts of reliability and validity were developed to ensure the quality of measures in the form of variables. Because qualitative research does not "measure" things and does not use "variables," those criteria are essentially irrelevant for the kinds of purposes that we pursue.
The citations that Karwan gave you summarize the current wisdom with regard to how to ensure quality ("rigor") in qualitative research. In addition to the 5 criteria that he used, you will also find triangulation, which Miguel used, on the lists from Guba and Lincoln.
The bottom line is that you want to assure people of the quality of the work that you did, and strategies suggested by Karwan and Miguel are typical ways that one does this.
Hi
Regarding the reliability, the following remarks need to be incorporated:
1. The reliability measures could be developed for the whole questionnaire or for each dimension of the questionnaire .
2. Reliability consists of several measures:
Item alpha reliability and split-half reliability assess the internal consistency of the items in a questionnaire – that is, do the items tend to be measuring much the same thing?
3.Split-half reliability on SPSS refers to the correlation between scores based on the first half of items you list for inclusion and the second half of the items. This correlation can be adjusted statistically to maintain the original questionnaire length.
4.Coefficient alpha is merely the average of all possible split-half reliabilities for the questionnaire and so may be preferred, as it is not dependent on how the items are ordered. Coefficient alpha can be used as a means of shortening a questionnaire while maintaining or improving its internal reliability.
5.Inter-rater reliability (here assessed by kappa) is essentially a measure of agreement between the ratings of two different raters. Thus it is particularly useful for assessing codings or ratings by 'experts' of aspects of open-ended data; in other words, the quantification of qualitative data. It involves the extent of exact agreement between raters on their ratings compared with the agreement that would be expected by chance. Note then that it is different from the correlation between raters, which does not require exact agreement to achieve high correlations but merely that the ratings agree relatively for both raters.
6. For more details, please refer to
Howitt and Cramer (2008). Introduction to SPSS, Pages 249-258.
Regards,
Zuhair
Regarding validity:
7.Still, this unified concept of validity is best understood and examined within the context of its four discrete facets: content validity, construct validity, criterion validity and consequential validity.
8.Several authors suggest employing the following four steps to effectively evaluate content validity: 1) identify and outline the domain of interest, 2) gather resident domain experts, 3) develop consistent matching methodology, and 4) analyze results from the matching task.
9.Several series of steps in which to follow when pursuing a construct validation study:
1) generate hypotheses of how the construct should relate to both other constructs of interest and relevant group differences,
2) choose a measure that adequately represents the construct of interest, 3) pursue empirical study to examine the relationships hypothesized, and 4) analyze gathered data to check hypothesized relationships and to assess whether or not alternative hypotheses could explain the relationships found between the variables.
10.Criterion validity refers to the ability to draw accurate inferences from test scores to a related behavioral criterion of interest. This validity measure can be pursued in one of two contexts: predictive validity or concurrent validity. In predictive validity, researchers are interested in assessing the predictive utility of an instrument.
11. For more details, please refer to,
Instrument Reliability and Validity:
Introductory Concepts and Measures
Amanda Jane Fairchild
James Madison University
Regards,
Zuhair
This might be very helpful also: http://www.jcreview.com/fulltext/197-1586090269.pdf
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire