Here are some criteria I look for as a journal editor for a journal that will accept some types of qualitative studies. These are in addition to the ones listed by Georgio

1) was the qualitative data collection conducted by a trained qualitative researcher?(e.g. If focus groups were conducted was the person appropriately trained to conduct focus groups beforehand - by whom? ). What experience did the person have?

2) were the research subjects appropriately selected to represent a broad spectrum of the target audience? (I.e. a group of college students rarely is an appropriate group for most situations). Are the population limitations mentioned.

3) were the questions asked as unbiased and non-leading as possible OR were the observations made in the least intrusive way possible to avoid biasing the data collection? Is the interview/moderators guide or other research leading document available for review to show its unbiased nature (I recommend it be published as supplementary information online in the journal I edit)

4) was the data independently reviewed and collated/categorized by several different individuals if at all possible to ensure an unbiased interpretation?

5) what alternative interpretations were considered? How was it determined that these interpretations were not the selected interpretations?

6) how has the researcher proposed that the results an be validated using additional research.

I am not aware of any official document dealing with results of qualitative research. I was chairman of a working group setting criteria for quantitative analytical chemical work (EURACHEM Guide no. 2), which is more straightforward than qualitative research. Personally I am a strong follower of Karl Popper’s suggestions. In this respect every statement has to be falsifiable! If the qualitative research follows this simple guide, it can be regarded as valid and in agreement with the reality (inter-subjective agreements on the presented statements). If a falsification is theoretically impossible, then the output of this research are simple personal opinions or matter of beliefs and should not belong to science.

I believe that answering to these 6 questions (from IJPM) gives clear indication about the quality of the work.

Have you made the aim of your work clear?

Have you explained the significance of your contribution?

Have you set your work in the appropriate context with sufficient background, and all relevant?

Have you addressed the question of practicality and usefulness?

Have you identified future developments that may result from your work?

Have you structured your thesis in a clear and logical fashion?

My understanding the Master of Philosophy should be given the same scrutiny that a PhD thesis is given! So, what Giorgio and Karl had indicated would be taken into account.

Regards

Thank you very much Theodora, Giorgio and Carl for the comments.

Giorgio presented very clear items suitables for all research work - so its like a common ground. It could be seen as a basic instrument and I agree with Giorgio that qualitative research must respond to these items too.

Carls "simple guide" is very fundamentally to science, and I think it is on the ground of the initial research question, students formulate to start their research work and when they clarify if their question is really a scientific one or a pseudo-problem. But once responded to this, we are searching for more criteria: You can find it in the quantitative research field, when you are looking for the quality of measurement (I think Carl, this is one of the main focuses in your area).

Theodora, thank you for this vision - we take into account the indications of Carl and Giorgio. How can we go further?

We are discussing in our group two different points of view: 1) adapt criteria from natural sciences and quantitative research to the field of cualitative research (reliability/validity/generalization) or 2) build proper criterias (transparency/autoreflexivity etc.)

Here are some criteria I look for as a journal editor for a journal that will accept some types of qualitative studies. These are in addition to the ones listed by Georgio

1) was the qualitative data collection conducted by a trained qualitative researcher?(e.g. If focus groups were conducted was the person appropriately trained to conduct focus groups beforehand - by whom? ). What experience did the person have?

2) were the research subjects appropriately selected to represent a broad spectrum of the target audience? (I.e. a group of college students rarely is an appropriate group for most situations). Are the population limitations mentioned.

3) were the questions asked as unbiased and non-leading as possible OR were the observations made in the least intrusive way possible to avoid biasing the data collection? Is the interview/moderators guide or other research leading document available for review to show its unbiased nature (I recommend it be published as supplementary information online in the journal I edit)

4) was the data independently reviewed and collated/categorized by several different individuals if at all possible to ensure an unbiased interpretation?

5) what alternative interpretations were considered? How was it determined that these interpretations were not the selected interpretations?

6) how has the researcher proposed that the results an be validated using additional research.

This might not be directly on point in terms of assessment rubrics, but I found this article on the intersection of qualitative and quantitative paradigms very useful when thinking about how the paradigms might complement each other:

Pope, C & Mays, N 1995, 'Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research', BMJ: British Medical Journal, vol. 311, no. 6996, p. 42.

I think this article could inform an approach to developing assessment rubrics.

Dear all,

great idea so far!

I just want to point out that we are speaking about a "masters thesis", not a PhD dissertation. It is at another level and the time given is (depending on your university rule) is 1/4; 1/6 of a PhD. I believe that this need to be taken in to account. Moreover a "master thesis" should cope with a more "practical" problem that a PhD dissertation and the student's analysis should be "practical enough" for a real life problem, at least for "industrial engineering".

Perhaps the criteria will have to start with an approved format or outline- Philosophical basis, research question, methodology, theory building. The basic criterion would then be following this outline. Next would be the acceptability of the philosophical basis to the evaluation committee (which would be guided by each of the committee member's perspective), the fitness of the research question to the philosophical foundation, the appropriateness of the of the method to answer the research question, and the novelty of the theory developed. Inasmuch as a qualitative research is measured in terms words (or actions), grammar, spelling, clarity of words, organization of ideas, sentences, paragraphs and chapters must be given heavy weight.

While most of other issues have already been highlighted, I feel reliability and researcher's bias are the two issues which affect acceptability of qualitative study.

Therefore, it would prudent to ask the students to use tools like NVivo which automates most of the components of qualitative research like interview, document analysis etc.

this semi-automation of qualitative research would address both the issues which I have brought out.....

Qualitative research at any level requires rigor. This includes the ability for the methodological procedures to be followed by another researcher. Although, the emergent findings may differ based upon the researchers interaction with the data. Master's level researchers need to demonstrate the ability to present a research design and procedural guidelines before the research begins. This is often best accomplished through the presentation of a prospectus (purpose, design, research questions, sampling, interview protocol, method of data analysis, and potential limitations of the study) to a research mentor, then submission to the Institutional Review Board. This issue as addressed earlier is that the researcher and the evaluators of the research should be trained in qualitative research.

A lot of good answers have been provided, but these criteria might also be of interest, from Richardson, Laurel. 2000. “Evaluating Ethnography.” Qualitative Inquiry 6 (2) (June 1): 253–255.

1) Substantive contribution: This research contributes to not only the greater body of academic literature surrounding the topic at hand, but also seeks to give voices to the participants therein.

2) Aesthetic merit: The thesis is presented in a way that is accessible, appealing, with a style that complements the material at hand.

3) Reflexivity: qualitative research, and especially ethnography, need to acknowledge the role and biases and influences of the researcher.

4) Impact: how the research affects the reader emotionally, intellectually, and otherwise.

5) Express a reality: Beliefs and attitudes are real to people, and as such they should not be minimized or essentialized by a researcher trying to present an unbiased analysis.

Dears

I want thank Rainer to raise the issue. Following the contributions I understand that my interest is not the same but let me see that the issue is really critic.

Actually I'm interested on the research impact from different perspectives: researcher, institution and society.

Some elements are common but some are not. By the way my are is management control, accounting and actuarial sciences.

I don't think the main problem is the dilemma quanti/qualy. I assumed it is an assumption applied the most it is possible due to the kind of impact and possible ways to access.

I think that main point is term: short, medium and long term. Here is a relevant problem due to the difficulty to gathering data and associate to the research contribution. Any additional insight?

Thank you for all the answers - they enriched our perspectives and the group discussions. Now we are finishing the work and in a few days I can present the results in a spanish version, further on we seek to make a translation..

Dear all. Thanks for an interesting discussion.

Based on my experience with qualitative method and my experience of supervising MA-students my answer to the initial question is:

1. Starting point

Theory and method is part of bachelor and master courses and the MA-thesis should reflect this training. If the thesis relies on qualitative method, courses in qualitative method should be part of the MA-students training.

A theoretical stand on qualitative method should be presented as a starting point of the MA-thesis. In general it will be expected that the student’s work reflects the theoretical position of its department. If the student decide upon a different stance than his/her department, the wise thing is to make sure that this is accepted by the supervisor before the student starts on his/her MA-thesis.

Qualitative method is regulated by ethical guidelines and the MA-student, in discussion with his/her supervisor, need to present (discuss) and implement these guidelines it his/her work. Ethical stance and considerations are to be presented in the introduction of the MA thesis.

Each department will (or should) have standards for quantity and formalities, i.e. how long the text should be, usage of illustrations, models, schemes, foot notes, list of references, how to refer to web-pages etc.

2. Material

Use of qualitative method is motivated by a willingness or need to gather substantial and complex information that cannot otherwise be identified.

Depending on the theme and research question the MA-student need to discuss with his/her supervisor, and later on to argue in the thesis, on why and how the interviewees were chosen. Some times one person will be enough for a particular project, for another project one would like to interview 5 or 10, for another it could be 30 etc. The underlying question is: who and how many do I need to interview to get ‘a full picture’ of the theme that I am to investigate, i.e. how do I gain material that is substantial enough to carry the analysis of the MA-thesis, to make it possible to answer the main question of his/her investigation?

4. Analysis

A main characteristic of all research is intersubjectivity, what T Portman nicely describes as the ability for the methodological procedures to be followed by another researcher. It must be possible to read and follow, step by step, the research process in the text/MA-thesis, to make the process open for discussion, including how conclusions were reached. How was the material gathered and interpreted? What is the relation between material, interpretation and conclusion?

To analysis qualitative material is an exercise in interpreting inter-textual and external contexts that contributes and may explain the meaning of the material/text. To decipher layers of meaning, the textual exegetical handicraft presupposes knowledge in the area the material is to represent. It might include cultural, social, religious, economic, historical, or other factors that is of importance to understand before on can reach a qualified interpretation. As such, the interpretative process is open ended. When the thesis is written, the text might be re-read and part of a quite different analysis leading to new conclusions.

5. Contribution

The MA-thesis should reflect that the student has understood the literature that s/he has been trained in on bachelor and master courses. In addition, a MA-thesis need to reflect in depth knowledge related to the theme of the MA-thesis. Usually this will be acquired before and during the MA-thesis writing process, guided by the MA-student’s supervisor. Contribution means that the MA-thesis is to represent new knowledge, either regarding research question, material, perspective, concepts or theoretical elaboration.

6. In sum:

- Criterion for starting point of MA-thesis is relatively straight forward.

- Criterion for use and creation of qualitative material will vary and depend on the department’s tradition and the supervisor’s qualifications. In general, if one wants to represent a different tradition than one’s department, one should investigate if this will be accepted by the department or not.

- Criterion on intersubjectivity is straight forward. Criterion on whether or not the interpretation is qualified, which areas of knowledge do the student need to included in his/her interpretation to make it a qualified reading is more open and will reflect the tradition of the department and the MA-student’s supervisor.