I think that Mixed Methods is much more about the use of the data, rather than the specific technique that was used to collect it. In particular, if you have research questions that amount to "how and why" and your open-ended data can address those questions, then I would certainly consider that qualitative research.

In your case, you mention triangulation, so that would imply that you could use your qualitative data and your quantitative data to produce results comparable that could be compared. That would certainly be an example of Mixed Methods research.

Think of it this way: if your questionnaire was exactly split 50-50 between close-ended and open-end questions, you probably wouldn't have any difficulty called that mixed methods. But what it were split 60-40, would that be good enough? Could you set the boundary at 25-75, is that enough? What if there was only one open-ended question, but it produced the most interesting and important results in the whole data set, would you deny the qualitative nature of that data and claim that it was quantitative merely because it came from a survey?

Hi, there is hardly an answer to find because the term mixed-method is used very differently by Researchers. Initially it was meant to have at least two different methods, a qualitative and quantitative Approach (for instance). However, Researchers tend to speak of qualitative research even when they use open ended questions in a quantitative Survey. The big difference between the open ended questions and a qualitative Approach is that the latter is interactive, the first not. You cannot react to the answers given in a quantitative Research. Therefore I would recommend not to Claim a mixed-method Approach.

I hope this helps

Hello Alicia, 

Actually, it is a very interesting question. However, although you are using both closed and open-ended questions in the same study and these questions are used for collecting specific data in your quantitative method, this does not explain 'why and how'. It is an enough reason to not claim a mixed-method design. I do recommend the following book. You might find it useful: 

Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications.  

From what you say, then at the moment, you have only used one method---survey, irrespective of whether you have used a combination of closed and open questions.

An example of a mixed methods  study would be for instance, combining survey with observation.

In particular, your study would become mixed methods if you combined your online survey with for instance, 1-1 qualitative interviewing and analysed these interviews according to the tenets of say, phenomenology or Grounded Theory.

Thank you everyone. I thought as much but great to have other confirm it. Best wishes, Alicia

I think that Mixed Methods is much more about the use of the data, rather than the specific technique that was used to collect it. In particular, if you have research questions that amount to "how and why" and your open-ended data can address those questions, then I would certainly consider that qualitative research.

In your case, you mention triangulation, so that would imply that you could use your qualitative data and your quantitative data to produce results comparable that could be compared. That would certainly be an example of Mixed Methods research.

Think of it this way: if your questionnaire was exactly split 50-50 between close-ended and open-end questions, you probably wouldn't have any difficulty called that mixed methods. But what it were split 60-40, would that be good enough? Could you set the boundary at 25-75, is that enough? What if there was only one open-ended question, but it produced the most interesting and important results in the whole data set, would you deny the qualitative nature of that data and claim that it was quantitative merely because it came from a survey?

I agree with David Morgan here. If the questionnaire yields answers that are a very comprehensive und can be analyzed properly with qualitative methods, I would consider this mixed-methods because you have to combine numerical and textual data on the same data set. 

I disagree - definitely NOT. The methods of data collection are also part of the methodology. To convert an open-ended questionnaire into a quant material is already a tricky thing (have tried) – it is then limited by the choice of a researcher and in case the theoretical frame and justification behind it is not adequate (as usually) it has very little scientific value.

If the goal is to conduct an open-ended research then the methodology should be composed as such also.

I didn't catch the part where she wrote about "converting" the open-ended questionnaire. I assumed the open ended questions would be analyzed qualitatively. In that case, depending on the material, the data analyis as part of the methodology does not have so different to analyzing a section of an interview. Hence, as David Morgan pointet out, it's difficult to answer the question "definitely" because the boundaries when qualitative methodology starts are not always clear cut. 

In exploring different methods for my PhD thesis, I have found this to be a difficult question with no simple answer as there is much disagreement over the boundaries between 'qualitative' and 'quantitative' methods. Does the way you collect the data have to be attached firmly to either label or can the method (in this case survey) not be seen as the best way in which to collect data which can then be analysed quantitatively and qualitatively?  For example, what if you meet face to face with a participant to carry out a survey which involves many closed questions? Does this make the method more qualitative? I think if you are asking questions in a survey online which elicit responses to 'how' and 'why' and contexts then its not the method of collection but the data that is qualitative. However, my reading has show that t'he use of mixed methods is also further complicated by ontological and epistemological concerns as to the type of knowledge you are seeking.   For a stimulating article on using mixed methods in social scientific and criminological research, the following is an interesting article by Shadd Maruna: https://www.researchgate.net/publication/227016120_Mixed_Method_Research_in_Criminology_Why_Not_Go_Both_Ways 

Chapter Mixed Method Research in Criminology: Why Not Go Both Ways?

Thank you for your contribution. I am working on an article that is about data collection, analysis and reading in criminology (social sciences).

I will read the attached chapter and comment then but...

Methodology and the following methods should not be used as synonyms.

My MM concept is based on e.g. K. NIglas’ works

Thank you for posting those.

It really depends on what you do with the data once collected.  Naturally quantitative data will be analysed for means/statistical significance/effects size as and where appropriate.  The open ended questions - if detailed/rich enough - could be subjected to some form of qualitative analysis.  For example you could do some very basic thematic framework analysis (look at Braun & Clarke, 2006 for thematic analysis methodology) and then as long as both quant and qual have been empirically collected and analysed, the write up could very well be mixed methods.

Alicia,  it does not seem to be a mixed method aproach to me. In my opinion, obtaining qualitative data does not mean a study is mixed methods. Using qualitative and quantitative methods is what defines a mixed methods approach. You used the survey and within this method, you obtained qualitative data. Of course, it depends of the whole setting of your study. I like to explain to my students, the difference between qualitative method, qualitative data, and qualitative analysis. Your qualitative data  is certainly very important but it was collected using a method that is not typically considered as a qualitative method per se (although some researchers argue that survey may be a "hybrid" method right at the border).  Also, remember that qualitative data can be analyzed and reported in quantitative ways ; can be quantified. I try to stay away from that when doing qualitataive research but know colleagues who support that quantification. For example, when we report  frequencies of times a certain topic was mentioned. I am just trying to illustrate this for you. You may find different opinions on this. Hope this helps .

I think Dina's response is correct here.

The issue of whether converting qualitative data into quantitative data remains controversial within the field of Mixed Methods Research. Note, however that the Journal of Mixed Methods Research does include such articles.

Generally, this involves using responses to open-ended questions as a basis for counting codes, and is known as quantification (or quantization). I myself do not think of that as Mixed Methods Research because it does not involve any qualitative research questions or results.

A useful way of thinking about your qualitative data Alicia, might be to think of it as 'small q' qualitative research rather than 'Big Q'.  I think it was Kidder & Fine (1987) who first made this distinction, but it's discussed in a number of qualitative methods textbooks (e.g. Jonathan Smith's; Braun & Clarke).  'Small q' research is seen as using qualitative date but not necessarily engaging with the epistemological & ontological assumptions common in 'Big Q' qualitative research.  Therefore it might have an approach to the 'truth' of data that is similar to (post) positivist quant research & might be concerned with issues of reliability etc.  

However, collecting qualitative data via a questionnaire is not necessarily at odds with the assumptions of 'Big Q' research.  It partly depends on how you set up your questionnaire, though as others have pointed out, using a questionnaire is likely to limit your understanding of context and you certainly can't probe novel issues that the participant raises. 

Do you need to label your research as 'mixed-methods' or 'small q' etc.  Is it not enough to clarify the aims addressed by the different parts of your questionnaire, the epistemological assumptions you are making about the different kinds of data collected and your rationale for using open / closed questions?

Good luck!

Is there an imperative for your research to be coined Mixed Methods ? I would state as others have pointed out that your research is quantitative (data produced via survey) but that you will do a qualitative analysis of a portion of the produced data.

Alicia, It is not a mixed method. That is just qualitative data which will be used in a quantitative study.  I think the whole logic of quantitative and qualitative approaches are unmixable. Quantitative studies begin hard in theory, looking for variables that you are going to confirm or deny.  Qualitative studies look to define categories based on theory but also on empiric data.  To sum, keep saying that your study is quantitative and if you want, explain in detail the way you analize that qualitative data obtained with open questions. In fact, you could transform the main categories obtained in those open questions in new closed questions, for a solid quqntitative analisys. Regards.

Hi, Dear All 

Thanks, for sharing very good question, regarding this question I hope below useful

Open-Ended Questions

Open-ended questions are exploratory in nature. As discussed with the “How do you get to work?” question, it allows for the respondent to provide any answer they choose without forcing them to select from concrete options.

How to Use Open-Ended Questions

Questions that are open-ended provide rich qualitative data. In essence, they provide the researcher with an opportunity to gain insight on all the opinions on a topic they are not familiar with. However, being qualitative in nature makes these types of questions lack the statistical significance needed for conclusive research. Nevertheless, open-ended questions are incredibly useful in several different ways:

1) Expert Interviews: Since questions that are open-ended ask for the critical thinking and uncut opinion of the respondent, they are perfect for gaining information from specialists in a field that the researcher is less qualified in. Example: If I wanted to learn the history of Ancient China (something I know very little about), I could create my survey for a selected group of historians whose focus is Ancient China. My survey would then be filled with broad open-ended questions that are designed to receive large amounts of content and provide the freedom for the expert to demonstrate their knowledge.

2) Small Population Studies: Open-ended questions can be useful for surveys that are targeting a small group of people because there is no need for complex statistical analysis and the qualitative nature of the questions will give you more valuable input from each respondent. The rule here is the group must be small enough for the surveyor to be able to read each unique response and reflect on the information provided. Example: A supervisor who is looking for performance feedback from his/her team of six employees. The supervisor would benefit more from questions that allow the respondents to freely answer rather than forcing them into closed-ended questions that will limit their responses.

3) Preliminary Research: As stated in the closed-ended questions section, conclusive research usually requires preliminary research to be conducted in order to design the appropriate research objects, survey structure, and questions. Open-ended questions can reveal to the surveyor a variety of opinions and behaviors among the population that they never realized. It is, therefore, incredibly useful to use open-ended questions to gain information for further quantitative research.

4) A Respondent Outlet: It is usually a good idea in any survey, no matter how large, to leave an open-ended comments question at the end. This is especially in the case of a survey asking closed-ended questions on attitudes, opinions, or behaviors. Forcing respondents to answer closed-ended questions asks them to fit in your box of options and can leave them with extra information or concerns that they want to share with you. Providing respondents with the outlet of a comment box is showing them the respect they deserve for taking the time to fill out your survey.

Weaknesses

There are a few drawbacks to open-ended questions as well. Though respondent answers are almost always richer in quality, the amount of effort it takes to digest the information provided can sometimes be overwhelming. That is why open-ended questions work best in studies with smaller populations. Furthermore, if your survey sample is a fraction of the population you are studying, you will be looking to find data which can be inferred on the overall population as statistically significant. Unfortunately, open-ended questions cannot be used in this manner, as each response should be seen as a unique opinion.

Best Regards

http://fluidsurveys.com/university/comparing-closed-ended-and-open-ended-questions/

I have to disagree with Stephen Phillip Clayton's suggestion that merely using a survey is synonymous with ontological and epistemological commitments. It seems to me that the kind of data you get, and the ways that you can use that data, have much more to do with the nature of the questions that are asked, rather than the overall interview format.

If someone wants to do exploratory research with a subset of the questions in a survey, and asks those questions in an open-ended manner, and analyzes them in an inductive fashion, then that strikes me as the core of what is meant by "ontological and epistemological" commitments.

Thank you everyone. What an interesting discussion! I appreciate everyone taking the time to respond and provide your thoughts - it has been very helpful. 

Most of the open-ended questions were exploratory and were asking a cohort about their opinions regarding barriers and facilitators on a particular issue. The other key open-ended question was asking all three cohorts about how they define a particular term. There is no intention to quantify these responses - they are undergoing thematic analysis. 

I come from a pragmatism stance (so thank you for your responses David Morgan as I have read some of your work!). Where I was struggling was that I understood that I had collected quantitative and qualitative data (although I know the latter is contested when collected in a questionnaire), that I had done so through a survey method, but I didn't know how best to describe the methodological approach that sat above those two things. I am in no means a post-positivist and my research design was a series of pragmatic decisions in order to best answer the research questions. The study investigates the use of evaluation (a pragmatic exercise in itself) in a nonprofit context (so one concerned with benevolence and humankind), which is an incredibly complex system, and for this and other reasons, it speaks to pragmatism for me. 

Pragmatism and mixed-method often go hand in hand, hence my original question, I suppose. 

However, as some have suggested, perhaps I can justify my approach without specifically naming it as quant or qual. 

Best,

Alicia

Alicia,

As you can tell from this discussion, a lot would depend on where you send your work for review. If you went to a largely quantitative journal, you will receive criticism for doing anything that even appears to look like a survey. In contrast, most journals that routinely include mixed methods would have no problem in accepting your general approach.

From a pragmatist point of view, this corresponds to the the assumptions that all knowledge is both contextual and probabilistic. In other words, the ability to apply what you believe depends on the context in which you are operating, and there is no one-to-one link between what you believe and the outcomes you will experience.

In this case, operating in a context that is governed by the traditional assumptions associated with qualitative research will have implications that are very different from operating in context that is more receptive to mixed methods.

For what it’s worth, I will add my two cents to this very interesting discussion.

First, as to the question whether this is a MM research, I think the question is misguided. It would be better to ask: may this research be called MM? Such an approach to this question removes it from the ontological domain to the discursive, and hence the answer may be (to some extent) whatever you decide. Certainly, if your research questions include qualitative and quantitative questions then the research may be rightfully termed MM, albeit of a poor design qualitatively.

That said, the main difficulty in answering your question is that we are lacking your research question. Open ended questions may be of a qualitative nature or a quantitative (and unfortunately answers to each may be Qual or Quan irrespectively of the nature of the question).

As for using the term MM for the research, you may be well advised to avoid referencing the more rigorous and methodologically meticulous approaches to MM (if indeed the research design is of a relatively poor quality from a qualitative perspective, which in part depends on the open-ended questions and their phrasing, as well as on your inclination to engage in actual interviewing or other method of data collection), and define your research on your own terms.

Take a moment to think what are your paradigmatic presuppositions, and do they correspond with a (post-)positivist or other (interpretative) paradigm. Ask yourself what are your true interests and what you wish to uncover in your study, and whether your open-ended questions enable answering these interests, and decide accordingly. If you would like to make qualitative discernments then your study may be termed MM, but if all you wish is to support your quantitative distinctions then I would not use the term at all, and just mention the use of open- and close-ended questions. Thus you may avoid the controversy altogether.

This is pertinent when the time of publication comes, for you do not want a poor qualitative research to be judged by a serious qualitative researcher. You might wish (if your inclination is more towards the positivist paradigm) to have a quantitative reviewer which will see the additional value of your design than its shortcomings.

Best of luck.

My original reaction was to say No, the open-ended questions are not enough to make this a mixed methods study. But I didn't want to go only on my opinion so I consulted with some research colleagues and they agreed. To be a mixed methods study, there needs to be a purposeful integration of the two types of studies, with three types of research questions: a qualitative question, a quantitative question and a mixed methods question. The purpose of open ended questions in a survey is to allow respondents to expand somewhat on the quantitative question that is asked. 

The beginning of this article gives a really good explanation of what mixed methods is. It is an easy and informative read.

https://pcmh.ahrq.gov/sites/default/files/attachments/MixedMethods_032513comp.pdf 

There are not enough options to combine them with a Lickert scale

My experience is also same as many other answers. In sort, we try to mix both qualitative and quantitative data in mixed method research. Open ended questions will help to gather qualitative data, but they will not help you to collect quantitative data. I would suggest not only a survey with both structured and unstructured questions, but to use other qualitative data collection methods including FGDs etc to gather qualitative data while structured questionnaire is using for quantitative data gathering.

Thanks to those who have provided additional responses to my earlier question. Much appreciated. 

I view that using closed and opened-ended questions online cannot be claimed both qualitative and quantitative methods have been used simultaneously and adequately. If the researcher has not directly interact with his or her research participants and has not engaged in probing certain questions or asking new questions based on what a research participant has responded to a particular question there is a possibility that the researcher will miss many important aspects of the qualitative information/ data. Furthermore, in every type of qualitative research direct interpersonal interaction between the researcher and the research participant, observing research participant’s facial expressions, gestures, and any other forms of no-verbal communications are very significant. In addition to recording the conversations/interviews, a qualitative researcher is supposed to observe and to keep notes about various issues while conducting a semi-structured or opened-ended interview. It is also important that the researcher has been able to build trust of the research participant in the researcher or in the research before asking the research participant for answering certain questions. This means such research will suffer from lack of in-depth-ness and rigorousness if there is no direct or face-to-face interaction between the researcher and the research participant.

 Pleas check these links:

https://www.surveymonkey.com/mp/open-ended-questions-get-more-context-to-enrich-your-data/

http://www.infosurv.com/open-ended-questions-qualitative-research/

http://www.simplypsychology.org/qualitative-quantitative.html

http://www.ccs.neu.edu/course/is4800sp12/resources/qualmethods.pdf

To all of you claiming that the method is not MM, could you please elaborate how do you determine this without seeing the questions?

For instance, I have a questionnaire I am working now to validate (quantitative), and added to the questionnaire, which refers to reactions to a given experience, are two open questions, both asking the person to share the story of experience, guiding participants to address, what it felt like? What were your thoughts? Who was involved?

Does this soliciting of narrative imply that the research may be considered a MM, if indeed the answers are narratives (in the strict interpretation of what constitutes narrative: temporality, plot, characters, etc.)? And, given that the analysis will be a narrative analysis (According to Lieblich, Tuval-Mashiach, & Zilber, 1998)?

I agree with Jacob Stein, that if the open-ended questions are carefully crafted to produce qualitative data, then that is exactly what you have, regardless of whether they were embedded in a survey.

Frankly, I can't understand all the dogmatism that what matters most is the surrounding survey format, rather than the data itself.

That's my point too!

Agree with David and Florian.

The five key questions that are relevant to this discussion are:

1) Evaluation may mean different things to different organisations. How does your organisation describe/define evaluation?

2) Some people may describe/define evaluation differently from the organisation in which they work. How do you personally define/describe evaluation?

3) What are the key barriers you face in leading evaluation in your organisation in the following areas? Please describe them in each box below.

(there are several options here re colleagues, organisational culture, processes etc)

4) What are the key things that assist you to lead evaluation in your organisation in the following areas? Please describe them in each box below.

(again, several options here)

5) In your opinion, please describe the three key barriers your organisation faces in relation to evaluation. Please use examples where appropriate. 

If you intend for your study to be mixed methods, then approach it as such. You can, for instance, perform quantification and statistical analysis of the items/scales represented by the closed-ended questionnaire items. Then couple that with qualitative content and thematic analysis of the open-ended questions, say from a phenomenological perspective. Then interpret and report findings from both streams of analyses together. That is an example of a mixed methods approach. So the short answer is: It depends on what you do. It counts as mixed methods if that is what you actually do with the data.

Agree with above.

My assessment is that your questions would be a good fit for a one-hour, semi-structured interview. In a survey, they are likely to produce much "thinner" data -- perhaps a sentence or two per question from most people. This is pretty much what most qualitative data analysis programs anticipate in what they describe as their mixed methods, so that should be a good fit to mixed methods in general.

As for more specific purposes, I agree with Anthony Waddimba that it all depends on what you intend to do with the data.

Hi everyone. Data collection is complete and unfortunately no interviews have or will be undertaken. I just wanted to post the questions to follow up on Jacob and David's questions to others in the thread re not seeing the questions yet making an assessment re MM.

Interviews would be a good way to build on the findings from the survey. Qualitative content and thematic analysis of the open-ended questions can give you a broad range of the views and perspectives that are held by your respondents.Within the responses to those open-ended questions you will not get "thick" descriptions that provide indepth description of any one theme. But you will get "thin" data laying out a whole range of explanations, which is a good start. Based on extant literature, your own philosophical perspective, theory, or on findings from the thematic analysis, you can then choose to do a more indepth focus on a select few of the emerging themes or categories by conducting follow-up interviews or focus groups.

From my experience, this is something that needs to be planned during the design phase. What theoretical framework and philosophical assumptions guided your questions? Was there an integration analysis plan? Using the qual data to answer a qual question and the quan data to answer a quan questions is not really Mixed Methods, it is more like multimethod. Here is a good article to read about the integration piece: https://www.researchgate.net/publication/51876404_A_Methodology_for_Conducting_Integrative_Mixed_Methods_Research_and_Data_Analyses

The lead author, Felipe Castro, is helpful to contact with questions.

Hope this is helpful.

Article A Methodology for Conducting Integrative Mixed Methods Resea...

Hi sisters and brothers

I agree with Tara Gwyn Bautista idea because our data collection tools are dependent on our research paradigm and theoretical framework in general and our topic and specific objectives of the study in particular. I may forward more of my opinion if I know your title and your specific objectives. Anyway I think it is better if you relook again your research paradigm, title of your study and its objectives.

I have followed this discussion with interest as I am also uncertain as to how best to describe the methods used in my PhD .....so I am not really contributing in the form of an answer. I am not even sure if anyone will see this. I am essentially a clinician and the purpose of my study is to explore the role of health visitors in supporting mothers with mental health problems with a view to developing an effective, feasible and acceptable intervention framework that could guide their practice in the future. (this latter part was added as a result of the findings from the first part) The first part of my study was to explore what health visitors are doing at the moment; the factors that influence their practice and what they think they should be doing in the future with respect to this aspect of their role. So I circulated an electronic survey and had 1637 responses to 40 questions including 9 open ended questions - I had 3,563 responses to the open-ended questions which amounted to 170 pages (A4) of free text. Apart from initially feeling totally overwhelmed by this volume of information, it does indicate to me that a lot of health visitors have very strong views about this topic and I feel a great responsibility to make sure that their voices are heard. I am thematically analysing these answers using NVIVO11. This data has given me a very rich picture of the realities of practice and health visitor thoughts about this aspect of their role and a much greater breadth and depth of information than I could possibly have hoped for from the closed questions. The two sources of data are very much interlinked - what is actually happening in terms of numbers but also why in terms of explanations. The second phase of my research is a two round real-time technological modified Delphi to gain consensus from a group of expert health visitors regarding the components of a health visitor-led intervention ( utilising what HV's told me in the survey combined with literature reviews of effectiveness, feasibility and acceptability). I thought I was doing a mixed methods multi-phase study but given the discussion above now I am not so sure. Any suggestions / advice would be much appreciated.   

My recommendation is the following and you have a very good database with this information it is pertinent to classify in criteria that allow to define the objective of the subject of your investigation with this it is necessary to close the questions with a scale of Likert to quantitatively measure the data and that You can tie them with relevant criteria and then compare with the opinion of the experts

Catherine,

One of the key issues in mixed methods research is how you integrate the results from the qualitative and quantitative portions of the overall project. Your statement that you are interested in: "what is actually happening in terms of numbers but also why in terms of explanations" sounds like what is known as a sequential, explanatory design or QUANT --> qual. In other words, you are using what you learn from the analysis of the qualitative data to explain issues that arose from the quantitative data.

Doing a Delphi would add a third step to this design, so you need clear statements about both how the earlier research motivated the Delphi and how the results of the Delphi will be integrated with those earlier results.

I think David's answer points you in the right direction here.

Hi Alicia,

I' m answering this without having read the previous responses (I don't have the time), so please forgive me if I'm beeing reiterative. 

Responses from open - ended questionaires are qualitative data as long as it produces textual data. 

You can use software like "MAXQDA Analytics Pro" to convert responces from your textual data to frequencies ( through coding process) and combine them with quantitative data and ask questions about your data. "Tableau" software is also a good way to represent data and visualize your results, asuming you have create tables with your data.

If you have time you can read the "QUALITATIVE TEXT ANALYSIS" by Udo Kuckartz SAGE publications. This will help you with the data from your open ended questionaires and how to manipulate them to create codings - categories and quantify them. 

Good Luck!!!

P.S.

If you achieve and combine your qualitative data with quantitative data together then the answer @ your question is definitely YES 

Think of your open-ended questions as questions of a structured interview. If you combine in your research surveys and structured interviews does it make your study a mixed methods one? I don't think so!

Maybe you should also consider your open-ended questions data analysis: was it performed with a priori categories?

This discussion has been valuable to me

I see the merits of David Morgan's suggestions, but I would respectfully add some other points that lead me to say "no". I don't think an open-ended question suffices to tap into the depth, complexity, and levels of meaning that other qualitative methods are capable of. Qualitative research isn't just allowing respondents to say what they will, which is exactly how far open-ended questions are able to take you, but also consists of engagement from the researcher. Engagement is more than just prompts in a survey, because it should be both (a) recursive, comparing what a respondent says/does later with what they say/do earlier, and (b) reflexive, relying upon the researcher to reflect on their own positionality and their own experience from the engagement itself. David Morgan's own pioneering work on focus groups exemplifies how a two-sided interaction that respects these principles can truly excavate layers of meaning in a qualitative study. Open-ended questions are ultimately one-sided, and so can't allow for such an engagement. In sum, they don't justify relabeling your study as MM.

In response to Anson Au, I would say that there is a difference between combining methods, which is what Mixed Methods Research is all about, versus doing either qualitative or quantitative research, which is what she is describing. In particular the supplementary, qual, part of a QUANT --> qual design might indeed be insufficient for a full-scale piece of qualitative research. At the time, it might well be adequate for generating hypotheses in a sequential explanatory design.

Mixed- method research requires handling the qualitative and quantitative research together to derive meaningful conclusions. There are many ways of combining as can be seen in the paper on Flexible Systems Methodology.

This is a dilemma for us as well. The problem has escalated due to different opinions on whether mixed methods approach can be adopted given the divergent research paradigms.

Nonetheless, we are inclined to follow up our quantitative data results with in-depth interviews of a few respondents to find out how? and why? We plan to use the sequential explanatory research design where we start with quantitative data collection via a questionnaire (which has 90% closed questions), followed by qualitative in-depth interviews using semi-structured and open-ended questions.

I agree with David Morgan that if your questionnaire was exactly split 50-50 between close-ended and open-end questions, you probably wouldn't have any difficulty called that mixed methods. In using mixed method you will triangulation you qualitative and quantitative data to produce results hat will answer your research questions.

Qualitative data can also be obtain from in-depth analysis of related literature.Analysis of related literature will provide qualitative data that will answer your research questions. Good luck. [email protected]

Interesting question. I am a qualitative researcher who is faithful and acknowledges the truth in a qualitative approach. However, I think anyone is also entitled to choose a recognized method of approach, and supported by honesty in the implementation. My advice, please read his book Cresswell which discusses the mixed method.

Are open-ended questions on surveys enough for a study to be considered mixed-method?

It is possible. I'd done that with one of my professors using MMR Explanatory Sequential Design - you can refer to this article:

Article Relationship between Team Satisfaction and Project Performan...

Just because you have open-ended research questions in a survey does not necessarily make your study mixed methods. In order for your study to be considered mixed method, the research design should invoke that various qualitative, quantitative and if possible geo-spatial approaches you are going to use to collect and analyze your data. Just because you have few questions in a survey that are open-ended does not make your study mixed method. I would like to encourage you to explore "Research Design: Qualitative, Quantitative, and Mixed Methods Approaches By John W. Creswell, J. David Creswell" on the use of mixed methods in research.

The short answer is no! The theoretical answer is that your research design should outline and justify the mixed methods approaches you are going to use, how you are going to analyze and triangulate your results such that the explanations captures the results collected via qualitative, quantitative or geo-spatial techniques.

No. The open ended questionnaire would cater for only the qualitative data. Closed ended questionnaire in addition would spark it with the statistical components filling the quantitative void.

Wow - such an interesting discussion - thank you all for your very helpful suggestions. What I have done with the 3,563 responses to the 9 open-ended questions ( over 104,000 words.. so not just one sentence reflections in many cases....and the volume of data is equivalent to the PhD thesis that I have to write... so it has felt rather unwieldy and overwhelming at times) is imported the text into NVIVO11 and then used the Theoretical Domains framework to categorise the determinant of practice and the TIDieR checklist to categorise what the respondents think should be included in the intervention PLUS additional inductive categories for comments that don't fit with either of the aforementioned frameworks. It has been a very interesting journey and I have learnt so much more from the free text responses than was evident from the answers to the closed questions. I am in the process of writing my findings up in the form of two articles - one about the use of the TDF and one about the use of the TIDieR checklist. Wordcount is again my enemy - trying to condense my findings into a journal length article is easier said than done!

I personally do not agree that we can call a study mixed methods just because an optional open-ended question was added to the quantitative questionnaire. This, indeed, really disappoints me. First of all, research question/s should require collection, analyses and even maybe syntheses of both research strands to be able to provide explanatory and elaborate answers to RQs. In some advanced MM designs, one strand is used to shape the rest of the research design while in some others one strand requires the other one to be able to make more sense of the data. With its rigorous data triangulation, MM design should be chosen to benefit from the strength of both research strands that complement each other and result in rich data. I strongly recommend for those interested in MM design to read John Creswell's books. In conclusion, MM design is a much more complex design than unfortunately most studies out there that allegedly claim to be. @John Creswell

Actually, I see so-called 'mixed methods' research as simply doing transparent research without the fuss of 'a new paradigm'.

I disagree that design and implementation is necessarily complex, although from the start, one must think about how the 'mixing' of say a qualitative method with a quantitative one will deliver a final integrated outcome----rather than simply 'tack' one on to the other with little thought.

From the 'mixed methods' perspective, adding an optional open-ended question to a quantitative questionnaire is certainly not 'mixed methods'---it's still quantitative research.

If it is done carefully, the use of additional open-ended questions can be quite useful for an "Explanatory Sequential" mixed methods research design. In that case, the goal is to gain a better understanding of the quantitative results, so open-ended questions can be included to help address that issue.

Note also that the sequencing in sequential designs is about how the data is used, not just when it is collected.

I agree with Junianita Sopamena above.

My understanding of Mixed methods design depends on the nature of the research inquiry. The mixed methods strategy highlights complementarity. If we pursue a qualitative inquiry and wish to simultaneously obtain statistical data, beyond the basic descriptive type, then we would need to propose a separate quantitative design which will complement and potentially answer the overall research question.

While it is fine to have a number of open ended questions within a quantitative study such as a survey, such a strategy cannot be called mixed methods because we are not using separate research methods which when combined answer an overall question.

Depending on the wordingof the open ended question it can provide invaluable data and insights that explains the data from closed questions better.

The down side is the generic open ended questions tagged onto surveys, this are poor survey design. When designing surveys I was taught to ask what data I wanted and then design the questions. Generic open ended questions can result in a ton of data which is unuseable.

Absolutely correct, the purpose always needs to be clear and remember one's responsibility to the participants and the potential for the research to have good effect.

This is capital NO please

You are correct mixed methods is still having serious challenges ie on the quantification of qual and Quan methods, the level of mixing etc let me send to you one of my articles on mixed methods

Try this http://www.academia.edu/36847174/A_Critical_Analysis_of_the_Mixed_Methods_Approaches_in_Impact_Evaluation_Experiences_from_Triangulation_Design

If you have only used survey method then it is not a mixed method study. Mix method can contain different methods like in media studies we use content analysis and survey only then we can call it mix method

The survey contains both open and closed questions. That makes it a mixed method. Survey design fits to both qual and Quan

Silas,

literally, you are correct; however, the open questions practically possible to include in a survey necessarilly return rather superficial data in contrast to say, that returned by an in-depth interview, which is why a survey containing open questions is still considered a quantitative method.

Therefore, an example of so- called 'mixed methods' would be a survey plus in-depth interview(s), the aim being the 'mixing' of both 'deep' QUAL / QUANT for a more comprehensive understanding of the researched topic than can be had from implementing either a quantitative or qualitative method alone.

A different mixed methods approach would be an explanatory sequential design (Quant --> qual), where a well-designed set of open-ended questions could be used to help understand the results of the larger quantitative portion of the overall project.

Philip

I agree with you but open questions are qualitative. Survey and case study designs are controversial because they sometimes fit on both qualitative and quantitative

Interesting answers, not sure that survey and case study designs are controversial - all research approaches require careful design and alignment of research questions and the purpose of the research.

The discussion and views above are quite interesting. Theoretically, the 'combination of closed and open-ended questions' is enough to be considered a mixed-method study. Mixing of quantitative and qualitative questions in proportionate ratio may well be said of a mixed method approach. This implies that the result obtained thus may be a balancing act, and with little mistake it may turn to dangerous inference.

For drawing a good conclusion one may further quantify or qualify the results such obtained and take them in the field of test. In other words, the results obtained from close ended questions or through quantitative technique, is capable of being converted into qualification and further tested for the same result we had arrived earlier. Similarly, the results obtained from open ended questions or through qualitative technique, is also capable of being quantified and further tested . By doing thus, I think we are following mixed method approach.

One more element comes into our aid while we analyse the results obtained from mixed method approach, viz. we may further test it by observation method, what is known as participant observation in ethnographic study. The conclusions drawn thus, is quite trustworthy. Now, coming back to your question, like online survey, these elements of the research tools of the mixed method approach may be missing or does not comprehend the scope. I am not sure, the results inferred thus, through online survey could be said of the results of the mixed method approach

For further clarity you may refer to book on Specialized Ethnographic Methods (Ethnographer's Toolkit book 4) Edited by Jean J. Schesul Margaret D. LeCompte (Alta Mira Press) Rowman &Littlefield Publishers, INC.

I tend to agree with Dr. Morgan that mixed methods studies get more at data usage than the methods themselves. Since we are discussing a survey, let's suppose that the survey contains Likert-scale questions that are designed to elicit purely descriptive information (e.g., percentage of individuals who believe x, y, or z). These questions would produce a "what," but not a "how and why." Open-ended questions, however, would be designed to provide the "how" or "why." To derive the "how and why," one would analyze the data for themes/categories, which is very different than simply reporting descriptive statistics. In another case, one might choose to utilize the Likert-scale questions to establish relationships or lack thereof between independent and dependent variables within a construct, but offer open-ended questions to provide explanation as to why the relationships exist, assuming they do. The short answer is "yes potentially" in regard to a single survey functioning as a mixed methods investigation.

As my post above, necessarily, a survey is a survey---regarded as a quantitative method----even if it includes open questions, as such questions within the constraints of a survey, return superficial answers compared to those returned by for instance, stand-alone open interviews.

However, combine these latter with your survey, and you have an overt 'mixed methods' project.

I'll repeat myself by saying it is not the literal method that used that makes something a mixed methods study. Instead, it is the type of data that is produced and how that data is used. Several mixed methods designs have a "small qual" component, and carefully planned open-ended questions on a survey may well be sufficient for this purpose.

Hello The questions raised as mixed will present a distortion in the results if they are descriptive variables, on the other hand when the answers are true or false they are qualitative variables.

Open Ended questions are considered qualitat data.its just that you haven’t asked them in a pi but have asked them in a questionnaire.This data abyways is qualitative so it will be considered a mixed study

Are open-ended questions on surveys enough for a study to be considered mixed-method?

Not enough. Reason being qualitative open-ended questions in survey are pre-set which can't achieve further deeper dives to collect the deepest data / insights until saturation - which a typical interview can possibly achieve.

Quantitative survey generally is prescriptive / deductive i.e. survey questions are pre-set based on construct / variable operationalization. Whereas qualitative open-ended questions are generally inductive that takes time to discover the quality depth of a phenomenon. Hence, you can't mix as what you stated in question as a mixed method research.

I carried out a large survey some years ago for a government public consultation. We analysed the likert responses and completed a submission on the set of questions Posed by the consultatio. I included a number of open questions around the various sub sections, but didn't have scope to include their analysis. A year later I thematically analysed one of the openended questions separately to the rest of the survey. That analysis was a small qualitative study in it's own right. There was nothing quantitative about it. Therefore if I can create a qualitative study independently from the quant part of the survey then logically a combination of the two would have to be considered MM, even if it does not meet the highest standard of a formal MM design.

I disagree with Han Ping Fung's statement that open-ended questions are "prescriptive / deductive" simply because they were asked in the context of a questionnaire. On the one hand, participants are free to answer as they choose, and on the other hand researchers are free to make inferences from those responses.

I do agree that open-ended questions do not allow for follow-up probing and that they may not provide depth, but that does not eliminate the possibility of the data being used for something other than a full-scale, self-sufficient qualitative study. In particular, that kind of data can be quite useful for a mixed methods Explanatory Sequential Design (QUANT --> qual), or a quantitatively oriented complementary design (QUANT + qual).

Furthermore the depth of data available depends on the design of the question and the context of the survey. Interviews may reach 'saturation' after 15-30 interviews - very much limited by the resources available to facilitate that. You really don't know if the next potential participant could have added a novel point. In contrast, survey may reach hundreds of participants and although the responses may only range from a sentence to a paragraph, a well formulated question should be able to elicit some depth even if to a narrowly defined concept. You may happen to be interested in understanding some particular fine detail.

The limitations of a survey open question do not change it from being fundamentally qual in nature and still providing valid contribution.

A survey is just a tool for eliciting data, the structure and analysis of the data is what defines a qual or quant study. How well the tool is designed, the collected data analysed and the results interpreted within the bounds of the study limitations contribute to the quality of the study, but don't change it from being qual or quant.

I agree with you Susan, but still stand by my last post, above.

In my survey (1599 respondents) I had 3569 responses (104, 249 words) to 10 open-ended questions submitted by between 10 and 45% of the respondents. Many of these were responses to comments following on from specific questions with the option 'are there any other comments you would like to make about x.' Whilst it has been pointed out to me that this is a rookie mistake in terms of asking too many open-ended questions, it is clear that the respondents felt very strongly about the topic and I have probably got more data from a greater number of respondents than I would have acquired from semi-structured interviews. I have thematically analysed the responses using NVIVO 11. I don't know how this could not be construed as mixed methods research. It is the richness and reality of the lived experience of practitioners ( and of course patients) that really makes the case for change. Having attended many conferences where research is presented alongside the lived experience of patients, talking afterwards to conference delegates, I think the presentations that make the greatest impact are the descriptions of 'lived experience.' In fact it does make me feel a little dejected as you make such effort to conduct methodologically rigorous research and present it as such but it just isn't as powerful as personal experience.

Catherine Lowenhoff I think your rookie status is reflected not so much in the number of question that you asked but in the weakness of your request: "'are there any other comments you would like to make about x." Instead, a better request would be something like: "please describe what you were thinking about when you answered this set of questions." In essence, you want each open-end questions to feel like it requires a response, just like every other item in the survey. Instead, your version made it too easy to skip over the open-ended questions, leading to what appears to be a relatively large non-response rate on these items (55% and higher).

FYI Among the options that others recommend are: "please say why you answered these questions the way you did" or "please give your reasons for answering these questions this way." I personally prefer to avoid asking "why" or assuming that people have easily expressed "reasons." Hence, I ask them to "describe" their thinking.

Personally - I think that would get a complete different set of responses and I don’t think your way of asking questions is any more valid than mine. I suppose it depends on the purpose of the research and the previous questions asked. If somebody asked me which of the following techniques do you use in the support that you provide ( based on literature to suggest that these are the techniques that are most commonly used) and maybe I did or didn’t use some of those techniques but maybe none of those listed techniques included the techniques that I did use, it would make me feel that the researcher wanted to hear what I thought if I was then asked ... are there any other techniques, strategies that you use that are not mentioned... so maybe my generic statement about any other comments did not do justice to the specific nature of the supplementary open-ended questions used in my survey. In my view...Please say why you answered the question the way you did would not necessarily get to the same outcome. I’m not sure that respondents like that sort of question. Describing your thinking is much more esoterica than just being able to say well I use some of those techniques but I also use others and here they are. And if I am trying to find out what it is that practitioners actually do then it is more useful to find out the techniques and strategies that they are using that may not be included in research studies or evidence/based guidelines.

David Morgan - thank you for taking the time and trouble to respond - Your comments are always thought-provoking and helpful

Hi Alicia , (as you can probably see from reading on) I am a complete novice to the world of research and have literally just starting reading around for my first piece of research myself so absolute respect to the above answers , and apologies for the lack of knowledge that im starting out with here. However this discussion did really get my interest as I have been searching for a similar clarification . I have also compiled a questionnaire with closed and open ended questions . The open questions asking the person to elaborate why they had filled in the quantative likert scoring questions as they had to try to provide some depth and increased understanding. I thought initially it would come under a mixed methods study , but then after reading around it became increasingly confused (and worried !) about calling it mixed methods as I purely wanted the open questions to expand on the closed questions and I was thinking of using quotes to illustrate the reasons why people had chosed the answers they had . I couldnt find anything that accurately described what I wanted to do .After searching I found an excerpt form a book on mixed methods . Actually it was Cresswell book . I believe it was from book by John Creswell, designing and conducting mixed methods research ( 2006) and it was one of four models described under Triangulation approach (2006 ) called the Validating quantitative data approach . It stated thus; 'Researchers use the Validating quantitative data model when they want to validate and expand on the quantitative findings from a survey by including a few open ended questions . In this model , the researcher collects both types of data with one survey instrument . Because the qualitative items are an add on to the quantitative survey the items generally do not result in a rigorous qualitative data set . However they provided the researcher with interesting quotes that can be used to embellish the quantitative data. For example Webb, Sweet and Pretty (2002)' It was an excerpt from a book about mixed methods Research so im assuming it was seen as a valid Mixed Methods Method! Ive just ordered the latest edition so I can see if it is still in there ! However it did provide me with some solace at the time, although now ive seen the above debate im querying it a little . I am aware that the debate and knowledge around Mixed Methods has evolved since 2006 and Im not sure if this helps you at all or at least could be commented on by the other people here . Again apologies for the lack of depth to my own knowledge. Im only just starting on this path myself .

Kerry Quorn I discuss options for using open-ended items in my book, Integrating Qualitative and Quantitative Methods. I do so in the chapter on what are now known as sequential explanatory designs, which are based on the pattern QUANT --> qual. In this design, the purpose of the follow-up qualitative study is to help understand the results from the core quantitative study.

Thankyou David that's very helpful . I'll have a read .

In my point of view that is not depend on the type of questions which you have used in the questionnaire. But in the analyzing techniques in your study. Even though those were open-ended, you can use cording methods to make them to numerical analysis.

I would recommend going through Cresswell's book on mixed methods. Using the phrase as this stage in response to the type of answers you are getting it may be risky to claim it technically a mixed methods study by design.

Better use quantitative study based on various types of instruments

In my understanding you have use only 1 method here. And I understand that your open ended questions is to answer your "why" questions. I have 3 perspectives regarding this matter:

1. You could be use your findings as mixed-methods technique analysis, we have several types of mixed methods: exploratory, explanatory, embedded... Your findings I think it's explanatory ones, where survey questions (as quanti part) are bigger than open ended questions (as quali/explanation part) please take a look to Creswell theory's. For this 1st perspectives you can analysis the data separately first quan and qual, then combine it together.

2. My second perspective is regarding your statement "triangulation" technique.. I think in this case you can't do triangulation, because you don't do observation/ or any qualitative data collection techniques that can compare and if it's triangulation sources, I also don't know if your findings is come for same/homogeneous participants or not.. in your case I think your open ended questions can be done by content analysis only.. and or develop to thematic analysis or grounded theory.. triangulation is used for validity in Qualitative research

3. My third perspective is.. usually I set at the beginning what I want to achieve after I conduct data collection, so it can make me clearly understand my direction. But I do understand why you make this open ended questions to answer your how and why after you get your survey data.. well good luck and feel happy for you as well that you have rich data.. next time I suggest you to consider to conduct Qualitative research data collection if you intend to use mixed method at the beginning.

I hope this answers can benefit for you. :)

I agree with Lafi Munira that this could be a sequential explanatory design in mixed methods (QUANT --> qual). In general, the goal in mixed methods research is to integrate the results from a qualitative and a quantitative strand, and I think that is definitely possible in this case.