You can look at alpha-cronbach after you have standardized the data in your hand. Your credibility has three major parameters. Alfa-cronbach coefficient method, method of correlation between question and whole (Item-total correlation) and reliability coefficient if question is deleted. You can do these via SPSS.

Thank you very much Mr. Ongel for your answer. I'll try with SPSS to see if I have the answer. What I am confused here is that I thought the Cronbach Alpha is applied for Likert questions only? What could I do if the questionnaire has both MCQs and Likert and even open-ended questions?

Please correct me if I'm wrong. 

Dear Phuong-Le,

The purpose of a reliability analysis is to reduce the dimensions of your data set. It is useful when you have a group of questions which are about the same thing. So you first need to decide whether you have a single group or multiple groups.

We recommend using a Principal Component Analysis before a Cronbach's alpha analysis. You are correct in stating that this technique is appropriate with Likert questions but not with questions yielding nominal or textual data.

You can not use this method for open-ended questions. (At least, I don't know such a method.) I suggest you to standardize your open-ended questions...

Hi Phuong-Le,

When we have multiple-choice questions, we will have binary answers (right or wrong or 0 and 1). The more appropiet test, for these cases, is the Kuder-Richardson KR-20.

Already in open-ended ones case you need create a scale by use "judges" for give hierarch values. for example. 0 = totaly wrong, 1 = have 20% correct, 2 = have 40% and following until 100%. The Judges have a important work. They need create a scale. After this, you can use the Cronbach alpha test.

Hope that help

Best Regards

Dirceu

Dear Phuong-Le,

you can test reliability of questionnaire by reliability test. For this purpose, take  a small sample and administer questionnaire among them tabulate the data . Again after some time. the same questionnaire may be administered among  the same set of respondents. Data obtained from both the count may be compared . If the response of  both sets are almost same say 95%  responses are same  then we can say reliability is 95% and accept the same.But all it depends on your acceptance level

For Reliability, different statistics are used, however, these methods are applicable or not for multiple choice questions, I am not sure. For multiple choice questions, face validity or content validity is more appropriate.

Cronbach's  (Alpha)

Average Variance Extracted

Construct Reliability

Indicator Reliability

KR20 and Coefficient alpha are the same thing, but alpha is generalized to work with polytomous item responses and not just dichotomous.  KR20 uses the term pq for binomial variance, which of course assumes binary data.

If you have a larger sample size and really want to evaluate the quality of the assessment, you should use item response theory.  The shortcomings of coefficient alpha, and classical analysis in general, are well documented.

My english is very poor. This is an automatic translation. I include the testo in Spanish in the end.

Open-ended questions should be codified (standardized) before analyzing reliability. One possibility that you can add to what you have already answered is that you do a first exploratory part with open questions with a reduced sample and then build that other closed for the definitive instrument.

I hope I have brought something useful.

Mi inglés es muy pobre. Esta es una traducción automática. Incluyo al final el testo en castellano.

Las preguntas abiertas deberían ser codificadas (standardized) antes de analizar al confiabilidad. Una posibilidad que puede agregar a lo que ya contestaron anteriormente es que Ud. haga una primera parte exploratoria con preguntas abiertas con una muestra reducidas y luego sobre esa base construya otras cerradas para el instrumento definitivo.

Espero haberle aportado algo útil.

There are many ways to measure reliability, including a very simple do we agree or not.  However, reliability is an important measure to a questionnaire's worth.  In other words, can you and a colleague administer the same (or part of) a questionnaire, or test, and get the same result.  I always have my students look at the published reliability of an assessment, for example, before administering.  What is an acceptable reliability measure is of importance, but difficult to quantify, in my opinion.

Thank you RG Friends

Thank you very much for your help!

It has been a pleasure to participate! The discussion is very interesting!

I think if u can categorize the open-ended question then it will be easy to to use cronbach's alpha to check the reliability for your questionnaire

For checking the reliability of questionnaire and response, you can do a pre-test/reliability test on a small sample say about 5-10% of the sample

Reliability test is feasible for the questionnaire which is mix of questions like: closed ended, likert scale, multiple choice, demographic. If yest then how to do can anyone help to perform it step wise so that i can understand. I am new in this field so please extend your great help.

Dependerá del cuestionario como evaluar su confiabilidad. Si mide dimensiones diferentes de un constructo, creo que lo mas recomendable es evaluar dimensión por dimensión. El SPSS tiene distintas formas de hacerlo y existen innumerables documentos que explican como hacerlo paso a paso, por ejemplo, https://www.uv.es/innomide/spss/SPSS/SPSS_0801B.pdf . Este es solo uno que busqué como ejemplo, no se si es el mejor.

You may do test and retest method and then analyse and compare the results. You may also use meta analysis to judge your results.

Dear Phuong-Le Ngo:

To assess reliability, they usually propose "the classic procedures evaluated by psychometrists: test-test, split halves, alternative or parallel forms and internal consistency measures." Cronbach's Alpha (AC) belongs to the last class and in my opinion it is recommended to evaluate the reliability of a test with homogeneous items, that is, when all items are linked to the same factor, since it is based, fundamentally, on in correlating all items with each other.

I think that in general it is convenient that all the items have the same scale, but this should not be the case, so you could also include multiple option items along with similar ones, but you should know if it is possible in your case. I think that at least they should be scales of the same type: ordinal, innervalar or reason (this contains subtleties that must be seen in each specific situation).

If the items don't respond to the same factor, I think you can still assess reliability with AC by taking each factor at a time and doing a separate assessment of them.

For the part that contains open-ended questions, an option might be to first categorize the responses and then use one of the specific forms above: test-retest, split halves, alternative or parallel forms, and internal consistency measures.

The ideal, I think, is to make an exploratory application with a smaller sample (at least 20 cases) and then, especially with the open part, build closed articles. Another option is to do a qualitative analysis of this part, which would use a mixed design. In a 2019 publication I make an analysis and a model to analyze validity in qualitative, quantitative and mixed studies (Echevarría, H. (2019). Research methods and inferences in Social Sciences. A proposal for analysis of their validity. Río Cuarto : UniRío Editora Http://www.unirioeditora.com.ar/?s=ECHEVARR%C3%8CA; http://www.unirioeditora.com.ar/wp-content/uploads/2019/05/M% C3 % A9all-research-% C3% B3n-e-inference-in-social-sciences-UniR% C3% ADo-editor.pdf).

I hope this is useful!

Cordially

Dr. Hugo Darío Echevarría

Estimado Phuong-Le Ngo: Para evaluar la confiabilidad, suelen plantearse “los clásicos procedimientos desarrollados por los psicometristas: test-retest, mitades partidas, formas alternativas o paralelas y medidas de consistencia interna”. El Alfa de Cronbach (AC) pertenece a la última clase y en mi opinión es recomendable para evaluar la confiabilidad de una prueba con ítems homogéneos, es decir, cuando todos los ítems se vinculan a un mismo factor, ya que se basa, fundamentalmente, en correlacionar todos los ítems entre si.

Creo que en general es conveniente que todos los ítems tengan la misma escala, pero esto no necesariamente debe ser así, por lo que podría incluir también ítems de opciones múltiples junto a otros likert, pero habría que ver si es posible en su caso. Creo que al menos debieran ser escalas del mismo tipo: ordinales, inervalares o de razón (esto encierra sutilezas que hay que ver en cada situación concreta).

Si los ítems no responden al mismo factor, creo que de todos modos puede evaluar la confiailidad con el AC tomando cada factor por vez y hacer una estimación de ellos por separado.

Para la parte que contiene preguntas abiertas una opción podría ser primero categorizar las respuestas y luego utilizar alguna de las formas mencionadas más arriba: test-retest, mitades partidas, formas alternativas o paralelas y medidas de consistencia interna.

Lo ideal creo, es hacer una aplicación exploratoria con una muestra menor (al menos 20 casos) y luego, sobre todo con la parte abierta, construir ítems cerrados. Otra opción es hacer una análisis cualitativo de esta parte, con lo que utilizaría un diseño mixto. En una publicación de 2019 hago un análisis y propongo un modelo para analizar la validez en los estudios cualitativos, cuantitativos y mixtos (Echevarría, H. (2019). Métodos de investigación e inferencias en Ciencias Sociales. Una propuesta para analizar su validez. Río Cuarto: UniRío Editora. http://www.unirioeditora.com.ar/?s=ECHEVARR%C3%8CA; http://www.unirioeditora.com.ar/wp-content/uploads/2019/05/M%C3%A9todos-de-investigaci%C3%B3n-e-inferencia-en-ciencias-sociales-UniR%C3%ADo-editora.pdf ).

Espero que esto se resulte de utilidad!

Cordialmente,

Dr. Hugo Darío Echevarría

https://www.ncbi.nlm.nih.gov/pubmed/30433882

If you are still looking for answers please look at the following attachment compiled by Dr. K. A. Korb. It has lots of information and practical guide of employing Split-Half Reliability

Really helpful.!

http://korbedpsych.com/LinkedFiles/CalculatingReliability.pdf

Not yet answered

Dear all,

I wish to have your precious help!

Data was collected from 340 respondents by using a structured questionnaire that has only 4 questions.

My question is:

What are the appropriate tests that measure reliability that can be done in such a case (4 questions)?

or

How can I defend that "not performing a reliability test in my manuscript text"?

N.B. I used a mixed-methods research methodology in three sequential phases (exploratory, descriptive, and explanatory). The mixed-methods research methodology consisted of two parts. 1) A quantitative cross-sectional study using a structured questionnaire, (my question about this part, a structured questionnaire has 4 questions). This was preceded and followed by 2) Two mutually-exclusive qualitative studies (exploratory and explanatory) using semi-structured questionnaires.

Thank you in advance

Yours,

Mohamed Bahlol

For open-ended questions this technique is not applicable

We cannot asses reliability using Cronbach's alpha. Cronbach's alpha provides internal consistency. to chek reliability administer the question twice and compute an ICC or do splithaft co-efficient. but the open end questions are not applicable for both

Dear,

There are many ways (tests, techniques) to assess the reliability of a questionnaire (Likert scale) (most used):

Cronbach's alpha - see the very didactic tutorial:

https://www.youtube.com/watch?v=AE5-IacZ8eA

and composite reliability - https://www.youtube.com/watch?v=8tyjdfpiAJQ

The alpha test is very sensitive to the sample size and the number of variables (items) in the construct. CC is more stable.

in both cases they must be calculated in each construct. Never for full scale (all variables).

Greetings and good luck

Dirceu

Si alguien lee Español, esta ponencia es una introducción al enfoque de la consistencia interna: Echevarría, H. La confiabilidad con el enfoque de la consistencia interna. Trabajo presentado en la XXV Jornadas Interdisciplinarias de la Fundación ICALA (Intercambio cultural alemán-latinoamericano). Conflictos sociales y convivencia democrática. Río Cuarto, 29 y 30 de octubre de 2020. Río Cuarto.

Gracias Echevarría

Saludos

Dirceu