Reliability testing for different types of data?

28 April 2023 2 2K Report

Hi,

I am looking for some advice regarding reliability testing. I have conducted a very small pilot study (6 participants) and need to test the reliability and validity of my questionnaire.

My questionnaire produces a range of categorical and continuous data, mainly in the form of Likert scales. My intention was to use Cronbach's alpha to test reliability, but it seems I cannot use this for categorical data.

Would it be appropriate to use Cronbach's alpha to assess the continuous data only and a seperate measure to assess the categorical data?

I wasn't sure if it were appropriate to use two seperate tests within the same questionnaire.

Thanks in advance.

Ronán Michael Conroy

Alpha can be used for ordinal scaled items. However, Cronbach's alpha isn't your problem, it's small sample size. You have no statistical power with such a small sample.

Small sample research on questionnaires should be focussed on think-aloud methods to explore the appropriateness, clarity and sensitivity of the questions and the response options.

I'd also caution that alpha is only interpretable if the items make up a unidimensional scale, and with items of various types I'm wondering if that's true in your case.

Stacey Provan

Hi Ronán Michael Conroy thanks for your reply.

This small sample was purely for piloting purposes to ensure there were no obvious flaws prior to final distribution to a much larger sampling frame. It’s for a dissertation project so the purpose of using statistical methods to test reliability etc was purely to demonstrate knowledge of such measures, but if you think alpha is not applicable then I won’t bother.

In regards to the different question types, one set of questions are asking about awareness/unawareness of certain aspects of a particular topic so asks participants to “tick all that apply” where as the others (where I was hoping to use alpha) are looking at the same items, but assessing participants confidence using a Likert scale.

I think it may be useful following data collection to apply these measures when i have a much larger sample… in which case would I still be able to apply two different tests to each question type or would it be pointless at that stage?

thanks again!

What test should i use in comparing the correlation between categorical and continuous variables?

Which is a lucid standard comprehensive book to guide on psychological scale construction and its standardization?

What is the most reliable source to get Impact factor of journals?

Is anyone familiar with the cyclic compressive fatigue testing (BS EN 12697 - 25) ?

Which index or sites are reliable in determining of plant species?

Looking for genomic DNA extraction kits from mouse tissue (centrifuge-based, not spin-column)?

Can we recognize signs of the "volatility clustering" property of a time series in its time domain plot?

How do historians determine the credibility sources in modern history?

How we can measure Kemp % in wool without using of benzene?

Which statistical test to use for comparison of different groups in a categorization?