Hello everyone,
I am currently conducting data analysis for a project using an existing large survey dataset. I am particularly interested in certain variables that are measured by 3–4 items in the dataset. Before proceeding with the analysis, I performed basic statistical tests, including a reliability test (Cronbach’s α), average variance extracted (AVE), and confirmatory factor analysis (CFA). However, the results were unsatisfactory—specifically, Cronbach’s α is below 0.5, and AVE is below 0.3.
To address potential issues, I applied the listwise deletion approach to handle missing data and re-ran the analysis, but the results remained problematic. Upon reviewing previous studies that used this dataset, I noticed that most did not report reliability measures such as Cronbach’s α, AVE, or CFA. Instead, they selected specific items to operationalize their constructs of interest.
Given this challenge, I would greatly appreciate any suggestions on how to handle the issue of low reliability, particularly when working with secondary datasets.
Thank you in advance for your insights!
First, I would try to find out why Cronbach's alpha is low. Usually, one major reason is multidimensionality of the items (i.e., the items measure more than one factor/dimension). You already conducted a CFA. The CFA should tell you whether the items measure one or several factors (i.e., whether a single factor model fits well and whether the items have decent standardized loadings). Cronbach's alpha is a measure for unidimensional items that are tau-equivalent (have equal unstandardized loadings). Alpha is not appropriate for multidimensional measures.
Also, listwise deletion is not recommended due to its restrictive assumptions (missing completely at random data) and the potentially massive loss of data when retaining only complete cases. A better way to handle missing data is by full information maximum likelihood or multiple imputation.
Christian Geiser Thank you for the suggestions! Someone told me using cross-national comparative data with many respondents (n > 30, 000) would decrease the reliability, is that true? I will try the full information maximum likelihood and multiple imputation you suggested. Thank you.
I would think that the opposite would be expected: A large, representative sample should result in a large amount of variability/variance (inter-individual differences). Since the reliability coefficient is a variance-dependent statistic (it is defined as the ratio of true score variance to total observed variance), having a sample with greater between-person variability would be more likely to result in a larger reliability coefficient, other things being equal.
See also my Youtube video in which I discuss 6 common reasons for a low Cronbach's alpha reliability coefficient:
https://youtu.be/7xk0mha-8zA?si=746SrEsjGivXpWWZ
- Badges
- Science topic
- Similar topics
- Social Science
- Media
- Journalism