Are you interested in how different items are associated, and/or differences in central tendency for items that are comparable (i.e., with the same response alternatives and where it makes sense to compare)? You say this is hypothesis generating. Can you expand on this phrase for your project?

Are any of your Likert-scored items part of scales, so that the individual items could be combined?

Hello George,

As Daniel Wright 's and David L Morgan 's responses might suggest, answers are simpler when the goal is known, rather than unknown! There are many possible ways to handle the data, but the choice of which is "best" will depend on the specific question to be answered.

1. One option would be to try to use item response theory to calibrate the full item set, such that you'd be able to make statements about which items were "more" or which were "less" challenging for respondents to endorse (if opinion/sentiment/attitude type stimulus) or to identify preferred answer (if knowledge type stimulus). It would likely make sense to separate the classes of items/questions/stimuli based on obvious, salient characteristics first. As well, IRT presumes unidimensional scales (though multidimensional IRT models do exist), so checking for dimensionality up front, via factor analysis, might make sense.

2. You can conduct simple, pairwise correlations of item responses, via the phi coefficient or Cramer's V. You'll have k*(k-1)/2 such correlations (with k items) to evaluate. That will address whether responses are related. A non-zero relationship may or may not be indicative of true dependencies among item pairs or sets.

3. McNemar's test could tell you whether, when people responded differently to each of two items, the difference was in a non-random fashion. Again, though, you'd have k*(k -1)/2 such tests to evaluate.

4. You can compare rate of endorsements (for dichotomous or polytomous) items via a binomial test, or the so-called, large-sample z approximation thereto.

Good luck with your work.

Dear Daniel, David and David,

thank you very much for your responses!

To clarify, I will briefly explain the study design: Our goal is to compare patient knowledge with treatment goal attainment in cardiovascular disease (CVD) and diabetes mellitus (DM). We performed the following assessment in ~200 patients, who suffer from both CVD and DM conditions:

1. Questions on their knowledge on CVD and DM in different categories (4-point likert scales, e.g. causes/courses of disease, treatment, complications: very well informed, well informed, not well informed, not informed at all)

2. Questions on their information needs for CVD and DM (4-point likert scales: very much, much, some, none)

3. Questions on treatment goals for CVD and DM (LDL-Cholesterol and HbA1c, nominal: right answer, wrong answer, no answer given)

4. Objective assessment (blood tests) of treatment goal attainment for both (LDL-C and HbA1c, binary: attained, not attained).

Additional questionnaire items were used to correct for education, participation preference etc.

Daniel Wright: I am interested to test whether

a. there are significant within-person differences between both conditions in 1) knowledge, including treatment goals, 2) subjective information needs, 3) objective treatment goal attainment

b. there is association of these features with treatment goal attainment and possible differences between both conditions

David L Morgan: From my understanding, none of the items are part of scales

David Morse Thank you very much for your time and efforts! However I could not fully follow you there, I am sorry. Could you specify for the described study design?

Thank you all in advance,

George

@Georg Wolff Does everyone in the sample of 200 have both CVD and DM?

Depends a bit how many items you have, but it might be worth first exploring if the data can be reduced to fewer diimensions (e.g., with Princals, a principle component model for ordinal data, this is in many packages, for example the Gifi package in R, or the PRINCALS procedure in SPSS) or if the variability is due to variation in a smaller number of latent variables (e.g., the partial credit model or rating model, in for example the mirt package of R). You might do this for sections of your survey. Then see if these components/factors make theoretical sense, and if they do the analysis to compare them is simpler than dealing with the large data set.

Hi George,

 If I understood your problem correctly, I will follow this approach:

Data:

Same 4-categories question/item (1=very well informed, 2=well informed, 3=not well informed, and 4=not informed at all) was applied two times to each respondent in your sample (N ~ 200). The first time the question/item was about CVD, the second time was about DM.

Problem:

To compare the responses to this item across both illnesses (CVD vs DM). The two responses provided by each respondent are not independent.

Solution:

I would analyze the data using a multilevel (two-level) ordinal regression model. To do this, you need to reshape your data from wide to long format. Therefore, your new dataset will contain N ~ 400 rows.

Basically, the new dataset (long format) will include two rows per respondent. In terms of the variables required for your analyses, the dataset would include: a) one ID variable that identifies each respondent, b) one “y” variable that will contain the responses to both illnesses,c) a dummy variable (let’s call “item”) that identify the illness that each row is showing (0 = response to CVD, 1 = response to DM), and d) any other covariate you want to include in your model (e.g., gender, age).

The regression coefficient for the “item” variable will answer your research question.

 I would use the same approach for the other type of items, but selecting the correct regression model. For example, for 2-point binary items, you should use a two-level logistic regression model.

Best,

Roberto

It appears that your team does not have the statistical expertise to follow some of the rather advanced advice you have already received. Instead, I would suggest that you do what most researchers do most of the time, which is to convert your groups of items scales that you can use as variables in your analyses.

In particular, the definition of a scale is a set of items that all measure the same underlying concept. The goal in combining those items into a single scale is to create a more reliable measure of that concept. Because there have been so many questions here about how to convert Likert-scored items into scales, I have compiled a set of resources on this topic which you can find at the beginning of the following RG question:

https://www.researchgate.net/post/How_can_one_help_people_create_simple_scales_from_Likert-scored_items

Thank you all for your help.

Viswa Chandu Yeah, all the patients suffer from both conditions, CVD and DM.

Roberto Melipillan Thats a nice solution, I can work with that! Thanks!

David L Morgan Thanks! Its true that we are new to this statistical field, thank you for the support. Thanks for the clarification on scales and for the link to your previous post on the topic!

Daniel Wright I understand, thanks for the suggestion!

Best, George

I think that you need to take a step back from all the technical recommendations you are getting and state the hypotheses that you want to test. In particular, you need to match each of your theoretical concepts to the actual variables (or scales) that measure that concept, and then specify which are independent variables (causal factors) and dependent variables (outcomes).

Once you have a concrete set of hypotheses that you want to test, we can give you better advice about to do so.