Hello Fatih Kaya

1) If you want to compare percentages in each question. (Minitab)

2) If you want to compare the Mean in each question. (SPSS)

Abolfazl Ghoodjani Thanks for the response. Could you share a link to the minitab?

Is it possible to use an odds:ratio approach to explore whether differences between the two studies are statistically significantly different from each other. Indeed, I was looking for an odds:ratio method.

I do not think you need to get OR.

The Two-Sample Proportion test calculates the probability value for you and that is enough.

https://support.minitab.com/en-us/minitab/20/help-and-how-to/statistics/basic-statistics/how-to/2-proportions/before-you-start/example/

Thanks. I will get the Minitab and try the method you suggested.

Consider the calculation of the effect sizes for both the replication study and the original study. Compare the effect sizes against each other and against standards such as those outlined by Cohen (1988).

David Sul how can I calculate the effect size? I compared the probability of two samples by using minitab suggested by Abolfazl Ghoodjani

Best,

Fatih Kaya, first, are there five rating levels per item or are there three rating levels per item? The data file you provide displays three: Disagree, Uncertain, and Agree.

It seems like you are trying to compare item proportions across the two samples. This is essentially responding to 30 different research questions. If the 30 items are intended to represent an overall construct, then you should be comparing the sum of the item ratings between the two samples (assuming there are no negatively worded items). By comparing the sums, you are asking about the difference in the overall construct you are measuring between Sample 1 and Sample 2.

Thanks David Sul . Yes, the data has three levels: agree, disagree, and uncertain. Each item in the survey is independent, so there is no construct and no sum score. I need to compare the percentages for each item across two studies. As suggested by Abolfazl Ghoodjani , I used 2 sample proportion test to calculate statistical significance. Then, you suggested calculating effect size. I think that I need to calculate Cohen's h. Is this right? I plan to report statistical significance and Cohen's h. Are these sufficient?

To calculate Cohen's h, I sed the formula below.

=2*(ASIN(SQRT(P1))- ASIN(SQRT(P2)))

Fatih Kaya , thank you for the clarification. My answer will apply to all 30 different analyses. First consider a chi-square test. This will account for the three rating options and will allow you to test whether there are different distributions between the two samples. If there are significantly different distributions, then you can continue with post-hoc analyses. Your statistical software tool should allow you to check a box to run the post-hoc analysis after first running the chi-square test.