In order to be able to use Friedman test, you need to have an observation for each respondent on each time.

If you aren't able to track the respondents from time-point to time-point, maybe something like the Kruskal-Wallis test ?

Hello Christian Erven. If Thom Baguley sees this thread, he will likely caution you to beware the Friedman test! ;-)

• https://seriousstats.wordpress.com/2012/02/14/friedman/

How many states are there? How many respondents within each state? Do you have the same respondents at each time point, or different samples each time? Thanks for clarifying.

Dear Bruce Weaver

Do im doing reasearch on 6 member states (Bulgaria, Hungary, Netherlands, Denmark, Spain and Italy). N= approx. 1.000. the Eurobarometer surveys are anaonimous but ive argued that given the relatively small N in respect to an entire population, its can be assumed that the samples are independent as they are selected at random for each new survey. So I want to investigate 2 dependent ordinal variable (measured at a likert scale) which are the “public support for national covid-19 policy“ and the public support for EU covid-19 policy. I want to measure them seperately, against 3 time points (3 different surveys with different respondents) to see if the support has significantly changed durinf the covid pandemic. I am guessinf that my independent variable (the time points) is categorical? Thanks for the help! The assumption for Mann Whitney test (and i think also the kruskall wallis test) is that the distributions of the dependent variable needs to be identical on the independent variable? But I dont understand how that works. So can I use a kruskall wallis test or is a different test better suited? Mann whitney only works for 2 groups (time points) while I have 3

There's no reason to beware of the Friedman test. It's not an imposter. It's exactly what it purports to be. One thing I like about the Friedman test is that it can handle truly ordinal data, unlike some other test that might be used in similar situations.

Christian Erven , because you aren't able to track the same respondent across time periods, you'll be treating the time points as independent of one another. So, you won't be looking at tests or methods that deal with dependent observations or repeated measures observations. (So, Friedman's test isn't applicable.)

Can you say more about the what the dependent variable is like from the survey. Is it a single Likert-type question? Or do you have a composite variable composed of several Likert-type questions?

I think what would be best --- though potentially more complicated --- would be to use a single model for each dependent variable, that includes as independent variables both the member state (if you have data separated this way) and the time period. This would give you the ability to look at if the changes are different among different member states. What model might be appropriate depends on the nature of the dependent variable.

Sal, as I understand Christian's data, the measurements over time can/must be treated as independent. But there are two measurements of the outcome variable for each respondent, own country vs EU, and those observations are not independent of each other: The two measurements of the DV (EU vs Home Country) are clustered within respondents. As I see it, Christian might consider a mixed effects (or multilevel) ordinal logit model with the following explanatory variables: Country, Wave, EUvHC (i.e., EU vs home country). It sounds like he wants some interaction terms too--e.g., the Wave x EUvHC interaction to ask if the difference between EU and Home Country ratings is changing over time. To ask if the nature of that two-way interaction varies by Country, add the Country x Wave x EUvHC interaction. Etc.

Christian Erven , if you say what software you use, someone may be able to give more detailed advice about how to estimate such a model.

The question with the Kruskal-Wallis test and identical distributions is a somewhat complex one.

I think I understand the issue, but no warranty is implied. :)

When sources say that the distributions of the underlying populations of the groups have to have identical distributions, I think that's taking the easy way out. If the populations have identical distributions, then the test is a test of location. It's a test of medians; it's a test of means; it's a test of every other measure of location. It's obvious if you have two identical distributions where one is shifted to the right, that the location is shifted to the right. But that's not very helpful in most cases.

In general, the K-W test assesses if the values of observations in one group tend to be larger than in another group. When we use the test, this is usually what we are interested in.

Conover, Practical Nonparametric Statistics, 3rd, lists the test assumption this way:

Either the k population distributions are identical, or else some of the populations tend to yield larger values than other populations do.

This is a little complex to parse, but basically what it's saying is that if you want the test to truly have an alpha value of 0.05, you have to meet this assumption. If you are sampling populations with different distributions, but where one doesn't have larger values than the other, you might get a p-value < 0.05 for the K-W test on the samples more than 5% of the time.

So, one way to make the test really statistically valid is to start with the assumption that the population distributions are identical. But who wants to do that?

Practically speaking, I don't think this complication has any real impact on our use of the test. If we conclude from a significant result that either some of the populations tend to yield larger values than other populations or the distributions aren't identical in some way, that's okay. Or we can just accept the inflated alpha value if the distributions aren't identical.

Bruce Weaver , yup, I see that now. Thanks. Yeah, that would be a better way to approach it rather than treat the measures separately.

This is very different from your original question with a single DV. You have a square matrix for each country and each time point. You likely have several research questions both about particularly countries liking their own policies more or less, but also about the transitions across time (e.g., do the X learn to like EU more after national spikes). I think the current question prompt may not get the responses that will be most helpful, and certainly as a guide for other people searching for help for their problems it will not be of much use. Therefore it might be best to delete this question, probably do some searching for square matrices/log-linear models/longitudinal, perhaps consult someone, and then start a new question.

thanks everyone for your very valuable insights. There seems to be a slight misunderstanding however. So ill explain it once more but now maybe more clear (sorry for the inconvenience).

I have three Eurobarometer surveys; conducted in April, June and October. It can be assumed that the respondents are completely different from each survey hence I have to treat them as independent samples.

All three surveys contain among other questions, the following two separate questions:

How satisfied or not are you with the measures your government has taken so far against the coronavirus pandemic?

How satisfied or not are you with the measures the EU has taken so far against the coronavirus pandemic?

The questions are measured on a 5 point likert scale. I have the complete datasets in SPSS so I can filter on country.

What I want to know if the change for a specific question and a specific country is significant.

e.g. public support in Spain for the national covid-19 policy changed over the three time points, now is this change significant or not?

Normally with just two time points I would run Mann-Whitney U tests byut this can not be done in this case given the multiple time points

Christian Erven , if you want to analyses each state and question combination separately, say, each with a Kruskal-Wallis test, you can do so. (12 tests total ? ). You may want to use Dunn(1964) test as a post-hoc test to compare the three time periods.

What some of the responses here are suggesting is that you are better off constructing a single model that includes both questions and all states. You could still separate out the effects for individual states and questions. If you are not experienced in specifying this kind of model, it may not be obvious how to do so.

I'd also recommend you not place too much emphasis on the p-values from analyses, but also to consider and report some kind of effect size statistic. You could use a formal effect size statistic (like Vargha and Delaney's A between individual times within states). Or simply report what you think makes sense, like proportion of responses that responded 4 or 5 for each time. The idea is that, if you get a statistically significant result, but it's only a difference of 1% of the respondents, that may not be practically important.

Sal Mangiafico thank you again for your help, it is highly appreciated. Initially I did not plan to use statistics at all and go for a more qualitative approach; plotting the variables in a graph and describe what patterns I see. My supervisor however wanted me to use statistical analysis to validate the differences.

I would love to use some kind of conprehensive model that captures all data at once rather than having to conduct many seperate tests. However, unfortunately my statistical knowledge is extremely limited having only ever followed one introductionary course..

But given that you agree that a kruskal-wallis test can be used in this situation I will use this test.

Also, thank you very much for your advice Regarding the post hoc Dunn test and the effect size tests, I havent heard of them but in case they are (fairly) easy to conduct theough SPSS I will for sure look into them.

thanks alot again everyone for your help! I am still struggling with the statistical part of my masters thesis but for sure less than before!

First, I strongly recommend against doing these separately. If I were a reviewer I would reject without re-submission (an editor should desk reject it).

But, if you plan to do something like a separate Kruskal-Wallis (btw Christian Erven , most people's names are capitalized and it is rude not to when appropriate) three things. 1. You can't compare them easily, so saying this is significant in Summer but not Spring, doesn't tell you if there is a difference between Summer and Spring. Similarly for the EU v national questions. 2. If you take this approach, look up adjusting p values. 3. As said above, try to avoid reaching conclusions from p values.

Overall, if you are planning this for more than a class exercise, I recommend taking further statistics courses and then returning to this data set.

Christian Erven ,

The Dunn test is an appropriate post-hoc the K-W test. I tend to identify it with the year 1964 just because there's a Dunn test described in 1961 that is similar but isn't used with the K-W test.

I'm not sure what post-hoc options SPSS offers for the K-W test. It sounds like there's an option for pairwise Mann-Whitney tests (which isn't a great approach), and something called the Dunn-Bonferroni test, which I assume is the Dunn test with (an option for ? ) p-value correction for multiple tests.

The following post mentions the Dunn test as a post-hoc for the K-W test in SPSS, and be helpful with the post-hoc analysis.

https://www.researchgate.net/post/Should_i_report_significance_values_or_adjusted_significance_values_when_reporting_pairwise_comparison_findings_from_a_Kruskal-Wallis_test

Friedman test is appropriate to measure repeated time variables, so for ordinal or categorical variables it is not feasible.