Dear Anna,

This cold happens because the post-hoc test is too restrictive. If you have only 3 IV levels (3 groups), try the LSD (least significant difference). If you have 4 or more IV levels (4 groups), try the Tukey HSD (honestly significant difference).

Good luck with your analysis!

Geht man davon aus, dass Deine Analyse wohl-überlegt und für Dein Problem sinnvoll war (ist), dann ist das Ergebnis einfach, dass Du keine der getesteten Hypothesen ablehnen kannst, ohne die geforderte FWER einzuhalten. Mehr ist da nicht zu interpretieren. Punkt.

Aber: Insgesamt scheint mir diese Art der Analyse für dein Experiment ziemlich ungeeignet zu sein. Meinst Du mit t=21 21 verschiedene Zeitpunkte? Mir scheint, die Tests sind nicht entweder nicht sinnvoll oder nicht notwendig.

Wenn es wirklich 21 Zeitpunkte sind: hast Du kein kinetisches (funktionales) Modell von der zeitlichen Änderung (z.B. linear, quadratisch, exponentiell, logistisch, usw)? Wenn Du einen solchen funktionale Zusammenhang hast (oder vermuten kannst), dann ist es sehr viel besser, die Modellparameter (z.B. die Steigung für einen linearen Zusammenhang oder eine Rate für einen exp. Zusammenhang usw) durch eine entsprechende Regressionsanalyse zu schätzen.

Wenn du keinen funktionalen Zusammenhang angeben kannst oder willst: welche Zeitwerte werden denn verglichen? Jeder gegen jeden? Jeder gegen den ersten? Jeder gegen seinen Vorgänger? Oder eine besondere Auswahl? Welchen Grund haben all diese Tests? Sind die getesteten Hypothesen tatsächlich alle voneinander unabhängig (was ich stark bezweifle, aber ich kenne die Details ja nicht)?

Gegebenenfalls gibt es dann auch andere Verfahren, um eine FWER zu kontrollieren. Statt Bonferroni sind eigentlich grundsätzlich step-up oder step-down-Verfahren wie nach Holm, Benjamini und Hochberg der "einfachen" Bonferroni-Korrektur vorzuziehen, weil sie weniger konservativ sind als Bonferroni. Dann könnte sich ggf. auch ein multiple-to-one-Verfahren wie Dunnett eignen, Tukey für all-pairwise war von Jeroen schon angesprochen, auch ein sequentielles Testverfahren könnte sinnvoll sein [d.h. man beginnt bei t2 vs. t1, und macht den nächsten Test t3 vs. t2 nur dann, denn H0 beim vorigen Test nicht abgelehnt wurde] usw.

This case occur frequently, and that not refer to something wrong , F test and post hoc have different hypothesis, you can get significant effect in F test but not significant in post hoc and reverse .

For more comments please visit the link:

https://www.researchgate.net/post/Is_it_possible_to_get_non_significant_results_in_post_hoc_test_when_we_got_the_significant_result_in_ANOVA/2

Good Luck

Dear Jochen,

Thank you for your suggestions.

I use english because my answers and yours could be heplful for others.

Well, we collected 21 times data for six different entities. We have relative frequencies. Our questions is: Is there a growing linear trend in these frequencies over time? Each year against each year of measurement of six entities. We conducted univariate repeated measures ANOVA with polynomial contrasts.

I found with SPSS 24, that there is a significant linear trend in my data. When I tried to compare the measure points with each other to see if there are differences between these measure points, the pairwise comparisons were non significant.

I can't conduct post hoc tests here because there is only time as the important IV with 21 levels. But still I want to reduce the FWER .

What do you suggest? How would a regression look like?

First start plotting the data (response against time). What do you see?

If that looks like a releationship between response and time that can be modelled by some function (in the easiest case this will be a simple linear function), a regression will clearly be the best tool. The model should include a time:group interaction to be able to model differences in the time-corses between groups other than simple offsets.

Consider the simples case of a linear relationship. The model coefficients will be an intercept giving the (expected) response of the reference group at t=0. The main-effect for "group" will give the offsets of the intercepts for the other groups (this is usually not interesting, but required for the model), the main-effect for "time" will be the slope of the response of the reference group, and the interaction will add coefficients giving you the differences of the slopes of all other groups to the slope of the reference group. These are actually the interesting things. A standard statistics software will provide you with accompanying stats like standard errors, confidence intervals, and p-values testing the null-hypotheses that the estimated coefficient is zero. If your 6 "entities" are 6 groups, this will give you directly the 5 comparisons of groups #2-#6 against group #1 (the reference group). If you want to compare other groups, some software allow to specify the contrasts. If not, you can re-analyze the data using group #2 as reference, then group #3, and so on to get all the possible pair-wise comparisons.

Regarding the FWER: actually you cannot control the FWER because you do significance testing, and there is no way to commit a type-I error (H0, by definition!! is always logically false, so not-H0 is always logically true, and if you take "rejecting H0" as "accepting not-H0" you can never be wrong). What you get from a single significance test is Pr(t>tobs|H0) for some test statistic. Is you do a multitude of tests you might be more interested in Pr( t > at least one of all the tobs | H0), and this probability is obtained (or estimated) by applying multiple testing corrections, like Bonferroni, Holm, Hommel, Hochberg, Benjamini, etc. If this applies to your scientific question, then you surely can do this.

-

PS: If you want to be very particular, you might think about the kind of response you have in order to identify asensible error model (it might not be normal). You talked about relative frequencies, what suggests that you would have a binomial error model. However, if the relative frequencies are based on a large number of counts, a normal model would be an appropriate approximation. If the the relative frequencies are getting close to 0 or 1, a beta-error model would be better, though.

Thank you very much for the fast reply!

One last thing, let me be clear: We don't have groups, we just have six cases measured 21 times. So the problem is always a really small sample size.

The easiest way to solve the problem, Jochen described,  is probably a correlation between response and year - this is what is meant by a regression analysis.