Vincenco, as far as I know do the "classical non-parametric tests" you are suggesting assume *continuous* data. Cateogorical ordinal data actually violates this assumtion. The problem may be tolerable in some cases (e.g. for data with many categories and few ties), but I think it is not a good idea to give such a general advice. Further, the suggested MWU test will test a location difference only and only if all other moments of the distributions are similar. Again, violations might be tolerable in some cases, but again this should not be given as a general solution for such problems.

I think you can use Chi-square for both analysis (2 by 2 and 2 by n tables).

Hi Jyrki, There are a lot of possibilities. For a 2X2 table, I guess you can never go wrong with Fisher's exact test. It doesn't have a lot of computational overhead and has no problems with imbalance or sparse cells. Traditionally, ordinal variables have been handled with non-parametric statistics, such as the Kruskal-Wallis test. In the case of just two groups this reduces down to the Mann-Whitney U test. With at relatively smallish sample like this, you would probably have the option of exact statistics. See StatXact at www.cytel.com. Another option is ordinal logistic regression. The thoughtless option for ordinal 2*n when n>2 is just a simple contingency table with a chi-square test because it throws out the information about the ordinality. That's a big difference. You get all kinds of wrong results that way. You also don't want to toss out information by taking ordinal scales and crunching them down to dichotomies just to be able to get odds ratios or something else familiar (you do get odds ratios with ordinal logistic regression). Bob

- but how to test if these OR:s differ significantly?

Calculate the confidence intervals (CIs) for the two ORs. When the ORs are not withing the CIs of the respective other gender, there is a significant difference.

-Is there a more sensitive method than just compairing OR:s?

You could set up an appropriate Poisson ("logistic") model using three predictors together (treatment, symptoms, AND gender). The ORs you have would be represented by the interaction of treatment:symptom, the difference in ORs would be represented by the trheefold-interaction of treatment:symptom:gender.

- I do not wish to be guilty of data torturing by transforming the 3 - 5 item scales arbitrarily into a dichotomous one.

Dichotomization is a torture, but not "advantagous" for you. I mean to say: it is much more difficult to identify an effect in dichotomized data instead of in the original, ordinal data. Thus, IF you are able to convince others that there is an effect even in the dichotomous data, then everything is fine (if i can hit a target when wearing dark sunglasses in the night, one would believe that I would also be able to hit it when there was more light). There are models for ordinal response data: ordered logistic models, as Robert said. in contrast to Robert I'd not suggest to use "traditional" methods like non-parametric statistics, because the don't provide effect estimates.

Btw, Robert, could you explain why with an ordered logistic model on a factor with 2 levels one would "get all kinds of wrong results"? I know that the direction is lost, but I think you get the same results (OR) like for a "standard" logistic regression. Maybe I am wrong - I would be greatful for any clarification.

Dear Jurky,

A covariate (age, race, sex and so on ) influences the response. So we need adjusting procedure for such a covariate. The stratified analysis of contingency tables is appropriate.

For stratified contingency table 2 X 2: Breslow-Day test for homogeneity of odds ratios ( or its modimacation). If significant difference not found, you can calculate common odds ratio.

For stratified r x c tables: generalized Cochran-Mantel-Haenszel test.

Breslow NE, Day NE (1980). The Analysis of Case-Control Studies. IARC Scientific Publications No. 32 Lyon, France.

If You use R - package epiR utilizes those tests.

Another way for 2 x r table - to fit logisic regression with interaction.

Dear Jyrky,

using the Mantel-Haenszel test you can compare the odds ratios of several 2x2 tables. The Mantel-Haenszel analysis provides two closely related pieces of information. First, it provides statistical tests of whether the odds ratios are equal (homogeneous) or unequal (heterogeneous) across strata (each individuale table, in your case the 2x2 table for males and the 2x2 table for females). Second, it provides an estimate of the odds ratio of the treatment/exposure variable, adjusted for the strata variable (i.e. estimates of the odds ratio of exposure/treatment to disease after removing the influence of sex).

The assumption of the test are: a) observations are independent from each other [each observation comes from a different subject, that the subjects were randomly selected from the population of interest, and that no specific group of subjects is purposefully omitted], and b) all observations are identically distributed [they are obtained in the same way]. That is your case, I think.

For hypothesis testing, use the Heterogeneity Test to test the hypothesis that all odds ratios are equal. Use the MH C.C. (Mantel-Haenszel confidence limits and hypothesis test with continuity correction) hypothesis test to test the hypothesis test that all odds ratios are equal to one. Use the Robins confidence limits (which have shown to be more accurate that any of the others presented).

For ordinal data comparisons, you can use the classical non parametric tests (i.e. Mann-Whitney U or Wilcoxon Rank-Sum Test for Difference in Medians [for 2 independent samples] or Wilcoxon Signed-Rank Test [for 2 paired samples]). Trasforming ordinal data to dichotomous data is not advisable since this procedure can introduce some biases (i.e. some item of your scale could not be clearly categorized as yes or no and your arbitrary decision could make the difference influencing in some extent data interpretation).

I hope this can help.

Best regards

Vincenco, as far as I know do the "classical non-parametric tests" you are suggesting assume *continuous* data. Cateogorical ordinal data actually violates this assumtion. The problem may be tolerable in some cases (e.g. for data with many categories and few ties), but I think it is not a good idea to give such a general advice. Further, the suggested MWU test will test a location difference only and only if all other moments of the distributions are similar. Again, violations might be tolerable in some cases, but again this should not be given as a general solution for such problems.

Hi Jyrki,

I disagree with all the ideas suggested this far in this thread.

There is a method that will allow you to test all the hypotheses you may have on the data you have. It has the advantage of a model comparison method, since it allows you to compare all the models you might want to test on your data. In this it is superior to the chi2 approach, which only tells you that all is not "the same" without telling you where is the difference.

The method consists in comparing all the models you want (through the Bayes factor or the BIC associated to each model) and thus finding the most likely model for your data. The computations are ridiculously simple, what is a little bit more complicated is the model comparison logic. By this I mean to say that it is you who must define the models to be compared in order to answer a question. This is the most tedious step. Once you're done with it (and God knows you must verify thoroughly that you've done this well), computation is a piece of cake and you proceed with the model comparison till you reach the best model for your data. Once you are there, you "read" what that model "says" and you have your answers. I even know of a R package the brute-force tests all the models for your data (an approach that spares you the pain of defining all the possible comparisons) and gives you the one most likely model for your data.

Let me know if you want to go down this path, and I'll lend you a hand if you need.

Cheers,

SCM

Sebran, these procedures just calculate how well the data fits to the model, in other words: how likely the data would be obtained given a model. This is not equivalent to "[...] gives you the one most likely model for your data". This is especially wrong when the models to be compared are not justified a priori and theoretically. Eventually, any model comparison/selection by means of statistics is superfluous. If there are several justified models, then each has good reasons to be like that and these models might all be appropriate under certain circumstances. A "selection", especially based on some statistic, is a step back, throwing away good thoughts about the models.

Jochen,

I was alluding to a procedure that uses Bayesian inference, so, NO, it is NOT about "how well the data fits to the model" (as you say), which would be p(data|model), but it is about how likely different models are given the data one have, i.e., p(MODEL i | data).

Fisherian statistics are about the probability of the data given a model (i.e., the null model). Bayesian statistics are about the probability of any model one wants to consider given the data at hand (included the probability of a null model, a feature of this kind of statistics that allows to conclude to the null hypothesis if the model corresponding to it is the most likely).

Now that it is clear that I was talking about Bayesian statistics / Bayesian inference, the way to go is to compute a value corresponding to the probability that a given model is the most likely for the data (e.g., its Bayesian Information Criterion, BIC) and do this for all the possible models one can build on that data (and here by data I refer to the levels of the variables considered, not the counts). The most likely model for the data (i.e., the one with the lowest BIC) will be found in this way and then one has only to read out what this model "says" and conclude.

And it is Serban, not Sebran.

Best wishes,

-- SCM

Sorry, Serban for the typo. I see your point. This is my point of view, feel free to correct me: If it were a real Bayesian analysis, you were absolutely right. Using the BIC for automated selection, however, is not Bayesian (not just because the statistic is called "Bayesian"). A real Bayesian analysis would require to set up a sensible prior for each tested model. As long as there is no external justification of the probability or value of a particular model, the BIC is not principally different to AIC, F, R² or any other statistic (each having different properties in detail).

Maybe I am wrong. Then I really apprechiate to be tought otherwise.

Best wishes!

Jochen,

You are right when you say "A real Bayesian analysis would require to set up a sensible prior for each tested model".

The point is that when you have no idea what would be the "right model", you set all priors to the same value.

So, if all in all there are 5 models to be compared, you set the unconditional probability, p(M i) to 1/5= 0.2. This is your unconditional (or a priori) probability that any of the 5 models be the right model BEFORE you look at the data. Once you look at the data, you can compute the posterior probability for each model, i.e., for each and every model its probability of being the right model for the data.

You may look at this this way: when you know nothing about the data all models are equally good, then, a posteriori, the probability for each model is re-evaluated by taking into account the data. The a posteriori probability of some of the 5 initial models may go down, that of some others would go up, but there will always be a model whose a posteriori probability will be greater than that of all the others.

You may note that I have not spoken in terms of BIC, but you may just as well reason in terms of lowest BIC instead of in terms of the highest posterior (or a posteriori, if you like) probability, these two things are interchangeable (see Schwarz, G.E. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.) So the analysis I was proposing is a genuine Bayesian one.

I think you might reply that setting all priors to the same value is odd. I would say it is not odd when you ask a question for which there is not a lot of data giving you the priors. And I would add that if you knew the priors you would have ended with the same conclusion in most of the cases, at least if you have gathered enough data to be able to draw a conclusion on which is the best model in the first place (and in that case, the probability of the model given the data would be bigger and its BIC even lower).

Cheers,

Serban

Thank you all for what has been said so far. It is very educating, although now realizing the complexity of my problem, I'm beginning to feel small and insufficient... Anyhow this has enormously helped in grasping the problem and I'm really grateful for all comments.

Ok. Now back to the roots: You can look at my latest papers to find out what we are doing. In the present problem I have two groups: cases with ADHD and cases no ADHD. The cases originate from a cohort with birth risks, such as asphyxia. In addition, I have a third group consisting of controls with no birth risk with or without ADHD. ADHD is somewhat more common (or less misdiagnosed) in boys, so there are more boys than girls in the ADHD group. The question is: are boys in our groups phenotypically different from girls e.g. in terms of aggressive behavior?

An hypothetical example:

1. Controls:

50 boys: 0 with ADHD, 10 aggressive

50 girls: 1 with ADHD, 5 agrressive (ADHD case not aggressive)

2. Asphyxiated without ADHD:

50 boys: 10 aggressive

50 girls: 7 aggressive

3. Asphyxiated with ADHD

50 boys: 20 aggressive

30 girls: 6 aggressive

We know from previous research, that the prevalence of aggressive behavior measured by this particular (hypothetical) test corresponds to that of the general population, but we have no apriori information of the ADHD population. We do not know the sensitivity of specificity of the test method, but it is assumed to be the same for girls and boys. So, are boys with ADHD more aggressive than girls with ADHD, or is it just asphyxia (and possibly misdiagnosis?

n.b. We are dealing with psychological tests with several questions generating an obvious multiple testing problem too, but let's leave that aside for now...

Jyrki,

I understand your example EXCEPT for the Controls, where it seems two things (i.e., with/without ADHD and agressive/not agressive) seems to be mixed. Or is it that what you call Controls aro those who were not asphyxiated? Please enlighten me on this point before I start writing down models.

Cheers,

SCM

If what you call Controls aro those who were not asphyxiated, then:

- there are 8 groups to be compared with respect to their probability of being aggressive (boy/girl * asphyxiated/not asphyxiated * ADHD/not ADHD)

- there are 4140 models

- the best model among these is the one that puts all the groups together except the group of boys who were asphyxiated and have ADHD

- this model has a Bayes factor of 81.1364, which is strong evidence in favour of this model ("strong evidence" is a way to speak of the size of the Bayes factor, see e.g. Jeffreys, 1961, app. B, cited in Kass & Raftery, 1995)

- this model predicts a probability of being aggressive of 0.404 for the group of boys who were asphyxiated and have ADHD and a probability of being aggressive of 0.167 for all the other groups

- it seems that the conclusion for this set of data is that if you are a boy and asphyxiated and with ADHD you will have a higher probability of being aggressive ; it is an interactive effect, since :

* boys in general are not more aggressive than girls

* asphyxiated kids in general are not more aggressive than non asphyxiated kids

* kids with ADHD in general are not more aggressive than kids without ADHD

* asphyxiated boys in general are not more aggressive than asphyxiated girls

* boys with ADHD in general are not more aggressive than boys without ADHD

* asphyxiated kids with ADHD in general are not more aggressive than non asphyxiated kids with ADHD

As you see I couldn't wait to supply you with an answer, though as I said in my previous post, I'm not sure that I understood correctly what's going on in the Control group. If I understood well, then here is your analysis and its conclusion. Otherwise, please correct me.

Let me know how things are. Hope to hear soon from you.

Cheers,

SCM

PS: took me about 2 minutes to enter the data and run the analysis :)

Hi Jyrki,

I see in Your example response variable=aggressivity

and three predictor: sex, ADHD and asphyxia

I reproduced Your data and fit logisic regression

The results are:

for sex (male): OR=2.115(1.133-3.948 95% CI) p=0.01869

for ADHD: OR=2.132 (1.058-4.3 95% CI) p=0.03433

for asphyxia: OR=1.189 (0.5583-2.533 95% CI) p=0.6533

Sex and ADHD are independent predictors for that example.

Asphyxia does not "work".

Of course this logistic regression does not reflect frequnces of predictors in population.

Best wishes!

Serban, I really appreciate your interest in this. You understood the case correctly, and in an intuitive way your cakculations make perfect sense. The important questions remaining are then what magic did you use to calculate this, and how to tackle multiple comparisons? In other words, the important question is if females with ADHD are different from males with ADHD in excess to vive la différence! And if they are, in which respect and to what extent.

Jyrki,

There's no magic involved, just a R package called AtelieR that has been developed by my colleague Yvonnick Noël. It is extremely easy to use... if you speak French. Yes, unfortunately, the package is in French for now. We use it to teach labs to our psychology students. It computes the Bayes factor for each and every of the models (this is why I can tell you that there are 4140 models) and then gives you the model with the highest Bayes factor (in Bayesian slang, this is "the most likely true" model)

I am not sure though to understand what you mean when you say "how to tackle multiple comparisons?" Do you mean comparisons that integrate many factors (e.g., boy/girl and asphyxiated/not asphyxiated and ADHD/not ADHD)? If this is what you mean, the answer is the following:

- you create as many groups as the product of the number of levels of all your variables

- this would be 8 here, with boy/girl and asphyxiated/not asphyxiated and ADHD/not ADHD

- for each of these groups, you count how many aggressive kids there are (and how many non aggressive, too

The previous 2 steps would yield something like this (sorry for the dots, it seems I cannot make a nice table here):

.........................................................aggressive ..........not aggressive

boy asphyxiated ADHD.......................20............................30

boy asphyxiated notADHD..................10............................40

boy notasphyxiated ADHD....................0..............................0

boy notasphyxiated notADHD.............10............................40

girl asphyxiated ADHD..........................6.............................24

girl asphyxiated notADHD.....................7.............................43

girl notasphyxiated ADHD.....................0...............................1

girl notasphyxiated notADHD................5.............................45

- you enter the data and have it processed with AtelieR, it will give you the "correct model" for your data, and its predictions

- you interpret that model.

With your data, the best model is one that says only the first group is different from the others and all the others groups are not different one from another, something I will note as (1)(2,3,4,5,6,7,8)

This means that in order to be more aggressive one has to be a boy, asphyxiated and with ADHD.

Try to imagine what your model would be if for instance there was only a main effect of gender. Can you produce my notation for such a model?

.

.

(if yes, try to do so, I'll put the answer bellow)

.

.

.

.

.

.

.

if no, keep scrolling, the answer is bellow

.

.

.

.

.

.

.

.

.

.

.

.

.

It would be (1,2,3,4)(5,6,7,8).

I hope you get the logic of this. Though this was an easy one: it is about a main effect.

Try to generate the notation for other hypotheses.

In the end you will become comfortable with this notation and also with what means a notation like this that AtelieR yields.

If your question is whether "females with ADHD are different from males with ADHD in excess to vive la différence", the answer is clearly no (at least with the data set you provided), since this involves the comparison of groups 1 and 3 on the one hand, and 5 and 7 on the other hand, and the model says (1)(2,3,4,5,6,7,8), which means that you cannot put together 1 and 3, since 3 is the same as 5 and 7, while 3 is different from 1.

Or, if you like it better, with a new analysis with only the groups 1, 3, 5, and 7, the best model is not (1,3)(5,7) but (1)(3,5,7). But this is just an idea to convince you of the result, not a very good one if you want to see the whole picture (it would be like carrying out a lot of t-tests instead of a Fisher's, if you see what I mean). You need to consider all 8 groups for that (and it would of course tell you the same thing on the comparison of 1, 3, 5, and 7).

Let me know if this is what you meant by "how to tackle multiple comparisons?" and if I got it wrong sorry for explaining things you already understood, and please let me know what you meant.

Cheers,

SCM