My first thought is to run a logistic regression, where your dependent outcome is indeed binary, and your independent variable of interest is the treatment assignment (a categorical variable with 8 levels). In Stata you would estimate this model first, followed by margins to determine which of the treatments was most effective. It would look something like this:

webuse lbw

logit low age lwt i.race smoke ptl ht ui

margins race

margins race, pwcompare(effects) mcompare(bonferroni)

So here we use the lbw data and specify a logit model. The variable we are interested in here is race, which has 3 categories. We run margins after estimation and see that the category "black" has the highest proportion of low weight births. To determine if this group is statistically higher than the other two groups, we rerun margins making pairwise comparisons with a Bonferroni adjustment for multiple tests. We see from the output that there is no group that has a statistically higher proportion of low birth weights than any other group (at the P

Dear Matthieu

You may use the Proc Logistic in SAS for similar results like that proposed for Akkur and Ariel. But If your sample of persons was produced by a sampling design you may obtain better inference on the sampled population using a procedure that takes care of the sampling design, the weight of the observations and other variables availables for each person. In SAS you have the Procedure Surveylogistic that let you to introduce the design information and to correct adequately the p values with an adjust because of the deff (design effect).

The conventional Proc Logistic may be used when you are applying your treatment on all your available population or you are applying your treatments on a sample obtained with a Simple Random Sample Design. 

Besides, if the persons have differences of education, sex, age, profession, place of residence, etc. you may introduce this information to improve the comparation of the treatments.

I hope this help

Thank you very much for the help ! I'll do the logit regression, then. Just a side question: I might have to present my work not only to economists, but also to psychologists and market researchers, who might not be as familiar with econometric models as economists. Do you think it could be a problem ?

Guillermo, I don't think I am following you here. In my case, I am doing this experiment on a sample that is "semi-randomly" selected from the french population ("semi-randomly" in the sense that, since I am conducting my test at the end of a phone survey made by an institute, the quota method has been used to ensure representativity according to socio-demographic variables - so they don't just call random people). How would I introduce this information into a model ? At least theoretically, then I'll try to figure out how to do it in Stata.

Yes, I always add socio-demographic variables into my models, just to be sure.

Here is a non-parametric approach that doesn't require the use of reference groups and dummy-variables: http://optimalprediction.com/files/pdf/V2A27.pdf 

Matthieu,

In response to your follow-up question:

"I might have to present my work not only to economists, but also to psychologists and market researchers, who might not be as familiar with econometric models as economists. Do you think it could be a problem?"

This is not a econometric problem, but a general statistical one. Thus, people from any discipline should be familiar with this common approach and interpretation of results.

Ariel

Do you have any beliefs about how the different conditions compare ( prior to seeing data )? If so, this will inform your testing.

As far as econometricians often using different phrases from statisticians, psychologists, etc., this is true, so be careful and if you are really worried get one of them to proof read. 

Dear Matthieu

In your first question you say:

"Last but not least, I have a very large sample (4000 subjects in total), and I have noticed that the Chi-square gives me very low p-value even when, intuitively, I would say there is no significant difference"

This matter is related to my affirmation about the deff of the Sample Design. For example I have recently analized data of a probabilistic sample of secondary students. They were selected in a random sample of schools with all their fellows in the same course. The deff design was for most variables near to 40. This means that a sample of 4000 is equivalent to a Simple Random Sample of 4000/40=100, then many Hypotesis, that with the Proc Logistic or with a Contingence Table Test, would have a little p value, with a procedure that takes care of the Sampling Design, they had a large p value.

I hope this help

                  Guillermo

Hi, Matthieu,

When you have many samples, you can detect relatively small differences between Groups. These small differences may seem irrelevant from a practical Point of view - therefore you may have the Impression that "this cannot be significant".

However, using the logisitic Regression you can easily check the Magnitude of the effect with the Odds-Ratio. For exampe, if he Odds-Ratio for a certain Group is 2, belonging to this Group doubles the probability of the Event you are interesed in (your binary Response variable). Odds-Ratis below 1 indicate lower probability  - compared to the reference Group. An Odds-ratio of 1 indicates no effect

Ok I think I get it, thank you very much for your answers and your solutions everybody.

One last thing while we're at it. If I have a categorical DV instead of a binary one, what should I use (assuming the IV is also categorical) ? Someone told me to do a multinomial logit, does that seem good to you ?

Dear Matthieu

If your DV is ordinal, you may use a logistic model with a Cumulative Response logit Model that take care of the ordinal nature of your response, but if you DV is nominal then you may use a logistic model with the generalized logit model. Your IV may be continuous, discrete or categorical.

The DV is not ordinal. And the probability of outcome may vary across the trials, although I assume this variation is random, so that might be a problem if I want to run a multinomial regression.

Dear Matthieu

I think that in your problem the generalized logit model within the logistic analysis is adequate.

Ok I'll look into it, thank you very much for your help !