Put them in the same model, the interaction of the event and the group will provide what you want. You can compare the models, with and without that interaction, in lots of ways depending on the specifics of your modelling. You can also plot the predictions including uncertainty intervals.

It seems to me yould might want to run the Cochran Q test or the McNemar test, which are for related samples on nominal or categorical variables. McNemar is for 2-samples (e.g., before and after), Cochran Q is for k-samples. In attachment an R script for both tests. It is for the data in Siegel on p. 173

Sidney Siegel and John Castellan, Nonparametric statistic for the Behavioral Sciences, 2nd Edition

Daniel Wright Thank you for replying. I made an interaction between each of the predictors and the event and put all the data in one model as you mentioned but my research boss thinks that this is rather daunting because I have a huge amount of variables in one model as independent variables (I have the original predictors, the event as dummy, the interaction between each predictor and the dummy and the control variables).

He thinks that there should be a single econometric test that examines whether there is a significant difference between the coefficients of each variable across the two models (model for the before and model for after the event).

Jos Feys Thank you so much for suggesting the two tests and sending the code. As far as I understood the Cochran Q test or the McNemar test require a 2x2 matrix which means it requires one of the samples to be controlled and the other one to be treated and then I test against the event.

In my case I cannot create a 2x2 matrix where there is controlled versus treated and before versus after the event. I only have 1x2 dimension: before and after the event. I do not have a controlled sample (all the data after the event is affected by the event).

Please correct me if I am misunderstood.

Zeinab Elrashidy wrote that "[her boss] thinks that there should be a single econometric test that examines whether there is a significant difference between the coefficients of each variable across the two models (model for the before and model for after the event)."

Is your boss asking for one omnibus test for all variables? If so, I think you would have to use a hierarchical approach to carry out something like a Potthoff analysis. On Step 1, include all the variables of interest, and also include an indicator for Time (0 = pre-event, 1 = post-event). Then on Step 2, include all of the interactions with time. The likelihood ratio test comparing Step 2 to Step 1 would give an omnibus test that might make your boss happy. But I imagine that if it turns out to be statistically significant, he/she will then want to look at the interactions for the individual variables anyway!

HTH. Apologies if I've misunderstood.

PS- For some discussion of Potthoff analysis in the context of OLS regression, see this document by Karl Wuensch:

http://core.ecu.edu/psyc/wuenschk/MV/multReg/Potthoff.pdf

I am not sure what "econometric test" means, but if you a mean a single hypothesis test (i.e., one p-value) to see if as a collection all the interactions "significantly" improve the model, then what I said earlier would work (and this is spelled out by Bruce Weaver ).

Not sure which software you are using, but I will assume R. Let g be a variable for the two groups, and x1, x2, ..., xk be the other covariates and store these as a single object X (e.g., X

Here is the above with toy data

> set.seed(924)

> g X Y m1 m2 anova(m1,m2,test="Chisq")

Analysis of Deviance Table

Model 1: Y ~ X + g

Model 2: Y ~ X + g + X:g

Resid. Df Resid. Dev Df Deviance Pr(>Chi)

1 995 1303.8

2 992 1293.4 3 10.421 0.01531 *

---

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

>

Bruce Weaver Thank you so much for your elaboration. No, not on the aggregate level. Rather on the predictors level, individually.

I will illustrate the full situation below.

Daniel Wright Thank you so much for illustrating and writting the codes. Yes I am using R. I am not trying to test the overall two models against each other. Please find exactly what I am trying to find below.

Full illustration:

Logistic regression with 4 predictors, controls and error.

y = 1/(e^-(alpha + beta1 x1 + beta2 x2 + beta3 x3 + beta4 x4 + betaj controlsj ) + ui

Data includes: t1 is 4 months before the event and t2 is 4 months after.

If I run 2 regression models (exactly as the regression model I wrote above) I get two regression results, one for sample t1 and another one for sample t2 .

By using my eyes (no statistics involved) I can see that there is a difference between the magnitude and significance of each beta across the two models on respective variables.

For example: in model 1 (based on t1 ) the odds ratio for x1 is 1.563 significant at 0.01 and in model 2 (based on t2 ) the odds ratio for x1 is 1.956 significant at 0.01. Without using any statistical test to support my eyes, I would say that the influence of x1 on the odds of y occurring (y being 1) is higher after the event. Same goes for the other 3 predictors. I am comparing the odds ratio as well as its significance for EACH predictor across the two models. So comparing the influence and significance of 4 predictors in model 1 against their respective in model 2. So, the odds ratio and significance of x1 (model 1) against the odds ratio and significance of x1 (model 2) all the way till x4 .

I understand that using my eyes to judge the difference is totally not reliable nor valid so what I did is I used interaction as if the event is a moderating variable. Thus, I included the 4 variables, the controls, the event as dummy and the interaction between each of the 4 variables and the dummy event in 1 model over all the data. However, after reporting my results, my research boss does not think this is the best way to test the difference between the influence of each of the predictors on the outcome variable before and after the event. He thinks there must be a test that reports the difference between the influence of each of the predictors on the outcome variable across the two samples without using moderation. And he also doubts that the usage of moderation is suitable in my case.

I hope I delivered a clearer view of the research. I really searched a lot for the test that he asked for but I cannot find a test that compares the coefficients of variables across two logistic regression models.

IMO, your boss is asking for a test that either does not exist, or that will be equivalent to the suggested test of interaction if it does exist.

Bruce Weaver I totally agree.

Zeinab Elrashidy Hi there.. i just saw this threat and actually I'm also having the exact same issue. Can i ask you how did you manage to do it in the end? Best regards.