It sounds like it would be fine to do a separate regression for each dependent variable, appropriately corrected for multiple comparisons.

I don't think it's necessary to try to get too fancy with this. It's better to use a simple model (or even just a clear graph of the data) and be clear about its limitations, rather than to use advanced stuff that you and your readers may not fully understand and thus might make mistakes on or misinterpret or not recognize the limitations.

My personal view (which the person or people grading your masters thesis may or may not hold): the purpose of a masters thesis is not to present a perfect piece of research, it's to show what you've learned. Thus, I think it's fine to present a simple statistical analysis, or even no statistical analysis, and give a discussion which clearly indicates what kinds of conclusions you can make and what the limitations are, and shows that you know what you need to do next. Personally, I would give a higher grade to a thesis which has a simple and less-than-ideal statistical analysis but shows a clear understanding of how it could be improved, than I would to a thesis which has a very fancy analysis but doesn't really show much of a learning process or awareness of what could be done better.

Hi Jacob Jarl, really neat masters thesis! Much like Stephen Politzer-Ahles mentioned, my suggestion would be to go for the simplest possible approach, and build up. I've included some suggestions below.

- I don't think you need to be using correlation. Your treatment variable is dichotomous (did I get treatment A or did I get the control), and your outcome is presumably continuous, so the difference of means is the most appropriate bivariate test for you.

- I really don't think hierarchical regression applies here.

- You've call it an experiment; just to confirm, were the participants randomly assigned to a treatment group vs. a control group? If participants were equally likely to receive the treatment as the control, then you don't need additional control variables; you can just use a bivariate test; there should not be any intervening variables. But, if participants were NOT randomly assigned, then this is an observational dataset, and you should consist using models to control for intervening variables.

- Focus on just one outcome first, the one you really care about - Purchase intention, I assume.

- Run a few targeted bivariate tests. What is the difference in mean purchase intention between people who received treatment A and people who received the control? What about for people who received treatment B and people who received the control?

- Next, especially if your subjects were not randomly assigned, build your evidence base by controlling for other effects. (And yes, definitely include all people who were involved in your study, including the control.) You had six treatments, right? So for each treatment, make a separate regression model, where the cases include people who got treatment A or the control, for example. Make a treatment variable that shows 1 = they got treatment A or 0 = they got the control. (Do NOT mix in people who received other treatments.) Then, add any of your extra control variables for gender, race, education, income, whatever.

- Formula should look like Purchase Intention ~ Treatment A + gender + race + other controls.

- Compare results from your bivariate test and your model. Does your treatment variable have a statistically significant effect on your outcome? Does that effect remain significant after you include any necessary statistical control variables? Report those.

- If you used a model, confirm that you've used the correct model. Usually, OLS is fine. If your outcome is binary (1/0), use a logit. If it's categorical, use a probit. If it's a count (0, 1, 2, 3, but not -1 or 2.3), use a poisson or negative binomial.

The same to @Timothy Fraser

Hi Timothy Fraser ! Thank you for your extensive reply!

The participants were randomly assigned to each group.

I have a questions with regards to how we can investigate interaction effects. Is it the right way to multiply our treatments (dummyvariables) with involvement (which we believe can moderate)?

Question 2, how should we test interactioneffects?

Best regards,

Jacob

Further! How do I control for other factors in a bivariate test? The bivariate test we have used is that of Anova. How can I control for other differences? worth mentioning is that we don't get significant results, when we perform ANOVA without any other control variables.

Hi Jacob,

1. Bivariate tests involve only 2 variables, so you can't control for other effects with a bivariate test. However, since you randomly assigned people to the treatment and control group, you shouldn't technically need control variables. (The beauty of randomization!)

ANOVA would certainly let you compare across multiple treatment groups (but still doesn't control, but again you don't need controls given your research design). However, ANOVA just shows you a single statistic representing the overall deviation in the outcome among each group. So personally, I recommend the difference of means, which is much, much clearer to interpret. You should do one difference of means test (t-tests) per treatment, using just the cases that got that treatment and the control.

2. The reason you randomly assigned people to the treatment and control groups was to find out the direct effect of the treatment, not to find out an interaction effect. So, how to alter your research design to make it work? Here are a few ideas.

1. Filter your data to just cases involving your control variable of interest. Eg. Filter to just women in your dataset. Then repeat the difference of means test.

2. Using the normal dataset, run a regression, involving your outcome, your treatment, and an interaction with your control variable. (Interaction effects are just multiplication, but most statistical software have specific buttons for it.)

3. It is okay that you are not seeing statistically significant results. A null result like this is common. If you want multiple predictors, and you have a good theoretical explanation for why a third variable would intervene in this process even after random assignment, you should turn to multiple regression. However, again, since you randomly assigned members to the treatment and control group, you shouldn't really need regression and its statistical controls.

If you're getting statistically insignificant results, this would be a good time to consider why. Eg. Is there basically no relationship? Or is there a small relationship? Is the sample size very small, thus inflating the standard error and reducing statistical significance? Each of these are things you can write about.

If sample size is an issue, you could repeat the t-test/difference of means using an Exact Test (permutation test), which is unaffected by sample size.

But really, I wouldn't worry too much about the null result and just go ahead and report it. The difference of means in a randomized control trial is pretty much the gold standard for causal inference in stats. Instead, look at a few things: 1. How big is the estimate effect of the treatment on the outcome? 2. How statistically significant is it? Eg. How certain are you? P < 0.05? P< 0.10? P

Hi Timothy Fraser and thank you once again!

I understand!

One question with regards to ANOVA, why can't I run an Anova, and use a post-hoc if i find singificant results? What difference will a t-test give me if I don't get any significant results in an ANOVA? IS IT POSSIBLE TO GET SIGNIFICANT RESULTS IN A T-TEST BUT NOT IN AN ANOVA?

Further, I want to compare the treatment in relation to the other treatments as well, thus I do find Anova to be most suitable... What are your thoughts?

Hi Jacob,

Yes, I think it is worth trying t-tests first, and ANOVA second. Think of it like using slightly different lenses to compare outcomes. I think you could get statistically significant results from a t-test but not an ANOVA; if the group means from some treatment groups are quite similar to one another, it might (maybe) reduce the F statistic.

- When you use a t-test, you are comparing whether the outcome differs enough among a single treatment group and a control group to produce a statistically significant T-statistic.

- When you use ANOVA, you are comparing whether the outcome differs enough among three or more groups to produce a statistically significant F-statistic.

In other words, ANOVA is really a second-level test for your experiment. First, you probably want to find out, does my treatment matter, and THEN test do many treatments matter.

Either way, the t-test gives you a more useful estimate. You can line up the difference of means for each of your treatment variables in a nice table and compare them. How much bigger is the effect size for treatment A than treatment B? An ANOVA test won't do that; it just gives you a single F-statistic describing the variance among group outcomes. So for that reason, I find it easier and clearer to use t-tests in this kind of situation.

I think it's best to focus on transparently reporting the results of each level of analysis, whether they are statistically significant or not.

Finally, about your comment: "Why can't I run an Anova, and use a post-hoc if i find significant results?" My guess is that you mean, why can't I run a multiple regression model, using post-hoc control variables? In short, you certainly could. My comments above were more about the "why." You've already used randomization to eliminate intervening variables pre-treatment. In other words, because you randomize who received treatment A and the outcome, women had the same likelihood of receiving the treatment as men; older respondents had the same chances as younger respondents; etc. In practice, this should create very balanced covariates among your treatment and control group, meaning that your treatment and control group are very, very comparable. Statistical controls are only useful if they are controlling for a source of variation that might affect the outcome. So if your covariate balance is good, then you should be good to go. No need for statistical controls.

On the other hand, you could always take a more practical approach: try it both ways and see what happens.

Good luck!

Hi again Timothy Fraser

Can't put enough emphasis on my gratitude towards you and that you are helping me so much!

I have just completed a T-test analysis of each and every treatment group towards my dependent variables - with no significant results. However, some treatments were close to that of 0.05, where they scored 0.06. Is this interesting to report and analyse, despite not being significant?

If I follow your advice and present this first in my empirical data section, concluding that my treatments had no intended effects on the variables in relation to the controlgroup; Is it interesting to present an Anova analysis?

I am thinking it could be interesting considering that there can be differences between the different endorser types.

Do you agree with this reasoning?

Kind regards,

Jacob

Hi Jacob, a p-value of 0.05 or 0.06 is definitely worth reporting! It just means that you're 94% confident about the observed effect, not 95 or 99%. That's great news. Those are absolutely important results worth reporting. If there weren't really a big effect, you'd probably see p-values like 0.15, 0.35, or higher. But in my opinion, if you find something below the 0.10 level, that's worth discussing. If you had a larger sample size, you would probably end up with a more statistically significant result. Congrats! Your masters thesis has identified some interesting potentially causal findings. (And we probably know that they're causal because of your randomized control trial, since you controlled the treatment, not some other process.)

You can visualize your difference of means by taking the estimated difference of means and adding or subtracting 1.96 times the standard error. That will give you your 95% confidence interval.

Having found somewhat-statistically effects, you could consider some follow-ups.

Check out the difference of means for each treatment. How big is it? Which treatment produces the largest difference of means? You could even make a visual of this, showing the estimate and confidence intervals for each of your difference of means tests. This is great for comparing which treatment is more effective.

Next, you could use ANOVA - but it's important to mention again that ANOVA really answers a different question. This would tell you if the group means of the treatments and control groups are different enough from one another overall. So if two treatment groups are really close to one another, you might not get a big F-statistic. If you have six treatment groups and a control group, that's going to be a lot. Don't be surprised if you get a statistically insignificant result.

Either way, I think the most convincing thing to do is go visualize these results.

Density plots are commonly used to visualize the distributions of several different treatment groups. If you can visually prove that the treatment groups and control groups are quite far apart, your analysis will be even more convincing. Same with t-tests.

I hope that helps.

Hi!

In our analyses of the t-test, we get different significance levels.

Given that we have only tested towards the control group, can we analyse the different levels of significance levels between treatment groups?

For example, treatment A got sig level 0.5. Treatment B got sig-level of 0.06.

Can we analyse the differences between these treatments effect in relation to control, when we are not directly comparing the two in a statistical test?

Timothy Fraser I think that in order to measure the differences in treatment effect for two treatmetns, i need to perform an F-test?

As I explained previously. I have 6 treatmetns in total. Two treatments come from the same category (i.e. I have two different experts, two different celebrities).

I have performed the T-test in relation to control, and now I want to evaluate the difference within the categories. How do I do that? What test should I perform?

Hi Jacob,

Yes, an F test / ANOVA will test whether the mean outcome for three or more categories are generally different from one another. That would be the appropriate comparison between multiple treatment groups.

But in general, you've already got most of what you need anyways.

You know that treatment A had a pretty statistically significant effect of Beta = something (p

Hi Timothy Fraser

I would be most grateful if You can take a look at the how we have written the empirical section now as it is. After speaking to our supervisor, he advised us not to go with the T-test, as it would be sufficient to go with Anova - thus we are not changing this.

Further, we have performed a linear Regression with the "ENTER" method instead of stepwise introduction. I am wondering if this is appropriate and if we have presented the right coefficients etc. Please, Please, please help us with making this section perfect. Attached, you find the empirical section.

Thank you so much in advanced, and you can look forward to being mentioned in our acknowlegements in the paper!

Hi Jacob, I'm afraid I can't open the file (and I think ResearchGate is more about crowdsourcing questions than editing, anyways). Nevertheless, two quick comments:

1. If your advisor says ANOVA, then great - that will also work, as we've discussed above. (But yes, t-tests also definitely work here.)

2. Yes, linear regression is fine here, and yes, definitely don't use stepwise regression unless you have a reason for it. I believe "Enter" in SPSS just refers to the default method. As for reporting regression results, the standard things to report include:

-R2: What percentage of the variation in your outcome explain? (0=worst, 1=perfect, somewhere in between is most likely).

-Beta: As your predictors increases by one, holding all else constant, how much does your your outcome increase by? Use Standardized Beta Coefficients if you want to compare effect sizes.

- P-values: How likely is it this Effect is just did to sampling error? Small p-values are ideal, add they give greater confidence in the external validity of your results.

Timothy Fraser , I have a question for you. In one of my regressions, i find multicollinearity. However, only for the dummyvariables and the interaction variables.

I have read online that you can ignore multicolliniarity when you are using dummy-variables and interaction variables. Is this correct? What are the implications of having multicollinear data?

Hi Jacob,

Multicollinearity is certainly problematic, because it will bias your beta coefficients. The question is, how much do you have?

There is a statistic SPSS and most stats software will produce for you, called the variance inflation factor (VIF), for each coefficient in your model. A high VIF indicates a variable that is extremely correlates with others. Generally, a VIF above 10 will lead to biased coefficients. In contrast, a VIF of below 2.5 is fantastic - no problems there. Because many variables are highly correlates, many social scientists focus on getting the VIF down so it is not near ten (eg. 6 or 7 or even 8 is high but more acceptable then above 10.

To reduce collinearity, you need to either drop variables with high VIF from the model OR transform them (square them, log transform them, etc.) so that they no longer are highly correlates with other predictors.

When using interaction effects, VIF is always high, and it is not really possible to correct for it, since interaction effects add collinearity. Social scientists usually present one model without interaction effects, that has good VIF, and then the same model but with interaction effects. This helps demonstrate that your model is built on a firm foundation, even though interactions naturally cause collinearity.

When using categorical variables, VIF does tend to increase, but not always. I've seen two approaches.

First, when categorical variables are involved, statisticians recommend that you transform the VIF to get the GVIF^(1/2*df), where df refers to the number of categories in your variable, minus one. But, that new measure isn't exactly on the same scale and the original VIF. If you square the GVIF^(1/2*df), you can now read all adjusted VIF scores on the same scale. These scores tend to be much more reasonable.

Second, it is always with considering, did I remember to make one of the categories a baseline category? That is to say, let's say I have a dichotomous variable like above age thirty vs. below age thirty. If you format this in SPSS as a categorical variables, the program will handle it properly. On the other hand, if you enter it as TWO DIFFERENT DUMMY VARIABLES, then that will cause severe collinearity. The dummy VARIABLES for one category must always be left out. That becomes the baseline category (the intercept). While SPSS does this automatically for categorical variables, it's an easy mistake to make if you are manually creating dummy variables.

I hope this helps.

Hi Timothy Fraser , I am glad to inform you that you are now being mentioned in our acknowledgement of our thesis.

I have some questions regarding factor analysis. We saw that from 10 items, 2 factors scored above eigenvalue of 1. However, we only want to use the ten items as one variable. The only argument I can present now is that the cronbach alpha level for the ten items score above 0.7. Is there any other argument i can say to motivate my action of only using the ten items as one variable and not two?

Regards,

Jacob

Hi Jacob,

Thanks for your email. It sounds like you went and applied factor analysis to your data. I don't think factor analysis is relevant for your data. I apologize if I was unclear.

Variance inflation factor (VIF) scores are an automatic output from regression in SPSS. They are not related to factor analysis. If you click the box for collinearity diagnostics in SPSS, you'll get a VIF column in your table.

See here for detailed instructions on how to get VIF. https://www.statisticssolutions.com/testing-assumptions-of-linear-regression-in-spss/

Considering that i used 3 or more items to measure my variables, I used factor analysis to strenghten caims of construct validity.

Then I used Cronbach alpha to check for reliability. Why would this not be appropriate to do when creating my variables?

Hi Jacob,

Yes, that's fine. I had not heard you mention using factor analysis before. I'm afraid I have little to share on factor analysis, and would suggest you consult other sources.

If you really want to be sure that the way you construct a variable doesn't affect your causal inference, then you can just do the following: test both.

Run the model once using just one factor. And then run the model once using two factors. You could even do a third model, where you just take the average of your items and use that as a predictor. Then, take these three models and compare the results. Do the effects of the variable differ? How much?

Simpler, more easily understandable measures are always better. If you don't have to use factor analysis, then don't. Personally, I prefer taking the mean of rescaled variables, because it's really clean what it means.

A good way to validate any measure is to compare its associations with known correlates. If it correlates highly in the same direction as expected, great! If not, then not so great.

Hi Timothy Fraser , Just checking in to hear what disadvantages exist with hierarchical regression. Do you need to have a good argumentation for why you choose a hierarchical regression approach? I am asking as I will oppose students who ahve performed this, and thye have not mentioned anything regarding pros and cons.

Hi Jacob,

Hierarchical regression is sometimes useful, and pretty simple. So, you just make one model, and then you repeat the model with an extra predictor, and you compare the variance explained in each model using an F-test. If model 2 explains a large enough additional amount of variation, then the F-test produces a statistically significant p-value. You would use hierarchical regression if you want to get the most parsimonious model equation for explaining your outcome.

You would usually use it if you have too many predictors and too few cases - use it to reduce your number of predictors to just the set of predictors that really explain variation. I've used it recently for a dataset of mayors, where I didn't have that many mayors to analyze, and so it was important to reduce the number of predictors to maximize my predictive power.

On the other hand, you might not want to remove predictors if they play a meaningful intervening role between two other predictors. So, for example, let's say you're predicting whether someone turns out to vote (DV), based on their age, gender, race, income, political partisanship, and political ideology.

Hierarchical regression might show you that political ideology does not explain much of voter turnout after accounting for political partisanship, since they are very similar concepts. So that would be a good use of it!

But on the other hand, it might also say, hey, income doesn't explain all that much extra after accounting for age, gender, and race - since these three variables often are closely related to income. But income is still an incredibly important theoretical concept in elections, and that might adjust the beta coefficients for age, gender, and race. So, that might be a reason to keep income in.