Are the dependent variables all measured on the same scale? That is, does a 10 on one equal a 10 on the others ?

the right statistical tool is discriminant analysis (Diskriminanzanalyse). A logistic regression of the two test conditions on the three "dependent variables" (being now predictors) should work.

Sal Mangiafico they are all measured on a 7-point scale, yes.

The attached excerpt from Frank Harrell's regression textbook (2nd Ed.) supports Horst Müller's suggestion to use binary logistic regression with group as the DV.

Since the power rating is indicated as the dependent variable, I would advise against flipping the variables to make the test condition the dependent variable, in order to use, e.g. logistic regression.

Instead, here's what I think.

If the three "power" variables are all commensurate, they could be viewed as a single measurement, e.g. "Response", measured on each of three power scales.

This leaves a single DV, and PowerType is entered into the model as an independent variable.

The difference between the Response for each of the three PowerTypes within each test condition may be the answer to the question in the original post.

For example, in the attached image, Group is analogous to the test condition. Here, it is clear that the effect of Group is not the same across Power Types, and that it is the difference in Force Lightning that is the important difference.

R code for the example follows. (Just the plot, not the model and analysis).

Data = read.table(header=TRUE, text="

PowerType Group MeanResponse se

'Force lightning' Jedi 1.1 0.85

'Force lightning' Sith 6.2 0.80

'Light saber' Jedi 5.1 1.00

'Light saber' Sith 4.8 1.30

'Force illusion' Jedi 3.8 0.85

'Force illusion' Sith 3.6 0.92

")

library(ggplot2)

pd = position_dodge(.2)

ggplot(Data, aes(x = PowerType, y = MeanResponse, color = Group)) +

geom_point(shape = 15, size = 4, position = pd) +

geom_errorbar(aes(ymin = MeanResponse - se,

ymax = MeanResponse + se),

width = 0.2, size = 0.7, position = pd) +

theme_bw() +

theme(axis.title = element_text(face = "bold")) +

ylab("Mean power rating\n") +

xlab("\nPower Type")

I like that approach, Sal. But as you noted, only "[i]f the three "power" variables are all commensurate".

"which has the biggest effect"

This can mean a few things. Is it Effect, and in cause? If it is association, is it largest bivariate association or conditional on the others? Once these are addressed the scale issues described above may be applicable. +1 to Sal Mangiafico for Jedi reference!

Daniel Wright the test subjects read one of two texts (randomized, of course), in which a person reacts in a different way. Afterwards they rated the protagonist on three different scales.

Therefore, I would like to find out which of the scales is influenced the most in it's rating by the different reaction/framing. Was that worded understandibly?

Sal Mangiafico thank you for your answer. I, too, highly appreciate the example you decided to choose.

How would I deal with the whiskers/standard deviation? I've already conducted a t-test for each variable (with adding Bonferroni's correction) and received a highly signifikant result for each one. Therefore, none of the whiskers should touch in an illustration like the one you did, correct? Will I just compare the different means and look at the biggest difference? Or will I just examine the difference between the "inner whiskers" (e.g. around 5.4 - 2 = 3.4 in the force lightning example)?

Marie Deublein ,

For my example, the best approach would be to start with a single factorial model (two-way anova in this case). This would give you a p-value for PowerType, TestCondition, and their interaction. The whiskers would probably be derived from the confidence intervals or standard error of the marginal means (or least square means) ---- practically, depending on what software you are using and what output it produces.

Using a single model is probably advantageous for you when compared to using three t-tests.

That being said, there's no reason why you couldn't produce a plot like this given the analysis you already have. You already have the arithmetic means calculated for each of the six cells. You have some options for what to use for the whiskers. Probably the standard error of the mean would be the most appropriate for each. You could also use the confidence interval for each mean. In these cases, yes, you are right, that the whiskers probably will not overlap with each PowerType. You could also use standard deviation for each cell, if that will make more sense for your audience.

Let's assume you measure the size of the effect in the 1-7 scale units (you could divide by sd if you want, or other ways, but ...) and the difference in means is the measure you want. Then find the means for the three variables for the two groups. Subtract them, and then the one with the biggest is showing the largest sample difference. If you are interested in particular inference questions, then you can do things like confidence intervals and ANOVA. I think some of the responses above might be thinking that you were interested in whether the power variables CAUSE different amounts of the group variable, but since this is randomized nothing (hopefully) is causing group allocation other than chance. You might re-word the original question:

Want to finding out on which of three variables my grouping variable has the largest effect.

Sal Mangiafico wrote:

Since the power rating is indicated as the dependent variable, I would advise against flipping the variables to make the test condition the dependent variable, in order to use, e.g. logistic regression.

Sal, the example Harrell used in the excerpt I uploaded was an RCT with two (randomized) groups. So in the usual types of models one might use, Group would always be an explanatory variable, not an outcome variable. Even so, Harrell argued that turning things on their head and making Group the outcome variable in a binary logistic regression model is a sensible thing to do in the context he described. And to address one of Daniel Wright's comments, I don't think Harrell is suggesting at all that the variables he treats as explanatory in his binary logit model cause group membership. (I've attached the excerpt again so that if anyone did not read it earlier, they don't have to go find my earlier post.)

Sal also wrote:

Instead, here's what I think.

If the three "power" variables are all commensurate, they could be viewed as a single measurement, e.g. "Response", measured on each of three power scales.

In my previous post, I said that I liked your suggested analysis if the power variables are commensurate. But I now find myself wondering what you would suggest if they were not commensurate. Would you entertain the binary logistic regression model Harrell recommends in that case? Or are you against it in principle because it makes Group the DV?

To be fair, when I first read Harrell's suggestion about this in the 1st edition of his book, I described it to a senior statistician in the department I worked in at the time, and his initial reaction was that it was wrong-headed, because the error structure would be different with Group as the DV. But after I told him the source of the suggestion was Frank Harrell's book, he backed down a bit. I do know that appeal to authority is not a valid form of argument. But I also know that Harrell has a very solid reputation when it comes to regression models.

Cheers,

Bruce

Bruce Weaver ,

EDIT: I was having some problems getting satisfactory results for the logistic regression method, but I finally figured out that it was related to low variability in my toy data in some way. Dunno. Was strange to me.

Once that was sorted out, yes, I like the logistic regression approach, and it's not bothered if the different measures aren't commensurate. (For example, even if they are similar types of measurements, imagine one is measured is kilograms and the rest are in grams).

My factorial approach won't be satisfactory if the measures aren't commensurate.

I have a couple of questions about the logistic approach though. 1) What statistic is used to compare to see which measure has the larger effect, or however we want to express what we are interested in here? 2) What if there are different numbers of observations for different measures? How to build the model? This would also affect the p-value, if that's our statistic of interest.

In the case of measurements that aren't commensurate, or unequal sample sizes across measures, another approach might be to use the separate t-test approach, and then calculate Cohen's d for each measure, to compare the (standardized) effect size for each measure. I think that would work out okay.

Hi Sal. You raised two questions about Harrell's logistic regression approach.

1) What statistic is used to compare to see which measure has the larger effect, or however we want to express what we are interested in here?

How does one determine the relative importance of explanatory variables in any regression model? I don't think there is general agreement on how to do that. A recent issue of the Stata Journal included an article on "dominance analysis", which is one way to address the question, I guess.

Article Determining relative importance in Stata using dominance ana...

And for you useRs:

• https://cran.r-project.org/web/packages/dominanceanalysis/dominanceanalysis.pdf

2) What if there are different numbers of observations for different measures?

Bearing in mind that Harrell's example was a two-group RCT, I think he was considering the situation where there is no missing data. I've never read anything about missing data in this particular context. But I suppose that multiple imputation could be used, assuming the missing data were missing at random (MAR) or missing completely at random (MCAR). YMMV. ;-)

As already noted in the comments above, logistic regression and flipping dependent and independent variables is certainly no standard procedure in this type of study. But without knowing more about the three dependent variables, their measurement properties and their statistical as well as hypothesized/theoretical relations, more concrete and appropriate suggestions are not that easy. Considering the discussion up to here, I think a more descriptive standard approach could be more appropriate.

Assuming that there is no strong hypothesis, which DVs respond most to the test conditions, the separate calculation of Cohen's d is certainly a good idea. This would offer not only reasonable and comparable estimates of effect size, but also a well interpretable standard statistic for the question here.

Have you considered using the full information model in, e.g., LISREL or other modeling programs, to create a new dependent variable (perceived power) and then test the effect of group on that new measure. One of the outcomes of these models is the total effect, which will give you a coefficient that describes the differential effect of group on each specific outcome measure that make up the composite dependent variable. This approach will also let you know if the assumption that the three measures create a composite scale is also a valid assumption.

I suppose to do a canonical regression if other variables support it. If you are running a regression then better to use the beta coefficients.