Ifigenia Aslanidou Can you elaborate a bit what you mean by positive and negative ratings and positive/negative models? Are participants rating something and you've separated positive (favorable) ratings and negative (unfavorable) ratings and analyzed each subset of data separately? Thanks.

Hi, Blaine Tomkins

I am using a bipolar likert scale from -5 to +5 to rate noice annoyance (-5 would be most annoying than a reference noice, and +5 most comfortable). The ratings consist of negative and positive numbers, so I thought of creating another two models (linear regression), one considering only the negative ratings and one only the positive. Maybe this could give me some information if the ratings where well distributed along the negative range and the positive range as well. I hope that helps a bit more..

Ifigenia Aslanidou It's generally not a good idea to split your dependent variable scale. By only considering scores at a select interval of the DV (called restriction of range) and examining separate models, you can easily be mislead to incorrect conclusions. For instance, this can produce illusory correlations (correlations that are only observed in a subset of the data). It can also cause you to miss a correlation that is present with a model containing a full range of scores (but not in either subset of scores).

Ifigenia -

You said that "...the combination of them has the highest p-value." Larger sample sizes generally decrease p-values. I don't encourage looking at p-values at all, but this is curious. Do you somehow have a larger standard error with a larger sample size? That is odd.

I agree with Blaine, by the way, that you don't generally want to split your data in this manner, though the experiment seems to have been interesting. It has something to do with your respondents and why you can't always use an 'average' to impute for a missing value, I'd say.

Cheers - Jim

Thank you both for your answers!

These were some of the results:

Negative and positive data (15 samples): mse=0.0479, rsquare_adj=0.944, pval=9.4364*10^-8

Only positive data (7 samples): mse=0.0468, rsquare_adj=0.813, pval=0.0592

Only negative data (8 samples): mse=0.0094, rsquare_adj=0.965, pval=7.09*10^-4

Greetings

Ifigenia Aslanidou The R2 values seem remarkably high. How many predictors did you include? The model might be overfit to the data.

I have used 3 predictors. Using only the first 2 predictors the rsquare_adj reached 0.9174.

Is there anyway to identify this overfitting?

I think its important to mention that I normalized both the independent and dependent variables and then averaged the dependent one.

Without normalizing and averaging, the rsquare_adj reached 0.76.

When you say 'normalized' do you mean you transformed the data somehow? If so, which transformation did you use?

This sound very vague, can you please just provide the models and the complete results with an explanation what the dependent and independent variables were??

Blaine Tomkins , I used the command in Matlab , normalize(A) which returns the vectorwise z-score of the data in A with center 0 and standard deviation 1.

Rainer Duesing , as I mentioned above, I am rating the noise annoyance (so the rating result is my dependent variable).

I had 10 test persons who rated 15 noises. So lets say I have my rating matrix, in which a row represents a test person and each column the noises (rating matrix 10x15). I normalized along each row (for each test person) and then calculated the average rating of each noise.

For the independent variables, I have 3 predictors or else psychoacoustic parameters of the noises, which I also normalized since they are different.

What kind of results would be helpful to your answer?

Greetings

Blaine Tomkins I also calculated the predicted rsquare which was close to the actual one. So I guess there is no overfitting..?

This really does not make any sense to me and it remains unclear a) what you are trying to achieve and b) what the statistical model was, please show your regression formula for example.

1) What was the rationale to z-standardize within each person?? You said in your first post something about "positive and negative ratings", but for whitin z-scores, this does not make any sense, unless you specifically asking for positive and negative deviations from the ipsative mean value. For example, participant one has the raw ratings -5, -4, -3. After z-transformation this becomes -1, 0, 1. The next participant has the raw ratings 3, 4, 5, after transformation, this also becomes -1, 0, 1. So, what does this tell you? Only the relative postions within the participants, but not the absolute valence. Is that what you want?

2) What is now the "average rating of each noise"? This cant be the mean of all ratings, since this would be across all noises. So, for what did you calculate the mean of each (of the 15) noises and what did you do with it?? If you calculated the mean across the z-values of all 10 participants, the mean would only indicate, which noise would have the lowest relative rating. E.g. all participants would give the same rating 3, 4, 5, then the first noise would be positve +3 , i.e. a pleasant one, but on z-scores it would be the most unpleasant noise of all with -1, but you cannot infer that this is unpleasant, since you shifted the mean. Therefore, a negative z-score could still indicate a pleasant stimulus.

3) What about the IVs? What are "predictors or else psychoacoustic parameters"? How many? How were they normalized? Also within or between participants?

See, your description is absolutely vague. Please share concrete information and maybe so screenshots.

Rainer Duesing the goal is to find a connection between subjective and objective perception of noises. I can't really show you the formula or the betas since it's an ongoing project.

The goal of normalization is to get comparable betas and find out the relative effect of the 3 psychoacoustic measures (X1,X2,X3). About splitting the ratings in positive and negative, I realise now that you are right and its not something that can be done using normalized data but only the original or maybe averaged ones (averaging ratings of each noise). But since it was mentioned that it can be misleading, I will not take it into consideration..

The ratings were normalized within each test person and the IVs (X1,X2,X3) within each X.

e.g DV or ratings [10x15] normalizing along each row

IVs [15x3] normalizing along each column

I hope its more clear now.

My last attempt: it is stil not clear what your statistical model is! It really doesnt matter if you explicitly write down the variables you have or just generic names to express your model as formula, if you do not want to openly share your specific design. I really wonder: How did you incorporate the 15 ratings?? Did you calculate 15 different models? One multivariate model? A multilevel model? The latter one for example could be interesting, because you have repeated measures and I assume that the stimuli have some property (e.g. loudness) in which you can order them (although your sample size is so small that I would not really trust any model, but that is a completely other question and you said that you were only trying something out).

After mentioning the betas: why are you explicitly interested in the standardized metric and not the raw metric? Standardized regression coefficients are also dependent on the variance of the predictors, therefore, it is not true that you can use them to assess the "importance" of the IVs or which is "best" (c.f. Darlington & Hayes, 2016 for example).

I thought it was clear that I created one multiple linear regression model which looks like this: y=bo+b1x1+b2x2+b3x3

About the betas:

Thank you for mentioning that.

I standardized because its easier to interpret the effect on the DV.

So the importance of the IV is not obtained because of the difference in standard deviations of the IV? Why do I then get better Rsquares?

Is it a better alternative if I compared the adjusted R squares of 3 different models to see which IV 'matters' most? e.g. 1. only with x1 2. only with x1,x2 and 3. x1,x2,x3

I am afraid, your doing something fundamentally wrong! The R square does not change, if you use unstandardized or standardized variables, nor do the inferential tests for the predictors change, the only thing that changes are the regression coefficients. I think this comes from your wrong "standardization" process!!

And you still did not clarify, how you incorporated your 15 ratings. I understood it the way, that the ratings of the stimuli were the dependent variable(s), but since you have 15, did you calculate 15 regressions??

I would highly suggest to consult a local statistician to help you with your problem, since you seem to lack some fundamental knowledge.

Besides that: if you want to "compare" the importance of different predictors, it is not unseful to calculate separate models for each predictor, since you miss their interrelation. One approach would be semiparial correlation coefficients or dominance analysis. But please, additionally to my advice concerning a local statistician, you may have a look at my reference in the last post Darlington & Hayes (2017) about this topic.

Darlington, R. B., & Hayes, A. F. (2017). Regression analysis and linear models. New York, NY: Guilford.

Well, between the original data and standardized data I actually have a small difference in the Rsquare (0.76 and 0.82), in comparison to the averaged data and standardized-averaged data where the Rsquare stays the same (0.95).

Let me explain what I mean by original/averaged data:

Original data

Rating data or DV: 10 test persons rated 15 noises, which gives a matrix of 10x15.

Since I have 10 test persons I reversed each row into a column, having eventually a matrix of 150x1.

IVs : normally a matrix of 15x3 (since I have 3 IVs). In order to match the DV, I repeated the IV values vertically, which gives a matrix of 150x3.

Average DV data

DV: Averages of each 10 ratings, which gives a matrix of 15x1 (so I'm averaging the 10 ratings of each noise)

I hope this gives you a better overview..

Thank you for your reference, I am looking into it!

It seems like your "Original data" approach violates the independent assumption of linear regression as you will likely have non-zero correlations between observations from the same person. It would be more appropriate to use a repeated measures ANOVA or a multilevel model (also known as linear mixed model) to account for the nesting of observations within person. (The multilevel is more flexible and to be preferred if the IVs are continuous).

The averaged data don't have this problem, though arguably the R^2 inflates the proportion of variance explained artificially as you have analysed means per person (which necessarily ignores some of the variability in the raw data). This is is an OK approach in a sense unless you interpret as the 95% variance accounted for as variation in the raw scores rather than the means.

I'm presuming the standardisation is of the raw scores within subjects and hence standardised averages or averages may well be identical except for scaling.

To add to Thom's answer: your original data approach is wrong, not only because you ignore the dependecies/correlations between your data (the ratings within each person are more correlated, as between persons and the linear model assumes independence, as Thom mentioned), but if you do not consider this (as in a multilevel model, which I also mentioned above) your model will treat your data, as if they were from 150 independent participants, where you only have 10!! That is not how it works!

The averaged version, with a resulting 15x4 matrix seems more appropriate, but I would surely check for the inter-rater agreement/reliability before blindly using the data/ratings. If the raters do not agree upon the evaluation of the specific stimuli, what does the average rating tell you? I would say "not much" For metric ratings, it is recommended to use e.g an Intraclass Correlation Coefficient (ICC), but there are several available and to decide, which one is appropriate for your application is a completely other issue.....

Oh, this was not my intention at all. Thom Baguley Rainer Duesing

Thank you so much for pointing out my mistake. I am really grateful for your help!

I did some research on ICC, and would like to hear if my train of thought is correct:

1. Interrater reliability study , 2. Same set of raters for all subjects, 3. Randomized sample of raters -> Two-way random effects, 4. Measurement protocol based on mean of multiple raters (since I want to consider the averaged data) ->Mean of k-raters, 5. Consistency is more important. (Formula mentioned below in the picture)

Using this logic, I got an ICC of 0.9804, whereas using the absolute agreement, 0.9742. This should be enough to be able to use the averaged data, right?

Worth-mentioning: In the experiment, the raters had to unknowingly rate all the noises twice cause I wanted to check for consistency. I wasnt expecting from the raters to be able to rate exactly the same all noises, since the sequence of the noises in each session was random and there could be numerous factors to take into account (e.g. maybe concentration?).

The original data are therefore the result of the mean of each raters' double ratings. It seems that in the second rating session the negative ratings increased a little.. Does this averaging also hinders getting reliable results?

Source of the picture: Article A Guideline of Selecting and Reporting Intraclass Correlatio...

Just to tack on a bit more to what others have implied above. Recall that converting raw scores to standardized (z) scores does not change the shape of the distribution. That is, it will not make non-normal data more normal.

Also, it may be a mistake to normalize this type of data. Psychoacoustic judgments are rarely normal or linear. More often they approximate a power function or sigma function. Take a look at the plots to see if this is the case.

Rainer Duesing, ifigenia says she normalized within each person, did I get that wrong?

so it makes sense that she gets different coefficients of determination, the models will not even be the same:

could it be that she is actually removing the within effects, as you would using a within-estimator.

Ifigenia Aslanidou that transformation will make your inference invalid because you are not taking into account the variation of that (the algorithm will treat that as fixed).

• a pvalue of 0.04 for the F test is actually pretty high if you ask me, and it really easy to obtain when the data is pure noise
• you say that you get "the best" results when you perform certain transformations. These measures (especially pvalues), should not be used iteratively to select a model.
• You should fit a model that is statistically appropriate and draw conclusions from the estimates (i.e. bis/small effect) and their variation, not fishing for pvalues or Rsquared

Blaine Tomkins ,

thanks for your comment. Plotting each three predictors against the ratings (raw data), looks like I am getting respectively a linear distribution.. Only with the first predictor I tried both a linear and a quadratic fit (polynomial curve fitting)..

Stefano Nembrini thank you for making aware of this! But I am bit puzzled though, what do those stats (p val, R2) then offer you? I am acquainted with their meaning, but not how to use them. Are these just secondary in importance to the variances and just shouldnt be used for comparison?

Greetings

Ifigenia Aslanidou There doesn't appear to be much of a relationship between predictors 1 & 2 and the DV. Perhaps plotting the DV (ratings) on the Y-axis will help illustrate effects more clearly. Out of curiosity - what were the predictors?

Blaine Tomkins wouldnt this mean that their contribution alone isnt that significant in contrast to the combination of them?

I am not sure I can reveal the name of the predictors, but I would be glad to ask if it's permitted, if doing so changes anything.

Greetings

I am confused by your figures. Didnt you say that you plotted the predictors against the raw ratings (your DV)? Why is the scale of the y axis changing all the time? If you have three predictors on the same DV, the y axis should be the same scale. Or did you put the DV on the x-axis? If yes: "why" and then your regression lines do not make any sense, since the regression from x on y is different from y on x. There are so many inconsistencies that I really doubt that your approach is correct or that you have found anything meaningful. I really suggest to consult a local statistician, because I would not rely on the correctness of your approach.