Were the questions answered at two distinct time points, or were they interspersed (and are the questions ones that people would remember the response for?). Suppose the data are

trial time quesno persno answer

1 1 1 1 c

2 2 1 1 c

3 1 2 1 a

4 2 2 1 d

I don't know what the ratings are so I left that categorical. Are you treating this a multilevel model with question, person, and time as random variables?

Hello Daniel Wright , thank you very much for your comment!

The responses were acquired using a rating scale from -5 to 5 and represent the annoyance of noises. They were repeated within the same test unknowingly since it is very unlikely that the participant will remember a specific sound thus his response to it.

I actually calculated two models:

I averaged the two responses from both trials and then averaged again for each noise, so that in the end I would only have 15 observations. I also checked the ICC before using aggregated data and interrater reliability was very high.

I used for the aggregated data (15 responses) a linear regression model 'Responses~x1+x2+x3'.

But for the raw data (10x15 responses after trial averaging) a model with crossed random effects "Responses~x1+x2+x3+(1|noises)+(1|raters)"

So I was hoping now to calculate the SEM and SEP for the CIs and see if the second trial responses are influential or not.

I hope this helps!

Not sure what the x1, x2, x3. What are they?

I'd avoid the trial averaging unless everyone does in fact give the same values. If you are willing to make a few assumptions, the typical approach would be in R would be a partitioning variance model like:

library(lme4)

m1

The Xs are the psychoacoustic quantities.

The ICCs(A,k) for each participant resulted in over 0.9 with an exception of one participant who had 0.82.

So to check for unreliability between trials, one should remove the parameters (Xs) and regress with only the grouping variables for each trial (2 models then)?

I am working with Matlab but I guess the same stats will be generated..

Thank you for the article, just requested it! Statistics is not my field at all, so thank you for help!