IYH Dear Adam Clifton

Based on your description, I suggest applying Generalized Linear Mixed Models (GLMM) followed by post hoc pairwise comparisons with Bonferroni adjustment. This approach allows you to incorporate multiple responses from the same participant, handle categorical variables, and determine statistically significant differences between conditions.

Example how you could structure the GLMM:

Once fitted, execute post hoc pairwise comparisons between HRTF types to investigate significant preferences. As suggested, I would adjust p-values accordingly using Bonferroni corrections to maintain strict control over family-wise error rates.

Additionally, calculate the proportion of consistent selections made by each participant for each condition, excluding those who consistently chose the same option regardless of presentation. Then run chi-square goodness-of-fit tests comparing expected frequencies (equal distribution) versus observed distributions derived from participants' consistent preferences.

Kindly be aware (or your prof may tell you :D) that your current sample size is relatively small, drawing definitive conclusions is still possible after properly accounting for dependencies and handling multiple comparisons.

Maybe a word on Bonferroni: The so-called Bonferroni adjustment serves to counteract the issue of multiple comparisons, reducing the chance of falsely rejecting null hypotheses. Essentially, it establishes stricter criteria for concluding statistical significance, thereby maintaining appropriate error rates in the presence of multiple tests.

Why is this needed? When executing post hoc pairwise comparisons after fitting a GLMM, running multiple tests simultaneously, increases the risk of encountering false positives. To mitigate this risk, we apply a corrected alpha level determined by dividing the desired family-wise error rate (Type I error rate applicable to the entire set of comparisons) by the number of tests performed.

Typically, you would choose α=0.05 or α=0.01 as baseline significance threshold. In cases w multiple comparisons, we set α_bonferroni=(α/number of tests). For example, suppose you intend to compare six means with a baseline α=0.05. In that situation, you'd compute α_bonferroni=0.05/6≈0.0083, meaning that you now require stronger evidence (smaller p-value) to claim significance.

Bonferroni is a safe conservative choice (however sometimes too much so, especially when tests exhibit low dependence) However, it guarantees strict control of the family-wise error rate. Other alternatives are Šídák or Holm–Šídák: Compared to Bonferroni, both Šídák and Holm–Šídák demonstrate reduced conservativeness ie lesser reduction in statistical power. In practical applications, choosing between Šídák and Holm–Šídák depends on the degree of dependence amongst the tests, the cost of committing type I or type II errors, and the complexity of computations tolerated.

Maybe you could use one-sample tests of propotions, i.e. you test if the observed proportions are different from the expected.

In attachment an R script for such a test.

In SAS :

data a;

input pictures $ freq;

cards;

pic1 50

pic2 27

pic3 23

;

proc freq data=a;

weight freq;

tables pictures / testp = (33.33 33.33 33.33)

testp = (50 27 23);

Thank you both for your answers! I'll try my best to answer you both separately.

Daniyel Yaacov Bilar I agree with the chi-square goodness of fit [G.O.F.] tests comparing the expected and obtained frequencies.

I'm a little unsure with GLMM, and I've been trying to get my head around it the past few days. My statistics knowledge is not the best and after going to a drop in session at my uni the other day, they were also unsure whether this would be the right path to go down. As the option the listener selected (the HRTF) WAS dependent on the distance and voice stimulus used, I didn't control which option they picked. They did not give a value of plausibility for each, they just picked the one they thought was the most plausible and then gave it a score. So I don't know if it's as independent as my initial post suggested?

If you've got any recommended reading or videos that explain the process of GLMM I should go through in more detail, I'd be very grateful. Perhaps I've missed a very useful resource. I'm struggling to understand it in SPSS at the moment too.

I'm also a little unsure on completely disregarding the listeners who gave the same option multiple times? Some of my lecturers have said to be very careful when doing so stating good reasons as to why, and as this does not have a standard, like the ITU-R BS.1534 MUSHRA. Which suggests "to exclude a participant from the data if they rate the (maximum quality) hidden reference lower than a score of 90 for more than 15% of ratings". I'm a little cautious to reject those listeners. If, again, there's a resource online that I've missed I'd be grateful to be pointed in the right direction.

Jos Feys I agree with the chi one sample (I think it's very similar to a G.O.F. test or the same thing?) I've been doing a general overall G.O.F. for all the frequencies combined, and then breaking it down into the various independent variable (of 0.25male, 0.25female, 0.5male, 0.5female). Weirdly enough there's a significant result when they're all combined but no result when it's done separately? Perhaps this is a result of including the listeners who gave the same option as mentioned above. I intend to do this test again and exclude their results this time to see if there's a noticeable difference

Thank you both for your help! I think I'm heading in a better direction with it now.

IYH Dear Adam Clifton I made you a LLM AI bot tutor for GMML .. try it, just ask the bot questions like you would ask a human tutor, and the bot will customize answers for you. https://hf.co/chat/r/-oByrif