Could you perhaps clarify a few things about your question for me?

Firstly, you write "Our hypothesis is that a certain psychological condition will be associated with increased "Yes" responses for the Unrelated and Remotely related trials (whereas the condition has no effect on the closely related trials)."

Conceptually, do you think that this is because patients cannot differ between related and unrelated trials (i.e., lower discriminability), or because they have a bias towards judging trials as related (i.e., lower criterion)?

Secondly, I am not sure how you plan to calculate SDT measures when one of your independent variables seems to determine whether something is a signal or non-signal trial. To calculate any SDT measure, you need both signal and non-signal trials. So you could calculate a d' or c per participant by including both Remote and Close as signal trials, but it would not be possible to, for example, calculate a d' for just the Close trials. Is this something you aimed to do? Or did you want to calculate one number per participant, across all trials?

Lastly, you write: "However, since we a priori expect that both "Unrelated" and "Remote" would be higher, the standard d' parameter might not be suitable (i.e., if both %H and %FA go up)."

What exactly do you mean by 'would be higher'? Do you mean the number of 'yes' answers would be higher? And how could there be a false alarm when these trials are, by definition, signal trials?

Without answers to these questions I am unable to provide an answer with confidence, but so far it appears to me that the Signal Detection framework may not be suitable for your design. It may make more sense to treat the percentage of 'yes' answers as your dependent variable, and use nonparametric tests to account for the nature of this variable.

Hello Martin,

You have an an interesting experiment.

I don't think that you can cast the analysis and interpretation of your entire experiment into the theory of signal detection amid noise.

In order to do that, you need to clearly define a consistent “target” class (signal sought) and a “noise” class (signal to be rejected). In signal detection theory, the subject is assessed for their ability to be CORRECT about the presence or absence of a target.

By extension to your case, I would say that the target is to be CORRECT about sematic relations, and a HIT should be registered when the subject is CORRECT about semantic relations. This means that the target is not always a YES response (because it is obviously the incorrect response for UNRELATED word pairs), and a HIT is not always YES response (as your example of hit rates seems to have it).

Elaborating further along this line:

So far as detection theory goes, moreover, your example hit-rate data should actually be the CORRECT-rate:

Healthy: Unrelated = 90 %, Remote = N/A, Close = 90 %

Patient: Unrelated = 70 %, Remote = N/A, Close = 90 %

(N/A = not applicable)

The only data amenable to signal detection theory analysis (and its d parameter) is then the UNRELATED word-pairing. The patient has a lower sensitivity setting on their sematic-proximity-detector than healthy people when given UNRELATED words, which is what your hypothesis is saying.

But your example hit-data (in which YES responses constitute a hit) clearly supports your hypothesis more than that. Even the REMOTE data supports your hypothesis, because it is the subject's inclination to respond YES that is ultimately at issue in your hypothesis rather than the subject's ability to be correct (which is the domain of application for signal detection theory).

The best approach in the analysis of your entire data set may therefore be hypothesis testing rather than signal detection theory. The application of hypothesis testing in this experiment requires further thought.

Ronald

Dear Sarah and Ronald,

I am very grateful for your detailed and insightful response!

To be precise, we expect that patients judge trials as related when they are not (“Life–Spoon”) or when the relation is weak (“Life – Scythe”), but they are not biased when the relation is strong (“Life-Death”). So I think about it as a kind of discriminability (patients have less discrimination because they hit on noise and remote relations). The expected patter is like this:

Condition: %YES Healthy / %YES Patients

Close: 90% / 90%

Remote: 30% / 50%

Unrelated: 10% / 30%

Thus, we expect some kind of “indiscriminability” in patients.

All participants went through all three trial conditions. I would like to have “Close” as something fixed – a “baseline” so I can calculate SDT(-like) measures for Remote (Remote [coded as noise] against Close [coded as signal]) and Unrelated (Unrelated [coded as noise] against Close [coded as signal]). I understand that the interpretation of the former would be presumably different from the later, but this is actually the question – whether this coding make sense or I should approach it differently.

So as I see it, I think I will try to split the three conditions into three separate analyses:

(1) Unrelated (noise) vs Related (signal)

(2) Remote (noise) vs Related (signal) – keeping in mind that “noise” is something different than in (1).

(3) Unrelated (noise) vs Remote (signal) – keeping in mind that the “signal” means something different than in (1) and (2).

Of course, I will also test the hypothesis using Logistic regression (having “Signal strength” as the independent variable) to estimate an intercept and a regression parameter for each subject and perhaps use this estimates in the further analyses.

Thank you for your thoughts and any further advice,

Best,

M.

I think you need a different approach. I would suggest that you separate your experiment into 2 parts:

1. Hypothesis Testing:

=================

The contrast you ultimately want to draw in this experiment is PATIENT vs HEALTHY, and you will make this comparison under three different conditions CLOSE, REMOTE, UNRELATED. Those three conditions can be treated independently, with the biggest patient-healthy contrast occurring for UNRELATED, and the least contrast for CLOSE.

Your hypothesis (H1) is that patients have notably less syntactic discrimination than healthy. The null hypothesis (H0) to be rejected before you can assert your hypothesis is that patients and healthy subjects have the same syntactic discrimination performance.

H0 cannot be rejected for CLOSE because mean patient and healthy scores are the same. If your subject sample size is large enough, however, then H0 can be rejected for UNRELATED, and it might be rejected with less confidence for REMOTE, and you can in either case go on to assert your intended H1. The CLOSE data set gives you no information to support H1 because it fails to discriminate patient vs healthy.

That is roughly the route of hypothesis testing. It can take a number of variant forms.

2. Detection Theory

===============

You also want to assign a syntactic performance score to healthy and patients, and contrast patient vs healthy performance score. It is the design of this score that is being addressed by the d parameter in detection theory.

REMOTE data cannot be used in detection theory because it has no ground truth (no generally accepted consensus about what is the right answer, yes or no, in any REMOTE trial). So ignore the REMOTE data here.

Use your data to estimate the following for EACH subject considered separately :

PD_H = prob of detection for healthy = portion of YES for CLOSE

PFA_H = prob of false alarm for healthy = portion of YES for UNRELATED

PD_P = prob of detection for patients = portion of YES for CLOSE

PFA_P = prob of false alarm for patients = portion of YES for UNRELATED

Plot points (PD_H,PFA_H) on a PD vs PFA for all healthy subjects.

Plot points (PD_P,PFA_P) on another PD vs PFA for all patients.

Fit a parametric curve (in the least square error sense) through the points for healthy subjects. The curve to be fit is the receiver operating characteristic (ROC) curve for a given d parameter assuming Gaussian distributions for target and non-target classes (as usually done in detection theory applied to perception). Let D_H be the d parameter that gives the best fit to the healthy subjects.

Do the same for patients. Let D_P be the d parameter that gives the best fit to the patients.

Now you have a best-fit d parameter for the set of healthy subjects and another for the set of patients. Your hypothesis H1 requires D_H > D_P, as it will be for the data that you have for UNRELATED trials. D_H and D_P will give an indication of the each subject class (healthy and patients) to make syntactic discrimination correctly.

Ronald

Roland,

much thanks, I really appreciate your help and definitely going to try it!

M.