With a discrimination task I know how to work out whether performance for a particular participant is better than chance or not. This is just done by computing the observed accuracy with the expected frequency of correct responses based on chance (dependent on the number of response options in the task) with the number of observed correct responses.
However with a signal-detection task I am not sure what to do. How does one determine if, for instance, a d-prime of 0.04 indicates responding that is better than would be expected by chance or not? I don't think that one can use the normal binomial formula (as with a simple 2AFC discrimination task) because the number of given present and absent trials given is not necessarily equal in a detection tasks; the expected frequencies of a random responder would also presumably differ depending on their level of bias towards 'yes' or 'no'. So the whole thing is far more complicated. Presumably this is a fairly common issue with detection tasks-so there must be a formula somewhere to deal with it..
I agree with both Andrew and Daniel. If you want to determine whether an individual's d' is "significantly different from 0" (or: determine the probability that this score was observed under the null hypothesis of chance performance), and don't want to make assumptions about the underlying distribution of the data (either the statistics themselves, or the hit/false alarm rates), a good way to go is to compute non-parametric bootstrap confidence intervals. Here is a paper that does this:
An evaluation of psychophysical models of auditory change perception. Micheyl, Christophe; Kaernbach, Christian; Demany, Laurent Psychological Review, Vol 115(4), Oct 2008, 1069-1083. http://dx.doi.org/10.1037/a0013572
However, if your goal is to filter out participants who performed poorly in order to get a significant p-value, this is not a good approach. Firstly, in practice removing data is rarely strengthens a statistical test, and in the worst cases could be considered cherry picking your data. Secondly, you say the subjects performed at floor "in one of the conditions." If you are talking about the condition in which you expected to find poor performance, I don't see how removing those individuals who performed the worst is going to help you. If you're talking about another condition, this would suggest that there truly isn't an interaction in your data.
It sounds to me like you need more data. Is it possible to collect more?
Some details about what the data are like that you used to estimate your d' value would be useful. But, if it is just hits, false alarms, correct rejections, and misses, you just have a 2x2 contingency table for that individual. There are lots of arguments about what p value to use (e.g., Fisher's exact, Yates' correction, Pearson's chi-sq), but if this is the situation which gave rise to your d', then this group of statistics would be worth exploring. Of course other sorts of data are used to create d' statistics, so if it is some other circumstance please proved more details.
The bigger point though, is why do you want to calculate whether individuals' d' values are different from 0? I can understand why you might suppose the sample was drawn from a population with d' ~ N(0,sigma) and want to explore this. This is why multilevel probit (or logit) models are often used in this circumstance. I guess this is just a different field than mine, but I would be interested to know of situations where this makes sense.
Thanks Andrew and Daniel for your advice.
@Daniel Wright. Yes my data is in a standard form with a 2x2 contingency table. What I am trying to do is re-analyse some old data I have on a signal detection task (data set contains about 25 participants). In this data I failed to find a significant 2 way interaction in an ANOVA analysis of d-prime scores (an interaction I expected to find). However for quite a few participants d-primes in one condition in particular were fairly near to zero. My concern is that the failure to obtain the statistical interaction was as a result of this possible 'floor effect' in the data with a number of participants. It may have been that- had the detection task been easier - a significant interaction might have resulted because the data would not be restricted by floor level.
What I want to do is re-analyse the data I already have in a principled way to see if floor effects on the task were an issue. I want to take out any participant I can identify who is not performing significantly differently from chance in any condition and then re-run the ANOVA on the subset of participants who pass this criterion. It is for this reason of screening out participants that I asked the question. I'm not sure if there is a more sensible way about achieving my aim, but I would be happy to hear if there is.
Hi Michael,
I would avoid that. First, you would be using "significance test" more like a measure of effect size (on effect size measures for the individual, there are a bunch of them in addition to d', see Table 2 of the paper below). If you want to see if the d' values come from a distribution that is a mixture of two distributions, there are methods for this. Do you have a lot of data for each individual? In general removing data from only one end of a distribution should be done cautiously if at all. There are procedures for removing data from both ends or lowering the impact of extreme data. Rand Wilcox has several excellent books on this.
It might be worth considering the approach described in the second have of the paper attached.
Happy modelling,
Dan
Article Functions for traditional and multilevel approaches to signa...
I agree with both Andrew and Daniel. If you want to determine whether an individual's d' is "significantly different from 0" (or: determine the probability that this score was observed under the null hypothesis of chance performance), and don't want to make assumptions about the underlying distribution of the data (either the statistics themselves, or the hit/false alarm rates), a good way to go is to compute non-parametric bootstrap confidence intervals. Here is a paper that does this:
An evaluation of psychophysical models of auditory change perception. Micheyl, Christophe; Kaernbach, Christian; Demany, Laurent Psychological Review, Vol 115(4), Oct 2008, 1069-1083. http://dx.doi.org/10.1037/a0013572
However, if your goal is to filter out participants who performed poorly in order to get a significant p-value, this is not a good approach. Firstly, in practice removing data is rarely strengthens a statistical test, and in the worst cases could be considered cherry picking your data. Secondly, you say the subjects performed at floor "in one of the conditions." If you are talking about the condition in which you expected to find poor performance, I don't see how removing those individuals who performed the worst is going to help you. If you're talking about another condition, this would suggest that there truly isn't an interaction in your data.
It sounds to me like you need more data. Is it possible to collect more?
To determine if the d values for each observer in each condition is significantly greater than zero (chance), you can use Marascuilo's test ( Marascuilo, L.A., 1970. Extensions of the significance test for one parameter signal detection hypotheses. Psychometrika 35, 237–243).
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics