I am currently conducting my final year dissertation which is looking at the efficacy of common police lineup practices. However, before I launch my study, I have put out a pilot study which is assessing whether any of the photos I have selected for the lineup are incongruous for any reason, or if my target photos are more salient or conspicuous than the others, which may cause them to be selected at a higher rate.
In my pilot study, participants viewed the same lineups as will be in the final study (8 lineups in total, with 6 photos per lineup), but were asked to indicate whether any of the individuals within the lineups stand out for ANY reason, such as the quality of the photo compared to the others, the background of the photos or any other reason. If participants selected a lineup member, they were also asked to state why they chose that specific member.
Looking at the results, I can see by tallying up responses that they are not being selected at a higher rate than the others, however, I am unsure of how to correctly analyse the data as I know a tally isn't an appropriate method of analysis. Would I need to input these results into SPSS for example, or is there a more simple approach?
Thank you in advance!
This is a two part solution (in my opinion).
The first part would be to use the free text responses to extract a discreet list of predictive factors. If you have open ended text responses ("describe why") then it would be possible to build a list of common reasons given for making a selection from the comments.
If you treat "Photo Selection" (or not) as the Independent Variable (Selected = 1 if Yes, else = 0 if No) this would give you the second part needed: a target variable to be predicted. The dichotomous binary (yes/no) variable lends itself to several different predictive models. If you take the list of common reasons and dichotomize them (yes/no), this would be a classic Binary Logistic Regression (BLR) of Selected (Yes/No) using dichotomous reasons for selection as predictors (color, sharpness of focus, background, race of the person in the photo, Age, piercings, tattoos, and so on). Also it would be advisable to control for the possible co-variate effects of the demographics of the person making the selection (age, race, gender, educational attainment, socio-economic status, etc.).
However, BLR requires approximately 20 observations (20 rows of data) for each variable in the model. If you have five categorical dichotomous predictors in your model, you will need at least 100 observations of the selection process. What you describe above is 6 x 8 = 48 observations of an individual photo being selected (or not). If you had 10 people, each viewing 48 individual pictures, that would be 480 observations, and that would likely result in a more robust model with more Dependent Predictors possible. What you will likely find, is that some of the variables in the model can be "trimmed" (dropped from the model) and this will improve the overall "c-statistic" of the model (Area Under the Receiver Operating Characteristic Curve) as it may become progressively more powerful as variables that do not contribute to model improvement are trimmed. The goal (as with all predictive models) is to achieve maximum discrimination (between Yes/No) with the fewest number of dependent variable predictive factors.
I hope this helps.
Being a pilot used to inform the development of the stimuli for your actual study, I think it's fine to focus mainly on the free text responses to choose which stimuli to keep, without worrying too much about using statistical methodologies. If you want to add another layer of control, you may ask a second independent reviewer to have a look at the participants' responses and choose which stimuli to keep, and then compare their choices with yours.
In case you really want to provide evidence for the claim that there are no differences among your pictures you might want to consider a multinomial test. However, you would not want to use frequentist approaches (i.e., chi-square test ) because these do not provide evidence for your null-hypothesis.
You then might want to use a bayesian multinomial test. I recommend using JASP which is similar to SPSS but can perform also bayesian tests and is free to use:
https://jasp-stats.org/wp-content/uploads/2020/11/Statistical-Analysis-in-JASP-A-Students-Guide-v14-Nov2020.pdf
Hi Matthew - could you please share key pointers of your research design, sampling procedure, and the criteria you applied in implementing 8 total lineups with 6 photos per lineup? In addition, please indicate if your data collection instrument contains questions with closed or open response options? Such pointers would help in the appropriate decision criteria.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS