I'd like to compare results from different MVPA procedures. In particular, I used cross-set decoding (i.e., support vector classification) and cross-validated prediction (i.e., support vector regression). Thus, from the first type of model I get a measure of classification accuracy (minus chance), while from the second type I get a measure of prediction accuracy (z-scored correlations). I want to test whether the accuracy of my models differs in a set of ROIs and, if so, which model has the highest accuracy in each ROI. What would be a proper way to compare these two different accuracy measures (i.e., how should I transform one measure to make it comparable to the other)? I'd really appreciate any reference to relevant method papers or practical advice. I ran my analyses with The Decoding Toolbox (Hebart, Görgen, & Haynes, 2015). Thanks in advance!
My answer would be: You don't. If you want to compare two models use a measure that is descriptive of the performance of both models. I'd suggest using Carsten Allefeld's cvMANOVA which provides you with D which is a measure of explained variance which can be used to evaluate both categorical and regression models. Whether a particular comparison of two models is 'fair' (in terms of model complexity etc.) is a different question, however.
Indeed, it wasn't a good idea to compare the output of the three models since they look for different patterns in the data (i.e., they rely on different types of similarity). In the end, I reconsidered the classification models and translated them in a SVR procedure that was comparable with the third model. Now, I have values in the same (z-scored) metrics that I can compare.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training