Hello,
I am working on machine learning algorithms using EMG and IMU (Inputs) to map joint angles (outputs). I have used RMSE to evaluate model performance because RMSE penalizes outliers more heavily compared to MAE.
However, I am also curious if there are metrics to determine the performance of the predicted slopes (flexion vs. extension). For example, if a user is performing knee flexion, then the predicted slope should follow the same direction as knee flexion. Conversely, if the predicted slope was in the opposite direction of knee flexion by incorrectly predicting knee extension, how would we 'penalize' this prediction? In this case, should I use a binary classifier (flexion/extension) and plot a Receiver Operator Characteristic (ROC) Curve?
Hello David,
First, I agree that RMSE is the right choice for the reasons mentioned. Second, the evaluation choice depends on whether the magnitude, not just the sign, of the slope is important. I think it might. If so, then standard regression tools are in play, and you can also introduce a logistic regression term into the cost function if you wish to especially penalize the sign of the prediction. The mixture of these two terms can be weighted by some scalar of your choice. Otherwise, if only the sign is important, I agree that an ROC would be valuable. I am curious if the prediction accuracy could be a function of the angle as well, in which case you might want to repeat the calculations for a variety of angles.
Thanks! I agree that the magnitude of the slope is also important. Perhaps we can have an RMSE for the actual prediction angle and another RMSE for the slope? Is there any problem with performing the same evaluation for the derivative of the same value? In this case, we would obtain RMSE for joint angles and RMSE for joint velocity.
David Stephen Hollinger
I suggest reading these articles:
1. Article Evaluating the use of machine learning in the assessment of ...
2. Article Prediction of the performance of artificial neural networks ...
3. https://www.nature.com/articles/s41467-020-17807-z
- Badges
- Science topic