My hypothesis is that a certain mathematical model will make more accurate predictions than humans. The model will output probabilities of a future event happening. Whether the event actually happened will determine success. I also have data on humans making those predictions (binary) and success rate. How do I compare accuracy of the model with humans in a scientific way?
Are both humans and mathematical models significant predictors?
I would run mathematical models and humans as binary predictor variables in a binary logistic regression for the binary predictions, and possible use the same predictors in a linear regression if success rate is a linear, scale variable. Then you could use beta coefficients (exp (B) in logistic regession) to assess which of your predictors is better in regards to predicting your criterion variables.
These links should help: 1) linear regression: https://statistics.laerd.com/spss-tutorials/linear-regression-using-spss-statistics.php; 2) binary logistic regression: https://statistics.laerd.com/spss-tutorials/binomial-logistic-regression-using-spss-statistics.php
Hope this helps :).
Lee.
Do multiple simulations wirh whole dataset. For example, if you have 100 events in total, create 20 different train sets by randomly choosing 50 events each. Then you can ask humans to make predictions and run your model on each trainset and average results (model vs humans). Now you can conduct a t test.
Or, you can threat all your humans as different groups, collect their predictions on each trainset but not averaging them with other humans' predictions. Then you can conduct a pairwise comparison test to check if your model outperforms every human in all cases.
Lee John Curley For humans the only thing I know is what they predicted (event will happen vs will not happen) and whether the event actually hapened
Hazar Altınbaş The prediction data for humans is binary (event will happen vs it wont)
Run the binary logistic regression with categorical predictors (e.g maths coded as one and humans coded as two), you can then compare which one was the better predictor from the output
Dear Talha,
The best way to compare accuracy of the model with humans is to compare which one was the better predictor from the output as Lee Curley suggested. If you don't have real data at the output to calculate the prediction error, you can use cross-validation technique.
More details you can find in my book: Motzev, M., Business Forecasting - A Contemporary Decision Making Approach, Eudaimonia Productions Ltd., 2016, ISBN 978-619-7209-17-4 .
If you are interesting in this topic, I can send you the chapter about Forecast Accuracy for free because this is not against the copyrights.
Best,
MM
By comparing the prediction results with reality for specific intervals and showing the degree of sincerity of mathematical prediction
Credibility in fact is neither in the program nor in the evaluation of researchers
Besides logistic regression, multiple regression with categorical predictors does the job as well
You can perform an analysis of MIMIC structural equations based on latent variables (eg behaviors, attitudes, beliefs, etc.) that allows you to model a proposed model by considering the prediction of multiple categorical variables that also allows you to distinguish differences between each caractaeristic or indicator of the proposed model in relation to each categorical variable.
- Badges
- Science topic