Often in environmental science we find ourselves having to work with large, complex models that we didn't write ourselves, that we don't fully understand, and which may have known problems that would be difficult to fix. I wondered if anyone has considered appropriate and robust methods for working with such models, in order to get the results we want without having to start from scratch? One of my favourite papers is "Six (or so) things you can do with a bad model" by James S Hodges (Op. Res. 39(3) 355-365, 1991), which is along these lines. For example, even if a model is wrong, you can use it to generate ideas, for training, or to store data. Usually we pretend that our models are without flaws, but I would be interested in modelling approaches that are valid even when the model is not. Any ideas?
Hi Simon,
In many time, the problems are originally related to the data that the algorithm uses. Hence, my advice is to clean your data up and run many algorithms with a fixed seed number under to the evaluation of k-fold cross-validation, for instance.
HTH.
Dr. Samer Sarsam
Although I have seen a few models that were essentially useful as training or educational tools, models for scientific level work often were developed to apply in some specific circumstances, and it is the responsibility of the user to understand, verify and validate its applicability to another area. It may be wishful thinking to expect a model to not have some flaws, limits, assumptions, etc. that suggest caution in applying without some degree of checking, verifying, validation, adjustment. But to your point, the model code and working of the model could be a training tool to learn from example on how to write code to address complex data and calculations needed to achieve some modeling results or products. Many of us more or less have learned the Arcmap basics, but have had to play with resource coverages to achieve some desired analysis, products, maps, etc. There is a degree of challenge and satisfaction to develop some estimates or reasonable answers to complex questions. There can be a degree of satisfaction in inputting data into a model, and getting it to successfully run. Perhaps the difficulty is does it make any sense, or how can I fix it to make sense?
https://adaptivesystemsinc.com/blog/adaptive-approach-handling-messy-big-data/
https://www.ibm.com/developerworks/library/ba-cleanse-process-visualize-data-set-1/index.html
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.390.9210&rep=rep1&type=pdf
I guess at the very least the model needs to give "reasonable" results, to some degree. Which is not a trivial thing to achieve with some messy models. If the model results are subjectively reasonable then they can be the basis of discussion and learning and exploration, even though they might be wrong?
I have been doing a lot of work with Bayesian Calibration, which is an amazing tool that you can use with fast models. This allows you to quantify your uncertainty, and see how much information the calibration data has for determining parameter values and predictions. Generally model input and structure error are a little harder to handle. I haven't considered similar approaches for slow models. I guess sensitivity analysis is the equivalent, although this tends to be overly linear IMO.
On reflection, I suppose any useful model must have some degree of correspondence with knowledge/data about the real world system. We do need some basis for anchoring it in the real world, whether this is process description or calibration data. Sometimes we can more aware of a model's shortcomings than its strengths...
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training