Hi Tiziano,

Have you considered evaluating your model's out-of-sample predictive validity / cross-validation? This boils down to refitting your model to the data while randomly leaving out a subset of observations (a data point, a cluster of data points, or a random number of data points), and then have the model predict the data that were left out (and compare the discrepancy with the observed, left out data). There is a lot of literature on this technique.

If out-of-sample validation is not an option (due to e.g. small sample size), you may want to consider posterior predictive checks. This entails simulating new data from your model's posterior (given the original covariates in the data, if any), and comparing the simulated and observed data by means of a discrimination test. If your model has a hierarchical structure, you should consider mixed posterior predictive checks (i.e. including resampling of random effects / hyperparameters). A major caveat is that (mixed) posterior predictive checks tend to be conservative, i.e. they don't detect discrepancies between model and data as well as cross-validation (because the simulated data are based on all the observed data).

Best,

Luc

Hi Tiziano,

In my view a very good (perhaps even the best possible) approach would be to use sets of known sample-ages: plonk them in and see what comes out. As shown by case studies of this type, run under default condiitions, Bayesian Sequencing optimizes the dating precision at the cost of accuracy :

https://journals.uair.arizona.edu/index.php/radiocarbon/article/view/3836/3261

Bernhard Weninger

*********************************

I am not going to give an answer. Rather, I am curious to know more about Luc Coffegng's reply.

I know something like the approach he described, and that is used to evaluate, e.g., Logistic Regression models. I am wondering how that can be applied in a Bayesian chronological modeling situation. 

I like Bernhard's suggestion of doing an out-of-sample validation.

For people who are interested in posterior predictive checks (PPP), I've put some links at the bottom of this message, including

• A blog post on PPP by Andrew Gelman
• The original PPP paper by Gelman et al
• Another good PPP paper by Marshall and Spiegelhalter
• A paper on the application of mixed PPP in a multilevel logistic model
• A short paper by Gelman discussing mixed PPP

The general principles of PPP can be used in any Bayesion regression model. I'm not an expert on chronological modeling, but I don't see why it wouldn't work for that too. Gianmarco, are there specific issues that you are thinking about?

I think that one of the great things about PPP is that it help us understand how our model deviates from the data rather than just providing a criterion for rejecting/accepting a model. As George Box put it: "Essentially, all models are wrong, but some are useful".

http://andrewgelman.com/2009/02/07/confusions_abou/

http://www.cs.princeton.edu/courses/archive/fall09/cos597A/papers/GelmanMengStern1996.pdf

http://www.ncbi.nlm.nih.gov/pubmed/12720302

http://dx.doi.org/10.1051/vetres/2009013

http://www.stat.columbia.edu/~gelman/research/published/STS235A.pdf

Hello!

Thanks for your reply, and for the link provided, that I have not checked yet. Bayesian statistics' details are, admittedly, too deep water for me. I, for me, do grasp the basic principles and the underlying mechanics, but my background is not so extensive to let myself play around with Bayesian modeling independently from the Bayesian radiocarbon modeling facility (i.e., OxCal).

As for modeling in general, I have recently come across the problem of evaluating Logistic Regression models and I am in due course in deepening my knowledge on the topic. And I have indeed found an interesting R package for that purposes (i.e., validation and calibration).

Cheers

Gm

http://cran.r-project.org/web/packages/rms/rms.pdf