There is a clear distinction between exploratory and confirmatory research. In exploratory research, we explore our data to discover interesting trends or associations, perhaps by analysing it in multiple different ways. However this multiplicity of analytical approaches mean we cannot control the Type I error rate, so such exploratory research cannot have evidential value. Confirmatory research on the other hand involves making specific, well-defined predictions that are directly tested through the research. The paper attached to this question, originally published by de Groot in 1956 and translated by Wagenmakers in 2014, gives a better description of this distinction between exploratory and confirmatory research.
I have a large set of secondary data (collision records for a country, going back for 20 years). My question is whether it would be ok to carry out both exploratory and confirmatory research using this dataset. First you would randomly select half the data and use this to carry out the exploratory analysis on. This would lead to the development of specific hypotheses which would then be tested using the other half of the dataset that was not involved in the original exploratory research.
Any comments or thoughts are gratefully received.
Hi Jim
This is a very interesting question and I would also like to know the answer and read the opinion of different experts.
In machine learning literature, it is a common, well established and acceptable practice to divide the data into subsets such as training, cross-validation and test sets. The model is trained and fine tuned on the training set and its effectiveness is shown on the test set. The underlying null hypothesis which is evaluated on the test set is that the trained model is not better than making a random guess.
Is it possible that you can also use the same approach for your collision dataset? Such as, propose a model on the first half which predicts some specific statement about the unseen future collision and test it on the second half.
Usman Rashid
Hi Usman,
Thanks for the response, it's interesting to know this type of data-splitting is used in machine learning. This is essentially the kind of thing I had in mind. My plan though would not be to use data from period x to then predict period y, but to take a subsample of data from periods x and y, explore that data, then make specific predictions or hypotheses which I would test using the data from x and y that I did not originally sample. Hope that makes sense! I'm trying to get a sense of whether this is a sensible thing to do or not, and any precedents for this type of approach (e.g. machine learning, as you've pointed out).
Thanks,
Jim
In such a situation, I would call my research an exploratory study. Instead of focusing on hypothesis testing, I would focus on illustrating the different relationships in the data with descriptive statistics, fit regression models and report estimates with confidence intervals for different variables.To evaluate the goodness-of-the-fit, I would use partial R-squared for variables and AIC information criterion for model comparison instead of relying on p-values.
Such an analysis would generate a rich set of useful hypothesis for the future research. But as I said earlier, it's an interesting question and I want to know more.
Hi Jim,
I have used the approach that you mentioned (splitting the data set) when I was checking the predictive performance of a modelling technique used. I used it in instances where I had to make predictions on whether a driver should get a warning depending on their speeding/acceleration profile when approaching a signalized intersection. In these cases I was working with large amounts of data so splitting the data set did not majorly affect the power of my sample.
In my opinion, as long as you specifically the state the goals of your analysis be it confirmatory or exploratory before presenting your results/conclusions it should be okay. I agree with Usman Rashid about calling your research exploratory or confirmatory. I can also see a situation where you can run a hypothesis test (look for differences) on the data and see what factors have major affects on your dependent measures and then run a exploratory analysis for by splitting data set as a supplementary analysis and then stating that more data is required to substantiate the evidence from the exploratory modelling. This essentially lays the foundation for a future analysis.
I feel like there is no wrong answer as long as you clearly specify the reasons for selecting a specific analysis prior and mention the limitations of your analyses.
Hope this is helpful.
-Raj
- Badges
- Science topic