Hi Jim

This is a very interesting question and I would also like to know the answer and read the opinion of different experts.

In machine learning literature, it is a common, well established and acceptable practice to divide the data into subsets such as training, cross-validation and test sets. The model is trained and fine tuned on the training set and its effectiveness is shown on the test set. The underlying null hypothesis which is evaluated on the test set is that the trained model is not better than making a random guess.

Is it possible that you can also use the same approach for your collision dataset? Such as, propose a model on the first half which predicts some specific statement about the unseen future collision and test it on the second half.

Usman Rashid

Hi Usman,

Thanks for the response, it's interesting to know this type of data-splitting is used in machine learning. This is essentially the kind of thing I had in mind. My plan though would not be to use data from period x to then predict period y, but to take a subsample of data from periods x and y, explore that data, then make specific predictions or hypotheses which I would test using the data from x and y that I did not originally sample. Hope that makes sense! I'm trying to get a sense of whether this is a sensible thing to do or not, and any precedents for this type of approach (e.g. machine learning, as you've pointed out).

Thanks,

Jim

In such a situation, I would call my research an exploratory study. Instead of focusing on hypothesis testing, I would focus on illustrating the different relationships in the data with descriptive statistics, fit regression models and report estimates with confidence intervals for different variables.To evaluate the goodness-of-the-fit, I would use partial R-squared for variables and AIC information criterion for model comparison instead of relying on p-values.

Such an analysis would generate a rich set of useful hypothesis for the future research. But as I said earlier, it's an interesting question and I want to know more.

Hi Jim,

I have used the approach that you mentioned (splitting the data set) when I was checking the predictive performance of a modelling technique used. I used it in instances where I had to make predictions on whether a driver should get a warning depending on their speeding/acceleration profile when approaching a signalized intersection. In these cases I was working with large amounts of data so splitting the data set did not majorly affect the power of my sample.

In my opinion, as long as you specifically the state the goals of your analysis be it confirmatory or exploratory before presenting your results/conclusions it should be okay. I agree with Usman Rashid about calling your research exploratory or confirmatory. I can also see a situation where you can run a hypothesis test (look for differences) on the data and see what factors have major affects on your dependent measures and then run a exploratory analysis for by splitting data set as a supplementary analysis and then stating that more data is required to substantiate the evidence from the exploratory modelling. This essentially lays the foundation for a future analysis.

I feel like there is no wrong answer as long as you clearly specify the reasons for selecting a specific analysis prior and mention the limitations of your analyses.

Hope this is helpful.

-Raj