Literature shows that field data (e.g., entry flows and time gaps) were used directly for roundabout capacity model development. In one of my later works, I used empirical data (straight from the field) to develop a capacity model. Still, I received questions like "How do you tell your data is reliable to develop a model/Do you carry out the data cleaning?".
I’d like to know if these data must be pre-processed before they are used for model development. Or it depends on the nature of each data type/discipline?
I am from the Traffic Engineering field, focusing on roundabout studies. The data I deal with the most include traffic flows, time gaps, speed and roundabout geometry.
Your opinions matter to me. Thank you.
Regards,
bong
Hello
I guess fieldwork is the most direct way to collect data for this kind of work. It is a classic and well-establish method among research and practicioners. I recommend to explain the reasons you choose to collect data in the points and the period of time you collect it. Related to data cleaning, you could conduct a simple data analysis looking for paterns in the data to find outliers in your measures. Using your domain knowledge you could identify this kind of data and delete it to obtain a "clean" dataset.
Good luck
Yes, field traffic data typically requires pre-processing. This is because the data collected from traffic sensors and cameras can be noisy, inconsistent, and contain missing values or outliers. Pre-processing techniques such as cleaning, filtering, and imputation are often used to address these issues and prepare the data for analysis. Additionally, feature scaling and normalization may be applied to ensure that the data is on a consistent scale and suitable for certain types of machine learning algorithms.
Thanks Santiago Cardona & Abararahemad Khalak
Could you suggest some cleaning methods? I used the interquartile range method, but it doesn't seem to "clean" data well. I have attached an actual field data sheet. Perhaps you can recommend one based on that?
Thanks.
1.Z-score method:
2. Statistical tests:
3. Domain-specific rules:
4. Data visualization:
, it's important to choose a data cleaning method that is appropriate for your specific data set and analysis goals. You may also want to consider using a combination of methods to ensure that you are effectively identifying and removing errors and outliers from your data set.
Thank you
Abrar
Yes you have to scan data for confidence intervals of accuracy required such as 95percent ti exclude unreliable data,then start your analysis with the rest one
No yiu have to scan them for cinfidence range as errors might be introduced due to some bad photography
Yes, If you take poit data (Gps data) may it contain inconsistent or unfit with road data
Thanks, Abararahemad Khalak, Abdulkhalik Al-Taei and Temesgen Haile Hayidso, for the response.
Abararahemad Khalak, Yes, I will consider which method is appropriate for my data.
Abdulkhalik Al-Taei, Perhaps, you could suggest some methods for the confidence range scanning?
Thanks.
- Badges
- Science topic