I am working on a project in the field of traffic data analysis. we are working on a routing app for vehicles with considering traffic data.
Imagine each street has got data-set gathered by users like this:
exact date and time | density of traffic | speed of traffic | weather | next_half_an_hour_speed (label)
we can derive "day of week", "Hour" and "hour quarters" from date column and add to features. the data is inserted for almost each quarter intervals.
for example:
2019-01-03 00:12:23 | 20 | 61.54 | Rainy | 23.12
2019-01-03 00:26:28 | 54 | 31.13 | Rainy | 40.12
Now I want to ask your suggestions on this problem which I am about to explain.
In uploaded file "1.png", You can see a plot of "label (next half hour speed) histogram" and in the second file "2.png" you see predictions (predicting next half-hour speed of traffic based on sklearn.svm.SVR in python) in contrast to test values.
Now based on these observations I come to conclusion that because of my imbalanced samples in speed feature (which has the most linear correlation with label and it also has the same histogram as label) in range of [22, 55], the predictions in this range are that terrible.
Do you think this conclusion is correct?
I was wondering if anyone could help me find a good source to use (like using SMOTE tools) for my problem.
" Do you think this conclusion is correct? "
Yes, as you have little data for middle speeds, the prediction is uncertain, leading to many "quite false" point estimates.
It is striking that the observed speeds are either quite low or quite high. Speeds inbetween seem to be got only in transitions from high to low speeds (and vice versa), so these speeds are not stable and may not be reflected much by the current situation (what makes it even more difficult or impossible as well as unmeaningful to predict these values).
For such data, a simpler "classification" into high-speed vs. low-speed would be more appropriate, to my understanding (what might be very wrong, though). The "in-between" speeds would be classified with accordingly similar probability to "high" and "low".
I would take this observation to investigate why these intermediate speeds are so rare and find a plausible real-world explanation that would then help to better model the data.
Jochen Wilhelm
Thanks for the answer. I have got an Idea to tackle the problem, but I am not sure if it's working or not.
add a categorical feature to data and set "class A" for speed in range of [22, 45] (where data points lack) and set the other data as "class B".
Then I use "SMOTE" tools to oversample class A or generating synthetic data for class A.
could this be a good solution if there were no problem with data observation?
I don't know. I still think the relevant problem (is the speed high or low) can be tackled in a considerably simpler way, unless there is some valuable information content in the slight speed differences within these groups (what I think is not the case). But since I don't understand the details of your research, I might be wrong. It's just my impression "from outside"...
if you are interest in SMOTE method, check it in the this website https://towardsdatascience.com/machine-learning-multiclass-classification-with-imbalanced-data-set-29f6a177c1a. For the detail, you could check the imblearn module in python.
There are already some methods to deal with imbalanced regression problems. For instance, you can use SMOTE for regression :Conference Paper SMOTE for Regression
You also have an R package (UBL) available on CRAN (https://cran.r-project.org/web/packages/UBL/index.html) with more pre-processing methods that allow you to deal with this type of problems.
This package has SMOTE for regression implemented as well as several other alternative pre-processing methods.
- Badges
- Science topic