I have NYC taxi trips dataset contains multiple attributes like (pickups and dropoffs) coordinates, DateTime also, trip distance and so on, indexed by "tpep_pickup_datetime" coulmn as datetime64[ns] data type, I extracted some features from date time pickup and drop-off columns like month, day, hour and other. So I am focusing on DateTime and location to do trip time prediction.
Concerning the problem that faced me, it is the problem of converting the dataset into a fixed time series intervals (1 or 10 mins as an example to get 1440 time bins for each day) to be ready to LSTM input, let me reflect the essential point of the problem, I have tried to do a resample the dataset based on pickups-time, but the dataset contains more features, so it is difficult to convert it into a sequence time series with fixed interval lengths. Because the data contain a lot of trips at the same time (such as Sunday at 8:00-9:00 am approximate 2990 trip) but they are from different places.
So the main problem which is briefly: "How do I convert or prepare a taxi dataset to a time series with the fixed intervals?".
Thank you advance,
You need to specify your problem more precisely.
If I get you right, you want your data set in a form where:
one column is your variable of interest ('trip-time),
one is a time variable (day, month, or year)
and other columns are factors which may affect 'trip-time', for example:
'trip distance'.
How to transform your data set in this desired form depends on how the information you describe is currently available to you. E.g. if the information you state exists in an EXCEL file, then you either manually or using excel functions, should transform it in the form you wish.
I think that you need to be aware that in this conversion you will necessary lose some information. If I understand well, you want to group trips in bins (i.e. each 10 minutes). However, I think you need to decide if you want to put together all the locations, or if you want to group the trips by time AND space (e.g. grouping departures and arrivals by hour). Moreover, you need to consider how to deal with trip duration.
I think you should better describe the purpose of this conversion and what is the information you really need.
Firstly let me say thank you Michel Noussan , yes I will necessary lose some information if I follow (resample () ), but I need other solution if that is possible to get time series with fixed intervals to input it on LSTM, and use long or short historical data.
Then actually the dataset contain trips from many places with different timestamps, so may be I need to map or some other thing to get a matrix based on the trips started from the same location in map cells as example, so may I need a map, but actually I don't know if this is good idea and what kind of map needed.
Firstly let me say thank you Pandelis Mitsis yes you got me right, and my dataset in CSV file so I process it with Python.
- Badges
- Science topic
- Similar topics
- Geoscience
- Cartography
- Mapping