I want to process a very large file, let's say 300 GB, with Python and I'm wondering what is the best way to do it. One limitation is that my algorithm needs the whole file in memory, so a solution with efficient memory handling is required. Any ideas from your experience or knowledge? (frameworks, tools, platforms, libraries... are welcome)
Thanks
What kind of file are you talking about ? For csv files frameworks liked pandas usually provide functions for batch processing.
But as per your specification that isn't possible ? I hope you are aware that "my algorithm needs the whole file in memory" actually sets the hard limit for your computer to actually have >= 300 GB RAM. So are you simply asking for how
to load a file with the least possible overhead in python ?
In general I think we (or at least I) need a bit more information regarding your problem.
Dear Antonio Jesús Banegas Luna
In my point of view, the use of GPU programming can be helpful for situations like this.
Useful link:
https://towardsdatascience.com/python-performance-and-gpus-1be860ffd58d
Best regards.
Benjamin Mayer yes, I work with csv files. Do you think other formats, such as pkl, could be easier to handle? I could work with other file format.
Let's assume I want to apply some sort of feature reduction algorithm that needs all the data to be in memory. As an example, if I wanted to find all the correlated columns in my dataset, I'd have to load every two columns, calculate the correlation ratio between them, free some memory and try another pair of columns. But, since I already have an algorithm to do so on the whole dataset, I would like to run it in one shot.
By the way, every column contains around 10,5 million rows. I know this is not a very clear question but I hope this info help to clarify it.
Thank you all for your answers.
Antonio Jesús Banegas Luna
Maybe this helps:
https://stackoverflow.com/questions/54709939/problem-getting-correlation-matrix-from-large-csv-file-with-python-pandas
But from what I understand it probably doesn't. If so it sounds like you will have to either do this on a computer with more ram (if you have access to a computer cluster they usually have a few big memory nodes with around ~1TB RAM).
Or you will have to somehow adapt your feature reduction to work in batches on your input data.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python