Hello, dear RG community.
Personally, I have found Xarray to be excruciatingly slow, especially for big datasets and nonstandard operations (like a custom filtering function). The only suggestion how to speed up the things that I have found on the Internet is to use numpy. When I adjusted my code accordingly (i.e., used numpy), I laughed so hard because I had to convert almost every single piece of Xarray-based code to a numpy-based code. Still, the remnants of the Xarray-based code kept slowing me down. I went ahead and wrote a crazy piece of code combining Dask and Xarray and numpy and, finally, increased the speed to some acceptable magnitude. That was such a pain.
Pandas, of course, are essentially the same speed-wise. And I couldn't find anything else to handle named arrays in Python other than Xarray or Pandas (I work with multidimensional arrays, so I need Xarray anyway).
I read the docs for Xarray. The authors say the reason for Xarray is to be able to work with multidimensional arrays. I can't fully comprehend that. Why not just add this functionality to Pandas? I could understand if they started such big of a project for some big idea, but just add multidimensional functionality that should've better been added to Pandas to spare users time learning two different data bases seems like not a good justification to me. To say nothing that Xarray has ended up being as slow as Pandas.
I think that a good justification for starting a new data base project for Python is to make it really fast first and foremost. I think a new data base project that will follow numpy example must be started: when the code base is written in lightning-fast C/C++ and then Python wrappers are added on top of that.
I am wondering if anybody is aware of such an effort. If so, when should we expect the release?
Thank you in advance.
Ivan
I understand your frustrations, pandas is known to be slow and even slower when the dataset is quite large. You might want to consider exploring DuckDB and Polars as potential alternatives. Both of these libraries have gained attention for their speed and efficiency.
https://pola.rs/posts/benchmarks/ shows the TPC-H benchmark results for several libraries. The benchmark test was done for the latest versions of the following libraries:
- Polars
- pandas
- PySpark
- DuckDB
- Dask
- Modin
What you'll see is that both duckdb and polars outperform everyother library on almost every metric.
Brahma Reddy Katam
, your answer doesn't improve on Gershinen Shanding's answer and it doesn't directly answer my question. I know about Dask - I mentioned it in my question. I was, specifically, asking about C-based data set handler for python. Gershinen Shanding provided metrics and concluded that either DuckDB or Polars are the way to go. I checked Polars: it is based on a low level programming language and can handle multidimensional arrays. That's why I accepted Gershinen Shanding's answer.You, in turn, just briefly listed some options for datasets handling in Python. There is no value in your answer as it is given right now.
And I suspect, you just created it in ChatGPT.
I'll give you some time to say something in your defense. If you don't have to say anything, I will report your answer as a spam.
08 August 2024 update: I reported your post @Brahma Reddy Katam as irrelevant and asked RG to delete it.
- Badges
- Science method
- Similar topics
- Methodology
- Crystallography
- X-ray Diffraction