What is the most efficient way to deal with big data in Python?

How to compare two groups with only two measurements?

Hiiiii everyone! I have an enquiry on statistical analysis. I was looking for many forum and it's still cannot solve my problem. I want to compare means of two groups of data but only with two...

03 March 2021 8,796 3 View

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10?

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10? Since, the growth curve from day 2 to 10 won't be like the "Logistic curve" it might not follow logistic...

03 March 2021 1,401 3 View

How to increase a model accuracy ?

dear community, my model is based feature extraction from non stationary signals using discrete Wavelet Transform and then using statistical features then machine learning classifiers in order to...

03 March 2021 6,994 5 View

1. What is the impact of having different scales in a survey? and how can we solve this problem before and after data collection (Literature)?

1. What is the impact of having different scales in a survey? and how can we solve this problem before and after data collection (Literature-based reflection)? Thank you for your time and for...

03 March 2021 2,870 3 View

What degree of missing data in a particular case/participant results in Expectation-Maximisation no longer working?

I'm dealing with a mediation model and am using the PROCESS module in SPSS. Due to SPSS and PROCESS being limited in the imputation methods - being unable to handle multiple imputation - the other...

02 March 2021 4,362 1 View

Can NFL theorem be valid in infinite search space in ML?

NFL theorem is valid for algorithms training in fixed training set. However, the general characteristic of algorithms in expanded or open dataset has not been proved yet. Could you show your...

01 March 2021 1,189 3 View

Does anyone know how to plot my station to a section plot on Ocean Data View (ODV)?

I made a vertical section plot with WOA .nc file on ODV, and now I want to plot my stations (csv. file) in it. Does anyone know how to import my points to the section? They are two different...

01 March 2021 3,610 1 View

How to address this issue in the analysis of Real-Time PCR results?

To dear Researchers, I was analyzing a series of concentration for estimation of Real-Time PCR efficiency. The concentration was 1:10. I used MS-excel to evaluate Slope. The result of slope was -8...

01 March 2021 8,683 4 View

How to arrange coordinates and extract climate dataset out of MPI-ESM-MR netcdf files?

Have a nice day everyone, I'm stuck in extracting datasets out of MPI-ESM-MR netcdf files because the latitude ranges from -3*10^6 to 3*10^6 and longitude ranges -5,9*10^6 to 5,9*10^6. Plus, the...

28 February 2021 912 3 View

Is there any Machine Learning/AI algorithm that lets you predict with incomplete features without missing value imputation process?

For instance, the model is trained with A,B,C,D,E features to predict F target. Then, I want to make prediction on a new data, but with only A,C,D,E is known without doing missing value imputation...

28 February 2021 1,556 6 View

Artur Jordão

Dear Mohammad Nahangi,

I recommend you to use the Pandas library and process these data in batches. In addition, this link provides a set of data science libraries in python:

"http://dataconomy.com/2016/10/big-data-python/"

I hope this be useful to you.

Mohammad Nahangi

Artur Jordão I have been working with pandas and other libraries in python and I know of their capabilities. My problem is that I have a huge data base (2TB). Importing and handling such a huge file in python is very challenging and inefficient. I am trying to look for possibilities for data compression or other solutions. Thanks for the answer though.

Thomas Hoppe

Dependent on the type of analysis you like to perform, it may suffice to draw a representative sample from the data. Otherwise, if you need to get simple exact counts, you may set up a HADOOP infrastructure and parallelize your analytics via map/reduce.

Ahmad Hammoudeh

In addition to Thomas Hoppe's answer,

I would recommend using cloud computing services (AWS, google cloud, ...etc). they provide suitable solutions to this problem. HADOOP and Spark are available on the cloud as well.

https://cloud.google.com/solutions/big-data/

Danesh Shokri

As Mr. Ahmad mentioned, you can use cloud computing which fortunately every university at least have one of them especially at electrical engineering, mathematical, physics colleges. I will never suggest cloud computing service online because you have big data and transfer them as so much time-consuming. you better find one of cloud computing system at your university and use it.