What is the most efficient way to deal with big data in Python?

More Mohammad Nahangi's questions See All

Which is suitable for use with Python? MySQL or SQL Server?

Which is suitable for use with Python? MySQL or SQL Server? What is your suggestion?

01 March 2021 3,422 3 View

Sample modeling in ABAQUS software and its relationship with the real world?

About crack modeling in ABAQUS: Mode I (pure-tensile cracking) and Mode II (pure-shear cracking), these assumptions may be accruing in theoretical approach (ABAQUS) not practical in Lab....

19 February 2021 6,156 1 View

absorption= 1-mag(S(FP1:1,FP1:1)S(FP1:1,FP1:1))-mag(S(FP2:1,FP1:1)S(FP2:1,FP1:1)) is it correct for absorption ?

hi everyone I am simulating a unit cell structure in HFSS using Flouquet port having two modes for absorption I have used this formula:- absorption=...

17 February 2021 9,178 2 View

An article on the Iranian state's civil liability in the COVID-19 matter needs supervision; Is anyone interested in collaborating?

Hello. An article on the Iranian state's civil liability in the COVID-19 issue has been written in collaboration with a friend and a comparative look at law and biology. The article is ready to be...

15 February 2021 2,906 2 View

Inter-rater / inter-coder reliability for thematic analysis?

In my research, I have used metaphorical elicitation tasks to investigate learners' conceptualization of speaking fluency. In the analysis phase, I used thematic analysis to make themes out of...

10 February 2021 4,719 11 View

How to define two different phases with different species?

Hello all, I am trying to model CO2 being dissolved in a bubble column filled with water. For mass transfer definitions, I need to define two phases: for instance, liquid and gas, where the liquid...

08 February 2021 2,722 3 View

What is the type or grade of ethanol used in SOL-GEL?

I'm preparing materials that will be use in SOL-GEL to produce silica gel, I will use Tetraethyl orthosilicate (TEOS) as precursor, and ethanol as solvent , but when I search to buy ethanol, the...

01 February 2021 5,153 4 View

Where to buy optically clear, low viscosity, refractive index 1.5-1.4 Silicone Oil ?

For research purposes, it is required to have optically clear, low viscosity silicone oil with a refractive index of 1.5-1.4. It is going to be used in optoelectronic research. Online purchase is...

01 February 2021 9,160 3 View

Can any one provide information about medicine, epidemic and critical issues or it is mandatory to be an expert ?

Due to the pandemic of coronavirus, the most source that the people depend on was social media. There is no guaranteed solution that is biased from political opinion. An example of that the...

01 February 2021 7,567 3 View

How can solve/disperse silicon nanoparticles in water?

I want to add silicon nanoparticles to my hydrogel, but when I sonicated these particles with water to disperse them in water, they agglomerated on the water's surface. How can I disperse them in...

26 January 2021 213 4 View

How to compare two groups with only two measurements?

Hiiiii everyone! I have an enquiry on statistical analysis. I was looking for many forum and it's still cannot solve my problem. I want to compare means of two groups of data but only with two...

03 March 2021 8,796 3 View

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10?

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10? Since, the growth curve from day 2 to 10 won't be like the "Logistic curve" it might not follow logistic...

03 March 2021 1,401 3 View

How to increase a model accuracy ?

dear community, my model is based feature extraction from non stationary signals using discrete Wavelet Transform and then using statistical features then machine learning classifiers in order to...

03 March 2021 6,994 5 View

1. What is the impact of having different scales in a survey? and how can we solve this problem before and after data collection (Literature)?

1. What is the impact of having different scales in a survey? and how can we solve this problem before and after data collection (Literature-based reflection)? Thank you for your time and for...

03 March 2021 2,870 3 View

What degree of missing data in a particular case/participant results in Expectation-Maximisation no longer working?

I'm dealing with a mediation model and am using the PROCESS module in SPSS. Due to SPSS and PROCESS being limited in the imputation methods - being unable to handle multiple imputation - the other...

02 March 2021 4,362 1 View

Can NFL theorem be valid in infinite search space in ML?

NFL theorem is valid for algorithms training in fixed training set. However, the general characteristic of algorithms in expanded or open dataset has not been proved yet. Could you show your...

01 March 2021 1,189 3 View

Does anyone know how to plot my station to a section plot on Ocean Data View (ODV)?

I made a vertical section plot with WOA .nc file on ODV, and now I want to plot my stations (csv. file) in it. Does anyone know how to import my points to the section? They are two different...

01 March 2021 3,610 1 View

How to address this issue in the analysis of Real-Time PCR results?

To dear Researchers, I was analyzing a series of concentration for estimation of Real-Time PCR efficiency. The concentration was 1:10. I used MS-excel to evaluate Slope. The result of slope was -8...

01 March 2021 8,683 4 View

How to arrange coordinates and extract climate dataset out of MPI-ESM-MR netcdf files?

Have a nice day everyone, I'm stuck in extracting datasets out of MPI-ESM-MR netcdf files because the latitude ranges from -3*10^6 to 3*10^6 and longitude ranges -5,9*10^6 to 5,9*10^6. Plus, the...

28 February 2021 912 3 View

Is there any Machine Learning/AI algorithm that lets you predict with incomplete features without missing value imputation process?

For instance, the model is trained with A,B,C,D,E features to predict F target. Then, I want to make prediction on a new data, but with only A,C,D,E is known without doing missing value imputation...

28 February 2021 1,556 6 View

Artur Jordão

Dear Mohammad Nahangi,

I recommend you to use the Pandas library and process these data in batches. In addition, this link provides a set of data science libraries in python:

"http://dataconomy.com/2016/10/big-data-python/"

I hope this be useful to you.

Mohammad Nahangi

Artur Jordão I have been working with pandas and other libraries in python and I know of their capabilities. My problem is that I have a huge data base (2TB). Importing and handling such a huge file in python is very challenging and inefficient. I am trying to look for possibilities for data compression or other solutions. Thanks for the answer though.

Thomas Hoppe

Dependent on the type of analysis you like to perform, it may suffice to draw a representative sample from the data. Otherwise, if you need to get simple exact counts, you may set up a HADOOP infrastructure and parallelize your analytics via map/reduce.

Ahmad Hammoudeh

In addition to Thomas Hoppe's answer,

I would recommend using cloud computing services (AWS, google cloud, ...etc). they provide suitable solutions to this problem. HADOOP and Spark are available on the cloud as well.

https://cloud.google.com/solutions/big-data/