Dear fellows.
this question might have been raised here before. However, I didn’t find it.
my question is:
what is the best way to perform statistical analysis on a big amount of data?
when I have more than 10 variables and over 5500lines of numbers in excel sheet?
I am aware there are SPSS and R studio.
I wonder if anyone did use Python for it or any similar? was this accurately represented?
what are your preferences and why would really help!
Thank you in advance.
Regards Ana
Anastasiia Grymak if you want a Python solution, give https://www.statsmodels.org/stable/index.html a look. As Jos Feys mentioned, R can handle this too https://www.rstudio.com/categories/rstudio-ide/, as can Julia https://docs.julialang.org/en/v1/stdlib/Statistics/, it depends on what language you're most experienced in. All three are open source solutions, so no pricey / closed software needed (unless your institution provides those).
That size of dataset isn't a problem anymore. My StataSE takes up to 2·1 billion observations and just under 33,000 variables.
Both R and Python are languages, and you may not want to learn a language in order to do some stats. As alternatives, consider jamovi and JASP, both based on R, and Orange, based on Python. These have excellent user interfaces and make reproducible analysis the default. Data are hot linked to results, so changing the data changes any calculations that are affected and updates all results that depend on that data.
Orange is brilliant for machine learning, and JASP is very strong on Bayesian statistics (and has very good learning materials for them). We've been very pleased with jamovi which we use with undergraduate students, but which is capable of some pretty sophisticated advanced statistical models.
Thank you very much Dr Ronán Michael Conroy for your valuable reply! I will have a look into those!
For big data analysis, SPSS reduces your speed of your action. If your dataset is offline (your analysis is not real-time) Stata is the best choice. Otherwise, you can consider R or Python.
I have started recently using python with Jupiter notebook. Stackoverflow and chatGPT are also very useful learning sources.
Dear, I think R is the best software. Not only because it allows you to apply all the tests, techniques and statistical procedures, but also because being associated with a community of knowledge, it allows you to access tutorials, forums and a lot of free, shared and democratic resources that allow you to advance your project. But also, it is flexible in the sense that its codes can be applied in display formats to share information or for others to replicate what you have done. In open science philosophy, it is highly recommended. However, if it is about psychometrics, there are other more specific software for each test. Regards
The R language is the best alternative because besides being free software, their packages (libraries) are developed mainly through researchers that belong to high-quality universities/institutions worldwide. Good luck!
Performing statistical analysis on a large dataset with more than 10 variables and over 5500 lines of data can be efficiently done using various tools. While SPSS and R are popular choices, Python has also gained traction for data analysis tasks. Libraries like Pandas, NumPy, and SciPy in Python provide powerful tools for handling and analyzing large datasets. Jupyter Notebooks can be particularly useful for creating interactive and shareable analyses. The choice among SPSS, R, or Python often depends on personal preference, the complexity of your analysis, and the specific tasks you need to perform. SPSS is known for its user-friendly interface, R for its statistical versatility, and Python for its flexibility and integration with other data science tools. Consider your familiarity with each tool, the complexity of your analysis, and whether collaboration with others is a factor in making your choice.
- Badges
- Science topic
- Similar topics
- Statistics
- Survey