Have you used any programming language for data analysis?
Hi Mir,
I have primarily used Perl and Python when scripting the transformations of data from one form to another, and Scala when reducing a map in a map-reduce system such as Apache Spark or Amazon's EMR.
I hope that helps :)
Hi Mir,
In addition to what David wrote, it is worth looking at Bash and Ruby. The first language I use for preparing data.
Good point, Dominik. Most command-line projects have a bit of Bash in them to tie them together, or start a run consisting of many scripts. You can even use Bash alone but are likely to outgrow it.
Ruby is a perfectly acceptable alternative to Perl or Python. The only reason to choose one over the other when getting started is familiarity or preference for syntax. I even knew one guy who used PHP to transform data :)
As you progress, the community around a language can help a lot, especially if they can help answer language-specific questions or provide libraries that tackle difficult problems. The depth of libraries in the Perl and Python communities are why I suggested those two initially.
To be more clear, a lot of the data I have worked with was initially in spreadsheets or relational databases so the first task was to transform the data into some other form that I could work with. Often that meant converting CSV or TSV into RDF, another relational database, or a map-reduce system. That's where scripting languages can help you.
Then, you can use another language to perform the reduce step of the map-reduce jobs. That's when I use Scala, etc.
To complete the list, if you need to perform statistical evaluations of your data, then R is your friend. R is the current power tool for statistical analysis.
Hello everyone @David Wood and @Dominik Tomaszuk have you done any data analysis with astronomical project?
Hi Mir,
I designed a control system for an astronomical observatory once, but never analysed astronomical data. Sorry!
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Software