Hi Mir,

I have primarily used Perl and Python when scripting the transformations of data from one form to another, and Scala when reducing a map in a map-reduce system such as Apache Spark or Amazon's EMR.

I hope that helps :)

Hi Mir,

In addition to what David wrote, it is worth looking at Bash and Ruby. The first language I use for preparing data.

Good point, Dominik. Most command-line projects have a bit of Bash in them to tie them together, or start a run consisting of many scripts. You can even use Bash alone but are likely to outgrow it.

Ruby is a perfectly acceptable alternative to Perl or Python. The only reason to choose one over the other when getting started is familiarity or preference for syntax. I even knew one guy who used PHP to transform data :)

As you progress, the community around a language can help a lot, especially if they can help answer language-specific questions or provide libraries that tackle difficult problems. The depth of libraries in the Perl and Python communities are why I suggested those two initially.

To be more clear, a lot of the data I have worked with was initially in spreadsheets or relational databases so the first task was to transform the data into some other form that I could work with. Often that meant converting CSV or TSV into RDF, another relational database, or a map-reduce system. That's where scripting languages can help you.

Then, you can use another language to perform the reduce step of the map-reduce jobs. That's when I use Scala, etc.

To complete the list, if you need to perform statistical evaluations of your data, then R is your friend. R is the current power tool for statistical analysis.

 Hello everyone @David Wood and @Dominik Tomaszuk have you done any data analysis with astronomical project?

Hi Mir,

I designed a control system for an astronomical observatory once, but never analysed astronomical data. Sorry!