In order to start, and if you are familiar with OOP languages, I would suggest Python, equipped with the panda, matplotlib, numpy and scipy packages.
R is easier, more intuitive, several packages are already developed and graphics are supported by the beautiful ggplot2, but projects becomes soon difficult to handle and performance are inferior compared to Python.
Actually i agree with Pietro Fordra,( depends on the data size) plus your formulation of data. For example i'm not working with statistic but i use it inside my problem of estimation and signal processing so actually i use my own code on Matlab it is not really an industrial-approach but as well-known its powerful and flexible. for example i followed the paper of Fayyad [Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth, From Data Mining to Knowledge Discovery in Databases, Article.] in my paper of data-mining:
where i used Matlab-Gui to apply data-mining methods and algorithms with special code we have developed.
more industrial and direct-get software as suggested before are (Weka Tool and R).
In general Data-mining term is wide and multidisciplinary aspect so by determining your formulation of desire data, data size, and desired result you can get the most useful software which fits your demands.