QGIS and GRASS GIS are open source alternatives to ArcGIS. On the math side, GNU Octave is an open-source alternative to Matlab. As someone already said, R has packages for handling spatial data, specifically the SP and RGDAL packages.
I agree with you Alber. QGIS and GRASS are very good options. And we should include ILWIS (ITC) , a raster based GIS that has a strong academic background on Photogrammetry an Remote Sensing from the ITC experiance. There was an academic version that can be downloaded from the ITC site. QGIS is easy to use and has the capability to import imagery from very different sources, like ERDAS, etc.
R is free and open source and easy to use. Also has a large community of developers and many documentation. With R you never need any proprietary software.
Put on Google "spatial pattern analysis in r" and take a look.