I dot think there is a standard software particularly for this.Try to get/write a Matlab code for the same. There must be some existing codes with few researchers in your own institute for some similar purposes, grab them and modify them to meet your requirements.

You can easily clip the raster in QGIS, however, I am not sure if QGIS support mat files, anyway you can import them using the r.in.mat grass function in QGIS. after that, you can follow any simple tutorial to clip a raster, like this: https://gis.stackexchange.com/questions/10117/clipping-raster-with-vector-boundaries-using-qgis

.mat file must be created using MATLAB. SO I suggest you to try to use MatLab code to extract data within the shapefile

Refer to this file exchange link

https://in.mathworks.com/matlabcentral/fileexchange/71500-maskregion?s_tid=prof_contriblnk