GEO5 and/or FLAC3D can be used. Other reliable software can also be used, it depends only on the level of accuracy that you want to expect. Machine Learning Techniques, Analytical and Empirical methods can be taken into account. It is suggested to select and try two methods in order to compare results and get high precision, as rock density on slope stability is a hot topic.
First of all, it is up to your target aim of rock density instability analysis. if you looking for discontinuities density, using DEM, DFN can be useful. you also can use the 3DDGM model for high precision modeling. if your aim is rock blocks deformations, using the FDM, BEM or FEM-DEM can be recommended.
Is the mean of density in soil slope at the open-pit mine, is overburden?. If yes; regarding the geological condition of the site using Plaxis or Phase2 can be useful.