rocks absorb infrared spectrum, unlike the soil, which presents a low absorption in this area. think to do band ratios by combining three bands NIR, MIR, FIR (B4,5,7).
You can separate the rock and soil based on soil moisture which you can derived from infra red and thermal bands. Barren rock do not have the moisture and giving the bright signature where as in the case of wet soil, we can get the bit darker signature. Most important aspect is that based on the soil (chemical and physical) properties the slight variations of spectral signatures.
From experience: any unsupervised classification of mid-resolution imagery will included a class corresponding to bare rock, if the latter is present in the area of interest, or more than one class when various types of bare rock (e.g. dolomite versus granite) occur at the surface. Presumably you know your area of interest well enough to locate some bare rocks on the image, a topographic map or along a road to allow for a supervised image classification. Attached is a case; secondarily we lumped the grass and the bare rock classes as neither of them was the target of our research. But the rock class stood out.
Article From monospecific to mixed forest after fire?: An early fore...
If you are using an MS sensor like Landsat, or EO1 ALI you should add the tasscap moisture index and Brightness index as ancillary dataset in your classification using layerstack option in ERDAS then classify again using decision tree or support vector machine classifiers. Also soils may contain organic matter on top but rocks not. You can check the Soil organic matter definition using satellite sensors subject on search engines.
If you are planning to use a hyperspectral data first of all check the USGS spectroradiometer database to find the which wavebands is capable to find soil organic matter and go for it.
In fact, I looked for publications about this topic. I did not find anything. If you know about any references or name of the authors who are publishing in this area, please let me know.