Is Kubelka-Munk transformation necessary in order to make the reflectance signal proportional to absorbance in this case and correct the roughtness problem? Is there another procedure to investigate the layers deposition?
KM units were derived for the thick layer (no light comes under the layer) of partially transparent particles. Two processes were considered: absorbance and scattering. Seems to me the LBL films should be thin and transparent enough to suppose, that the reflectance from bottom Al substrate is more significant, that scattering. So, it is more physical sense to get an absorbance as A = log (100%/R%), and the optical length is d = 2h/sinQ, Q is an incidence angle in your integral sphere. In this case it works just like mirror reflectance.
In some integral spheres you can cut-off mirror reflectance, but in the case of smooth film and support you will get too low signal.
The procedures of investigation depend on the layer thickness. An ellipsometry can be mentioned for example.