The output of a VNA is S11 and S21, S22 and S12 if required. These are vectors, or representations of complex numbers. The reflection loss or return loss is the square of the magnitude of S11. The return loss is just one number (a scalar) and doesn't contain the phase information that is in S11. It is usual to measure S11 to get the reflection loss. You can measure it without measuring S11, but then your apparatus would be a Scalar Analyser, not a Vector Network Analyser. It is necessary to measure S11 as a vector in the calibration procedure to eliminate imperfections in the VNA hardware. This makes VNA measurements more accurate than scalar analyser measurements.
It is also important that you know what you mean by the reflection loss of a material - do you mean the free-space reflection loss for a plane wave at normal incidence? If it is polarisation independent this can be measured in an air-filled coaxial line by replacing a section of the air by the material, but not by putting the end of a coaxial line onto a lump of the material. It is also possible to measure it using an antenna to get an approximate plane wave hitting the material and reflected back. It would need to be compared to the reflection from a very good conductor in the same place.
VNA can do the measurements for the reflection of materials, but in most cases you have to construct a specific measurement system or, at least, a fixer, and sometimes you are also required to do some data processing. There are many kinds of measurement system, dependent on dimensions(length, width, thickness ), states (solid, liquid, or gas) and EM characteristics (isotropic or anisotropic, linear or nonlinear) of the material sample you want to measure.