I think, one of the simplest method is to apply a triangle-shaped input signal and evaluate the form of the output signal. However, this simple measurement does not allow a quantitative evaluation of the degree of non-linearity.
Since the THD is a measure of non-linearity, the classical method is to use a sinusoidal input and measure the amplitude of the generated harmonics.
Thanks you for your answers. I actually mean that I would like to find books or papers which mentioned about nonlinearity and approaches to measure it between nonlinear systems
I used to do research in the field, dealing with characterization/quantification and compensation of nonlinearity. I can refer you to my recent paper in MSSP, or paper by D. Rijlaarsdam or P. Nuij (and the references therein). Our approach used frequency domain method to quantify the nonlinear effects of a certain class of nonlinear system.
Please see the fille attached for our work on quantification of nonlinear effects and performance of nonlinear systems. Especially focus on chapters 4 and 5.
An up to date link to references on this matter can be found on http://www.davidrijlaarsdam.nl/website/?page_id=26 as several papers and extensions were published / developed after publication of the thesis. Good luck!