In Rayleigh's method, the number of terms used is only one. In Ritz's method, the number of terms used is more than one. The basic principle is same in both methods, that is, minimize the total potential. Hence, Ritz's method is also known as Rayleigh-Ritz method.
Some people use Rayleigh-Ritz and Ritz method interchangably. According to Leissa [1] the Rayleigh-Ritz method is a more restricted Ritz method that Lord Rayleigh used for finding natural frequencies of vibrating systems. Another good reference is [2].
[1] Leissa, Arthur W. "The historical bases of the Rayleigh and Ritz methods." Journal of Sound and Vibration 287.4 (2005): 961-978. http://www.sciencedirect.com/science/article/pii/S0022460X05000362
[2] Gander, Martin J., and Gerhard Wanner. "From Euler, Ritz, and Galerkin to Modern Computing." SIAM Review 54.4 (2012): 627-666. http://www.unige.ch/~gander/Preprints/Ritz.pdf
In Rayleigh's method, the number of terms used is only one. In Ritz's method, the number of terms used is more than one. The basic principle is same in both methods, that is, minimize the total potential. Hence, Ritz's method is also known as Rayleigh-Ritz method.
Ritz method is a direct one. In some sense you also use the variational form in the Rayleigh method since you need to compute both the potential an kinetic energy of the system.
'Rayleigh's Principle and its Applications to Engineering' by G. Temple and W.G. Bickley, Dover Publications ( It is likely that the book is out of print).
"Comments on the historical bases of the Rayleigh and Ritz methods" http://www.sciencedirect.com.www.sndl1.arn.dz/science/article/pii/S0022460X08005476
Yes. Rayleigh's method can be used to determine the fundamental frequency only. The accuracy of the result depends on the chosen admissible function. Raleigh's method gives the value of frequency which is higher than the exact value. To get higher frequencies, one need to take more than one term and apply Ritz(Rayleigh-Ritz) method. In R-R method, the contribution of each term is different and establishing the criteria for selection of significant terms is a challenging Research problem.