This is what we called a "mixed optimization problem" in a series of papers I wrote with Postlethwaite & Gu some 20 years ago. The idea is quite simple - you can choose any cost index (objective function) and search through the space of LQR-optimal controllers to find the Q & R that minimizes the cost function. Look on my website for papers and some s/ware: http://public.cranfield.ac.uk/eh3081/
The general approach actually works better for H-infinity methods rather than LQR, in particular Glover & Macfarlane's method because the optimal controller can be calculated without gamma-iteration
I am not a big expert in the field of a vibration problem. However, I can assume that if it is necessary to find two matrixes of Q and R then each individual in GA shall have two chromosomes. The first chromosome should be created on the basis of Q array elements, and the second – on the basis of R. If some matrix is the symmetric, then for formation of a chromosome it is enough to use the elements located above the principal diagonal, differently – all array elements are drawn out at line. It is possible to use still approach when genes of a chromosome are matrix columns, but it requires the appropriate programming. Fitness function of the GA is created on the basis of goal function of a vibration problem. Let goal function of the problem is as F(Q, R) -> min. Then fitness function of the GA can have an appearance of FF (GA) = 1/F (Ch(Q), Ch(R)), where Ch(Q) – a chromosome for Q, Ch(R) – a chromosome for R.
You can refer to following articles which used LQR and GA for vibration control:
Amini, F., Hazaveh, N. K., & Rad, A. A. (2013). Wavelet PSO‐Based LQR Algorithm for Optimal Structural Control Using Active Tuned Mass Dampers. Computer‐Aided Civil and Infrastructure Engineering, 28(7), 542-557.
Roy, T., & Chakraborty, D. (2008). GA-LQR based optimal vibration control of smart FRP composite structures with bonded PZT patches. Journal of Reinforced Plastics and Composites.
Chen, D., Zheng, S., & Wang, H. (2012). Genetic algorithm based LQR vibration wireless control of laminated plate using photostrictive actuators. Earthquake Engineering and Engineering Vibration, 11(1), 83-90.
Schulz, S. L., Gomes, H. M., & Awruch, A. M. (2013). Optimal discrete piezoelectric patch allocation on composite structures for vibration control based on GA and modal LQR. Computers & Structures, 128, 101-115.