I'm getting old and I'm not sure whether or not I've learnt any theories about solving polynomial matrix exiations, like the second order equation XTAX+BX+C=0.
Can anyone fresh my mind up or suggest where to learn more about this?
for just the system you wrote down read about quadratic forms and quadrics. For more general equations read about Gröbner bases. The most convenient way for guys who are getting old (you) or already got old (me) is to play arround with Mathematica, guided for instance by the tutorial 'Complex Polynomial Systems' and enjoy the progress that happed since our time at university.