One way to know an active agent in an herb is by isolation and characterisation of the chemical constituents. HPLC  and some other techniques can help

If you mean "phytochemical screening" you have to use "Trease and Evans Pharmacognosy" or use  articles and papers. 

Good luck 

I think you should first do extraction of the plant material and then do phytochemical screening of the extract. Based on the presence of the major phytoconstituent like alkaloids, steroids etc, you shoud then do HPLC for the major phytoconstituents of the the particular group of the compounds.

Good. However, will HPLC be able to quanlitatively determine both referencely confirmed and novel pyhtochemicals. Withal, I am of the opinion that GCMS will be more accurate and elaborate.

LCMS or GCMS are the best options after proper extraction method based on the compound type.