Find a copy of the book "Phytochemical Methods: A Guide to Modern Techniques of Plant Analysis" by J.B. Harborne. It is an indispensable tool if you're involved in phytochemical screening and analysis, component characterization, and compound bioactivity studies. Good luck!
non-chromatographic techniques such as immunoassay, which use monoclonal antibodies (MAbs), phytochemical screening assay, Fourier-transform infrared spectroscopy (FTIR), can also be used to obtain and facilitate the identification of the bioactive compounds
First you need to find out the phytochemical composition of the active fraction. According to the presence of various phytoconstituents, you can select how can you isolate the particular chemical compound and then find out methods for its characterization. Normally for characterization you need to perform IR, NMR, Mass and elemental analysis.