You could identify compounds in your plant extracts using full scan GC/MS combined with searching the obtained spectra against a library (i.e. NIST or Wiley). Its not necessarily a turn key approach, however; you'll need to make sure you get reasonable chromatography, specificity, and responses for the compounds in order to have a chance at identifying them. LC/MS could be helpful for larger, less volatile compounds. There aren't libraries for LC/MS, unless you have a nice high-res MS/MS type system, so it can be a bit more difficult to interpret the data if you don't know what you are looking for.
First, look up your plant in the literature to see what has already been found, also look at the genus level. This will narrow the list of compounds (still the possibility you found something new). But one can see if the mass spectrum matches compounds already found.