HPLC in general will give you better resolution and separation, since you have more parameters to manipulate to get to the optimal separation i.e. gradient, mobile phase, ph and eventually different types of columns. For qualitative analysis and quantitative separation HPLC is a superior option. Keep in mind that HPLC is usually using reverse-phase while HPTLC is usually normal phase, some compounds may show better selectivity on normal phase chromatography.
HPTLC is suitable one. if u have standards(eg.Quercetin used as a standard for flavonoids) HPLC may help you but only preparative HPLC will useful for your isolation process. analytical HPLC help you in quantitative estimations of particular type of compound.
They can both work well. Most important, do some research on the plant and see what has already been found on the genus level as well as the species. This will give a sense of what has been found and allows you to screen against these compounds.
TLC is good for comparison against standards. It is also very good for developing a solvent system if purifying antibiotic compounds because of the ability to run bioautograms where the the compound elution could be observed by a lack of growth on an agar plate. TLC has a variety of stationary phases including silica, alumina (acidic, basic, and neutral), diol, amine, and C18. One can also run a variety of solvent systems with it (Normal phase include: hexane/ethyl acetate, hexane/dichloromethane, hexane/toluene, hexane/ether, hexane/ethanol, dichloromethane/methanol, and others) and the combination of solvents and stationary phases provides a lot of selectivity- the ability to resolve different compounds. The pH of the solvents can be altered by adding modifiers such as TEA, ammonia, formic acid, or TFA.
Column chromatography is useful if doing assay-directed fractionation, as opposed to antibiotics. The TLC shows many spots, but one doesn't know which spot confers the biological activity, or even if the compound is visible. I've come up with a column screen using "wide polarity range chromatography" which is useful for open columns as well as the automated flash systems I used (just use a step gradient for manual columns). It can be adapted for HPLC, although I prefer running crude extracts on cheap columns. The solvent changes ensure nearly everything elutes from the column, and provides a sense of the active compound(s)' polarity. Links below.
HPTLC is important to check the components of your extract and to have an idea the system solvents to apply later on with more amount of your extract in a HPLC.
You can use the HPTLC to check the complexity of your sample and can perform different visualisation techniques on a single plate to get plenty information as to what classes of compounds are present.
You do not say what activity you are looking for but you could choose a visualizing agent that shows up particular compounds, but be careful as you are using very small sample sizes and you may not be able to see where the spots/bands are on the plate if they are present in very low concentrations.
The best information is due to multiple staining that can be used and for scouting for the best H/UPLC conditions for your extract.
HPTLC also gives the ability to run/test gradient conditions and if a separation only moves a short distance up the plate you can dry it and try another solvent mix as the mobile phase.
Each technique has its pros and cons. Chromatography is always a compromise but you make the best of what you have and the time and separation that is acceptable in the end.