I would start out by making sure that your compound does not reduce MTT on its own as some plant extracts do this. Incubate your compound with MTT for 2 hours and check to make sure that purple colour does not develop. If your compound does not reduce MTT, then start with the MTT assay. The acid phosphatase assay can also be used. The LDH-release assay will tell you whether your compound is causing cell lysis, but secondary necrosis can happen if your compound is left on your cell for too long. To differentiate between apoptosis and necrosis, use the Annexin V/PI assay, but again, make sure that your compound is not inherently fluorescent (many are). You can begin to tease out the mechanism if you desire, or do preliminary in vivo work. I would start by testing your compound in tumour xenografts in immune-deficient mice. Intratumoural injections can be done if you want to do a proof of principal preliminary study.
You can also submit your compound to NIH and they will screen it against a library of cancer cell lines. If you do it yourself, be sure to include normal cell cultures, such as mammary epithelial cells, PBMCs, human umbilical vein endothelial cells, dermal fibroblasts, etc, and calculate the EC50 for both cancer cells and normal cell cultures. Good luck!
Initial step to find out the in vitro screening is to do MTT assay on cancerous cell lines. Take your desired cancerous cell and find out the EC50 and at the same time also check the cytotoxicity of your compounds on non cancerous normal cells like HEK-293 to ensure the general cytotoxicity of compounds. After that you can do the LDH release assay to differentiate between apoptosis and necrosis.
to check anticancer activity of plant extracts first you should study their effect in vitro on cancer cell lines ( using MTT and SRB assay) then in vivo using tumor model such as Erlich's ascites carcinoma model or you can induce cancer in rats or mice using chemical carcinogen and here you can check the anticancer activity of your compound before and after induction of cancer to study preventive, ameliorative and therapeutic effect