Before any in vitro or in vivo study, you should test if your plant extract does not reduce MTT on its own since some of plant extracts do. To test that, Incubate the extract with MTT for 2 hours and check that a purple color does not develop during this time period. If the extract does not reduce MTT, then commence with the MTT assay.
For testing the anticancer activity of plants the in vitro effect of extracts obtained from the plants should be studied on cancer cell lines (using MTT and SRB assays) then in a later stage in vivo testing using tumor model such as Erlich's ascites carcinoma model or inducing cancer in rats or mice via chemical carcinogen can be conducted. In addition, the anticancer activity of the extracts before and after induction of cancer can be studied in terms of preventive, ameliorative and therapeutic effect
Furthermore, the acid phosphatase and the LDH-release assays can be conducted where the latter gives information whether the extract is causing cell lysis, To differentiate between apoptosis and necrosis, the Annexin V/PI assay can be used, make sure that the extract is not inherently fluorescent . Preliminary in vivo study. can be conducted in tumour xenografts in immune-deficient mice.
Submission of the extract to NIH for screening against a library of cancer cell lines can also be helpful.
Alternatively,you can conduct this test by yourself using normal cell cultures, such as mammary epithelial cells, PBMCs, human umbilical vein endothelial cells, dermal fibroblasts, etc, and calculate the EC50 for both cancer cells and normal cell cultures.