It would be helpful if you could be more specific in your question. However, a first step would certainly be to assess cytotoxicity of your compounds. You can find a detailed protocol in the link. These protocol in the link are designed for nanotechnology and delivery systems, but will also be helpful regarding assessing anti tumor activity if you use cancer cells. Which cancer cell you want to use is largely up to you and will depend to what you can get access. One cancer cell line used quite often is MCF7, a great cancer cell line or A2780, which is a ovarian cancer cell line, but there are many more. Cytotoxicity will be assessed according to protocol in link. A true anticancer activity you would have if you see no or limited cytotoxicity for "normal" cells (e.g. fibroblasts) as opposed to strong cytotoxicity in cancer cell lines. Hope this helps