For extracts, you can indeed follow the suggestion of Isabelle.
For pure compound, you can test six concentrations from 0.001 to 100 µM.
Be aware that your choices about the cancer cell lines to be tested can markedly impact on the 50% growth inhibition concentrations (GI50s) of your compound(s) of interested, with sometimes more than a log of magnitude in GI50s for the same compound on different cell lines. Especially if you are working with pro-apoptotic compounds, or with extracts from which you want to isolate pro-apoptotic compounds.
As suggested by Isabelle for extracts, you should also use a "positive" control for your compounds (see the attached articles).