You should do a preliminary experiment, maybe... 10, 5, 2,5, 1,25, 0,6, 0? If your compound is a highly effective inhibitor you should use a big range of lower concentrations. Good luck!
Generally, you need to know the concentration where there is maximal effect, e.g., all cells are death, and a concentration with an undetectable effect. Then choose a range of fractional values, such that the points are roughly equidistant on a log scale, e.g.. each conc is 1/4 of the next higher one in your series (like 16-4-1-0.25-0.0625-0,016-0 µM).
This is needed to be able to use the 4-parameter logistic regression model for calculation of the IC50. There is commercial software to do this calculation for you, and at least one R-package also exists. This needs the data in a specific format though, best checked out before setting up the experiment. I ended up programming my own script.
javascript:
It may not always be possible to get an IC50 though, if the maximal effect is less than 50% below untreated values, or there is a threshold below which nothing much happens, and above which all cells become necrotic e.g.