Any metabolic cell viability assay (MTT, resazurin, ..) would help you to plot a curve for concentration/viability % to calculate IC50.
Generally, we used 30% of the IC50 to start studying the activity of a tested compound.
But when you find a promising compound in the early screening, you should study the effect of different doses (NOT ONLY 30% of IC50) and different incubation time too.
I definitely agree with the colleagues above. I would like to add another vital dye which suits your needs: neutral red (see Nat Protoc. 2008;3(7):1125-31. doi: 10.1038/nprot.2008.75.
Neutral red uptake assay for the estimation of cell viability/cytotoxicity).
I guess that the first point to test is a dose response curve, to measure for example lactate dehydrogenase release or trypan blue exclusion using a wide range of concentration (0.1, 0.3, 1, 3, 10, 30, 100 mM) and using two different incubation times let say 2-4h and 24-48h. Then you have also to determine the effect of a non lethal dose on cell proliferation, simply by counting the number of cells or measuring DNA content.
As a personal experience: Lactate dehydrogenase does not work so well with long time incubation 24-48H for some compounds due to the deterioration of the released LDH, even with numerous controls...