If you are using IC50 (LC50) dose of the drug candidate, it is expected to kill 50 % of the cells.
I will suggest you determine the appropriate LC50 dose and the time of drug exposure, so that each time you estimate cell viability assay you find 50 % cell death.
Then you can go for, Annexin V/7AAd staining, that will indicate that your molecule is inducing apoptosis or necrosis. You can also go for the DNA fragmentation assay.
The drug which is inducing apoptosis in a physiologically relevant dose should be the first choice than the other.
After determining these you can assess the other hallmarks of the cell death. neither of the Bax/Bcl-2, Cytochrome C or Caspase 3 indicate apoptosis or necrosis.
Well, herein the cytotoxicity assay type and the repetition of the experiment are the important factors. Do not forget different cytotoxicity assays might give different results and also you must ensure that your substance did not interfere with used cytotoxicity test’ agent. If you are sure that your IC50 is realiable enough, then we can discuss other things. IC50 is the dose of the substance which inhibits the proliferation of half of the cells. It does not necessarily need to kill the cells. If you find high IC50, this means that the cell line is more resistant to drug1 than drug 2. If the drug 2 trigger less apoptosis, then it must kill the cells with different cell death pathway more than apoptosis. In the Annexin test, find the proportion of total add vs apoptotic cells. If the total death more than the cells undergoing apoptosis. Then, you can suspect from other cell deaths, too.
it is a bit hard to give a proper answer without neither knowing the IC50 values you are talking about nor the concentration at which you've carried out your experiment, though:
1. how high is the high IC50 you are talking about (1)? xx or xxx uM or you are in mM range? If IC50 is too high, then your cells are most likely dying from toxicity rather than drug efficiency, which might explain the positive apoptotic results.
2. is it possible that a. you are adding too low amount of compound (2) resulting in the low response, b. having delivery issues (not all drugs are cell permeable), c. your target is redundant or have no major role in the studies cell line (you either need an additional stress factor or you need to use another cell line - have you tried another one?)
3. have you carried out a dose response assay? i would recommend starting by running one.