Although there may be multiple factors that could account for your observations, the simplest explanation would be that at "lower" dose of the compound there are fewer targets and as the concentration of the compound is increased the number of targets are increased. In toxicology, it remains a challenge to identify targets at different doses of a compound.

Very nice answer by Dr. Divaker Choubey .

Is this a uniform cell population? Often the answer involves mutations or presence of multiple phenotypes. A dose-response curve should provide a hint as to the answer.

Based on your results you are most probably dealing with the phenomenon that is Hormesis, although I am not aware of such clear negative effect (at a low dose) as the ones you report. Shortly (and not precisely; look up papers on the subject, especially by Edward Callabrese and Suresh I. S. Rattan for more detail) Hormesis describes the dose-response to toxic compounds by a biological system as U-shaped or invented U-shaped based on the endpoint measured (similar to your results), and not as linear. This would make small doses mildly toxic (mostly unmeasurable), intermediate ones potentially beneficial, and high ones toxic. 

I support Dr. Divaker Choubey's answer as the most probable mechanistic explanation for why this negative effect takes place at the small doses (and might explain why it is so clear), but some of the negative effect might also be down to the cells incapability to induce their stress response pathways (see papers by Suresh I. S. Rattan) when dealing with very small amounts of stress, possibly resulting in a gradual and debilitating build-up of damage. The intermediate dose, however, induces a stress response in the cells, leading to the clearance of the damage and an overcompensation by the cells, which again leads to an improvement in cellular growth (compared to non-stressed cells).

Thank you for all your explanations. There is an exact ∩ shape in the growth curve. I will read atricle Peter suggested. By the way, the cell pouplation is uniform.    

What assay is being used: clonogenic, growth rate, something else?

MTS assay was used as a cell proliferation assay

This assays the activity of some mitochondrial dehydrogenases and may or may not reflect true viabiltiy (clonogenicity). There can be delayed toxicity or repair. See what happens if you use another criterion of growth. I recall a study where the investigators were assessing the toxicity of reactive oxygen species. The MTT assay reported that sodium azide could reverse toxicity, but it only reversed ROS formation.  Sodium azide is itself quite toxic.

Could the compound you are using be precipitating at higher doses? Or interacting with MTS reagent . You may need to try other viability assays to confirm your results

The compounds are highly soluble in the medium so there is no chance that the cemical precipitated. For your second question, MTS reagent containing medium is applied to the cells after removing compound containing medium at the end of incubation period.

You are right that I should try additional proliferation assay to be sure about the findings. However, I am now trying to know whether it is possible to decrease cell viability or suppress proliferation at low doses for any compound while higher concentrations increases. 

I am not an expert of the cytotoxicity of a chemical compound but found your observation interesting from the perspective of cytotoxicity of ionizing radiation. At a cellular or molecular level, the shape of the radiation dose response relationship is not necessarily linear, and provided below are some examples.

Irradiation of normal human diploid cells at 2 or 5 cGy has been reported to stimulation of cellular proliferation . At 0-6 Gy, whereas p53 accumulation, p53 phosphorylation at serine 15 and increased p21 expression increased with increasing dose, ERK1/2 phosphorylation was bichasic as it increased at 2 and 5 cGy, decreased and increased again at >1 Gy. Such stimulated proliferation at 2 or 5 cGy was alleviated with AG1478 (epidermal growth factor receptor tyrosine kinase inhibitor) and PD98059 (MEK inhibitor). 

https://www.ncbi.nlm.nih.gov/pubmed/11454682

Other low dose phenomena include low-dose hypersensitivity (cells are more effectively inactivated at low dose than at high dose), bystander effect (the effects occurring in cells that have not themselves been irradiated but have received signals from irradiated cells), and adaptive response (cells primed with low dose become radioresistant to subsequent higher dose radiation).

Effects are also different among dose rates. Depending on some endpoints and radiation quality, the lower the dose rate the lower the biological effects (dose rate effect), or conversely, the lower the dose rate the higher the biological effects (inverse dose rate effect) 

Viability is not the same as clonogenicity, and provided below are five example approaches for evaluation of cellular proliferation but at different endpoints.

[1] A simplest way is to directly count cell numbers at different days after plating.

[2] A colorimetric assay based on MTT, XTT, MTS, WST-1 or WST-1 measures the enzymatic activity of mitochondrial dehydrogenase (succinate-tetrazolium reductase) that converts these tetrazolium salts into purple colored insoluble formazan. A similar assay employing resazurin or AlamarBlue can be used both colorimetrically and fluorometrically. The assay of this type evaluates the mitochondrial metabolism.

[3] Tritiated thymidine uses radioisotope and evaluates the DNA synthesis. A similar assay employing other thymidine analogues BrdU, IdU, CldU or EdU does not need the use of radioisotope. There have been S phase markers available, such as Ki-67, PCNA, DNA topoisomerase II, and histone H3. Evaluation of E2F (required for a transition from G1 to S) or other proliferation-related markers may also be useful.

[4] A dye exclusion assay measures the membrane permeability and uses colorimetric (e.g., trypan blue, Erythrosin B) or fluorescent dyes (e.g., propidium iodide, ethydium bromide).

[5] A colony formation or clonogenic assay measures the ability of single cells to form colonies with 50 or more cells, usually in about two weeks after plating.

Hi Selami Demirci and Mr. Peter Jorgensen,

I am using the cell proliferation assay for Human T and B cell by stimulating lymphocytes with PHA-M and CPG respectively. The final objective is to test and compare the inhibitory activity of the cell proliferation by cyclosporin-A and other two phytochemicals. I am using WST-1 reagent as a detection reagent for end point analysis.

Although I have a satisfactory results for B cell proliferation assays, In T cell assay I have the similar inverted U dose response (Higher toxicity at low doses of the compound) and the conversation from Mr. Peter and Dr. Divaker helped me to understand the effect could be due to the "Hormesis".

1.What is the alternate to test the efficacy of these compounds at these low concentrations?

2.Use of CCK 8 instead of WST-1 can help improve the results?

3. @Selami Demirci:  Have you been able to overcome the problem and able to perform the assay furhter? If so, what have you done to improve the results?

Regards,

Dharmendra