After screening of potential drugs in vitro, what is best way of determining the concentrations to use in animal model, like mouse model for example, using IC50 values?
It depends a lot of the type of administration you are going to use. If it is a per os administration I would first try to figure out if there is available data about the logP of the substances to be assayed. In that way you will have an approach of the absorption rate you would have so you could stimate plasma concentrations.
There is also a need to see plasma protein binding to estimate unbound drug concentration. I've seen several cases that unbound concentration at Cmax or AUC correlated with in vitro potency, however, there has been also a several cases that no correlation has been found.
I am not a fan of the LD50. Most of compounds, especially drugs, can have their adverse effects in small doses. In vitro studies allow to determine the impact of the minimum dose at the cellular level. Please read more about so-called hormesis dose response
In reality, PK data are often not available and an educated leap from in vitro to a first in vivo study needs to be made. Two suggestions when you know little about the specific compoundin your possession:
1) Look in the literature for in vivo studies that used very similar types of molecules.
2) Do a range-finder (pilot) study with a few concentrations that are e.g. 10-fold apart.
We cannot predict the range of concentrations to be used from the IC50 values what we get in cell line.
As Mr. J Thomas Sanderson said, you need to do pilot study.
While setting your dosage, you have to take one dose for example 1000mg/kg and give it to the animal and if you don't find any clinical symptoms of toxicity, you you can go the higher dose and find the MTD (maximum tolerated dose). When you take different groups for different dosage concentrations, you make sure that the concentrations are in the log 3 range. You can get the ED50.
The sample size in a group also varies based on the type of experimentation do you work.