In docking studies, we generally dock a ligand in a binding cavity to find-out the orientation of the ligand and the score of the protein and ligand complex. but How will we decide that it is an inhibitor or agonist.
it depends on the site you are using for the docking study. For example, docking in the catalytic site will prevent enzyme-cofactor interaction resulting in an inhibitory activity. Docking in the activation site can have many possibilities, for example, if the ligand competes/prevents/slows down enzyme-substrate interaction, then the ligand acts as an inhibitor. Anyway, you can use literature sources to understand the mechanism of action of the enzyme and the nature of its binding site. Moreover, the parent molecule that you use for virtual screening is either an inhibitor or activator of this enzyme, then the compounds that you will get from this screening having structural similarities with the parent molecule should act similarly.
Molecular Docking techniques cannot answer that question. You may use bibliography to study molecular mechanism of the ligands. Sometimes the activity as an inhibitor or agonists depends on the formation of a key Hydrogen Bond, other of the volume of the ligand… Sometimes you can us experimental structure data oa de active or inactive form of the protein…
I think the best way is to us complementary information from experiments and bibliography.