Dear Mahadev,

If you wish to know whether your ligand is working as an inhibitor or activator for the receptor, you need to perform a comparative study of your ligand with one known inhibitor and activator of the receptor. And if you get resemblance in the binding sites then you can say the action of ligand.

And yes Athavan Anand has said,  Autodock is one of  the best free software for docking studies. If you can have access then Schodinger Glide is the best.

Best Wishes,

Piyush

Dear Mahadev,

This situation requires that you perform comparative studies of experimentally known activators and inhibitors separately. Please also ensure that a review on the structural requirements of your studied protein is available, if so, they may have highlighted the properties very specifically and distinguishing properties between activator and inhibitor.Then, check your chemical structure and come to conclusion.

If literature does not exist, a set of activators and inhibitors can be used to develop pharmacophore model to assist in this discrimination test (I wont think if a set of molecules are reported, then there is a high probability of explanation of possible mechanism of action in literature). Docking can be misleading as the activator and inhibitor may share similar chemical patterns and may conform to similar binding modes. Hence, an exhaustive sampling of dock generations must be required if you use Autodock. Commercial options are available subject to your availability.

Best Wishes, 

I think calculations of binding energy or binding affinity of the docked molecules could provide a good indication for potential activity of the molecules. Autodock methods are quite However, many other factors, such as solvation issues,good. ADME properties etc also play important roles toward activity. Ultimately, experimental validity is necessary.