You cannot infer such information from the protein-ligand interactions alone. But if you have an example of an agonist and an antaginist for your receptor and know how they bind, then you can hypothesise that a new compound that interacts with the receptor more like a known agonist is an agonist. And the one that interacts like an antagonist would be an antagonist.
Agonists stabilize those receptor conformations that triggers the downstream signalling events. Such conformations have higher affinity to certain signalling proteins (such as, G-proteins or beta-arrestin for G-protein coupled receptors). Antagonists usually bind to the same pocket (and, hence, outcompete endogenous agonists), but the receptor conformation that they induce is not optimal for binding signalling proteins and, hence, does not trigger the downstream signalling cascade.