I have done an RMSD validation for such protein by docking the two ligands one by one, but each docking results in large RMSD value compared to the reference.
you have docked already as per your target protein. So, if you target one native ligand site at a time for docking and validate by its RMSD. Secondly, you can validate your docking results for the second native ligand as per the binding site.
In validating ligand binding poses for a protein with two distinct binding sites, employing a site-specific approach is crucial. Docking both ligands simultaneously, considering the unique characteristics of each binding site, provides a more accurate assessment.
Unfortunately, when I docked the ligand one by one to each binding sites, the RMSD value compared to the reference is always large, it can be more than 30.
Is it possible that the large value of RMSD is because it is compared to the reference in the other binding site?
Deka Prismawan Hi! First of all, you can perform docking of multiple ligands in one run simultaneously against one protein with PyRx software. Second, we check RMSD for any control ligand so that we get possibly the best binding affinity between protein and it's control through docking analysis. In case of other ligands (like the ligands you are investigating), you just need to know the location where the control binds. It is very rare to get a RMSD lets say less than 3 for investigational compounds based on the position of control, because the the structure of the control and investigational compounds are different. I can suggest, not to check RMSD for other compounds. A visual representation of the bond formation between investigational compounds and protein at the same domain/pocket as control would be better.