Try optimizing the ligand via energy minimization and applying appropriate force field

You should have proper parameter files (in pdb format) before docking your molecule. If you are using Autodock, then you can download the parameter files from Scripps Research Institute website.

First try to assign correct bonding. Co-ordination bonds should be treated diffrently.