It would be better to include all the hydrogen atoms of the ligand or receptor in order to consider the atomic charges of the whole system. If you are only considering hydrogen atoms of the polar group, ignoring the hydrogen atoms of others would not affect your results much. Both ways can be tried.