I usually respond to every point mentioned by reviewers. You need to pay a special attention to those points that are critical. And you can thank the reviewers for general observations that do not require any changes to your manuscript.
It is best to respond to both specific and general issues raised by the reviewers. Be polite in your responses and thank them for their careful reading of your manuscript.