Most of the mass of what you think of as "matter" comes from protons and neutrons, which are baryons. While the interaction with the Higgs field explains the mass of the quarks inside the baryons, most of a baryon's mass comes from the binding energy of the gluons that hold it together.