Only two useful answers... "Sub-scale and Subscale"
But subscale is more grammatically correct in general. For sub-scale (hyphened) means a "name" given to a particular scale within a scale. The sub-scale (hyphen) is placing an emphasis on something that someone should take or make note.