You probably want to consider nanoindentation, but it would be good to know the material you are interested in characterising.
For engineering materials, you can find out more about modelling the contact deformation in "Contact Mechanics" by K. Johnson, and more about measurement techniques in "Nanotribology and Nanomechanics - An Introduction" by B. Bhushan.