First, you need a good theoretical model which is able to calculate the energy (mass) of a nucleus as a function of shape parameters. Then you can calculate that energy as a function of those parametrs and look for the minima in the multidimentinal space. The lowest (global) minimum corresponds to the ground state (its parameters determine the shape), next local minimum is considered as superdeformed one. If the fission barrier extends to big deformations and is high enough, sometimes the third (hiperdeformed) minimum can be observed in rotational spectra. Several, reasonably good models exist in the market. Generally all models are more or less phenomenological and calcultions require a big computing power.