Modal analysis requires a linear system. If your system is non-linear then the stiffness and hence the frequencies will be a function of load and you will need to perform a dynamic analysis to obtain the response.
After about one year I must acknowledge you for your answer to my question. Though, I could not model nonlinear torsional vibration in Abaqus but it is necessary to really appreciate your help. God speed you