Hi, I wouldn't say in all the cases that FOPID will outperform the traditional PID controller when controlling nonlinear unstable systems. But due to having more degrees of freedom in a FOPID controller the chances of achieving a better closed-loop performance are higher. Either way, it is always better to test which one fulfills your necessities having in mind that PID controllers have local validity and that far from the nominal operating point their performances will deteriorate and the closed-loop system response could even become unstable.
Maybe the following discussion is of your interest:
Yes, of course, then this link within the discussion I have referred you to could help you. Clearly it is not an analysis of nonlinear unstable systems but could give you an idea about how the stabilizing region of a FOPID controller looks like.
Conference Paper Generalized robust stability regions for fractional PID controllers
Yes, in most of the non linear dynamical system the two extra fractional degrees on integer and Derivative control schemes gives superior stability than integer order controllers.