In planetary gears , the best way to have a solution is to make analytical solutions for all included epicyclic gears, taking in mind the loads and the way of rotation of all members. You have to be carefull with the negative values of the solution of the system of equations.