This is a good question with lots of possible answers.

A worthwhile approach to consider in mathematical modeling a robotic arm is given in

https://www.researchgate.net/publication/263466578_Mathematical_Modeling_and_Trajectory_Planning_of_a_5_Axis_Robotic_Arm_for_Welding_Applications

Another good paper to check is

https://www.researchgate.net/publication/237576692_Optimal_control_based_in_a_mathematical_model_applied_to_robotic_arms

Conference Paper Mathematical Modeling and Trajectory Planning of a 5 Axis Ro...

Article Optimal control based in a mathematical model applied to robotic arms

You can refer to professor Utkin link is attached below:

https://books.google.co.in/books?hl=en&lr=&id=8IrLBQAAQBAJ&oi=fnd&pg=PP1&dq=sliding+mode+control+in+electromechanical+systems+utkin+pdf&ots=2UEJ9uv3_z&sig=2zN-u-XER1XEfu2g3X5ILoLmP4w#v=onepage&q&f=false

Or you can refer to this thesis: http://www.iitg.ernet.in/engfac/chitra/notes/thesis.pdf

My professor, Academician Miomir Vukobratovic was the founder of Belgrade School of Robotics. His work include the mathematical modelling of robot arm.

http://link.springer.com/chapter/10.1007%2F978-94-017-0397-0_2

http://facta.junis.ni.ac.rs/macar/macar2000/macar2000-28.pdf

https://books.google.rs/books?id=UaTbBwAAQBAJ&pg=PA121&lpg=PA121&dq=vukobratovic+robot+arm&source=bl&ots=dm3qpH0ohK&sig=rCABYRuObA7PU2LQw8c9jDBoFu4&hl=en&sa=X&ved=0ahUKEwio7sqS45rNAhVBvhQKHQtXBb8Q6AEINTAG#v=onepage&q&f=false

http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19890017188.pdf