The formula works for commuting matrices, i.e. if AB=BA. In the other case (AB)2=ABAB, and in general you cannot reorder the multipliers and get A2B2. As an example take
A whose first row is 0 0 and second row is 1 0, and
For the fraction "a" and two commuting matrices A,and B, is it "(AB)^a =B^aA^a" true. ? ( In particular, for a= 1/2 power is called the square root of a matrix and does it work ?)
The theoretical background for your question is the theory of functions of self-adjoint operators (matrices). The fractional powers are correctly defined for Hermit-symmetric non-negative matrices, i.e. for matrices A that satisfy condition $ \ge 0$ for all vectors x. For such a matrix A and for q = n/m and function f(t) = t^q the matrix f(A) is the unique non-negative matrix G that satisfies equation $G^m = A^n$. If two matrices A, B commute, then any functions f(A) and g(B) of these matrices commute as well. From this you can prove the desired formula $A^qB^q = (AB)^q$ for q = n/m just by elevating $A^qB^q$ in power m and checking that the resulting matrix equals (AB)^n.
when I recommended to clarify in what sense is understood the fractional degree of matrix, I meant exactly what Vladimir Kadets says in his last answer.
Let A, B be square commutative matrices of order m. Then for arbitrary integer n the equality (AB)^n=B^nA^n holds.
Proof. (AB)^n=ABAB... AB (n times). Since the matrices commutate we can move all Instances of the matrix B to the beginning of the last expression. We obtain
For two commutative matrices A,B of order n(integer), (AB)^n = B^n A^n. Which concludes that " Matrix commutation implies (AB)^n = B^n A^n", but what is about its converse ?