you open more brackets than you close;

don't use floating point 0.5 if you want it analytical, use 1/2;

I attached a Maple file

Dear Professor Slavchov,

Thanks a lot for your answer.

Hello Ercan: you already have an answer directly using integration but here is one from Maple's Statistics package making use of the fact that this is a probability distribution function:

with(Statistics):

X := RandomVariable( PDF = (x->(6/Pi)* (1-x^2/4)^(1/2)/((5^(1/2)+5^(-1/2))^2-x^2)) ):

PDF( X^2, x );

The result from Maple is:

piecewise(x