Although I can't prove it, I'm starting to believe that there is no simple expression for either (1) or (3). It looks to me like the simple expression for (2) is very special. Any comments?
at the best of my knowledge there is not a closed form expression to solve the integral you mentioned. However, a possible solution to obtain a closed form expression for (1) is to consider the series expansion of J_n ( see http://mathworld.wolfram.com/BesselFunctionoftheFirstKind.html eq. 42 ), multiply the terms of both the series ( J_n(ax) and J_n(bx) ) and simply integrate the resulting polynomials. I used such a "trick" to obtain a closed form expression ( even if expressed as inifinite summation ) for the cumulative distribution function of the Rice distribution ( in that case I used the series expansion for I_0 ). In my case it works quite well and I have very small error for the range of variable I was interested.
I'm wondering if summing such a series is any better (for accuracy or efficiency) than a numerical integration algorithm such as that provided in MATLAB (which I happen to be using); especially in light of the fact that a double summation results from multiplying the series expansions for Jn(ax) and Jn(bx). I suppose a series approach might be better if it converges quickly enough.
Depending on a and b, the integrand Jn(ax) *Jn(bx) can be somewhat oscillatory over the interval [0,1], which is a little worrisome. That's one reason I was hoping for an analytic solution to (1) and (3).
I'm thinking about conducting a numerical experiment using (2) that compares (a) the series solution, (b) the MATLAB numerical integration algorithm, and (c) the exact solution. That should give me a pretty good idea of what approach to use for (1) and (3).