A praise worthy attempt has been given to develop an universal interpretation of fractional derivatives by Podlubny. See,

I. Podlubny, “Geometrical and physical interpretation of fractional integration and fractional differentiation,” Fract. Calc. Appl. Anal. 5, 367–386 (2002).

But still this is an open problem as considered by the top guns in the field. See, 

J. T. Machado, F. Mainardi, and V. Kiryakova, “Fractional calculus: Quo

vadimus? (Where are we going?),” Fract. Calc. Appl. Anal. 18, 495–526 (2015).

However as far as interpretation in viscoelasticity and transport phenomena is considered, it can be found in my PhD thesis through its second, third, and fourth article.

https://www.researchgate.net/publication/309490737_Physical_and_Geometrical_Interpretation_of_Fractional_Derivatives_in_Viscoelasticity_and_Transport_Phenomena

See, https://www.researchgate.net/publication/308514869_Linking_the_fractional_derivative_and_the_Lomnitz_creep_law_to_non-Newtonian_time-varying_viscosity

https://www.researchgate.net/publication/311528051_Connecting_the_grain-shearing_mechanism_of_wave_propagation_in_marine_sediments_to_fractional_order_wave_equations?ev=prf_pub

For more details, see my latest PhD thesis (just 4 weeks old) which also includes historical development of the field. You will also find the answer to the "physical significance" part of your question in my second and third paper explicitly. Hope that helps.

Thesis Physical and Geometrical Interpretation of Fractional Deriva...

Article Linking the fractional derivative and the Lomnitz creep law ...

Article Connecting the grain-shearing mechanism of wave propagation ...