Hi,

Generally, a refining of grain size in the HAZ is usually a wanted procedure and accepted. This procedure can be achieved by suppressing the growth of austenitic (γ)

grains using particles of oxides and sulfides which are nanometer in size, contain elements such as Mg and Ca and are stable at high temperatures.

Please look at Microstructural Control of Weld Heat-Affected Zone of Steel

by Mn Depletion around Non-metallic Inclusions by Shuji AIHARA et al. Published in NIPPON STEEL TECHNICAL REPORT No. 91.

Yes, it is  rather  a normal situation since TMAZ is formed not only under the action of a FSW tool but unsder the action of different thickness metal flow layer and therefore its recrystallizationis retarded as compared to that of the NM. 

According to the sharp slope of strain, strain rate, and temperature from the BM to SZ, it is expected that the microstructural features will be different in various areas. It seems that in your case, lower temperature in TMAZ compared to that of the SZ may cause to less grain growth of recrystallized ferrite grains. Assuming that the heat input is enough for the dynamic recrystallization in TMAZ, too.