We have standard methods like Kern's method for calculating Reynolds number for shell side fluid, but for helical baffle shell and tube heat exchanger it is yet to be devised.
You can use the conventional form of Reynolds formula, Re = 4 m/ (pi) Dh mu
but use the more general form of Dh, which equals four times the net flow volume divided by the wetted area (take care: volume and area not area and perimeter)
The net flow volume is volume of shell - volume of tubes - volume of helical baffles