The formation of the gel phase is most likely due to the formation of high-MW dendrimeric macromolecules with the increasing yield (or extent of reaction).
This issue can be resolved either by "freezing" the reaction at a relatively low yield or altering its conditions to produce polymers with lower molecular weight. Another approach is to use a reaction medium that is capable of dissolving these hyperbranched polymers, however, this might necessitate the use of aggressive solvents.
As per conventional thinking more than two functionality leads to gelation. However, this is not the case as Flory predicted. The controlling factor towards soluble product are the extent of reaction (concentration), reactivity, temperaturer etc. You may find report on extensive study regarding this.
Had some success by reducing concentration and/or solvent change, as mentioned above. Protecting groups may also be effective but a little more involved.