I recommend clean all the glassware with freshly prepared HCl: HNO3 solution (molar ratio of 3:1) and water. It will help you to avoid undesired nucleation during the synthesis. Due to contamination, the gold colloids may also aggregate.
It might be contamination. Please note that the synthesis of AuNP is sensitive to impurities per my experience. I suggest cleaning your bottles with aqua regia & EtOH 3 times before you prepare.