Please see the link that answer your question.

https://www.cytodiagnostics.com/pages/gold-nanoparticle-properties

It sounds like your sample is changing over time, i.e. aggregating. This could be due to changing conditions in your preparation over time, and you may find the below article of interest: Article Banana peel extract mediated synthesis of gold nanoparticles

I do not know the chemical composition of your mixture after the nanoparticles ahve been formed, but I'm pretty sure that aggregation is taking placebecause of a progressive change in stability condition of "ALL" the ingredients of the mixture. Therefore, to try finding an answer to your question, I suggest you repeat the entire experiment and examine progressive stages of the mixture condition, over time at fixed storage conditions (e.g., keep in dark environment ata fixed T). At each stage you should not only measure the UV-Vis spectrum, but you shoud also extract a tiny sample of the mixture, drop-casting it on a suitable substrate (possibly hydrophobe if your mixture is in a acqueous host), and measuring an FTIR spectrum on that deposit to have a characterization of any possible change in the chemical bonds of the organics involved (with special attention to carbonil, carbosil, nitrile bands which are often involved in Au nanoparticles stabilization). At least this is what I would do.