It could just be microbial contamination, in which case you would need to remake it and filter sterilize it or autoclave it. If the growth happened that quickly in an inorganic phosphate buffer there's a good chance the water you used to make the buffer, the salts themselves, or the glassware is heavily contaminated with organics. Just something to keep in mind if there's no source of organic compounds in the recipe itself.
If the buffer recipe also contains magnesium, calcium, or iron salts these can form insoluble phosphate salts over time which will precipitate out and make the solution cloudy, which can also look like bacterial growth (but not puffy growth like a mold). In that case autoclaving would probably make the precipitation happen even faster and filtering it wouldn't make a difference.
The buffer wasn't filtered. Could this be the cause?
Yes.
Can we autoclave phosphate buffer to resolve this issue?
No. phosphate buffers by themselves are generally not autoclaved due to the common occurrence of precipitate formation. Filter sterilization might be a better option to consider.
If you leave phosphate buffer standing for long enough at room temperature, it will become cloudy with little creatures growing in it. Usually, one or two weeks of expiry should be sufficient without any problem of contamination. I prepare 10X PBS with short expiry (10 days) and do not face such problems of microbial growth.
Some measures you could follow to avoid buffer-growth problems.
1) Do not refill a buffer container.
2) Fill a clean container and replace it with a new one rather than refilling the same container when more buffer is needed.
3) Try to keep buffer expiration period short and regularly clean the containers.
4) Ensure that the container is tightly sealed to prevent contamination and evaporation.
5) Protect the container from direct sunlight and extreme temperatures.