Once the organism (Saccharomyces cerevisiae) reaches the peak of the log phase, follows the stationary phase, and then it enters the declining phase. Dead cells form pellets and settle at the bottom of your tube or flask (assuming you grow the culture in them). When the cells are dead and settled on the bottom, absorption of light (spectrometric measurement depends on this) reduces and results in decline in the OD600. So, your OD600 declines. Assuming that you make dilutions of your culture to measure the OD, if you plate dilutions of culture on TSA (or any other suitable media), you can see lowering of CFU after the stationary phase. However, it looks not so normal that the culture reaches declining phase so fast. More information on the growth condition, including pH, dissolved oxygen, etc. are needed to understand the issue properly.