That pH will not kill most species of clinically important bacteria, even neutrophiles. It may stress the cells out a little, but I think that's about it. Perhaps if you have prolonged (>4 hours) incubation of the cells in just that buffer without any carbon sources then you might reduce the viability a small amount. I say this, but there are always weird species out there, so it depends on the species. It wouldnt be hard to find out if your buffer kills them. What species are you looking at?
I am dealing with BCG (the vaccine for TB) which needs 21 days to form a visible colony. So i hope there's supporting view that it wont kill them as basis then I start CFU counting.
Yeah, if you're just using the buffer to resuspend the cells before plating for CFU then it should be fine. The time the cells would spend in the buffer isnt very long.