Not much of damage will be made to the cells. The viability ratio of the cells get affected only when the temperature goes beyond -50 degrees. The major concern with respect to your question is that the shelf life of the cells reduce when you get it back to -80. It may stay stable for maximum one year and this also depends on the handling and storage media used.
At least for some primary culture it will affect the cell viability. We sometimes keep pulmonary artery endothelial cells in -80C, because a lot of our young scientists are so busy and forget to put the cells back to liquid N2, for a few weeks and reculture them later for other experiments. When compare to the cells, directly from liquid N2, we see more dead cells and sometimes they refuse to attach to the bottom even after 2 days.
It is far better to store long-term under liquid nitrogen. Ice crystals restructure at temperatures above -115C. Below this temperature, a temperature called the glass transition temperature, solid water, and what is in it, is stable.
Short-term, many people use -80C to store cells.
I call long-term, indefinitely; short-term a year.
Viability and total recovery may be affected. However, most cell lines are robust and make take time to recover in culture once thawed. We've tested more than 50 cell lines at iQ Biosciences for our in-vitro bioservices and found that in most cases, not an issue. However, for good practice, we store all our banks at liquid nitrogen (vapor phase).
My research is on the physics as well as the biology of cryobiology. If you want the safest way of storing cells, store them below -115C. This can be done in the vapor phase of liquid nitrogen, but it depends on the tank. Some new tanks and most older tanks had temperature variation between the vapor just above the nitrogen and the vapor closer to the top of the tank. Some newer tanks address this issue and have vapor temperatures below -115C throughout the tank. Our and other collaborating laboratories freeze gametes and embryos. In the old tanks we observed a decline in the survival over time. Now, with these newer tanks, we get the same survival as when thawed just after freezing. I hope this was useful.