Paleoclimate records, from within the hydrological system itself, for instance ice core records, can help reconstruct climatic conditions for the last 800 000 years. The longest ice core on records was drilled at the site of Dome C, in Antarctica and highlight 8 glacial/interglacial cycles. Within the ice, the water isotopic composition is a good temperature proxy, while the CO2 concentration is archived within air bubbles trapped in the ice itself. You can read more about this for instance in this article: https://www.nature.com/articles/nature02599
There are many more ice cores covering different time spans, but so far, none go beyond 800 000 years, even though, there is an ice core project Beyond Epica - Oldest Ice which aims at getting up to 1.5 million years records.
You can find paleoclimatic records in marine sediments that extend beyond the 800 000 years. One of the reference is a stack from Lisiecki and Raymo, which covers 5.3 million years: Article Lisiecki, L. E. & Raymo, M. E. A Pliocene-Pleistocene stack ...
Sea level changes are also recorded by both marine and ice records, due to the strong interactions between ice sheet and sea level rise. In this regard, I would recommend you to read the special report on Ocean and Cryosphere of the IPCC: https://www.ipcc.ch/srocc/download/
It includes a comprehensive overview of past, present and future changes of the hydrological cycles for a changing climate.