The question is a little bit surprising. Dentin basically is a modified bone of dermal origin and keratin is a structural protein of epidermal origin. There are nearly nothing is common between these two structures.
I agree with Marc - the tissues are essentially unrelated. If you are thinking about the transition to filter feeding in mysticetes then you might start with these papers (below) or a popular book on whale evolution - there are several by Berta, Thewissen, DR Wallace and others.
Deméré, Thomas A., et al. "Morphological and molecular evidence for a stepwise evolutionary transition from teeth to baleen in mysticete whales." Systematic Biology 57.1 (2008): 15-37.
A bizarre new toothed mysticete (Cetacea) from Australia and the early evolution of baleen whales. Erich M.G Fitzgerald Proc. R. Soc. B: 2006 273 2955-2963; DOI: 10.1098/rspb.2006.3664. Published 7 December 2006