Several methods, based on different algorithm types, have been successfully

applied to model the complex relationships between taxon assemblages and environmental variables: weighted averaging regression – calibration based approach (ter Braak and van Dam 1989; Birks et al. 1990), weighted averaging partial

least-squares regression (ter Braak and Juggins 1993), maximum likelihood based approach (ter Braak and van Dam 1989; ter Braak et al. 1993), full probability based approach (Bayesian modeling) (Ellison 1996; Toivonen et al. 2001; Vasko et al. 2000), and artificial neural networks based approach (Racca et al. 2001; Köster et al. 2004).

See:

Racca, J.M.J., and Prairie, Y.T. 2004. Apparent and real bias in numerical transfer functions in palaeolimnology. J. Paleolimnol. 31: 117–124.

Tailoring palaeolimnological diatom-based transfer functions

Julien M.J. Racca, Irene Gregory-Eaves, Reinhard Pienitz, and Yves T. Prairie                                                                                                                             NRC Research Press Web site at http://cjfas.nrc.ca on 21 February 2005.

Abstract: This paper presents a method designed to build species-tailored diatom–environment models. Using a pruning algorithm of artificial neural networks, powerful species-tailored models constrained to water temperature, water depth, and dissolved organic carbon were developed from a 109-lake training set from northwestern Canada and Alaska. The reasoning behind the approach is that the implementation of a single, gradient-based, organism–environment relationship should only use species that are comprehensively influenced by the variable of interest. By pruning species according to their relevance to each of the three studied variables, the cross-validated performances of all three models were significantly increased, suggesting that nonrelevant species have corrupting influences and need to be removed. The removal of corrupting species also suggests that palaeolimnological transfer functions based on an appropriate subset of useful species are more independent.

If your samples are of Miocene/Pliocene age it is quite likely that most of the diatom species you'll find in them are now extinct, so you won't find them in modern samples. In which case, you will have no modern analogue of your fossil samples and the transfer function technique will not work.

I can't remember any publication trying to use the transfer function technique on such old samples. I think you can only try to surmise the past environment conditions by assuming that  the fossil species you find have similar ecology to modern species that are very close (same genus, similar cell size). See papers by J.P. Bradbury for examples of this applied to long sedimentary records, especially to Lake Baikal.

Bradbury J.P. (1999) Continental diatoms as indicators of long-term environmental change. in Stoermer,E.F.; Smol,J.P. (eds): The diatoms: applications for the environmental and Earth sciences. Cambridge University Press, pp 169-182.