You can try epipolar geometry which is explained in

Hi,

You may find a good approach either by: a) 3D location from full perspective model; or b) 3D location from Weak-perspective model. In a) the form of solution is to establish set a of linear equations that can numerically be solved (extrinsic parameters by Newton-Rapson method). For the second way b), a close form may be set.

A good information source is the book "Introductory techniques for 3D computer vision" by Trucco E., and Verri A., Try the chap. 11 (require previous camera model review in chap 2). The chapter 11 is devoted to the problem you are stating.

Good look!

See e.g. Negahdaripour, S. & B.K.P. Horn, “A Direct Method for Locating the Focus of Expansion,” Computer Vision, Graphics and Image Processing, Vol. 46, No. 3, June 1989, pp. 303–326.

also related to this:

McQuirk, I.S., B.K.P. Horn, H.-S. Lee, & J.L. Wyatt “Estimating the Focus of Expansion in Analog VLSI,” International Journal of Computer Visions Vol. 28, No. 3, 1998, pp. 261–277.

and

Horn, B.K.P., Y. Fang and I. Masaki, “Time to Contact Relative to a Planar Surface,” IEEE Transactions on Intelligent Transportation Systems, June 2007.

I think you can get the reply if you go through the first reference cited by Prof. Berthold. That's pretty informative.