The chapter on "The Geography of Imperialisms" in Peter Taylor's "Political Geography, world geography, nation state and locality" is probably very useful and is based on data compiled by Henige D.P. (1970) Colonial governors from the 15th Century to the Present, University of Wisconsin Press, Madison. Taylor's book has been re-edited several time with an additional author Colin Flint, who became the first author.
You might also contact Pieter Saey (formerly from the University of Gent, Belgium) who has complied such a list, but probably only in Dutch: [email protected].
The dataset by CEPII "GeoDist" is a bilateral dataset which contains information about every country in the world indicating colonial links with other ones. I think it can be very useful for you.