By which approach of study in plant science will lead to find the ancestral plant in a genus or family? what were the molecular biological experiments which ensure to predict the ancestral plant among genus or family ?
Botanists may approach this question differently, but as a general phylogenetic principle, no living taxon is ancestral to any other living taxon. Different taxa diverged from a common ancestor at some point in the past. The common ancestor does not exist past that point of divergence, but is split into two or more separate lineages with different subsequent evolutionary histories. Different lineages may preserve some or many features of the common ancestor, but none IS the common ancestor. If you want to know what the common ancestor might have been like, you do phylogenetic analyses to try to identify which characteristics the living taxa are primitive for the group, i.e., present in the common ancestor. You then partially reconstruct the common ancestor based on these primitive traits.
Skulan is right. Of course, there WAS an ancestor (population at least) but it is not there any more. Given that in the long run most organisms die without leaving descendants the chances of finding a fossil that is an actual ancestor is very very small, and in any case, how would you know? At best, you could say that this fossil seems to have all the characters of living species in a group that we regard as plesiomorphic.
I strongly agree with Mr Skuland and Mr Cameron. By definition of the term 'ancestor' it does not exist anymore after the split of its descendants. Therefore it is impossible to find the ancestor among the extant species.
But still there is a lot of methods available if you are interested to reconstruct a certain trait of the last common ancestor of a given group of species. There's different methods to reconstruct morphological, physiological or ecological traits of the hypothetical last common ancestor as well as the distribution and time, when its descendants split from the ancestral line. All of these analyses are usually based on a phylogenetic tree which itself is based on DNA data. If your new to this topic, I strongly recommend reading some recent reviews as molecular phylogenetics in plant science is in a big upheaval right now. You could start with two papers by Elizabeth A. Zimmer and Jun Wen (2013,2015), available at:
If you are planning to actually do a project in the field of molecular systematics, I also recommend looking for a cooperation with someone familiar with molecular phylogenetics and ancestral trait reconstructions as the methods are highly diverse and have lots of pitfalls.