It is very difficult to assess the impact of a published paper immediately. It takes time, sometimes even decade for work to be noticed and recognized.
The impact factor considers the previous two years, therefore the publications had some time to collect citations. But I think the 5-year-impact factor is a better way of evaluating a journals' impact because, like you said, it needs some time for a work to be noticed and to attract attention.
Thanks Andreas Meier. By the time the work is noticed and cited, the researcher loses precious time, he / she may lose funding, tenure post and several other good opportunities. But, is there a way out? I doubt it very much.