It can be due to a single, highly cited paper, see the story of the Acta Crystallographica Section A Journal Impact Factor outlined in the Scientific American blog http://blogs.scientificamerican.com/information-culture/2012/05/07/understanding-the-journal-impact-factor-part-one/
Just search for "Plant Molecular Biology Reporter" in Google Scholar between year 2011 and 2012 (or just follow this link: http://scholar.google.se/scholar?q=Plant+Molecular+Biology+Reporter&hl=sv&as_sdt=0%2C5&as_ylo=2011&as_yhi=2012 ). Look at the citations for each paper and you get your answer. Its because they published a lot of papers that were cited quite a lot, many over 20 times. Kind of impressive actually...
Did any of you ever submit to PMBR? You would know the answer. Count the journal self citations in articles from the recent years; you will find a minimum threshold ...
It is theoretically possible for a crowd of authors to jointly affect a small journal's IF. If so, I agree that the IF should drop again, but then we may see another small, low IF journal booming.