A few journals with impact factor (3 to 4) are publishing average quality papers whereas some journals with impact factor (1 to 2) are publishing the same quality or in some cases better quality papers. Hence, impact factor is not enough to measure quality of a peer reviewed journal.
It is not a very accurate measure, because it measures the citations to the journal articles, which is basically based on the accessibility. Nowadays, the accessibility to open access journals makes them getting relatively high impact-factors even though the quality might be less than the subscription journals.