This observation is correct. Many mathematicians in various institutions and countries resent the impact factor as a measure of scientific excellence since it does not reflect accurately the value of papers in their field. Mathematics is an exact science, but its citation dynamics is similar to the social sciences. One bibliometric rule is that every science field has a different citation dynamics, and the comparison should take place within fields and not between them.
Beside mathematicians, the most Critical of the IF are the medical profession researchers, and the strange thing is that their journals usually have the highest IF of all disciplines. This is just for the record.