This is a good question with many possible answers.

Keeping track of impact factors of journals is a bit like watching a football match.   The better teams that play each other often end up with a tie score at the end of the  game. 

With journal impact factor computation based on the number accesses and downloads,  citations and stiff review policies, it is not surprising that high-demand journals have high impact factors.    For mathematics journals, a true test of the worth of the journal is not impact factor but rather the range of results that are appreciated by mathematicians. The impact factor of a mathematics journal does enter into the picture in evaluating a journal.   Instead, it is the quality of the published articles where quality is measured in terms of results (theorems) found routinely in the articles.     Good mathematics journals are like the better soccer teams with matching scores at the end of the day.   It is the quality of the plays, not the score, that leads to satisfaction among those watching the game.

Over the years a great deal has been said and written about the impact factor. The attached two articles are two samples of these, which I believe are relevant to the question posed on this page.

http://www.nature.com/nature/journal/v435/n7045/full/4351003b.html

http://blogs.nature.com/news/2013/05/scientists-join-journal-editors-to-fight-impact-factor-abuse.html

This is a good question with many possible answers.

Keeping track of impact factors of journals is a bit like watching a football match.   The better teams that play each other often end up with a tie score at the end of the  game. 

With journal impact factor computation based on the number accesses and downloads,  citations and stiff review policies, it is not surprising that high-demand journals have high impact factors.    For mathematics journals, a true test of the worth of the journal is not impact factor but rather the range of results that are appreciated by mathematicians. The impact factor of a mathematics journal does enter into the picture in evaluating a journal.   Instead, it is the quality of the published articles where quality is measured in terms of results (theorems) found routinely in the articles.     Good mathematics journals are like the better soccer teams with matching scores at the end of the day.   It is the quality of the plays, not the score, that leads to satisfaction among those watching the game.

Impact factor is the result of citations of the research papers. I am not sure -- may be researchers in mathematics are comparatively less and so, citations are less.

Other researchers are not consulting mathematical papers as these are either not understandable or not directly applicable. 

Impact factor is calculated based on the number of citations an article of a journal gain in a year. The most impacted journals are the ones which are in fields of medicine and other sciences  in which huge amounts of publications are made due to funds and monetary incentives available from funding agencies/companies of profits.  Because of that, citations in multiplicities are integral and vital parts of the research and publications process.

But mathematics,  first it is a field which does not enjoy fundings like the other popular ones as it does not bring direct profits and second, mathematicians who publish results are fewer in number compared to the number of people in the other popular sciences and   in most cases are solitary who just want to do what they love, i.e., to study and get results not for popular consumptions but to enrich the knowledge of their field and third,  I do not think even we are interested in promoting works of others as well, probably it is within our mathematical DNA inherited from Newton and Leibniz.   

We can imagine the popularity difference between mathematics and these popular fields of journals of impact factors with 40 points or above and the most popular mathematics journal with impact factor of a point below 10.

Dear @Sudev, watch the discussion on this issue in Academia! Why is the impact factor of mathematical journals “often” lower than the impact factor of journals in other discipline?

American Mathematical Society issued a statement The Culture of Research and Scholarship in Mathematics: Citation and Impact in Mathematical Publications! 

The Misuse of the Impact Factor  by AMS member  brings good opinion about!

http://academia.stackexchange.com/questions/7214/why-is-the-impact-factor-of-mathematical-journals-often-lower-than-the-impact

http://www.ipm.ac.ir/doc/CultureStatement09v2.pdf

http://www.ams.org/notices/200707/tx070700821p.pdf

Very limited number of students opted for students, so less number of researchers. So,  Less papers, less citations, low impact factor.

Simply because Most of People Believe in what they see! in Mathematics we believe in what we prove even if its unseen. Been a Mathematician is to enjoy doing math without expecting any reward.