I do not think there is a bias in the calculation, what happens is that there is more funding for health research (Biological Sciences) and that probably makes that these journals have the highest impact factor. On the other hand, there are journals with low impact factor and good quality.

On the other hand, there are journals with low impact factor and of good quality, in this case the apparent discrepancy only depends on the scope of the study.

To say that the impact factor is biased is probably not entirely correct, but it is fair to say that impact factors are not easy to correlate between different fields. Lucero's comment above about relative funding may have some validity, but I think that a more important factor is the difference in citation and publication cultures in separate disciplines. A useful resource is the Essential Science Indicators Baselines citation page from the ISI Web of Knowledge. Unfortunately this is a subscription only page, but the link is: http://esi.webofknowledge.com/percentilespage.cgi. This page shows the percentiles for specific disciplines based on citations. So for example, the top 1% of publications (defined by citations) in 2010 across all fields had greater than 35 citations, in Agricultural science this figure was 18, In Mathematics 12, Chemistry 43, Biology/Biochemistry 40, and Physics 34. So with this in mind I would suggest that the impact factors are really only significant for relative comparisons within a subdiscipline, and have relatively less importance when used to make comparisons between disciplines.

I do not feel there is any bias to calculate IF of journals between physical science and biological science fields. If the same formula is used to calculate IF of Journals in all Science fields then how can it be biased. No need to say that it is the most important way (Journal citation report by Thomson Reuters) to evaluate the quality of Journals in science because it is accepted world wide officially. Now a days there are so many ways people are calculating IF and assigning to Journals. In many cases a particular organisation also calculate IF of all the Journals and recruits the staffs basing on this. NAAS rating in India is one of them. However, in such rating cases there are many limitations. For example maximum rating factor of 10 irrespective of the citation index. In this case, there may be difficulties to evaluate a Journal properly. In simple words, what ever a man will work in a day he will get maximum 10$ per day irrespective of his output by his talent. One man having less expertise, less quality and less talent may get less price but the person having the opposite quality must get proper price which may be higher. If the fact is under stood then in my opinion Thomson Reuters offered IF of Journals is the best to follow which is impartial to all field of Journals irrespective of Physical and Biological Science. On the other hand the low IF of physical science Journals may be attributed by number of research people and publications followed by citations. Since these factors are high in Biological sciences so IF of Biological Science Journals are higher than the former one. Also IF of a journal always not considered for the reputation of a journal. In a particular field the Journal may be at the top position irrespective of its low IF. For example BBRC is the Number 1 journal in the Thomson's JCR ranking for Biophysics in terms of Total Cites, Number of Articles and Eigen Factor ™ score. So in this case again the total number of Journals in this field and the publication followed by citation comes into count.

Thanks a lot you all guys

As chair of the publications cmte for a large and venerable scientific society I attended training sponsored by the publisher. I remember one of the speakers -- apparently an expert on the topic -- making the point that impact factors should never be used to make comparisons between journals in different areas. Use it only to compare journals that actually compete head-to-head for the same papers and readers.

Accepting this view means rejecting the premise of the question, or perhaps just answering it negatively.

@Pranaba. My answer is yes I agree that biological/medical journals have higher IFs than physical/engineering journals, and that if these are IFs are indiscriminately used then it does indeed result in bias.

So the standard solution to this problem is never to quote IFs directly. Rather, rank the journals in a field according to IF and refer to a journal as being ranked Nth out of a total of M journal in the XYZ field.

That is how to remove the bias.

Another metric worth considering is Eigenfactor (EF) rather than Impact Factor (IF), and in this case again you need to remove bias by ranking according to EF within a discipline.

You can easily do all these rankings on ISI Web of Knowledge. All the online tools are there to do it at the click of a button very easily.

You can even rank all ~8000 journals in the world and that is quite interesting. Nature is top no matter if you use EF or IF. If you rank by EF you get some interesting surprises, like Applied Physics Letters is 5th in the world, and PLoSONE is 12th in the world.

I quite like Eigenfactor as it gives weighting to how often journals are cited outside their own discipline, and it also removes journal self-citations. Journals that support multidisciplinary research naturally come out pretty high on the EF scale.

Thank you Derek.

see a long list of weaknesses of the h-index in an interesting paper by Claude Ziad Bayeh:

@Paul, whilst there certainly are many problems with how the h-index is applied and the meaning ascribed to it, the link that you've attached is a very poor way to go about trying to fix it.

As a matter of principle, we should all be unsurprised that distilling a scientist's importance down to a single number is always going to be flawed. The h-factor has taken a prominence that is worrying because of the ability to game it, and the feeling of being somehow objective, when it fact it is far from it.

However, the paper you have linked to, seems to propose replacing the h-index with their own made up j-index. The j-index has a bizarre and complicated number system that again, appears superficially to be objective, but is full of arbitrary weightings to various tasks. This includes an arbitrary distinction between three different classes of journals, each with their own weighting.

I was amused to note the weighting provided to having apparently multiple PhD and Masters degrees. Of course it is standard for most of us to have one PhD, and a few exceptional people have two (although there are usually interesting reasons for that in those few cases). I note that Prof (?) Bayeh has 3 PhDs and 2 Masters, apparently taking around 1 year for each.

Section 2 on the disadvantages of the h-index appears to have been lifted directly from the Wikipedia page (http://en.wikipedia.org/wiki/H-index), albeit without the citations that are on the Wikipedia page (hence I would recommending reading the Wikipedia in preference to the Bayeh paper).

I mostly agree with you, Andrew, in particular with your observation "The h-factor has taken a prominence that is worrying ...".

During the last days I have taken some trouble to get an overview of the current ("state of the art") research area of "scientometrics" and/or "bilbiometrics" (terrible words, but ok). First I was astonished to find so many proposals of quite different metrics and approaches (seems to be an area in which you can earn very quickly a very high h.index !!), but then upon closer look I was very disappointed about the low level of sophistication of almost all of those so-called metrics.

It seems, physicists/scientists like Hirsch, Bayeh and many of the other protagonists in this field have never heard of the theory of representative measurement, a respectable well-developed field born out of necessity in the behavioral sciences (exceptions are a very small number of papers by a select group of researchers amongst them Woeginger, who gave a nice characterisation of the h-index and some related measures, but then left the field for apparently the same reason I am discussing here).

As for Bayeh's proposal: it is utterly superficial and unfounded, being just a linear combination of a bunch of popular factors, without any justification for the selected set of factors nor the weights given to them. I wonder if he has ever heard of factor analysis ...

In conclusion: I have seen enough, and prefer to continue with more serious matters now. Too bad, that large parts of publicly funded science and research have already - way too early - embarked on the h-index, JCR and things like that. Researchers who take 'prominence' for an attribute of quality will not question any further and take their established role in funding policy and science management for granted. IMHO this is utterly distorting how good scholarship and research is recognized and rewarded. In such cases I always wonder: who is the winner in this game?

http://library.sfu.ca/collections/scholarly-publishing/scholarly-metrics

Dear Paul,

Thanks for this second post - we are indeed much closer in our opinions than your first post suggested.

The real issue here is that all of these metrics are attempting to quantify things that are essentially unquantifiable. Citations in themselves should not be what we are trying for as researchers, we should be trying to do good research and solving important problems.

Now comes the vexed issue - how we decide which problems are important and how do we decide what research is 'good' or 'outstanding'. We use citations as a proxy for this, but in reality, the only judgements we can make are either (a) historical (i.e., the signfiicance of particular research can only be judged long after the discovery, for example Veselago's work on negative index of refraction), or (b) subjective - we think that such and such a problem is important for various reasons that we justify to ourselves.

So because of this, we really need to be honest about the fact that judgements must perforce be subjective. We have an 'inkling' about what is meant by 'quality' and 'importance', and when we are considering people for positions, promotions etc., we must go back to the old-fashioned tasks that include: actually reading their papers, talking to them about what they do or do not know, and fostering creativity.

Distilling these factors to a single number must be wrong, although I'm not convinced that using multiple metrics would give much more insight.

As a final comment, I will relate the publishing output of one of the world's greatest living mathematicians - Andrew Wiles. According to the Web of Science, he has 18 papers and an h-index of 13 (I note, however that his Princeton bibliography - http://web.math.princeton.edu/WebCV/WilesBIB.pdf - records 24, but that isn't the point!). It is also demonstrably not true that every one of these papers is a classic as judged by citations, as even some of his top papers are only garnering on average around 4 citations per year.

So why is the low publishing, low cited mathematician considered one of the world's greatest mathematicians? And why is he deserving of funding? In 1995 he solved Fermat's last theorem. He only works on hard problems so one shouldn't be surprised that it takes a great mathematician a long time to get a significant result. And although he has a large number of students, he doesn't automatically put his name on their papers.

So now we get to the rub - you have a hiring decision to make: 18 publications since 1977 and an h-factor of 13, or a hotshot with an h-factor of 20 since 2004. When you are faced with a such a decision, keep Andrew Wiles in mind, and don't base your decision on h-factor alone!

Very nice example, Andrew, thank you so much! Quality judging is hard work, seems a lot of people try to avoid it by substituting numbers and other proxies for the real work. As a student of psychology, one of the first things I learned was NOT to rely too much on the outcomes of tests. Sometimes they are helpful, sometimes they are necessary, but only as one out of many "sources of evidence", e.g. talking with a client. Guess it is not much different in the hiring case you mentioned.