I agree. And I'll add.

They encourage people to write for a very narrow audience that limits the impact of their work.

They create too much power for a limited number of journals that abuse authors, especially the untenured. Those journals abuse their power by giving out R&Rs multiple times for a paper, knowing authors need the pub and will comply, only to reject the paper. A colleague had that happen back-to-back at two FT50 outlets when she was still untenured.

Hello

I agree with your points and would add as a subpoint of point 3 that those who are unmotivated will remain unmotivated. I say that as obviously, I am sure you have seen, some members are more motivated than others and some less. The ones who are “less” seem to remain so regardless of the rest of the department.

Thanks for your comments Paul, Hannes, and Joseph. Based on your remarks, I add the following to the list:

4. They attempt to motivate people who can't be motivated (Joseph).

5. They focus on where an article is published rather than on what is published (Hannes).

6. They reduce research impact by targeting a narrow audience (Paul).

7. They encourage editorial abuse (Paul).

To clarify my original statemennt, it is not the lists themselves that may have such effects. It is the administrative use of such lists for evaluating scholarly productivity.

Paul E. Spector Hannes Zacher : you are two respected names in the field. Maybe you should team up with Thomas E. Becker and/or Joseph C Lee to write a critique on Journal lists? It would help junior scholars like myself.

So far, those who have replied seem to agree that journal lists are pretty useless for assessing faculty performance. Some feel that using lists for this purpose do more harm than good.

SO, here are two related questions: 1. Do you know of any evidence supporting or not supporting the use of journal lists? 2. If your employer uses lists, to what extent do you feel free to voice your opinion about it?

Thanks for your reply, Jetmir, and for the Ramani paper!

Funniest exchange I witnessed in a hiring committee with people from the hiring university using a journal list and the external members not.

Internal member asked externals: "But how do you evaluate the quality of an applicant's work then?"

External member: "We read their work."

Silence :-D

A bit less tongue in cheek, I think

1. plenty of academics are overwhelmed with the demands on them. So when there are 3+ candidates to be reviewed, they are very grateful for getting some supposed quality triangulation and not having to read all their work in detail. Yes, it's bad but also a symptom of an overly strained system.

2. framed less negatively: Sometimes academics have to review candidates in a field where we have fairly little expertise. At the same time, journals are VERY different in their approach to developing papers and allowing papers of different quality to be published. So getting a list of journals that (supposedly) does a better job than others can help to categorise the quality of a paper (yes, yes, all caveats apply, i.e. the job a journal does in developing papers is not reflected in journal lists, and yes, one can also publish top notch stuff in journals that are less developmental).

2. especially business schools are often run like businesses, seeking to differentiate themselves. (and btw the audience, i.e. prospective students, also seem to go for that more quantitative rather than qualitative differentiation, i.e. rankings) Therefore, as soon as you put a metric in front of business schools, they'll manage accordingly, wanting to outperform others (recent examples to the contrary are law schools and Harvard Medical school decidedly exiting rankings - though there maybe be other competitive and not just ethical considerations behind that decision).

4. to some extent journal lists also make explicit a debate that is otherwise only implicit in a field (and btw, the implicit often comes out of left field for candidates that are not part of the "inner circle" - I speak from experience). For instance, Social Psych to the best of my knowledge has no "list" and yet it is still clear what journals are top and what hiring/promotion committees look for. So, a list may help a more candid discussion about what's in and what's not - and why.

As to your question how is it handled in your institution: I've been part of multiple institutions that use journal lists. Some claim a simply numeric approach, e.g. top25% IF in their field paired with FT50 and UTD, others argue that they consulted "experts". In any case, I always saw quite a bit of wrangling going on. Very naturally so, because a list determines what will be counted a success, financial/career rewards hinge on it, and to some extent it will (re)shape your identity. Hence, in my experience, even though managers of such lists (i.e. Deans of research) usually allow voice, they are very reluctant to make any changes because it opens Pandora's box. You'd have to argue for your journals in your specific case as an exception to the rule.

Puh, wrote a lot more than I wanted, but yeah... it's a ... big ... topic.

Ps: I define list here more broadly, not just externally hovered (FT50/UTD), but also any variations of these kept within an institution.

Great points, Niels! I'm of the opinion that universities should hire the very best people they can and then leave them the hell alone. The key productivity goal of administration should be to NOT demotivate faculty by overusing carrots and sticks (e.g., tenure, teaching assignments, favors). Again, thanks for your very thoughtful comments.

I agree. It is somewhat absurd teaching about motivation and then bringing in most business school's compensation/career schemes as a case how NOT to do it ;-)

I suppose academics same as other people are afraid they do not make the right choice. In response, they want to implement standard criteria for selection and subsequent control over others. Both relieve one of the burden of personal responsibility when hiring.

On a meta-level, it's the same dilemma everywhere in management, between trust and control. And as dilemmas go, usually both sides have good arguments - personal opinions notwithstanding. Because, let's be honest, while many academics consider themselves intrinsically motivated, I've also seen a fair share just taking it easy with no interest to further follow their curiosity. And sure, one could argue that is a result of the treatment they've had to endure... but I suppose then the whole US tenure track system could be looked at (beyond just journal lists). And likewise one could argue that would happen either way, journal lists or not, so why wreak havoc on those who are motivated.

Would be great to see actual, methodologically sound studies being done on the topic. Any recommendations?

Thank you for a fascinating and useful discussion. For Russia, this problem is very acute now. Our government has posted the national whitelist and going to use it in research evaluation. The method of compilation is much more primitive than that of the lists you mentioned. I have published an opinion piece on this topic, and of course I have referenced this discussion. The paper is available at Article Russian Journal Whitelist: Questions to be answered

Dmitry, thank you for the describing the Russian context! I appreciate your referencing this discussion in your paper, and hope that your paper promote more critical thinking on the topic. Tom

Thanks, Tom

wow, m glad someone has the courage to speak abt this nexus, which is ruining the life of many early researchers.

Thanks for your comment, Prabha. The fact that it requires some courage to speak about the issue tells us something about the state of affairs in many universities. Wishing you the very best!