I think that part of the problem is due to software that is often poorly designed and documented and therefore is usually difficult to install and/or use. For example I think to the myriads of R scripts and algorithm implementations (that usually requires Bioconductor) and works well with example data from the authors but give strange results when used with your data. This usually happens because of slightly different file formats, tables that should be sorted and are not and so on and so forth. Then, most bioinformatic tools are released and then the authors forget about them (since the paper has been published and so why we should care anymore?), in this way bugs and issues are never solved and, what's worse, never documented. Our group tries to keep released software always up to date and to this regard we prefer to release web-interfaces that are far more easily maintained on one hand, and assure that the user is always using the latest version on the other, but this is unfeasible for software working on huge NGS data. Naturally the pressure on "producing" new articles is always high in a research context and this makes the release of new software always a priority with respect to the maintenance of already released software and I think this is quite a problem and one hard to solve. And this explains, at least partially, why we have so many methods but very few "golden standard" tools that one can rely upon for benchmarking new tools.

I think that part of the problem is due to software that is often poorly designed and documented and therefore is usually difficult to install and/or use. For example I think to the myriads of R scripts and algorithm implementations (that usually requires Bioconductor) and works well with example data from the authors but give strange results when used with your data. This usually happens because of slightly different file formats, tables that should be sorted and are not and so on and so forth. Then, most bioinformatic tools are released and then the authors forget about them (since the paper has been published and so why we should care anymore?), in this way bugs and issues are never solved and, what's worse, never documented. Our group tries to keep released software always up to date and to this regard we prefer to release web-interfaces that are far more easily maintained on one hand, and assure that the user is always using the latest version on the other, but this is unfeasible for software working on huge NGS data. Naturally the pressure on "producing" new articles is always high in a research context and this makes the release of new software always a priority with respect to the maintenance of already released software and I think this is quite a problem and one hard to solve. And this explains, at least partially, why we have so many methods but very few "golden standard" tools that one can rely upon for benchmarking new tools.

I really believe on these points:

1) most of the tools are created from the specific needs of a certain group, because would be impossible to start with a total generic solution;

2) After those tools are developed, they are enhanced, but you never get the appropriated scenario to expand it to the more general cases;

3) There is not a lot of documentation for most of those tools, which turns their use more complicated;

4) The three points above are closely related to money and time: you need money, that will provide you the time to work specifically on the software. The main problem here is that build the software, in the context of bioinformatics, with exceptions of course, it is not the final objective of a research. The need to develop the software comes from the research, and it is usually included in the same effort.

So, yes we have too many tools, because there is no concentrated effort to build the bioinformatic software as the final objective (again, good to reinforce that exceptions exists).

Thomas, altough I'm just beggining on bioinformatics, I hope you are correct and I had the chance to be part of this field's development.

More tools please, and make them well known and well used.

I spend a bit of time at the start of each project I'm given to try and hunt for pre-existing solutions that will solve my problem (possibly with a bit of tweaking). It is always preferable for me to use a solution someone else has developed, because I then don't have to go through the tedious process of bug fixing that someone else has already experienced.

However, if a solution is not easy to find, or looks too difficult to change to work for my particular use case, then I go ahead and make an attempt on my own custom solution. I always try to make a point of stating "this is something written recently by me, so there will be plenty of bugs", but frequently the researchers I work for don't care too much about that -- a quick solution that works "sort of okay, given a salt cellar of assumptions" is a whole lot better than a detailed, generalised, robust solution that is never implemented. Of course, if I've done similar things before, then modifying and generalising my own code is much easier than working on someone else's code.

The best thing for a bioinformatics tool is open sharing, many eyes, and many users. The most frequently used tools tend to be more reliable, but that doesn't mean there's no space for anything else.

Thank you all for sharing your opinion on the subject.

To expand on Walter's point of limited drive to complete softwares, I'd say that beyond money and time, there might be a lack of enthusiasm:

Once the problem is solved, at least in the context of its original application, there is little incentive to develop and maintain the software any further.

Increased citation of the corresponding Applications Notes in Bioinformatics should be a good motivation, but although tools are used, they are rarely cited, e.g. my fantastic little GREVE online tool is being used by hundreds over the world, I answered all specific requests, but the corresponding article has been cited only once so far:(

Is it worth for me to spend the time and energy to prepare a robust, flexible and well documented, 2.0 version?

I agree with David that the more tool available, the more likely they are to address your own analytical issue. However too much is too much and I am afraid Thomas' separation of the wheat from the chaff takes so much of Bioinformaticians' precious time. Furthermore, it might not be the best tools surviving, but the easier to install and run, as well as those pushed by bigger groups. It is a jungle out there!

Your question is born out of a frustration, that is shared by many. The zoo of SW offers is dense and hard to explore without stopping doing your own research. The reviewers request to compare with competing methods is warranted, but one may suspect that the reviewer is the author of the competing method, and would be interested in boosting his citations. Everything is convolved, and the objective review remains an illusive ideal.

Buying a piece of SW is buying a black box. You have no idea what mechanisms has been implemented in the SW and what fudge factors the development team has been forced to implement, and the SW company will not share this info with you, because it is proprietary information. The alternative is to write your own SW, which only few workers are able to. Saving money and downloading somebody else's public domain SW also implies that you are likely to inherit whatever bugs is buried in the SW. The larger the code the more likely it is that it will harbour bugs.

My own vision is that we have to implement something equivalent to imageJ in image analysis, i.e. a public domain platform, where you have most of the functionality that people are seeking. In addition the user should be able to add his or her own plugins, that may solve very specific problems.

In order to achieve this we are faced with the ZOO of existing solutions, written in R, C, C++, MATLAB, Fortran as well as others. How can such SW pass inputs and outputs? - probably not, or only with severe difficulties. Thus I think there is a need to launch a public domain platform where some basic rules exist that secures compatibility between the SW submissions. If properly made it will gain importance as a resource that most people working in the field will recognize - if not it will vanish after a couple of years. A host institution is needed , it should be one rooted in bioinformatics, maybe EMBL?

According to my opinion, any effective bioinformatics approach should have good physico-chemical ground. If bioinformatics tool does not meet this requirement, it will be of little help in solving of problems such as protein-protein interactions, assessment of biological effects of mutation, functional mapping of proteins and DNA, selection of dug targets and drug candidates, etc. In concussion, the simple answer to the simple question: “Is there too many Bioinformatics tools out there, or is it simply too difficult to chose?” is NO, because in contrast to numerous “technical” bioinformatics tools, bioinformatics tools which are based on clear physico-chemical principles are very rare.

Jean-Baptiste, I agree that lack of enthusiasm is also a factor, but this could be solved with bringing people with the more appropriate skills to build the tools. I mean, maybe having specific budget on projects to have a computer science team members, that could use the appropriate methods to build the tools, that could invest in preparing them to be shared. I think the lack of enthusiasm occurs because if you have biophysics, biochemists, geneticists building software, no matter how good they are, this is not their field and they will do out of need, not because they like to work on that.

Toolbox, toolbox, toolbox. I think the solution to proliferating software (in many situations) is to try very hard to contribute to an existing toolbox rather than to create your own. This will allow others to more easily compare your method with others without installing and learning many different packages. It's not always possible, but should be attempted even if it's a bit more work because it ensures better accessibility and longevity for your method. Those interested in this question might be interested in the article I wrote with collaborators on software usability (and re-usability): http://www.nature.com/nmeth/journal/v9/n7/full/nmeth.2073.html

Look at this way, there used to over 50 independent auto manufactures in the USA alone, now there are essentially 2-3. Markets and competition will drive this, but it takes users (both working and failing) and time. We are still trying to get a handle on how to process the vast quantities of data. At one point during the human genome project sequence data was being generated faster than it could be assembled, and this was using the Sanger method!. People were (and some still are) applying microarray statistics on chip seq data, this was wrong, but we could make data before we understood how to process it. one innovation spurs a cascade of innovation.

Consolidation and mergers will happen, the tricky part will be to keep it free. There are some excellent software package out there that cost 1000's and are out of the reach of most users. R and bio-conductor try to keep the tools available to all. GEO tranche and others try to keep the data available to all.

My experience in "Resolving the discrepancies among nucleic acid conformational analyses" (http://www.ncbi.nlm.nih.gov/pubmed/9917397) tells me that the differences between similar programs may not as big as the authors claim to be. At the most fundamental level, any real difference in methods can be more clearly identified.

Yes, there are too many tools. But the big differences resides on the way you introduce your data, the visualization of results, is some cases the database used for statistical inferences and a few differences on the statistical approaches. I try to use the most cited free software, well documented and try to keep the same algorithm for data analysis for all the experiments. I also time to time evaluate new programs with a well characterized data.

I think this is a problem with the publish or perish mentality. At my university the number of publications a faculty has is more important that the impact factor of their publications. This encourages the rapid creation of new tools that are quickly abandoned, rather than the creation of a few good tools followed by long term maitenance. Since there is no incentive to think long term, there is less incentive to use good programming practices, open source code, or documentation. This also making new tools rather than contributions to existing tool if there are related but missing functionality to a current tool.

Let me add a bit, colleagues. Just an example of our project to develop Python library for proteomics data mining (available at http://pythonhosted.org/pyteomics/). We spent almost three years to developing the library and out of these years one year was spent for documentation only (ok, we were slow, probably). When we send the paper out to Bioinformatics, the journal rejected it without reviewing by saying that they have paper in press (in press at the time!) concerning the Python parser and that paper "dent the novelty". The paper in question was concerning just one single function among many we have implemented and without documentation. We would probably spend just a month to make this single function. So, no wonder that the bioinformatics niche is full of uncompleted and non-maintained tools if the major journals promote this kind of policy as well.

The answer must be that there are too many tools. I am still coming across new tools after I have completed the task. I am not saying the newly discovered tools can do the same job any better, or even do anything different. I may be offending some of the more experienced bioinformaticians but I do think that commercial companies have much more time and many more staff to concentrate on the development tasks in hand. I especially find CLCBio can fulfil most of the needs for my own are of medical research and if they don't have the solution already, you can rest assured that someone in the company is working on it. So commercial software is the answer for reliability, flexibility and user support. But there is a place for all the old tools, but you got to know where they are or you could spend valuable time hunting for them.

I should completely agree that the open-source software in bioinformatics is the only way to go. Relying on commercial SW is relying on the company's good will and capabilities to make necessary changes, on its programmers' turnover, etc., etc. Moreover, it is typical situation that a lab wants to adapt data processing SW for specifics of its own problems, incorporate something, etc. I would opt for the bioSW as open-source building blocks for in-house written application tools.

I agree with you, Thomas. This is a good model to follow. And the key to that is one: community. Some of the great achievements in bioinformatics area or computational biology was due to the community (remember, PDB was created because a community of crystallographers). We need to bring this same mentality to produce the software that will make the difference for us.

This is both a problem and a strength. I read a blog somewhere about this and the author pointed out that more people who write 'new' algorithms would serve the community better by forking existing code but that tends to be hard to push to PIs and publishers. Commercial SW has a major weakness in that they do not publish thier bugs nor do they alert users to critical issues discovered after a release. Also, depending on the CSW, they may or may not have a robust QC of a new release. Crowd sourcing bug discovery and having this in an open forum is a true strength of the open source model.

Yes the bioinformatics tools around are too many , but this should not be a problem if one should be able to think to the mathematical operation he would like to do on his data before going around and looking for the 'last method'....

Yes, their are too many tools, Major problem for a user is how to install them on his/her computer. We have integarted large number of software on single platform see http://osddlinux.osdd.net/

The question might be rephrased are those tools satisfactory for bioinformatics , I beleive this is normal to have this amount , that comes as answer of user diverse requirements and needs

It would be nice if someone would integrate open-source tools and perhaps create Amazon image with all tools pre-installed. This could be open-source as well as commercial project. And most importantly, this should include documentation, tutorials and plenty of examples. Documentation traditionally is a week point of open source, perhaps because writing docs is boring task (although some open-source software have excellent documentation).

I think you can decide which tools to use; the problem is what kind of results you can get from data uploaded.

A fast answer. My opinion is that you should not use a software were you can not see the implementation of the algorithms or modify them. I rarely use any software that dont have that requisite. About published code poorly documented. Well, I think is nice people publish their code. My suggestion, if you give it a try, but the code dont seems to work, then get in contact with them to invite them to collaborate. Just remember to invite them to contribute to your paper or a project proposal, they need something back for their effort.

Jose, I agree with you, but I think we need to go one step further on that: we need, as a community of bioinformatics users/developers, put our minds together and look for a solution to our common problem: rationalize the space of bioinformatic tools.

I agree there is a need for rationalisation. But which model to choose?

There are already some integrative solution with various pro/cons: R/Bioconductor, Ensembl, Galaxy, Taverna, ...

So if we want to rationalise the work, where do we find this COMMUNITY?

Should we focus on the big players, Broad, Sanger, etc (apologize for not citing your place;-) ?Start from the people in this discussion? Reach out some Association?

Maybe I should start another question...

I would say: let's start with the people on Research Gate. Let's discuss our pros and cons and create a simple manifesto of what we look in terms of bioinformatics tools. And then we can create an action plan of what we need to do. One of the revolutions on software engineering have started exactly like that (http://ag.ilemanifesto.org). I think we may start a revolution of our own... (sorry if I'm being too entusiastic, but one of the reasons to be on this area is the passion I have for extracting useful information from data).

I could give a good example I think. I work for center providing research infrastructure and applications for researchers and have started some time ago collaboration with RNA-seq team being software coordinator for big European project and having good software engineering background and rather low bioinformatics knowledge. But even in this case I am able to go deeper into the latter topics and provide software engineering best practices for research groups quite fast. Now we are extending to international collaboration providing competences we have on each site. This was possible mainly due to involvment of ICT infrastructure for research in communication, data storage and computing. Operators of this infrastructure understand very good research models (including high risk of transition from prototype software to product) and when focusing on reliability they are able to drive research. Of course this is only true when they are searching for deployment of new services for research (i.e. develop this inrastructure) and keep them running and not going only for publishing papers. In fact this is our point.

If you need some specific tasks, it is difficult to find a software that fits your exact needs. Currently, I was asked to search for some features in a human genome. I explored various software packages and online services, but no one fitted my needs. Since I have some programming experience, I wrote a program in C++ employing some functions of sequence analysis library Seqan C++. After few days of work I was wondering how well my program works, what it is possible to scan entire human genomes for multiple features at once in a matter of minutes on current desktop computer .

But my software still is not compiled, it runs from the development environment, because I need constantly to adapt it, try various options, algorithms, modify them, and the easiest way to do that is directly in the source code. To write some interface, even command line, would take me even more time than to design algorithms and and write the working software. Maybe, many of that published software were designed for very specific tasks, and some universality and user interface was added only to publish the paper. If there would be some good framework (preferably in C or C++ to work fast) with good user interface, graphical representation capabilities and possibility to use the libraries of choice, many programmers would use it and resulted software could be more friendly for users.

You sometimes get quite different answers to the same question using the same data but different tools. This is a bit frustrating but it would have been worse if there were only one tool, the answer from which we might think was "The Truth".

This question also makes me wonder:

"Are there too many sample preparation methods?"

"Are there too many sequencing platforms?"

(because bioinformatics is only part of the inference chain).

...then I start thinking

"Is there not enough time or money for stable, robust and appropriate measurement and analysis methods to be developed?"

I appreciate your frustration Jean-Baptiste. I'm not sure whether to say yes or no in response.

What I will say is that I think the pace at which measurement processes in molecular bioscience change creates many challenges, including burgeoning analysis software.

The rate at which these new processes change is on a par with the time taken to get a PhD or a postdoc. I think that that, coupled with the time and other competing pressures on researchers, leads to a situation where there is probably just enough time to think up, implement and benchmark a promising algorithm, but probably not enough time/funding/motivation to maintain and improve the algorithm or its implementation (at least in many cases).

Causes aside, I think it's clear that in many cases, we have to take into account variability due to

1. biology (the bit we are interested in)

2. sampling

3. sample preparation

4. measurement error (e.g., sequencing)

5. data analysis approaches (including different algorithms and parameters)

...so that we can get a sense of how robust our conclusions are.

...and if you accept that, that means all of us have to have an appreciation of the different links in the inferential chain. Wet lab scientists can't cross their fingers and hope that Algorithm X will do the trick; dry lab scientists can't pretend that the impact of the sampling, sample preparation and measurement processes can be discounted once a nice big file of numbers is provided for analysis.

Great question... and a deep one too!

With respect to "too many tools", something that is not mentioned here is the "survival of the fittest" tool. That is, as time goes by, the community seem to converge on a few tools for a specific analysis.

For example, for ChIP-Seq, numerous tools were developed in the years 2008-2010, but now there are only a few that are commonly used (eg MACS is very popular, and then maybe SISSRs and QuEST. Others? Don't see much of them...). So these things seem to sort themselves out after a while. I think the same will happen for RNA-Seq, which seem to converge on the Cufflinks-deconvolution or edgeR-count approach, depending on what you are after.

So what makes a tool survive? Easy-to-use, easy-to-interpret outputs, fairly fast, good user support and active user groups are key-elements.

Dear colleages,

this is a very good question, and one that probably comes to mind every time a researcher finds him/herself overwhelmed when looking for a tool to analyze his/her data. The comment by Dr. Razanskas is showing us an important problematic: oftentimes, if you have the programming skills, it is more straightforward to write your own program for a specific task than to use publicly available tools (due to difficulties in installation, incompatibilities, disagreement with the heuristics, etc.). But the hustle of streamlining the program for general use doesn’t pay, so the program doesn’t become public. This has two main drawbacks: the results produced by such a novel unpublished program will be difficult to compare, and probably many clever or sensible approaches to specific problems won’t ever get published.

I am wondering about the statement “these things seem to sort themselves out after a while”. This sounds to me like evolution; survival of the fittest. Not a bad thing, other than that evolution is blind and takes a lot of time. We are not blind, or are we? The publication rush may be blind, but I think most scientists are not blind, they just don’t have the time (and/or funding) to delve into the details of every algorithm. It seems to me also that very often programs are chosen by inertia instead of careful consideration of the problem at hand. The data to analyze is produced at such an incredible pace and results have to follow as quick. But there is almost no time to think about it (by this I mean to think about the interpretability and comparability of the results)… So I often find myself wondering, why slowing down scientific progress by trying so hard to go as fast as possible?

How can we change this? As usual the answer is to focus on quality instead of quantity; slowing down the process to more precisely scrutinize the analysis process. That is easier to say than to do in a money-driven world. But I think it would be an awesome idea to get together as a community to collaborate and produce something like a public database of bioinformatics tools, rated in the first place by the comprehensiveness of their documentation. Here people who used the program could write specific reviews and rate the program (like in Amazon :P). This is not perfect, but it is a first baby step. Having a centralized database citing or containing as many bioinformatics tools as possible would help us to more precisely see the dimension of the problem and to integrate individual efforts, building on each others experiences and ideas. This database would still be overwhelming and growing daily… but hopefully in a more organized and collaborative way, integrating both computer and life scientists in the process.

Of course I would be happy to spare some of my time for this endeavor… ;-)

> But I think it would be an awesome idea to get together as a community to collaborate and produce something like a public database of bioinformatics tools, rated in the first place by the comprehensiveness of their documentation. Here people who used the program could write specific reviews and rate the program (like in Amazon :P).

You mean like this?

http://seqanswers.com/wiki/Software

Here's another one:

http://omictools.com

The problem is not with the abundance of tools out there, but how users in the community aggregate their experience. I also find it's very time confusing to try out different tools and compare them. But if I can share my experience with the rest of us, the effort may be worthwhile. What we are lacking is a mechanism to share these experience, something like the Amazon environment where buyers review products. GitHub may be something for this.

Thanks for the links David. I think the forum at SEQanswers is one of the best places to look for help relating bioinformatics tools. Both of the pages present exhaustive lists of bioinformatics tools classified by methods/applications. This is already very useful, but it is only the first step for what can become a daunting search of the optimal program. The more graphically oriented STATSARRAY webpage indeed gives the option to rank the available programs. I may have come across this webpage in my search for tools to analyze RNA-seq data, but I did not notice there was a ranking option. This option is only visible in the listing once programs have been ranked already. None of the tools I am currently using has been ranked yet, not even Trinity which, as far as I know, is very widely used. Could this be because STATSARRAY is a commercial webpage?

As Xinkun Wang also mentioned, it is very useful to be able to share our experiences with specific bioinformatics tools so that not everyone has to start from scratch again (that makes work much more worth it). Doing this in a setting similar to Amazon, which makes it easy to have a general overview of tools and their ratings (linked to reviews), could make the application of these tools more efficient and effective. Actually there could also be separate rankings for the installation process, program stability, speed, output, etc. Just fantasizing here…

SEQanswers would in this case continue being the better option for in-depth discussions.

As far as explosion of bioinformatic tools, its good to have it as every new progrrame is up to date but I wonder there are too many programmes and scripts for analybsis but bioinformatics is also about interpreting biological data. It is thus or should be very important component of bioinformatic tool to explain to some extent what significance this software will bring to our current understanding of area. This is my thought, needs more enlightning thoughts on it.

In-depth knowledge of pro's and con's of available tools is important. It is better to have so many options to do one bioinformatics task. For a given experiment, if different tools gives slightly different answers but conclusions are same, it further strengthens the postulated hypothesis.

I can understand your point of view. And yes, sometimes it's really hard to choose. Well what, I want to do is to combine 3 or more tools (i.e. softwares) so that i can be sure of my results.

"interpretation and the interpretability (i.e. does the software really does what I think with my data?) of the results" : I would like to say you that you have to try a bioinformatic tool and see that for yourself.

Because on Researchgate, we can give you all logical arguments in this world, but, when you try it, you'll find how & why it is useful.

Thank you all for your contributions.

It looks like there is no escape from the plethora of Bioinformatics tools.

Having the pros and cons is consensually seen as the best way to ensure their proper use.

This is turn might be the best was to justify the existence of bioinformaticians. With bioinformatics ubiquity, everyone from biologist to computer scientist can consider him/herself as a bioinformatician. However only the expert, "pure", bioinformaticians can hold the expertise knowledge of when, how and what is the best tool to use in any situation.

So bioinformaticians seem to create these so many tools to exist;-)

May be I am late here. But the issues raised by Jean-Baptiste & many others here have also bothered me a lot. Compilation of such resources & systematic comparison seemed such an essential requirement in biology today - yet a very neglected task! In fact, realizing I can't get funding from traditional agencies, I started a company to address the issue. I think we have begun well with 5 years effort into this now: http://www.startbioinfo.com -

Startbioinfo.com is a public portal and all are welcome to contribute their scientific observations about the biological databases and tools.

By the way, this portal (startbioinfo.com) was developed only after a comparison with other existing similar portals, & realizing a need for a better one! There are different levels of compilations and comparisons. Literature search section forms a good example for in depth comparative analysis of software/databases from a user's perspective (also see: precedings.nature.com/documents/2101/version/3 - 3 years of work by our team with more than 50 searches). We have been able to make recommendations for more than 20 specific search-purposes/questions, across 53 search engines.

There are some section simple compilations, and a few which are catalogued better with feature validations. We have been constantly trying to up-grade every section to be equal to the 'literature search' section. Hopefully we will soon publish more such conclusions.