What "pains me" the most is generally not knowing which questions to ask. Once you come up with one, I don't see a way around tailoring the analysis to the nature of the question. You seem to be asking for a one size fits all strategy. The only such systems I know of walk around on two feet and drink a lot of coffee.

@John

Thank you for your thoughts - they are very much appreciated.

So in terms of features are you saying that such a system would need to:

1. help you formulate new hypotheses by giving you an automated visualisation of all the data-led relationships in a first pass, then

2. give you all the tools (automated and semi-automated) you need to investigate your hypotheses.

The system we are creating does just that - it helps you to formulate hypotheses and inspires to ask more 'what if...' questions.

Experience has taught us that analysis and statistics is often an iterative process, so the tools that we use should embrace this.

Would this help with your pain?

Does it answer your questions or raise more?

One semi-automated tool that I think many researchers would appreciate would be some kind of decision-tree functionality for determining the appropriate statistical analyses given the nature of the data. At its most extreme, this could be akin to the 20-questions apps that are floating around (Do your data consist of counts of things? Did you measure an individual multiple times?).

Another nice functionality would be a battery of quality control steps, such as testing for normality of residuals, if such a thing is assumed by the test in question. Many people forget such things after introductory stats courses and reducing the friction to using those would probably improve things all around.

One final suggestion would to simplify presenting the workflow in a methods section. For instance, you could output a list of tests and data manipulations with the reason (e.g. log transformed values so residuals would be normally distributed), with any key parameters and citations where appropriate. It would be good to avoid the temptation of attempting to format this as legible text, as it may be a few years before we get past issues of self-plagarism in methods sections.

Hope this helps. Have fun developing this tool!

@David

Semi-automated decision tree to select the appropriate statistical analysis:

Whilst a grand idea, this would be an awfully big job. At the end of it the tests and all the analyses should then be done automatically (otherwise there would be little to be gained), making this a mammoth task. Imagine trying to automate all the 10,000 or so tests in SPSS.

We're automating the relational family of statistical tests, which contain a relatively small number of tests, but it's a very large job automating it.

Nice idea though.

Quality control in automated stats:

You're absolutely correct, and in fact we include QC measures at all stages of analysis, all the way from data cleaning to visualisation of the 'story' of the data. This includes the ability for the user to reject outcomes that are likely to have arisen by chance, and the intelligent engine of the system to automatically adjust to the change in circumstances.

Simplify the workflow presentation:

One thing we have done in this area is to include the justification of each step with full explanations (in layman's terms). What we hadn't thought of was to show a visualisation of the workflow from 'question' to a specified, user-selected 'answer'. That's an excellent idea!

Thank you for your input David - it's very much appreciated, and if you have further questions, comments and/or insights, please feel free...

ps - we're having great fun developing this!

Well...no. That would not help with the pain. Here is a more concrete example. Many labs have generated extensive gene expression analysis data on tumors. You can examine correlations within the data until the cows come home but if it doesn't occur to you to draw correlations outside of *your* data with someone else's knowledge base (a good example would be the Gene Ontology database; another would be one of several drug-gene interaction databases) then you will never get to the "story" that such a combination can reveal. A clinical scientist will have an entirely specialized context. My point is, the math behind drawing reasonable descriptions of covariation is seldom the obstacle: insight comes from comparison with something external to your data, and that is the thing that takes months... or maybe years. The question is, "How do I design an experiment that says something new within this or that context?" This is the second most difficult question in science. (The first is, "How do I get someone to pay for this".) This is immensely difficult. So... if you want to make real progress, my advice would be, forget about being general, and focus on one immensely difficult problem at a time. Just sayin'.

@John

You're absolutely correct.

If you want to perform hypothesis-driven analyses, you first have to have a hypothesis, and conversely if the analyses are to be data-driven, you need to have the data.

If you in a situation where you neither have data nor a hypothesis, then no statistical package in the world can help.

You have R and Bioconductor to handle biological data. My doctor uses decision tree software in the office. What is your goal beyond packages already available? I think we need a better definition of what it is you are really after.

@John

For several years I worked as a statistical consultant in a large teaching hospital. I worked with over 100 researchers in areas as diverse as surgery, cancer research, molecular biology, pathology, bioinformatics, palliative care and more, and worked on well over 300 distinct databases.

What I noticed was that the same kinds of analyses were needed time after time, and I was required to find relationships between each of the variables (columns of data).

To find the 'story' of each of the datasets was extremely time-consuming and could take months for each dataset, depending on how big it was and how complex were the hypotheses. Most of the time was spent repeating the same few statistical tests (chi-squared tests, 2-sample t-tests, Kaplan-Meier analyses, etc) on different pairs of variables, and I soon realised that if the analyses were automated each of the tests returns results in milliseconds.

This was the beginning of the realisation that relational analyses could be completely automated.

Our vision is to create an automated tool for relational analyses that takes you from data to story in just a few minutes rather than months.

It will automatically clean and classify the data, select and run the appropriate statistical tests (univariate and multivariate) and create 1D, 2D and 3D visualisations of all the relationships in the dataset. The user will then have the facility to investigate each of the relationships in detail.

The program is in late-stage development and we are reaching out to the medical and biological research communities to try to find out how researchers in different fields might use such a tool.

We are particularly interested in what features might be useful and valuable, and as such we value your input highly and are very grateful you have spent the time to engage with us.

We would love to hear your thoughts and whether you consider that such a tool might be welcomed in your field of research.

Lee:

Your premise is intriguing.

It so happens that a company with which I am affiliated will be applying for an SBIR grant to assess the effectiveness of our Health Information Exchange platform in improving health outcomes in the treatment of cancer patients throughout their continuum of care. I would like to speak with you regarding the potential for your innovation to enhance the ability of clinical oncologists to improve health outcomes in pre- and post-cancer patients.

We can provide you with the data to feed your engine. What we need is an engine that can use the data in effective ways. Let me know your thinking about this.

Best regards

@Bruce

thank you for your interest in what we're doing.

We have several years experience in dealing with all manner of cancer research data and I would be delighted to discuss your initiative with you in greater detail.

Can I suggest that we continue this conversation in private?

Please feel free to email me at your convenience:

[email protected]