Meta-analyses and systematic reviews seem the shortcut to academic success as they usually have a better chance of getting published in accredited journals, be read more, and bring home a lot of citations. Interestingly enough, apart from being time-consuming, they are very easy; they are actually nothing but carefully followed protocols of online data collection and statistical analysis, if any.
The point is that most of this can be easily done (at least in theory) by a simple computer algorithm. A combination of if/thenstatements would simply allow the software to decide on the statistical parameters to be used, not to mention more advanced approaches that can be available to expert systems.
The only part needing a much more advanced algorithm like a very good artificial intelligence is the part that is supposed to search the articles, read them, accurately understand them, include/exclude them accordingly, and extract data from them. It seems that today’s level of AI is becoming more and more sufficient for this purpose. AI can now easily read papers and understand them quite accurately. So AI programs that can either do the whole meta-analysis themselves, or do the heavy lifting and let the human check and polish/correct the final results are on the rise. All needed would be the topic of the meta-analysis. The rest is done automatically or semi-automatically.
We can even have search engines that actively monitor academic literature, and simply generate the end results (i.e., forest plots, effect sizes, risk of bias assessments, result interpretations, etc.), as if it is some very easily done “search result”. Humans then can get back to doing more difficult research instead of putting time on searching and doing statistical analyses and writing the final meta-analysis paper. At least, such search engines can give a pretty good initial draft for humans to check and polish them.
When we ask a medical question from a search engine, it will not only give us a summary of relevant results (the way the currently available LLM chatbots do) but also will it calculate and produce an initial meta-analysis for us based on the available scientific literature. It will also warn the reader that the results are generated by AI and should not be deeply trusted, but can be used as a rough guess. This is of course needed until the accuracy of generative AI surpasses that of humans.
It just needs some enthusiasts with enough free time and resources on their hands to train some available open-source, open-parameter LLMs to do this specific task. Maybe even big players are currently working on this concept behind the scene to optimize their propriety LLMs for meta-analysis generation.
Any thoughts would be most welcome.
Vahid Rakhshan
There was a recent well-publicised event where an actual legal court case included legal documents prepared by AI that included supposed legal citations to cases that did not ever exist.
So, you have two problems:
(1) Constructing code that does actually work;
(2) Persuading others that you have code that actually works.
As ever meta-analyses rely on proper publication of research results. A recent BBC podcast included something relevant to improving this. See https://www.bbc.co.uk/sounds/play/p0hd42dt
starting at about 14:25. This should be freely accessible worldwide for a year at least. I think it says that, before a spotlight was out on the problem, only 50% of research trials in the UK had results reported, and that this rose to 90% given that a parliamentary investigation was then taking place.
David A. Jones it is a scary figure; this becomes even scarier noting that it is possible that a considerable portion of the 50% published portion are actually imaginary and fabricated (partially or even fully). Add another level of scariness to it, noting that journals as well tend to refuse from publishing neutral or bad results; journals too prefer more "delicious" significant p values! So all parties work hard to skew the results in the literature to the "bright side".
But this problem has nothing to do with AI-powered meta-analyzers, as it is a limitation to meta-analysis in general, either automated or done manually. Besides, some methods like Publication bias assessment already try to see if such a bias exists in literature or not.
Regarding AI incorrectly citing cases that don't even exist (i.e., AI hallucinations), this will be greatly solved by making the AI more conservative through at least 2 mechanisms:
1. User demand at inquiry: The current version of Bing (Chat) Copilot allows the user to use 3 different levels of "strictness". The most strict level has the lowest amount of AI hallucinations (the most accurate). There is a trade-off between AI's accuracy versus its so-called creativity. The user can maximize the accuracy when conducting meta-analyses or other scientific tasks.
2. More polishing: Besides, with better AI training together with larger models to be used in future, this problem will easily fade away to a great degree because even mundane models of future AI will be much more accurate than the most accurate versions of current AI.
Since you are being optimistic about what progress might be made, you might like to consider a scenario where you have several different AI engines constructed on different principles, and you do a comparison of the results to see the extent of agreement. This would be similar to studies of manual proof-reading of manuscripts as a check on how good a combined result is likely to be. Of course, the underlying problem would still remain.
I think eventually AI tools will be able to conduct systematic reviews with minimal oversight from researchers, but at present the AI tools available are hopeless. Predicting the future is notoriously difiicult and usually wildly inaccurate, but I do not think AI will be able to produce a systematic review for 30+ years.
As a side, producing a systematic review may be relatively easy, but producing a good systematic review is hard, and requires a lot of expertise and experience.
Thanks Joseph Whittaker . 30 years seems a lot. The exponential growth of AI's intelligence plus the exponential growth of hardware power make me think in a couple of years, they will get there.
Yes, they don't need to be perfect for now. Actually, many meta-analyses conducted by humans too are not perfect or even good, you as too pointed out.
- Badges
- Science topic
- Similar topics
- Psychology