What AI can and cannot do for evidence synthesis

A good systematic review can take a team many months, sometimes years. Much of that time goes on repetitive work: running searches across databases, screening thousands of titles and abstracts, and extracting the same details from hundreds of studies. By the time a review is published, the evidence has often moved on. It is easy to see why artificial intelligence is so attractive to anyone who works in evidence synthesis.

In my current work at Queen’s University Belfast, I help build a synthesis-ready repository of education studies and test AI-assisted tools against human decisions. That experience has left me optimistic, but also clear about where the limits lie.

What AI does well

• Prioritising screening. Machine learning can learn from a reviewer’s early decisions and move the most likely relevant studies to the top of the pile. Reviewers then find the important studies sooner.
• Classifying study designs. Classifiers trained on large sets of human-labelled records can reliably identify, for example, randomised controlled trials. Cochrane’s RCT classifier is a well-documented example, and it substantially reduces the number of records people need to check by hand.
• Drafting and checking searches. AI can suggest synonyms and help translate a search string between database interfaces. These are tasks where small syntax errors can silently lose relevant studies.
• Supporting data extraction. Language models can produce a first draft of study details such as sample size, setting or outcome measures, for a human to verify.
• Working across languages. Translation tools make it more realistic to include studies published in languages other than English.

What AI cannot do

• Take responsibility. A tool cannot be accountable for a review’s conclusions. If evidence is used to shape policy for schools, children or patients, people must stand behind it.
• Exercise judgement reliably. Appraising risk of bias, deciding whether an intervention really matches the review question, or weighing the certainty of evidence all require expert judgement and context.
• Guarantee accuracy. Generative AI can produce confident but wrong answers, including references to studies that do not exist. Every output needs checking.
• Stay the same. Commercial models are updated frequently. A tool that performed well last month may behave differently today, which is a real problem for reproducibility.
• Escape its training data. Most models are trained on material that is mostly in English and mostly from the Global North. Without care, AI could make evidence synthesis even less representative of research from Africa, Asia and Latin America.

A shared standard

In 2025, Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence published a joint position statement on AI use in evidence synthesis. Its message is clear. Reviewers remain responsible for their methods and findings. AI should only be used where it does not compromise rigour, and with human oversight. Any AI that makes or suggests judgements, such as screening, risk-of-bias assessment or data extraction, should be fully reported, including the tool, version, date and purpose. Reviewers should also show that a tool works for their particular review, if necessary by piloting it against human decisions.

How I approach it

In practice, I find it helpful to treat an AI tool like a new member of the review team: potentially very useful, but not yet trusted. That means:

• testing it on a sample that humans have already screened, and measuring what it misses;
• keeping a person responsible for every final decision;
• recording the tool, version, settings and prompts used, so that others can see and repeat the process;
• being open in the write-up about what the tool did and how its outputs were checked.

Used this way, AI can free reviewers from the most repetitive parts of the work and give them more time for the parts that need human judgement. The aim is not simply faster reviews. It is getting trustworthy evidence to teachers, policymakers and communities while it can still make a difference.