Writing Risk-of-Bias Notes That Reviewers Can Actually Check

Risk-of-bias assessment is the step in a systematic review where good intentions most often turn into unreadable tables. Two reviewers fill in a grid of “low”, “some concerns” and “high”, the grid goes into a supplementary file, and nobody — including the authors six months later — can reconstruct why a particular study was judged the way it was. This guide is about the notes behind those judgements: what to write, how much, and how to keep them consistent across dozens of studies. It also covers where an AI research agent such as Yila can help collect the evidence for each judgement without making the judgement for you.

Why the notes matter more than the ratings

Tools such as Cochrane’s RoB 2 for randomised trials, ROBINS-I for non-randomised studies of interventions, QUADAS-2 for diagnostic accuracy and the Newcastle–Ottawa Scale for observational studies all produce a summary rating. Those ratings feed into the certainty of evidence (for example via GRADE), sensitivity analyses and the discussion section. A rating without a reason is impossible to audit. Editors and peer reviewers increasingly ask for the “support for judgement” text, and RoB 2 explicitly expects it.

Good notes make three things possible:

1. Consistency. When the same issue (say, unblinded outcome assessment of a subjective outcome) appears in ten studies, the notes show whether it was handled the same way each time.
2. Resolution of disagreements. When two reviewers disagree, comparing their notes usually reveals that one of them missed a sentence in the methods section, which settles the matter quickly.
3. Updating. In a living or updated review, new reviewers can follow the original reasoning instead of re-assessing everything from scratch.

Anchor each note to the signalling question

Most modern tools are organised as domains, each with signalling questions. For RoB 2, domain 1 (the randomisation process) asks whether the allocation sequence was random, whether it was concealed until participants were assigned, and whether baseline differences suggest a problem. Write one short note per signalling question rather than one paragraph per domain. A useful pattern is:

Answer: Probably yes. Evidence: “Randomisation was performed using a computer-generated sequence held by an independent statistician” (Methods, p. 4). Comment: Concealment method not stated explicitly, but central allocation is implied.

The three parts — answer, evidence and comment — keep the note short and checkable. The evidence is a direct quotation with its location. The comment is where your reasoning lives.

Quote, do not paraphrase

Paraphrase is where errors creep in. “Assessors were blinded” and “assessors were not aware of the hypothesis” are not the same claim, and a paraphrase can quietly turn one into the other. Quoting the exact sentence, with page or section number, lets a second reviewer check the source in seconds.

When the information is absent, say so explicitly: “No information on allocation concealment in the article, the protocol (NCT01234567) or the supplementary material.” The phrase “not reported” on its own does not tell the reader where you looked.

Collecting quotations is exactly the type of work that an AI research agent can accelerate. Given the full text and the list of signalling questions, Yila can pull out candidate passages for each question — the randomisation paragraph, the blinding statement, the flow diagram numbers — with their locations, so the reviewer starts from the evidence rather than from a blank page. The reviewer still decides whether the passage actually answers the question and what the answer is.

Check more than the paper

Many judgements depend on information outside the main article:

• Trial registrations reveal whether the primary outcome changed between registration and publication, which is central to the selective reporting domain.
• Published protocols describe the planned analysis, helping you judge whether a reported analysis was pre-specified.
• Supplementary files often contain the CONSORT flow diagram, attrition by arm and sensitivity analyses.
• Companion publications may report the same trial with different follow-up or outcomes.

Record every source you consulted in the note. When a registration and a paper disagree, quote both.

Calibrate before you start

Before assessing the full set, have all reviewers independently assess the same three to five studies, then meet and compare notes question by question. Typical calibration outcomes are a shared decision rule (“for patient-reported outcomes in unblinded trials, domain 4 is high risk unless there is a specific argument otherwise”) and a list of agreed phrases. Write these rules into a short guidance document that sits alongside the assessment form. It prevents drift as the work stretches over weeks.

Keep the summary judgement mechanical

Most tools provide an algorithm from signalling-question answers to a domain judgement, and from domain judgements to an overall judgement. Follow it. If you override the algorithm, which RoB 2 allows, explain why in the comment field. Overrides without explanation are the first thing a careful peer reviewer will question.

Make the notes usable downstream

Risk-of-bias notes are not an end in themselves. They feed into:

• Sensitivity analyses, where studies at high risk of bias are excluded to see whether the pooled estimate changes.
• GRADE assessments, where serious risk of bias across the body of evidence lowers certainty.
• The discussion, where specific weaknesses of the evidence need to be named.

Structured notes make all three easier. If each note has a consistent answer field, you can filter quickly for, say, all studies with high risk from missing outcome data.

A minimal template

For each study and each domain:

• Signalling question number and text
• Answer (Yes / Probably yes / Probably no / No / No information)
• Evidence: quotation with location, or explicit statement of where you looked
• Comment: reasoning in one or two sentences
• Domain judgement and whether it follows the algorithm

Kept in a spreadsheet or a review tool that supports free-text support fields, this template produces notes that are compact, consistent and checkable.

Summary

Risk-of-bias ratings are only as trustworthy as the notes behind them. Anchor notes to signalling questions, quote the source with its location, check registrations and protocols, calibrate as a team and follow the tool’s algorithm. An AI research agent like Yila can speed up the most tedious part — locating and extracting the relevant passages from papers, protocols and registrations — so reviewers spend their time on the judgements that actually require expertise, and anyone reading the review later can see exactly how those judgements were reached.