How we grade evidence

Two separate questions

Most health writing collapses evidence into a single grade. That hides the most useful distinction there is. How much we trust the research and what the research found are different questions, and the combinations that look strange are the informative ones: a well-run trial can show that something does not work, and plenty of popular remedies have never been studied at all.

So every review carries both, and they never merge.

Evidence strength

How much the body of research can be relied on. Adapted from the GRADE framework used in clinical guidelines — adapted, not applied: GRADE is a formal process carried out by a panel against a protocol, and this is a simpler derivative of it.

High

Two or more well-conducted randomised trials answer the question directly, or a high-quality systematic review of such trials does.

Moderate

One well-conducted randomised trial answers the question directly, or a systematic review does but the trials under it are limited.

Low

Only observational studies answer the question, or the randomised evidence is small or at risk of bias.

Very low

The only evidence is small, uncontrolled, or measures a related question rather than this one.

No human evidence

No study in people answers the question. Laboratory and animal work may exist; it cannot show that something works in people.

When studies contradict each other, the strength drops one level, because a body of evidence that disagrees with itself is weaker than its best single study suggests.

What it shows

The direction of the finding, weighted by how strong each study is — a randomised trial counts for more than a case series, and a high-quality study counts for more than a poor one.

Benefit shown

Most of the stronger evidence points to a benefit.

Possible benefit

The evidence points to a benefit, but is not strong enough to be confident.

Unclear

Studies point in different directions.

No meaningful benefit

Most of the evidence found no meaningful difference.

Signal of harm

A well-conducted study in people reported harm, or enough evidence of harm accumulated across studies to be worth warning about.

Not studied

No study has measured this outcome in people with this condition.

One rule is deliberately lopsided

Showing that something helps requires evidence in people with the condition. Showing that something might harm does not. Laboratory evidence that a substance damages healthy tissue is a fair reason to say “this may do harm, and there is no good reason to use it” — while the same tier of evidence could never support “this works”.

That asymmetry is intentional. The cost of wrongly discouraging a home remedy is small. The cost of wrongly encouraging one is not.

How citations are checked

The hardest problem with machine-written medical content is invented references. This system is built so that inventing one is not possible rather than merely unlikely:

• Studies are never named by the software that summarises them. The search fetches records from PubMed first; the summariser is then shown one real record at a time and can only describe what it was given. A study that PubMed never returned has no route onto the page.
• Every figure is traced to a sentence. Each extracted detail — the number of patients, the comparison, the result — carries the words it came from, and software checks that those words really appear in that abstract. You can read them yourself on any study card.
• Every reference is re-checked against PubMed before a page publishes, and its title, journal and year must match.
• Retracted papers are detected and excluded from every count and conclusion. They stay listed and labelled, so you can see they were considered.
• Conclusions are calculated, not written. The two grades come from the rules on this page applied to the study table — not from a program’s opinion. Run the same studies through them and you get the same answer.

What those checks cannot catch is a misreading of a sentence that genuinely exists. Several independent checks test each claim against its quoted source for exactly that, and a page that fails any of them is not published.

What this is not

• Not a meta-analysis. These reviews count, classify and describe studies. They never pool results into a combined statistic, which would not be defensible from abstracts.
• Not a clinical guideline. Guidelines are issued by specialty societies after a formal process. This is patient education.
• Not medical advice, and not a substitute for being examined. A research summary cannot know your history.
• Not complete. Details are read from abstracts, not full papers. Anything a study reported only in its methods section is not captured.

Who is responsible for it

The search strategies, the grading rules on this page, and the checks that must pass before anything publishes were written by Allen Kamrava, MD, MBA, FACS, FASCRS, a board-certified colon and rectal surgeon in Beverly Hills.

The pages themselves are compiled by software running that method, and are not read one by one before publication. Saying otherwise would be the one claim on these pages that could not be checked — so it is not made.

Something here look wrong? Tell the office — corrections to the method improve every page at once.