How we grade evidence

Every review on this site is compiled by an automated search of the medical literature. This page is the method it follows. It is written out in full because a conclusion you cannot check is worth very little.

Two separate questions

Most health writing collapses evidence into a single grade. That hides the most useful distinction there is. How much we trust the research and what the research found are different questions, and the combinations that look strange are the informative ones: a well-run trial can show that something does not work, and plenty of popular remedies have never been studied at all.

So every review carries both, and they never merge.

Two kinds of question

“Does it work?” and “could it hurt?” are questions about a treatment. They make no sense asked of a condition. “Can a pilonidal cyst cause sepsis?”, “how long does it last?” and “why does it hurt?” have answers, but there is no benefit or harm in them to find.

So a review about one treatment shows what the studies found for it, on the scale below. Every other review — what a condition can lead to, how it behaves over time, why something happens, or which of several approaches to take — shows how well the question has been studied, in blue rather than in the green and red of a finding. Its answer is in the bottom line, in words.

Well studied: Strong research answers the question directly — the top of the ladder that suits the question (below).
Fairly well studied: Some solid, direct research, but not much of it.
Some evidence: Direct research exists, but it is limited — few studies, small ones, or imperfect ones.
Little evidence: A few case reports or small studies, or only research on a related question.
Not studied: No study in people answers the question.

Evidence strength

How much the body of research can be relied on. Adapted from the GRADE framework used in clinical guidelines — adapted, not applied: GRADE is a formal process carried out by a panel against a protocol, and this is a simpler derivative of it. Only studies in people that answer this exact question count toward it; every listing says how many of the studies found do.

Which research counts most depends on the question. For a question about one treatment, or a choice between treatments, randomised trials lead:

High: Two or more well-conducted randomised trials answer the question directly, or a high-quality systematic review of such trials does.
Moderate: One well-conducted randomised trial answers the question directly, or a systematic review does but the trials under it are limited.
Low: Only observational studies answer the question, or the randomised evidence is small or at risk of bias.
Very low: The only evidence is small, uncontrolled, or measures a related question rather than this one.
No human evidence: No study in people answers the question. Laboratory and animal work may exist; it cannot show that something works in people.

A question about what a condition can lead to, how it behaves over time, or why something happens is answered by following people, not by randomising them. There, studies that track patients over time lead, and case reports count when there are enough of them:

High: A good-quality systematic review answers the question directly, or two or more well-conducted studies that follow patients forward over time do.
Moderate: One well-conducted study following patients forward over time, any systematic review, or three or more studies looking back at patients' records.
Low: At least one study following patients over time or looking back at their records, or five or more case reports and series.
Very low: Only a handful of case reports or small series, or only research on a related question.
No human evidence: No study in people answers the question.

On a question about one treatment, when the studies contradict each other — some finding a benefit, others none or a worse result — the strength drops one level, because a body of evidence that disagrees with itself is weaker than its best single study suggests. Other questions have no benefit or harm for studies to disagree on, so the rule does not apply to them.

What it shows

On a review about one treatment: the direction of the finding, weighted by how strong each study is — a randomised trial counts for more than a case series, and a high-quality study counts for more than a poor one.

Benefit shown: Most of the stronger evidence points to a benefit.
Possible benefit: The evidence points to a benefit, but is not strong enough to be confident.
Unclear: Studies point in different directions.
No meaningful benefit: Most of the evidence found no meaningful benefit — no difference, or a worse result than the comparison.
Signal of harm: Studies in people that test this treatment directly reported injury or adverse effects — and Dr. Kamrava has reviewed the finding against the standard below before it is shown.
Not studied: No study has measured this outcome in people with this condition.

A harm finding is reviewed before it is shown

Telling someone a treatment may hurt them is the most consequential thing these pages can say, so it is the one finding that does not go up on the software’s authority alone. Before “Signal of harm” appears on a review, Dr. Kamrava reads the finding and the studies behind it, against this standard:

At least two studies in people that test this treatment directly report injury or adverse effects, and at least one of them compares groups — a trial or a cohort study, not only case reports.
Laboratory and animal work, and studies of a related but different treatment, cannot produce the badge. They can be described on the page; they cannot decide it.
Working less well than the comparison is not harm. It is evidence against a benefit, and is read as that.

A review that reaches that standard is held back, and Dr. Kamrava is sent the finding with the exact sentences each study reported. It goes live only when he approves it. The same holds while a review is being re-checked for whether its studies test exactly what the question asks: until then the page shows how well the question has been studied instead of a finding.

How citations are checked

The hardest problem with machine-written medical content is invented references. This system is built so that inventing one is not possible rather than merely unlikely:

Studies are never named by the software that summarises them. The search fetches records from PubMed first; the summariser is then shown one real record at a time and can only describe what it was given. A study that PubMed never returned has no route onto the page.
Every figure is traced to a sentence. Each extracted detail — the number of patients, the comparison, the result — carries the words it came from, and software checks that those words really appear in that abstract. You can read them yourself on any study card.
Every reference is re-checked against PubMed before a page publishes, and its title, journal and year must match.
Retracted papers are detected and excluded from every count and conclusion. They stay listed and labelled, so you can see they were considered.
Every study that decides a grade is checked against the question. A study of a related but different treatment — laser hair removal when the question is about laser treatment of a pilonidal sinus, or a freezing device when the question is about ice packs — is counted as a related question, not a direct answer, and its card says why.
Conclusions are calculated, not written. The two grades come from the rules on this page applied to the study table — not from a program’s opinion. Run the same studies through them and you get the same answer.

What those checks cannot catch is a misreading of a sentence that genuinely exists. Several independent checks test each claim against its quoted source for exactly that. A final review then reads the finished page the way a patient would, and checks that the answer, the grade and the studies behind them agree with one another and with the question. A page that fails any of these is not published.

What this is not

Not a meta-analysis. These reviews count, classify and describe studies. They never pool results into a combined statistic, which would not be defensible from abstracts.
Not a clinical guideline. Guidelines are issued by specialty societies after a formal process. This is patient education.
Not medical advice, and not a substitute for being examined. A research summary cannot know your history.
Not complete. Details are read from abstracts, not full papers. Anything a study reported only in its methods section is not captured.

Who is responsible for it

The search strategies, the grading rules on this page, and the checks that must pass before anything publishes were written by Allen Kamrava, MD, MBA, FACS, FASCRS, a board-certified colon and rectal surgeon in Beverly Hills.

The pages themselves are compiled by software running that method, and are not read one by one before publication. Saying otherwise would be the one claim on these pages that could not be checked — so it is not made. The single exception is a harm finding, which is not shown until he has reviewed it.

Something here look wrong? Tell the office — corrections to the method improve every page at once.