Risk-Based Code Review: Match Checks to What Could Go Wrong

Addy Osmani, Morgan Westlee Lunt, and Qing Wang describe a workflow where Claude reviews and fixes most changes before a human is asked to look at them ¹. The useful point is not simply that AI does an initial review. The source's sharper claim is that “did a human read the code?” is an imprecise proxy for identifying what could go wrong and choosing the right check ¹. One reading is that approval by itself is a weak control. If reviewers often catch routine issues such as missing null checks or broken contracts, some of that pattern-matching work may be automated before human review.

The useful move for engineering leaders isn't to automate approval. It's to figure out which checks catch which failure modes, and spend human attention on the questions those checks cannot settle. That is the core of a risk-based code review process.

What breaks when review means “someone read it”

Reading code is an activity, not a control. The objective is to identify what could go wrong and pick a check, automated or human, suited to catch it. Osmani, Lunt, and Wang frame “did a human read the code?” as an imprecise proxy for that objective ¹. A reviewer can read every line without knowing the intended behavior or the specific failure the review is meant to prevent.

A blanket approval requirement doesn't say which failure mode to look for, what evidence would satisfy it, or whether a different check could assess the risk faster or more reliably. It just says someone looked.

A practical starting point is to ask:

• What risk does this change introduce?
• Which checks can detect that risk?
• What's left over that only a human can judge?

That last question defines the reviewer's remaining job. The other questions help identify what can be checked automatically.

How to size up the risk of a change

A useful risk assessment should not rely on line count alone. A small diff that loosens an authorization check may be more consequential than a large refactor with well-specified behavior and good test coverage. Assess reach and consequence instead:

• Affected surface: which components, interfaces, or users does this touch?
• Reversibility: can you roll it back cleanly, or does it mutate persistent state?
• Data sensitivity: does it touch access, processing, or retention of sensitive data?
• Security impact: does it change authentication, authorization, input handling, or trust boundaries?
• Operational consequence: could it affect deployment, recovery, or service behavior under load?
• Uncertainty: is the intended behavior clear enough to test?

Record the risk alongside your confidence in the checks that cover it. A high-consequence change backed by a thin test signal merits more scrutiny than a sprawling change that's straightforward to verify, regardless of how imposing the diff looks.

A short change record makes that pairing explicit instead of leaving it in someone's head:

affected_area: "authorization middleware"
intended_behavior: "deny requests without the required role"
failure_modes:
  - "unauthorized access is allowed"
checks:
  - "authorization integration tests"
human_decision:
  required: true
  question: "Does the role boundary match the product policy?"

This format isn't drawn from a published standard. It's a practical template. Its purpose is to give the author, the automation, and the reviewer a shared description of what's risky and what's already covered.

Automated code review checks: match each check to a risk

Treat each automated check as targeted evidence for a specific failure mode, not as a blanket “green means safe” signal:

Risk	What catches it
Formatting, type errors, known static bugs	Formatter, type checker, static analysis
Known vulnerability patterns	Security scanning
Interface breakage	Contract or integration tests
Behavior change	Targeted tests for the new behavior and likely regressions

These pairings are illustrative, not a specification. Pick tools based on the failure modes your codebase actually produces.

Then split checks into blocking and advisory. Block integration only when a failure represents unacceptable risk. If a check is noisy, partial, or mainly useful as context, report its output for the reviewer rather than letting it gate the merge. A flaky or over-broad blocking check can train people to ignore it, which may be worse than not having it.

The workflow Osmani, Lunt, and Wang describe runs review continuously, with Claude reviewing and fixing most changes before a human is asked to look. They say this catches routine mistakes and saves team attention for higher-level questions ¹. The source excerpt does not provide the tool configuration or evidence behind that reported outcome, so don't assume the same result. Validate your own checks against your own failure history before relying on them to pre-filter changes.

Where humans still have to decide

Automation can report a failed test or a matched pattern. It cannot, by itself, determine whether a design fits the system's intent or whether a product trade-off is acceptable. Those decisions require context and accountability. Put a human in front of those decisions, rather than asking for undirected approval of every change.

Make the triggers for human review explicit:

• Changes to sensitive data or authorization boundaries
• Broad changes to dependencies or public interfaces
• Migrations that are hard to reverse
• Changes with weak automated coverage
• Disagreement between checks, or unresolved findings

Make the reviewer's job equally specific. Don't ask for a general thumbs-up. Ask the reviewer to weigh the stated risk, the evidence from the checks, and what's still uncertain. If the change is too poorly understood for that assessment, pause or escalate rather than rubber-stamping it.

The same logic of explicit limits applies to automation itself. For a related look at how teams define what an automated actor may and may not do, see how financial firms are defining boundaries for AI agents.

What continuous code review actually changes

Osmani, Lunt, and Wang describe review moving from a final gate after a pull request opens to a continuous process while code is being written. They say most changes have been reviewed and fixed by Claude at least once before a human is asked to look ¹. The source excerpt does not define the scope or proportion represented by “most,” or describe the tooling and operating procedure in enough detail to copy directly ¹. Treat it as a reported approach, not a benchmark. The same caution applies to headline demonstrations of agent endurance; see why a long-running demo is not a reliability case.

A workable version could run quick checks during editing, broader checks before integration, and human review against a short summary rather than a raw diff: intended behavior, likely failure modes, checks run, open issues, and the decision the reviewer needs to make.

That changes where the reviewer starts. Instead of an undifferentiated diff, they start from a risk claim and its evidence. The review still has to interrogate that framing: an author or an automated system can miss a failure mode or misdescribe the intended behavior, and a confident summary is not the same as a correct one.

Writing a pull request review policy

Build the policy around risk tiers, not specific tools. For each tier, specify the minimum automated evidence required, whether a human decision is mandatory, and who can waive or escalate it.

Require authors, or their automation, to state intended behavior and likely failure modes up front. That gives reviewers something concrete to check the evidence against, instead of rereading every line without a defined target.

Build in escape paths, because something will eventually not fit the policy:

• A required check is unavailable
• Checks disagree
• The change grows past its declared scope mid-review
• Nobody can confirm that a likely failure mode is covered

In each case, the safe default is to escalate, narrow the change, or add a check, not to treat missing evidence as a pass. A policy that doesn't say this explicitly may default to “ship it” under deadline pressure.

What to measure

Track whether checks are finding the risks they were assigned to find: findings by category, defects that escaped a check meant to catch them, time from detection to fix, and how often changes are escalated to human review. These are proposed measures, not results reported by the source.

Separate review effort from review quality. Reviewer time and turnaround can show where the process is consuming attention. Repeat findings and post-merge rework can indicate where a check is weak or a requirement is unclear. Reviewer notes can help reveal whether people are debating design and trade-offs or repeatedly catching routine mistakes that an automated check might be able to catch.

Don't infer better outcomes just because the workflow changed. The authors report that their approach catches routine mistakes and saves team attention for higher-level questions, but the source excerpt provides no quantitative results or operating detail to establish whether that holds outside their team ¹. Measure your own process before relying on the claim.

Rolling out risk-based review without weakening anything

Start by writing down your current review gates and naming the failure mode each one is meant to catch. Anything that consumes reviewer time without a named risk or decision attached is a candidate to cut or automate.

Pilot the clearer policy on a bounded set of low- and moderate-risk changes, and keep full human review on changes that are high-consequence or poorly understood. Compare the pilot against the old process using the proposed measures, not intuition. If a check throws repeated false alarms, misses a known risk, or leaves reviewers finding the same issue repeatedly, fix the check rather than simply adding another layer. If an incident surfaces a failure mode the policy doesn't cover, add it to the risk list and decide what evidence should have caught it.

If coding agents will be part of the loop, settle what they can access and run first. Our piece on local-model support in coding agents covers the hard questions.

The target isn't fewer approvals. It's a process where every check has a named job, every human decision has a stated reason, and missing evidence is visible, not quietly mistaken for safety.