Where Multi-Agent Code Review Goes Blind

Reviewing code with a system of agents is becoming common, and the question I find interesting about it is not whether a swarm of reviewers finds more bugs than a single reviewer does. It is how the swarm fails. There are two ways. The agents can look past something entirely, and the step that turns their separate findings into one verdict can wave through a file that should have failed. Finding bugs makes a good demo. Knowing where a detector you built goes blind is harder, and more useful.

To have something concrete to work against, I built a small consensus reviewer that runs. Two agents read the same syntax tree, each with its own job, and a verifier turns their findings into a single PASS or REJECT. The agents here are deterministic AST passes standing in for what a production system would run as separate models. The point is the shape of the pipeline and the way it fails, not parity with a real one. It uses the standard library only, no API key, and it is accurate enough to be worth attacking. The code is at github.com/ghostintheprompt/blind_spot_lab.

The blind spot a text match never sees

Start with the baseline, since a lot of code review still comes down to a pattern matched over source text. Here is a reasonable grep-style reviewer run against a file that contains two command injections.

$ python3 naive_regex_review.py fixtures/command_injection.py
REJECT
  L18   REGEX-CMD  possible command injection

It found one. The hit is os.popen("tar czf /tmp/out.tgz " + user_path), a command built with string concatenation, the classic smell sitting right in the text. The one it missed is three lines up:

os.system(f"ping -c 1 {user_host}")

Same vulnerability, same class, CWE-78. The f-string has no + for a text pattern to catch, so the reviewer walks straight past it. This is not a contrived case. It is how a lot of adversarial code, and a lot of machine-generated code, clears text-based review. The dangerous construct gets shaped so the matcher was never looking at it.

The alternative is to stop matching text and read structure instead. The reviewer parses the file into a syntax tree and hands it to an agent that reasons about what a call actually does. The command-injection rule is about ten lines:

sink = self._resolve_shell_sink(node, name, root)
if sink and node.args:
    module, func = sink
    shell_true = any(
        kw.arg == "shell" and isinstance(kw.value, ast.Constant) and kw.value.value is True
        for kw in node.keywords
    )
    dynamic = not _is_constant_command(node.args[0])
    if module == "os" and dynamic:
        self._add("CTX-CMD-INJECTION", "CRITICAL", "CWE-78", node,
                  f"os.{func}() runs a non-literal command string (injectable)")

It never looks for the text os.system(. It checks whether a shell sink receives a command that is not a pure literal. The f-string is an ast.JoinedStr, so _is_constant_command returns False and the rule fires. Both injections come back, each marked CRITICAL. The same gap matters more as agents start reviewing model-generated code, which the OWASP Top 10 for LLM Applications files under improper output handling. A reviewer that works by matching text gives the evasion away for free.

Reading structure handles how the string is built. It does not, on its own, handle how the sink is named, and the fastest way to learn that was to attack my own detector. Renaming os.system through an import, from os import system and then a bare system(...) call, slipped past the first version completely. That one has a clean fix: resolve imports statically and map the alias back to its origin, which the reviewer now does, so the import version is caught. Rebinding the sink to a variable at runtime, run = os.system, still slips, and that one I cannot patch the same way. It is the same limitation that sits under the second blind spot below.

The structural approach also carries into auth logic, which is closer to what a code-analysis platform works on day to day. The second agent flags a callback that exchanges an OAuth code for a token but never validates the state parameter, the authorization-code CSRF class, CWE-352. It leaves alone the clean version in the next file, which validates state with hmac.compare_digest. It flags a token compared with == instead of a constant-time check (CWE-208, a timing side channel) and a verify=False that disables TLS verification (CWE-295). Catching the bug is only half the work. Staying quiet on the correct code sitting right beside it is what decides whether anyone leaves the tool turned on.

The blind spots it keeps on purpose

The first blind spot is one any reviewer can have. The next two I left in deliberately, demonstrated and asserted by tests rather than hidden, because they are the ones specific to this shape of system.

The first is the slow accumulation of medium-severity findings. One fixture uses a broken hash, hashlib.md5, in four separate places. Four MEDIUM findings, each weighted 2, add up to 8. The reject threshold is 10. The verdict:

   PASS  subthreshold.py              score=8   findings=4

A module broken in four places clears the gate. Nothing in it is individually bad enough, and the verifier has no sense that a pile of low-severity findings should add up to something worse. The threshold is a named constant in the code, so that choice stays visible rather than buried inside a comparison. A real consensus engine has to make this call on purpose. This one shows what happens when the call gets made by inattention.

The second is that the reviewer does not track values. Every agent reads one file's syntax tree in a single pass and reasons about the shapes it sees directly. It never follows a value from where it is set to where it is used. That is why run = os.system slips, and it is also why a vulnerability assembled across function or module boundaries, tainted in one place and dangerous three calls away in another, sits outside what the reviewer can see. Catching either one needs dataflow and taint tracking, which this lab does not do and does not claim to.

Stating those limits is the useful part, and most of what I know about them came from attacking the reviewer rather than trusting it. Saying where a consensus of agents goes blind, and roughly what it would take to close the gap, is worth more than one more claim about how many bugs the thing finds. The tests cover all of it: the catches, the quiet on clean code, the import alias it now resolves, and the two blind spots it keeps. They run in well under a second with nothing installed. Clone the lab and you can watch the reviewer fail in the places it was built to fail.