Engagement Report: The Red-Team Suite That Never Ran

Third sample engagement report, this time on Clutch, a cellular threat-detection monitor for IMSI catchers and signal downgrades. The target this time wasn't the detection logic directly — it was the test harness built to validate that logic, adversary_emulation.py, which is supposed to simulate a rogue cell tower and confirm Clutch catches it.

Scope and Objective

Target: Clutch's red-team adversary-emulation suite, invoked as test_ios_remote_integration.py --red-team --authorized, which simulates a StingRay-style cell-tower downgrade attack and checks whether AdvancedIMSICatcherDetector flags it.

Objective: confirm the validation suite actually runs end to end and produces a real detection-rate number, rather than trusting that a test file with the right docstring does what it claims.

Findings

Critical: the suite could not be imported at all

adversary_emulation.py failed on a plain IndentationError before any test could run. The cause: an orphaned block of code sitting directly after a complete, working method (inject_stingray_pattern), with no enclosing function. It looked like the tail end of a second, abandoned method, a leftover from an edit that removed the function's opening but not its body. Python can't parse a file with a dangling block like that, so nothing downstream of it, the entire red-team suite, could even load.

Checking whether that orphaned block was reachable from anywhere else: it wasn't. The suite's one call site (DetectionValidationRunner.run_all) only ever invokes inject_stingray_pattern. The dead block was genuinely dead, not a second feature quietly going untested. Removed.

Critical: the one reachable method called a method that doesn't exist

With the import fixed, the suite ran and immediately threw AttributeError: inject_stingray_pattern calls self._detector.detect_anomalies(metrics), but AdvancedIMSICatcherDetector has no detect_anomalies method. It has analyze_advanced_metrics. The deleted orphaned block, ironically, had the correct method name. That's the clearest evidence this test path had never actually been exercised: the live code called a name that never existed, while the dead code next to it called the real one.

High: two constructor calls were built against a schema that had since changed

Two more runtime errors followed, each the same shape. CellularTower was being constructed with lat=, lon=, and signal_strength= keyword arguments; the real dataclass has no such fields; it takes location as a single (lat, lon) tuple and carries no signal_strength field at all. AdvancedCellularMetrics was being constructed with earfcn= and encryption_algorithm=; the real field is arfcn, and there is no encryption_algorithm field anywhere on the class. Both call sites were written against an earlier version of each dataclass that no longer matches what's on disk. Fixed by conforming the calls to the real, current schema rather than guessing at new field names.

Low: a missing-dependency warning pointed at the wrong library

A separate, smaller issue, found while setting up the environment to run the above: cellular_security.py prints "scikit-learn not available" whenever its combined try/except ImportError block fails, but that block also imports pandas, in the same try. With scikit-learn installed and pandas missing, the error message blames the wrong dependency. Anyone debugging that warning by reinstalling scikit-learn would get nowhere. Not fixed in this pass since it's cosmetic, not functional, flagged for the repo's own backlog.

Informational: raw packet injection silently assumed root

The emulation also sends a crafted UDP packet over a raw socket as a secondary, "actionable" touch, separate from the detection-logic assertion. On macOS this requires root (BPF device access), and the code caught ImportError for scapy being absent but not a runtime permission failure for scapy being present and unprivileged. Added a try/except around the send() call so an unprivileged run degrades to a logged warning instead of crashing a test that doesn't actually depend on the packet going out. The detection validation below ran entirely unprivileged.

Verification

Re-run live tonight, after all five fixes, unprivileged, no sudo:

RED TEAM VALIDATION COMPLETE
Detection Rate:  100%
[+] All expected threat types detected
Report:          forensics/validation/ADVEML-20261011-004907.json

The full JSON report shows all three expected threat types fired on every one of 8 injection cycles: TIMING_ADVANCE_ZERO, INVALID_PHYSICAL_CELL_ID, POTENTIAL_JAMMING, against a simulated forced protocol downgrade (MITRE Mobile technique M1011) with rising signal strength across cycles, zero timing advance, an invalid physical cell ID, and no neighbor cells, the signature of a StingRay-style rogue base station getting the simulated device to lock on. Zero missed detections.

Mitigations and Next Steps

A test suite that can't import is worse than no test suite, because its presence in the repo implies coverage that doesn't exist. Add this suite to CI so an import-breaking change fails the build immediately instead of waiting for someone to run --red-team by hand.
Dataclass field renames need their call sites tracked. Three of five bugs here were the same root cause: a struct's shape changed and a caller elsewhere in the codebase wasn't updated. A type checker (mypy against these dataclasses) would have caught all three at review time, before any of them shipped.
Fix the scikit-learn/pandas misattribution in cellular_security.py's dependency check: separate the try blocks so each warning names the actual missing package.
The detection logic itself, once actually exercised, performed correctly: 100% on the one technique tested tonight. That's a real result, specific to this technique and this run, not a claim about the detector's performance against techniques not covered here.

GhostInThePrompt.com // The test said nothing because it never ran. Once it ran, it worked.