Building an agentic harness that outlasts the model
AI Summary
Shopify shares insights from building an internal multi-agent security harness called Dispatch to scan massive Rails monoliths for vulnerabilities. Over five months of evaluating frontier models, they found that while newer models discover more potential findings, they also introduce more noise and false positives. To address this, Shopify's custom Ruby-based harness manages code partitioning, artifact generation, parallel hunting, adversarial verification using separate models, and automated PR generation. By enforcing strict verification constraints and leveraging differential scanning, the harness minimizes token costs, reduces noise, and proves vulnerabilities with real tests before notifying developers.
Why it matters
As frontier AI models rapidly evolve, relying purely on raw model capabilities leads to excessive false positives and developer fatigue. This article demonstrates how engineering robust, deterministic application harnesses with structured multi-agent orchestration, test oracles, and cross-model verification can sustainably secure massive codebases while outlasting individual model lifecycles.
Conversation 0
Be the first to share your perspective.