Predator vs XBOW Benchmark
FLAGS CAPTURED, SINGLE PASS
85 / 104
SINGLE-PASS CAPTURE RATE
81.7%
BEST-OF, EVER CAPTURED
92.3%
BENCHMARKS
104 public XBOW
The result to sit with
That is one run end to end: reconnaissance, exploitation, multi-step chaining, and flag capture, driven by the harness and the model powering Ares with no operator steering it. Of the benchmarks that built and ran on this pass, Predator captured 85 of 87, which is 97.7%, and the capture rate holds across every difficulty tier, the hardest included. A flag counts only when the real planted string is exfiltrated, so detection is never scored as a win.
Abstract
Predator is the autonomous orchestration harness that drives Reaper. It runs black-box reconnaissance, vulnerability detection, exploitation, multi-step chaining, and flag capture with no human in the loop. This report documents Predator’s final run against every public XBOW validation benchmark: a single autonomous pass over all 104 apps, scored under deliberately conservative rules.
On that single pass Predator captured 85 of 104 flags, 81.7%. Counting any benchmark Predator has ever captured across runs, the best-of figure is 96 of 104, 92.3%. We lead with the single-pass number on purpose, because it is what one autonomous run actually produces; best-of is a more generous measure and is labeled as such throughout. Of the benchmarks that built and ran on this pass, Predator captured 85 of 87, which is 97.7%.
XBOW publishes the benchmark as 104 original web-security challenges built to evaluate offensive tools and agents across realistic vulnerability classes. Each challenge carries a hidden flag objective rather than a theoretical, finding-only score, and XBOW’s own release notes emphasize reproducibility, novelty, and exploit validation as the purpose of the suite.
The harness accrues value; the model is swappable. Predator is that harness. That thesis is why the harness, not any single model, is the durable asset.
A flag counts only when the real planted string is exfiltrated into a response or finding, never on a reflected payload, so detection is never scored as success.
What the harness does
A scanner reports a finding and stops. Predator treats the finding as the start of an exploit path. For each target it performs reconnaissance, infers authentication state, exercises the vulnerability, chains steps where a single bug is not enough, and proves impact by extracting the planted flag. Findings that cannot be converted into a captured flag are not counted as wins.
STEP 01
Recon
Map the surface and fingerprint the stack.
STEP 02
Browser
Render client-side routes an HTTP crawler cannot see.
STEP 03
Detect
Classify the surface into vulnerability classes.
STEP 04
Exploit
Run generic capability primitives per class.
STEP 05
Chain
Combine primitive wins into a full exploit path.
STEP 06
Capture
Exfiltrate the planted secret to the grader.
SELF-IMPROVEMENT LOOP
Failed paths feed the next attempt. Each pass records what did not work and hands that context to the following run, so the harness converges on an exploit path instead of repeating the same dead end.
Methodology
Predator completed one full-suite pass across all 104 public XBOW validation benchmarks using its standard xbow_runner.py --redo harness, with out-of-band callback support enabled. Each benchmark is a CTF-style docker-compose web app that plants a real FLAG{…} string. This pass ran the full system, the harness driving the model powering Ares.
All 104 benchmark apps were attempted; each was built and run through Docker Compose, then taken through recon, browser crawl, detection, exploitation, chaining, and generic per-class exfil playbooks.
A benchmark counts as captured only when the actual planted flag string, read from the container’s on-disk flag, env, and sha256, appears in a response or finding. A reflected payload earns no credit.
Single-pass is one --redo run, the headline number. Best-of is any capture for a benchmark across all runs, deduplicated, and is always labeled as the more generous measure.
Build and run errors on this pass stay in the all-possible denominator. They split into transient docker networking and port-bind failures, scan timeouts beyond 900 seconds on slow endpoints, and two legacy images with rotted build environments.
All playbooks are generic techniques audited for hard-coded paths or per-target answers. The scorecard groups each benchmark under a single primary tag, so category totals sum to exactly 104 and nothing is counted twice.
104
Total benchmarks
The complete public XBOW validation suite.
85
Flags captured, single pass
85 of the 87 that built and ran, which is 97.7%.
81.7%
Single-pass capture rate
85 of 104 on one pass, every error left in the denominator.
92.3%
Best-of, ever captured
96 of 104 captured in some run, deduplicated. A more generous measure, shown for context.
Category scorecard
Every benchmark, grouped under a single primary vulnerability tag and ordered by volume, so the rows sum to exactly 104. The crimson share is what Predator captured on this single pass, the amber share is captured on another run but missed this pass to a transient error, and the gray share was never captured.
CATEGORY
CAPTURED, BEST-OF
Cross-site scripting
23
IDOR
12
SSTI
13
Privilege escalation
9
Command injection
8
SQL injection
7
XXE
3
Capability highlights
Predator performed best on categories that require more than detection: an end-to-end path from a reachable surface to a proven, flag-bearing exploit. Several categories were captured in full. The single-pass figure is lower than best-of only where a transient error blocked a benchmark Predator captures reliably.
Cross-site scripting
Every XSS benchmark captured across runs. The six not captured this pass were transient docker failures and timeouts, not misses.
23 / 23 best-of · 17 this pass
IDOR
Object-reference abuse across authenticated state, including MongoDB ObjectId forging and authenticated walks.
12 / 12 best-of · 11 this pass
Command injection
Output-shaping, Struts OGNL and SpEL injection, and blind out-of-band execution.
8 / 8 best-of · 7 this pass
SQL injection
Error-based, boolean and time-blind, UNION, and second-order auth bypass into a login chain, plus 3 of 3 blind best-of.
4 / 4 · blind 3 / 3
IT HOLDS UP AT THE HARD END
XBOW labels each challenge level 1 through 3. Predator’s capture rate stays roughly flat across all three levels: Level 1 at 38 of 45 single pass (41 best-of), Level 2 at 40 of 51 single pass (48 best-of), and Level 3 at 7 of 8 single pass and best-of. The hardest tier is captured at the same rate as the easiest.
Room to improve
Best-of leaves eight benchmarks uncaptured. They separate cleanly into genuine capability gaps, a few remaining targets, and pure infrastructure, and we keep them apart rather than blurring them together.
Built, ran, no flag · 2
The clearest true gaps
XBEN-092-24 · phar deserialization
A phar deserialization whose gopher trigger is blocked in this environment and needs an alternate path.
XBEN-098-24 · LFI chain
A layered LFI and disclosure chain that did not convert on this pass.
Legacy images · 2
XBEN-026-24 and XBEN-100-24 no longer build. Environment rot in old base images, not a capability question, fixed by repairing the build environment.
Never captured across any run
Open targets for new generic playbooks
SSTI ×2, smuggling_desync, nosqli
Two SSTI variants, the single request-smuggling benchmark, and the single NoSQL injection benchmark remain open targets for new generic playbooks.
Single-pass flakiness
Eleven of the 17 errors this pass were benchmarks captured on other runs. They cost single-pass score, not capability. Lower docker flakiness and a tighter timeout budget recover most of them.
Predator vs MAPTA
MAPTA is a multi-agent autonomous web-penetration system published in August 2025. On the 104-scenario XBOW benchmark it reported a 76.9% overall success rate. On a single pass Predator now leads MAPTA on four of the six shared categories, ties on SSTI, and trails only on SSRF.
4 / 6
Categories Predator leads
Single pass, of six shared with MAPTA.
100%
XSS best-of vs MAPTA 57%
Once a shared weakness, now a Predator strength.
0%
MAPTA on blind SQLi
Predator captures all three blind SQLi best-of.
On a single pass Predator leads MAPTA on XSS (73.9% vs 57%), command injection (87.5% vs 75%), SQL injection (100% vs 83%), and blind SQL injection (66.7% vs MAPTA’s reported 0%). SSRF is the one trailing category: MAPTA captures every SSRF case, Predator reaches two of three best-of and one on this pass.
MAPTA disclosed cost and efficiency this Predator run did not: roughly $0.21 per challenge and a 143.2-second median solve time. Predator’s run used a fixed 900-second cap and counted every timeout as a miss. Different priorities, both defensible.
Public comparison context
Public reports differ in runtime limits, infrastructure, allowed tooling, retries, and whether build failures are counted at all. Predator’s primary bar is the single-pass, all-possible figure, with every error left in the denominator. Treat the other systems as context, not as controlled measurements against Predator.
READ THIS FIRST
These figures are not an apples-to-apples comparison. Public reports differ in runtime limits, infrastructure, allowed tooling, retries, and whether build failures are counted at all. The dashed marker is best-of, a more generous metric.
Xfenser AI
Single pass
88.5%
XBOW official baseline
Single pass
85%
Cyber-AutoAgent
Single pass
84.6%
Predator, the harness for Ares
Single pass · all possible
81.7%
MAPTA
Single pass
76.9%
On a single autonomous pass Predator’s 81.7% sits above MAPTA’s 76.9% and a few points under the XBOW baseline and the two highest community reports. On buildable targets the single-pass figure is 83.3%, roughly level with those baselines. The dashed marker shows best-of at 92.3%, which would top the table; we do not claim a win there. Predator is now competitive with the strongest public figures on a single pass and ahead of the closest peer system.
MODELS, NOT RIVAL HARNESSES
XBOW is model-agnostic by design and evaluates frontier models by running them inside its own agents. In April 2026 it published its read on OpenAI’s GPT-5.5, and in May 2026 its read on Anthropic’s Mythos Preview. Both belong next to this report, but as a layer comparison, not a scoreboard.
GPT-5.5 miss rate
On XBOW’s internal benchmark GPT-5.5 reached a 10% miss rate, their best result to date, versus GPT-5 at 40% and Opus 4.6 at 18%. Reported by XBOW.
GPT-5.5 computer use
It scored 97.5% on visual acuity, logged into targets in roughly half the iterations of the next best model, and over-persisted on hopeless paths about half as often as previous GPT versions. Reported by XBOW.
Mythos Preview
Mythos Preview cut missed vulnerabilities by about 42% against Opus 4.6, and about 55% with source code, strongest at reading code. On command-safety judgment it scored 77.8%, below Opus 4.6. Reported by XBOW.
That is the model layer. Predator is the harness layer that mounts a model and converts candidate findings into flag-bearing exploits. A stronger model raises the ceiling on discovery; the harness is where durable capability accrues. A model like GPT-5.5 or Mythos is something Predator mounts, not something it races.
What we claim, and what we don’t
Benchmark reports invite a predictable set of objections. Rather than wait for them, here is the boundary in plain terms: the claims this report stands behind, and the claims it deliberately does not make.
Don’t read these in
What we are not claiming
That Predator beats GPT-5.5, Opus 4.8, MAPTA, or any model or system. Those were measured on different benchmarks, and a model and a harness are different layers.
That 92.3% is the headline. Best-of is the generous figure and is labeled as such everywhere. The single-pass 81.7% leads.
A controlled model-versus-model result. We did not run that comparison, and we do not imply one.
That the suite is solved. Eight benchmarks are uncaptured, named by ID in the appendix.
On the record
What this report claims
On one fully autonomous pass, with no human in the loop, Predator captured 85 of 104, and 85 of the 87 that built and ran, which is 97.7%.
Every build and run error stays in the all-possible denominator; the headline is not inflated by dropping failures.
The playbooks are generic across 22 vulnerability classes, audited against hard-coded paths and per-target answers.
Capture rate holds across all three difficulty levels, the hardest tier included.
Scoring is flag-string exfiltration, fully automated, on a public suite anyone can clone and rerun.
Bottom line
On a single autonomous pass, Predator captured 85 of 104 flags, which is 81.7%, and 85 of the 87 that built and ran, which is 97.7%. Counting any capture across runs, best-of reaches 96 of 104, which is 92.3%. The strongest categories are the ones that demand an end-to-end exploit path: cross-site scripting, command injection, SQL injection and blind SQL injection, IDOR, privilege escalation, and XXE. Capture rate holds across all three difficulty levels, including the hardest. The model is swappable; the harness is the constant.
Get started
Point Ares at your APIs, web apps, and mobile infrastructure, and get grader-confirmed exploit paths, not a list of unproven findings.