The same skill scanner reads 2.5% or 92.7%, and the benchmark decides which

Draft. Every number from our own harness regenerates with python -m score.report; see methodology.md for how the lenses are computed and corpus-sources.md for corpus provenance. External figures are cited to the table and paper they come from.

If you are picking a scanner to vet agent skills before they reach your developers, you have a reading problem. In the last six months at least five benchmarks have published a recall number for "can this tool catch malicious skills," and they do not agree. The spread is not noise at the edges. One Cisco Skill Scanner binary is reported at 2.5%, at 35.6%, and at 92.7% recall across three published setups — every one of those figures is sourced below. A practitioner reading two of those papers side by side has no way to tell whether one scanner is far better than another or whether the two teams simply measured different things.

They measured different things. This post shows the three knobs that move a scanner's recall without touching the scanner — using numbers from our own harness, where we control all three, and verified figures from the published benchmarks — and then places the result against the two measurement papers closest to it.

The same Cisco binary, six ways. Blue bars are static mode and differ only by which corpus they ran on; the orange bar is the tool's LLM mode on the same set as the 35.6% static bar. Figures from MalSkillBench (arXiv:2606.07131, Table 5), MaliciousSkillBench (arXiv:2608.19901, Table 3), and this harness.

The claim in one table

We run each scanner unmodified, at a pinned version, over one fixed corpus of 987 malicious skills plus 8 inert canaries (995 positives in all), and we keep every tool's full raw output. From that single run we read recall under four lenses — four choices of where to put the flag/no-flag line. Nothing about the scanner or the samples changes between columns. Only the line moves.

Read the NVIDIA row twice. The same scan of the same 995 skills is 87% recall or 21% recall depending only on whether you count the tool's own "not SAFE" recommendation or require at least one HIGH-severity finding. That is a factor of four, and it is entirely a reporting decision. A benchmark that quotes a tool's native gate and a benchmark that quotes a fixed HIGH cutoff will publish different leaderboards from byte-identical scanner output.

This is knob one: the lens. Every scanner has a different native gate — Cisco's is_safe, NVIDIA's recommendation, Snyk's numeric risk index — so a single "detection rate" per tool is not comparable across tools unless the benchmark says which line it drew. Most headline numbers quote the native gate, which flatters tools with a loud gate (NVIDIA's CAUTION counts as a flag) and punishes tools with a quiet one (Cisco's install gate fires on 4% of a corpus where its scanner noticed something on 15%).

The published benchmarks turn the lens knob too. MalSkillBench (arXiv:2606.07131, Table 5) runs one Cisco binary over one set of 3,944 malicious and 4,000 benign skills and reports 92.7% recall in the tool's LLM mode and 35.6% in its static mode — a 57-point swing from one configuration flag, with precision moving only from 71.4% to 78.5%. Same scanner, same samples, same paper. MaliciousSkillBench (arXiv:2608.19901, Table 3) ran NVIDIA SkillSpector with its model disabled (--no-llm) and measured 0% recall on the source-disjoint set where our run of the same tool's native gate reads 87%. Disabling a scanner's model is a configuration choice, and it is itself a lens.

We can turn that same knob in-house, with the backing model as the only variable. We run one scanner — Tencent's AI-Infra-Guard — in its LLM mode over our corpus and swap only the local model behind it. Backed by qwen3.5-9b it reads 5% (7/148) recall on home-ground and 8% (71/839) on held-out; backed by gpt-oss-20b the same tool in the same mode reads 33% (49/148) home-ground and 2.5% (21/839) held-out. Overall recall barely moves (78/995 vs 70/995), yet the split profile inverts: one model concentrates its detection on the well-known corpus, the other spreads it thinly and holds up a little better on held-out sources. Neither is good in absolute terms, and that is the point — "LLM mode" is not one instrument but as many instruments as there are models you can put behind it, and which one you picked moves the leaderboard as much as the corpus does.

Knob two: the corpus

Now hold the scanner and the lens fixed and change only where the malicious samples came from. Our corpus has two disjoint slices: a home-ground slice of 148 skills confirmed malicious in the wild and drawn from the kind of well-known public source a tool may have tuned on, and a held-out, source-disjoint slice of 839 malicious skills from sources none of these tools advertise.

Same tool, same version, same cutoff. The only thing that changed is the origin of the samples, and Cisco's recall falls from 68% to 5%. A benchmark built mostly from one well-known public corpus will report the left column; a benchmark built from held-out sources will report the right column. Both are "recall against malicious skills." Neither is wrong, and nothing in a one-line leaderboard tells you which you are looking at.

The published Cisco numbers line up exactly this way once you hold the tool in static mode and change only the corpus:

Our held-out slice is MaliciousSkillBench's source-disjoint set, so their 2.5% and our 4–5% are the same 839 samples read at slightly different gates: the corpus holds the number down to single digits, and the lens only nudges it within them. Move to a mixed generated set and the same static tool reaches 35.6%; move to its home-ground and it reaches 68%. That shared-corpus agreement is the credibility check in the first two rows of the table — two independent runs of the same tool on the same 839 samples, landing within a few points of each other — and it is what makes the disagreements above trustworthy. Snyk shows the opposite trap: we measure 42% on this held-out set while MalSkillBench reports 28.7% on its own mixed set — close-sounding numbers that are not the same measurement and should not be read as agreement.

Knob three: the label

The third knob is the hardest to see and the easiest to argue about: what counts as a true positive in the first place. Our positive label means attributed as malicious in the wild — a human or a registry concluded this skill was shipped to do harm. It does not mean the skill was observed to succeed in a sandbox, and it does not mean a specific dangerous capability was demonstrated. Those are three different ground truths, and a scanner's recall changes with which one the benchmark scored against. We document our definition and a two-rater adjudication plan in labeling.md; we have not yet re-scored the full corpus under a demonstrated-capability definition, so we report the label knob as argued, not yet measured on our side.

Two recent papers make the label knob concrete. After the Party (arXiv 2609.17274) built a human-adjudicated reference standard for the OpenClaw skill ecosystem and found weighted scanner sensitivity ranging from 21.67% to 61.06% against it — even "the same scanners" land in a wide band once a human decides what malicious means. ClawHub Security Signals (arXiv 2606.01494) is blunter still: its labels are the registry's automated verdicts, "not human-annotated ground truth." When the label itself is a scanner's output, a benchmark is partly measuring agreement, not detection.

Where this sits

Two papers already own most of the territory next door, and it is worth being precise about what they did so the contribution here is not oversold.

• ClawHub Security Signals runs VirusTotal, static heuristics, and SkillSpector over 67,453 skills from one registry and shows they barely overlap: any two scanners agree on at most 10.4% of their combined positives, and 81.9% of flags come from a single scanner. It attributes the split to attack surface — SkillSpector raises semantic agentic-risk advisories while VirusTotal raises malware-reputation hits. That is the disagreement, measured and explained, inside one ecosystem.

• After the Party runs three scanners over the OpenClaw ecosystem, builds a human-adjudicated reference standard, and reports the 21.67–61.06% sensitivity band above. That is the reference-standard contribution, inside one ecosystem.

Both look within one registry. Neither runs the same scanner versions across the several independently published benchmarks that practitioners actually cite against each other, and neither separates the resulting recall spread into lens, corpus, and label as distinct causes. That cross-benchmark decomposition — the fact that one Cisco binary reads 2.5%, 4%, 35.6%, 68%, or 92.7% depending on which of these knobs a benchmark turned — is the gap this harness fills. It is a measurement contribution, not a new detector, and we state it as such.

What to do with a scanner recall number

1. Never compare two scanners by their native gates. Re-score both under one fixed severity cutoff, or you are comparing how loud their gates are, not how much they catch.
2. Ask what corpus the number came from, and whether the tool's vendor could have seen it. A recall figure on a tool's home-ground corpus is close to an upper bound, not a field estimate.
3. Ask what "malicious" meant — attributed in the wild, demonstrated capability, or another scanner's verdict. These are not interchangeable.
4. Ask what mode the tool ran in. Cisco's own LLM and static modes differ by 57 points on one set; a scanner run with its model disabled is a different instrument from the same scanner run whole.
5. Re-run on your own skills. The only recall number that describes your risk is the one measured on the skills your developers actually install, under the cutoff your gate actually uses. That is what the harness and the CI linter are for.

The harness that produced every in-house number above is version-pinned and rerunnable. If you get different figures, that is the point: tell us which knob you turned.

Sources

• MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills — arXiv:2606.07131, Table 5 (Cisco LLM 92.7% / static 35.6% recall; Snyk 28.7%; 3,944 malicious / 4,000 benign).
• MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection — arXiv:2608.19901, Table 3 source-disjoint (Cisco local-behavioral 2.5% recall, FPR 1.1%; SkillSpector --no-llm 0% recall; 839 malicious / ~545 benign).
• ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree — arXiv:2606.01494.
• After the Party: Growth, Governance, and Security Scanning in the OpenClaw Agent Skill Ecosystem — arXiv:2609.17274.
• This harness: leaderboard.md, regenerated with python -m score.report.