Agent Skill Scanner Benchmark
An independent, rerunnable measurement of the security scanners that vet AI-agent skills. Every scanner runs unmodified, at a pinned version, over the same corpus. Each lens is computed from the scanner's own output: the native column is the tool's own verdict gate, the severity columns apply the same cutoff to every tool, so only those are comparable across rows.
Detection (recall) by lens
| Scanner | Version | Native gate | ≥ any finding | ≥ MEDIUM | ≥ HIGH |
|---|---|---|---|---|---|
| cisco | 2.0.14 | 4% (41/995) | 15% (145/995) | 5% (53/995) | 4% (41/995) |
| nvidia | 2.13.1 | 87% (870/995) ‡ | 40% (403/995) | 40% (403/995) | 21% (210/995) |
| snyk | 0.6.9 | 47% (470/995) | 47% (470/995) | 46% (460/995) † | 37% (368/995) † |
| tencent@gpt-oss-20b | 0.2.2 | 7% (70/995) | 7% (72/995) | 7% (70/995) | 7% (66/995) |
| tencent@qwen3.5-9b-mlx | 0.2.2 | 8% (78/995) | 8% (79/995) | 8% (78/995) | 7% (73/995) |
Errors (no parseable verdict, counted in the denominator): cisco 17, nvidia 0, snyk 9, tencent@gpt-oss-20b 9, tencent@qwen3.5-9b-mlx 10.
False-positive rate by lens
| Scanner | Native gate | ≥ any finding | ≥ MEDIUM | ≥ HIGH |
|---|---|---|---|---|
| cisco | 2% (9/545) | 4% (20/545) | 2% (13/545) | 2% (9/545) |
| nvidia | 69% (377/545) ‡ | 44% (239/545) | 42% (229/545) | 15% (81/545) |
| snyk | 24% (131/545) | 24% (131/545) | 14% (78/545) † | 9% (51/545) † |
| tencent@gpt-oss-20b | 2% (8/545) | 2% (9/545) | 2% (8/545) | 1% (7/545) |
| tencent@qwen3.5-9b-mlx | 0% (0/545) | 0% (0/545) | 0% (0/545) | 0% (0/545) |
† Snyk has no severity enum; its severity lenses band Snyk's numeric risk score and are provisional until confirmed against Snyk's published scale. Its native and any-finding lenses are exact.
‡ NVIDIA's native gate here is recommendation ≠ SAFE; the tool's install gate is a stricter risk-score threshold, not recomputed. The severity lenses are exact.
By corpus split
The same lenses restricted to each corpus slice: read a home-ground tool against the source-disjoint slice to see how far detection generalises.
Canaries
Detection (recall):
| Scanner | Version | Native gate | ≥ any finding | ≥ MEDIUM | ≥ HIGH |
|---|---|---|---|---|---|
| cisco | 2.0.14 | partial (1/8) | partial (2/8) | partial (2/8) | partial (1/8) |
| nvidia | 2.13.1 | partial (1/8) | partial (5/8) | partial (5/8) | partial (1/8) |
| snyk | 0.6.9 | partial (1/8) | partial (1/8) | partial (1/8) | partial (1/8) |
| tencent@gpt-oss-20b | 0.2.2 | partial (0/8) | partial (0/8) | partial (0/8) | partial (0/8) |
| tencent@qwen3.5-9b-mlx | 0.2.2 | partial (0/8) | partial (0/8) | partial (0/8) | partial (0/8) |
False positives:
| Scanner | Native gate | ≥ any finding | ≥ MEDIUM | ≥ HIGH |
|---|---|---|---|---|
| cisco | 0% (0/1) | 0% (0/1) | 0% (0/1) | 0% (0/1) |
| nvidia | 0% (0/1) ‡ | 0% (0/1) | 0% (0/1) | 0% (0/1) |
| snyk | 0% (0/1) | 0% (0/1) | 0% (0/1) † | 0% (0/1) † |
| tencent@gpt-oss-20b | 0% (0/1) | 0% (0/1) | 0% (0/1) | 0% (0/1) |
| tencent@qwen3.5-9b-mlx | 0% (0/1) | 0% (0/1) | 0% (0/1) | 0% (0/1) |
MSB source-disjoint (held-out sources)
Detection (recall):
| Scanner | Version | Native gate | ≥ any finding | ≥ MEDIUM | ≥ HIGH |
|---|---|---|---|---|---|
| cisco | 2.0.14 | 4% (33/839) | 5% (42/839) | 4% (38/839) | 4% (33/839) |
| nvidia | 2.13.1 | 87% (732/839) ‡ | 32% (265/839) | 32% (265/839) | 12% (99/839) |
| snyk | 0.6.9 | 42% (351/839) | 42% (351/839) | 42% (348/839) † | 35% (290/839) † |
| tencent@gpt-oss-20b | 0.2.2 | 2% (21/839) | 2% (21/839) | 2% (21/839) | 2% (18/839) |
| tencent@qwen3.5-9b-mlx | 0.2.2 | 8% (71/839) | 9% (72/839) | 8% (71/839) | 8% (66/839) |
False positives:
| Scanner | Native gate | ≥ any finding | ≥ MEDIUM | ≥ HIGH |
|---|---|---|---|---|
| cisco | 2% (9/541) | 4% (19/541) | 2% (12/541) | 2% (9/541) |
| nvidia | 69% (375/541) ‡ | 44% (237/541) | 42% (227/541) | 15% (80/541) |
| snyk | 24% (130/541) | 24% (130/541) | 14% (78/541) † | 9% (51/541) † |
| tencent@gpt-oss-20b | 2% (8/541) | 2% (9/541) | 2% (8/541) | 1% (7/541) |
| tencent@qwen3.5-9b-mlx | 0% (0/541) | 0% (0/541) | 0% (0/541) | 0% (0/541) |
MSB SRC002 (home-ground)
Detection (recall):
| Scanner | Version | Native gate | ≥ any finding | ≥ MEDIUM | ≥ HIGH |
|---|---|---|---|---|---|
| cisco | 2.0.14 | 5% (7/148) | 68% (101/148) | 9% (13/148) | 5% (7/148) |
| nvidia | 2.13.1 | 93% (137/148) ‡ | 90% (133/148) | 90% (133/148) | 74% (110/148) |
| snyk | 0.6.9 | 80% (118/148) | 80% (118/148) | 75% (111/148) † | 52% (77/148) † |
| tencent@gpt-oss-20b | 0.2.2 | 33% (49/148) | 34% (51/148) | 33% (49/148) | 32% (48/148) |
| tencent@qwen3.5-9b-mlx | 0.2.2 | 5% (7/148) | 5% (7/148) | 5% (7/148) | 5% (7/148) |
False positives:
| Scanner | Native gate | ≥ any finding | ≥ MEDIUM | ≥ HIGH |
|---|---|---|---|---|
| cisco | 0% (0/3) | 33% (1/3) | 33% (1/3) | 0% (0/3) |
| nvidia | 67% (2/3) ‡ | 67% (2/3) | 67% (2/3) | 33% (1/3) |
| snyk | 33% (1/3) | 33% (1/3) | 0% (0/3) † | 0% (0/3) † |
| tencent@gpt-oss-20b | 0% (0/3) | 0% (0/3) | 0% (0/3) | 0% (0/3) |
| tencent@qwen3.5-9b-mlx | 0% (0/3) | 0% (0/3) | 0% (0/3) | 0% (0/3) |
Per-behavior coverage
flag: the gate fired. noticed: a finding, but the gate said safe. miss: nothing. Multi-sample families show counts.
| Behavior | cisco | nvidia | skill-lint | snyk | tencent@gpt-oss-20b | tencent@qwen3.5-9b-mlx |
|---|---|---|---|---|---|---|
| dynamic-code-eval | flag | flag | miss | miss | miss | miss |
| env-read | miss | noticed | miss | miss | miss | miss |
| file-read-outside-skill | miss | miss | miss | miss | miss | miss |
| file-write-outside-skill | miss | noticed | miss | miss | miss | miss |
| http-request | noticed | noticed | miss | miss | miss | miss |
| instruction-file-read | miss | miss | miss | flag | miss | miss |
| malicious/source-disjoint | 33/839 flag, 9 noticed | 732/839 flag, 41 noticed | 372/839 flag, 20 noticed | 351/839 flag, 0 noticed | 21/839 flag, 0 noticed | 71/839 flag, 1 noticed |
| malicious/wild | 7/148 flag, 94 noticed | 137/148 flag, 4 noticed | 122/148 flag, 7 noticed | 118/148 flag, 0 noticed | 49/148 flag, 2 noticed | 7/148 flag, 0 noticed |
| network-connect | miss | miss | miss | miss | miss | miss |
| subprocess-spawn | miss | noticed | miss | miss | miss | miss |
Reference — our own linter (not a neutral entry)
skill-lint is this repository's own linter (ci/), run through the same harness. It is listed for transparency, not as a neutral entry: we wrote both it and the corpus.
| Scanner | Version | Native gate | ≥ any finding | ≥ MEDIUM | ≥ HIGH |
|---|---|---|---|---|---|
| skill-lint | 0.0.1 | 50% (494/995) | 52% (521/995) | 50% (494/995) | 16% (156/995) |
Method in brief
- Scanners are shelled out to, not reimplemented. Versions, exact flags and run dates are recorded in every result file.
- Every lens is recomputed from the scanner's own stored output; the native gate is what the tool itself returned.
- Canaries exercise one capability each with no payload, EICAR-style.
- Real samples are pinned by hash to a public dataset; nothing is executed or redistributed.
- LLM-backed scanners are one row per model; output is not deterministic and is reported as a single run.
- The benign set is small; treat FPR as indicative.
Full methodology, corpus provenance and raw per-sample scanner output
are in the repository (docs/methodology.md,
docs/corpus-sources.md, results/).