Methodology

How the leaderboard in leaderboard.md is produced, so that someone else can rerun it and get the same tables. Everything below is as of the run dates in the run log; this file is updated with every rerun.

What is measured

Each scanner is run, unmodified and at a pinned version, over the same set of skill directories. For every directory we keep both the scanner's own verdict and its full raw output. From that raw we derive, for every tool, its findings on one common severity scale, and then apply the same set of lenses to every tool.

A single detection number per scanner would not be comparable across tools, because each tool's own install gate means something different (Cisco's is_safe, NVIDIA's recommendation, Snyk's "any risk index", Tencent's SARIF level). So we report, for each tool, the fraction it flags under four lenses:

• native — the tool's own install gate, as the tool documents it. This reproduces each tool's own headline number.
• ≥ any finding — the tool reported at least one finding at LOW or above.
• ≥ MEDIUM / ≥ HIGH — at least one finding at that severity or above. These apply one cutoff to every tool and are the only columns comparable across rows.

The native lens is read from the stored verdict; the severity lenses are recomputed from the stored raw output. Because every run already saves the raw output per sample, the lenses can be recomputed from an existing results/*.json with no scanner re-run (python -m score.report).

A sample that produced no parsable verdict is an error, and never counts as flagged under any lens. Errors stay in the denominator: a malicious package the scanner refuses to load is still a package the user is not warned about.

Native gates, by tool

NVIDIA's documented install gate is a stricter threshold on a 0–100 risk score (block above 50); we report its recommendation as the native gate and do not recompute the score gate, so the native column overstates what SkillSpector would actually block on. The severity lenses do not depend on it. This is a deliberate choice, kept after checking its size: on the source-disjoint slice recommendation != SAFE fires on 87% of malicious and 69% of benign skills, almost all of them CAUTION on INFO-only findings, whereas a BLOCK-only gate fires on about 1% of each (see Replication check below). The recommendation is what a user sees in the tool's output, so it stays the native lens; the ≥ MEDIUM lens is the number to compare across tools.

Severity normalization

Each tool's findings are mapped to info < low < medium < high < critical:

Snyk exposes no severity enum, only a numeric score per risk index. We band that score onto the common scale (score ≥ 800 critical, ≥ 500 high, ≥ 200 medium, ≥ 1 low; the bands are in score/lenses.py). These bands are provisional — they decide Snyk's ≥ MEDIUM / ≥ HIGH numbers and have not been confirmed against Snyk's published risk-index scale, so those two Snyk cells are marked with a dagger (†). Snyk's native and ≥ any-finding lenses do not depend on the bands and are exact.

The scoring also keeps a coarser per-capability view for the canaries: flag (the native gate fired), noticed (a non-informational finding but the gate said safe), miss (nothing). Several scanners report MEDIUM findings and still recommend SAFE; "noticed" is what makes that visible.

Metrics

• Recall (detection) = flagged ÷ samples with expected: flag, per lens.
• FPR = flagged ÷ samples with expected: clean, per lens.
• Errors = samples with no parsable verdict, counted in the denominators above.
• A run that did not cover the wild set is partial: its recall is shown as counts, never as a percentage, so a canary-only run cannot be misread as a full-corpus score.
• Per-behavior coverage = flag/noticed/miss for each canary behavior, and flag/noticed counts for the wild family.

Corpus

Two parts, both under corpus/. Details and licenses are in corpus-sources.md.

Canaries (corpus/canaries/, 9, in-repo, authored here). Each exercises exactly one capability with no payload: inert by construction, in the spirit of EICAR. They measure whether a scanner flags a capability, independent of any known family.

Manifest (corpus/manifest.yaml, 151 entries, by reference). No sample bytes are committed. Each entry carries a public source, a pinned commit, and a sha256; corpus/loader.py fetches into the gitignored corpus/cache/ and deletes anything whose hash differs.

• 3 benign: Anthropic's internal-comms, mcp-builder, pdf skills at anthropics/skills commit dbd4588f, pinned file by file. These are the false-positive side.
• 148 malicious, real-world: the SRC002 block of MaliciousSkillBench (protectskills/MaliciousSkillBench at 893afd60, data CC-BY-4.0), which is MaliciousAgentSkillsBench's set of skills found malicious in the wild on skills.rest and skillsmp.com. Pinned by the archive's sha256. Each is one behavior family, malicious/wild, because the dataset carries no per-sample capability labels we would trust.

Source-disjoint manifest (corpus/manifest-msb-source-disjoint.yaml, expands to 1,380 samples, by reference). MaliciousSkillBench's held-out test split: sources SRC009 / SRC011 / SRC012, 839 malicious + 541 benign (four benign rows are excluded because the dataset publishes redacted text for them with no hash to verify; corpus-sources.md names them). Labels, hashes and split membership come from the pinned benchmark_manifest.csv; the text comes from the Hugging Face dataset at revision d4b42ce5 and is written to the cache only if its sha256 matches exact_hash. This is the out-of-distribution slice; the SRC002 manifest above is the in-distribution ("home-ground") slice. Results carry a split field, and the leaderboard reports every lens per split.

The samples are never executed. Every scanner here is static or LLM-based and only reads files.

Scanners and how they are invoked

Adapters shell out to the real CLIs and parse their native output; they do not reimplement any rule. Each result file and leaderboard row records version, exact flags, and run date.

Tencent has no detection logic of its own beyond the prompts it sends, so its row is named tencent@<model> and every model is a separate row: a local qwen3.5:9b-mlx on Ollama, and OpenRouter's free nvidia/nemotron-3.5-super-vl:free. The tool's default model (deepseek-v4-flash) has not been run; it is not free.

Snyk and Tencent send sample content to third parties. All manifest samples are already public, and the canaries are ours.

Results as of 2026-10-29

The tables in this section are the home-ground run (156 expected-flag: 148 wild + 8 canaries; 4 expected-clean). The source-disjoint slice was run the same day; its per-split tables are in leaderboard.md under By corpus split, and this section is to be rewritten around the two slices. Snyk's source-disjoint run hit the public Agent-Scan daily quota on 561 samples (reported as errors); runner.run --retry-errors rescans exactly those once the quota resets. The 2026-11-02 retry cleared the quota backlog, so Snyk's held-out figures are now final. The 9 errors that remain (6 source-disjoint, 3 home-ground) are permanent parse failures, not quota: Snyk refuses a SKILL.md with no name in its YAML frontmatter ("could not inspect skill"), and those stay in the denominator like any sample the scanner cannot load.

Detection (recall) by lens

False-positive rate by lens

† provisional Snyk band (see Severity normalization). ‡ NVIDIA native gate is recommendation, not its numeric install gate.

Reference — skill-lint (ours), not a neutral entry. skill-lint is this repo's own linter; we wrote both it and the corpus, so it is kept off the comparison above. For reference its recall is 78% / 83% / 78% / 61% across the four lenses, with 2/4 FPR.

Per-behavior coverage

The full flag/noticed/miss grid per canary behavior is in leaderboard.md. In brief: presented with a single capability and no payload, no scanner flags a loopback connection, an outside-skill file read, a subprocess, or a temp-file write. Only eval (Cisco, NVIDIA) and the prose-only file-read instruction (Snyk) fire. On the wild family: Cisco 7/148 flag (94 more noticed below its gate), NVIDIA 137/148, Snyk 119/148, Tencent @ Qwen 11/148.

What the numbers say, and what they do not

• The ranking survives a consistent lens. Applying the same ≥ MEDIUM cutoff to every tool — not each tool's own gate — NVIDIA flags 88%, Snyk 72% (provisional), Cisco 10%. The gap between SkillSpector and skill-scanner is not an artifact of comparing a strict gate to a loose one; it holds when both are read at the same severity.
• Cisco's findings are mostly low-severity. Its ≥ any-finding recall is 66%, but that collapses to 10% at ≥ MEDIUM: the 94 wild samples it "noticed" below its gate (largely PIPELINE_TAINT_FLOW) sit below MEDIUM. The generous lens does not rescue the number.
• Recall is bought with false positives. NVIDIA's 88% comes with 2 of 4 benign Anthropic skills flagged at ≥ MEDIUM (50% FPR); at ≥ HIGH that falls to 1 of 4 as its recall falls to 71%. The trade-off is visible per lens, which is the point of reporting them side by side.
• What fires on the wild set is "download and run". NVIDIA's top rules were remote fetch-and-execute and tool chaining; Snyk's were unverifiable_dependencies, third_party_content_exposure, suspicious_download_url. This corpus is dominated by skills whose SKILL.md pulls a payload from a URL, so detection here is largely detection of that one pattern.
• Errors are malformed packages. All 8 Cisco and 3 Snyk errors are SKILL.md files with missing name frontmatter, no SKILL.md, or unparsable YAML. Both tools refuse to load rather than scan; we count these because the user still gets no warning.

Published comparisons

Every number below that includes SkillSpector or Cisco was published by someone other than NVIDIA. NVIDIA has published no comparison of SkillSpector against other scanners: its repository carries one script, scripts/compare_scan_accuracy.py, that compares two SkillSpector revisions on a human-adjudicated corpus kept outside the repo, a release regression gate rather than a benchmark. Static-only rows, recall / FPR:

Read together: the widely quoted "0% static" for SkillSpector is the paper's BLOCK-gate figure; at ≥ MEDIUM the space and this benchmark agree. Every published Cisco static number sits a few points above ours, with the space's source-checkout rule pack the outlier. None of the three publishes code and per-sample results together: the paper gives a table, Liatrio a post without code or raw output, the space data files whose reproduce steps live in the Cisco scanner's own tree. Liatrio and SkillSecurer (arXiv 2609.14079) also compare the tools LLM-backed, where both exceed 90% recall at 20–28% FPR; those are not static measurements and are not reproduced here.

Replication check against a published static run

One other static run over the same MSB source-disjoint split is public: Vineethsain's Hugging Face space Cisco Skill Scanner vs NVIDIA SkillSpector (2026-10-16), a Cisco-side study whose reproduce steps live in the Cisco scanner's own evals/ tree. Its static arm ran Cisco at source commit 4f8bec3 and SkillSpector v2.11.2 over 1,368 records both tools scored; we ran the pip release 2.0.14 and SkillSpector 2.13.1 over 1,380. Same split, same four lens shapes, independent harnesses (docs/figures/replication-vs-space.png):

SkillSpector replicates within three points at ≥ MEDIUM and ≥ HIGH across two harnesses and two versions; the shipped-gate row differs only in the lens definition (above). Cisco does not replicate: the space's numbers are higher at every lens. Two differences are known and either could explain it: the rule set in play (a source checkout with the "shipped core rule pack" versus the pip release) and the any-finding lens (ours starts at LOW, theirs counts INFO). Resolving it needs both sides pinned to the same release; until then our Cisco row is the lower of two published measurements. Both harnesses also see SkillSpector report most MSB records as partial analysis (1,043 of 1,384 in the space), because each SKILL.md references files the dataset does not ship; coverage of the shipped text is still complete.

Known limitations

• Home-ground corpus — the biggest caveat. The wild set is one dataset (MaliciousSkillBench SRC002) with one dominant behavior (curl | sh). A scanner tuned to that pattern scores well here and may not elsewhere; independent vendor comparisons report far lower static detection for the same tools on a source-disjoint corpus, which is consistent with dataset overlap being the dominant factor in these numbers. The source-disjoint slice now measures exactly that gap (see By corpus split in leaderboard.md); read the home-ground tables as "detection of the MSB-SRC002 pattern", not "detection in general". It is still one benchmark's held-out split, not a non-MSB corpus.
• Small negative set. Four benign samples cannot support a precise FPR; it is reported because it is what we have, not to one decimal.
• Snyk severity bands are provisional. Snyk's ≥ MEDIUM / ≥ HIGH numbers depend on a score→severity banding we chose, not confirmed against Snyk's own scale. Its native and ≥ any numbers are exact.
• NVIDIA's native gate is its recommendation, not its install gate. The score>50 install gate is stricter; the native column overstates what it would block. Compare tools on the severity lenses, not native.
• LLM rows are not deterministic. Rerunning Tencent on the same model gives different findings; its rows are one sample each. On an earlier run Qwen justified "benign" verdicts by quoting the canaries' own SKILL.md (which says they are inert) — an LLM scanner that trusts the skill's self-description is scoring the author's honesty.
• The Nemotron row is partial. Run only on the 8 canaries: OpenRouter's daily free quota does not cover 156 requests and three canary scans hit the tool's 300 s streaming deadline. Kept as partial rather than dropped.
• Behavior labels for wild samples are the dataset's. We take "malicious" from MaliciousAgentSkillsBench and do not relabel the full set. What that inherited label asserts (attribution in the wild, not demonstrated capability) and how a sample of it is adjudicated with a second rater are in labeling.md.
• Scans are not sandboxed. All scanners are static or LLM-based and read files only; runner/sandbox.py does not yet exist and must before any dynamic scanner is added.
• The SkillSpector OSV lookup is a network call that cannot be disabled and whose results depend on the day it runs.
• Scanner defaults, not tuned configs. Each tool is run as a user would from its README. Vendors may ship stricter policies; we measure the default.

Reproducing

uv venv -p 3.12 .venv
uv pip install -p .venv/bin/python -e ".[cisco,nvidia,snyk,tencent,dev]"
.venv/bin/python -m corpus.loader verify     # fetches the manifest into corpus/cache/
uv pip install -p .venv/bin/python datasets  # the source-disjoint slice is HF-hosted
M="--manifest corpus/manifest.yaml --manifest corpus/manifest-msb-source-disjoint.yaml"
                                              # omit $M to scan the canaries only
.venv/bin/python -m runner.run --scanner cisco $M
.venv/bin/python -m runner.run --scanner nvidia $M
SNYK_TOKEN=... .venv/bin/python -m runner.run --scanner snyk $M
LLM_API_KEY=ollama LLM_BASE_URL=http://localhost:11434/v1 LLM_MODEL=qwen3.5:9b-mlx \
  .venv/bin/python -m runner.run --scanner tencent $M
.venv/bin/python -m runner.run --scanner skill-lint $M
.venv/bin/python -m score.report              # recomputes every lens from results/*.json

loader verify downloads a tarball of live malicious skill packages into a gitignored directory, and --manifest is what puts them in front of a scanner. Do both in a disposable VM; without the flag the runner touches only the inert canaries. The static scanners take minutes over the full corpus; the local 9B-parameter LLM took about three and a half hours for the 160-sample home-ground run alone, which is why the Tencent rows do not yet cover the source-disjoint slice. Both slices must come from one run per scanner (--manifest is repeatable), because the report takes the latest result file per scanner. Snyk's public tier has a daily quota below the corpus size: rerun with --retry-errors results/snyk-<date>.json the next day to fill in the samples it refused.

Because every lens is recomputed from the raw stored in each result file, changing the scoring or adding a lens does not require re-running any scanner — only python -m score.report. Every cell in the leaderboard traces to the tool's own JSON or SARIF in results/<scanner>-<date>.json.

Scope

This repository measures scanners. It does not author malware, build evasion techniques, or redistribute samples; the full boundary is in CLAUDE.md. Canaries are inert and labelled as such in their own SKILL.md. Real samples are cited by hash and public source only.

Run log