Label schema and adjudication plan

What the positive label in this benchmark means, where each positive's label comes from, and how a second rater validates a sample of it. This is the layer the precision/recall numbers stand on: a PR curve is only as trustworthy as the ground truth under it. Provenance of the sources is in corpus-sources.md; scoring and lenses are in methodology.md; this file is about the per-sample label itself.

Scope note: the positive set here is the 148 real-world malicious skills in corpus/manifest.yaml (MaliciousSkillBench SRC002 = MaliciousAgentSkillsBench, arXiv:2602.06547). The source-disjoint slice inherits the same schema; its labels come from the same benchmark's frozen label column and are not re-adjudicated here.

What a positive label means — and what it does not

Every positive carries expected: flag. That is a single, inherited, binary judgement, and it is important to say precisely what it asserts.

• It asserts: this skill identity was attributed as malicious in the wild by MaliciousAgentSkillsBench — found on skills.rest or skillsmp.com and judged malicious by that project's process, which we take as given and do not reproduce.
• It does not assert: that the skill demonstrably executes a malicious behavior under any defined capability, nor that running it would achieve harm, nor that we have independently confirmed intent. The upstream dataset carries no per-sample capability labels we would trust, so we do not claim any.

This distinction is the one the evaluation-critique literature turns on: "When AUC 0.998 Is Not Enough" (arXiv:2606.22864) shows that a detector can score near-perfectly against labels that mark the presence of something flagged rather than a demonstrated malicious outcome, and that the two are not interchangeable. Our positive label is a presence/attribution label, not a demonstrated-capability label. The benchmark therefore makes the narrower, defensible claim:

We measure whether a scanner flags skills that were independently attributed as malicious in the wild, at a stated operating point, against a benign set drawn from official skills. We do not claim to measure detection of demonstrated exploit success.

Reviewers who would otherwise reach for 2606.22864 are answered by stating this up front rather than implying the stronger claim.

Provenance recorded per positive

The chain is already in the manifest; this names it so it is auditable end to end. For each positive we can state:

So a positive's label is traceable as: our id -> UASSB origin id -> SRC002 tarball (hash-pinned) -> marketplace slug, with the label definition and authority fixed here. No step invents a label; every step points at where the prior one came from.

Open item this surfaces: 148 vs 157

The research proposal cites "157 confirmed-malicious skills" from arXiv:2602.06547; the manifest uses 148 (the complete, unsanitized SRC002 packages that survive hash verification). The 9-identity gap must be reconciled against the paper before the count is quoted in a write-up: it is most likely sanitized/duplicate/withdrawn identities, but that has to be checked, not assumed. Recorded here so the labeling pass owns it.

Capability taxonomy (for the adjudicated sample)

The dataset gives one bit. To report recall by capability — the useful view for a defender — a rater tags each sampled positive with the behavior classes it exhibits. The classes are the ones the benchmark's own inert canaries already exercise, so the taxonomy is grounded in what we can independently measure, not invented for this file:

• file-read-sensitive — reads a file outside the skill's own tree
• file-write-outside-skill — writes outside the skill's own tree
• network-connect — opens a socket / connects to a host
• http-request — makes an outbound HTTP(S) request
• subprocess-spawn — spawns an external process
• instruction-file-read — reads agent instruction/config files
• prompt-injection — text aimed at steering the agent, not the user
• obfuscation — encoded/hidden payload (base64, unicode-tag, etc.)
• none-visible — rater sees no malicious capability (tests the inherited label)

A sample may carry several. none-visible is deliberately available: if a rater cannot see why a skill was flagged, that is a finding about the inherited label, and it is recorded, not discarded.

Adjudication protocol

Purpose: validate the inherited positive label and add capability structure, on a sample, with a reported agreement statistic — the minimum a reviewer expects and a thing no published scanner comparison has done.

1. Sample. 50 of the 148 positives, drawn with a fixed seed (reproducible, below). Roughly one in three, enough to estimate a proportion to about +/- 10% and to compute agreement.
2. Raters. Two people rate independently, blind to each other and to the scanners' verdicts. (Rater identities are Martin's to assign; the ratings are a human judgement and are not produced by this tooling — see below.)
3. Per item, each rater records: (a) concur the skill is malicious — yes / unsure / no; (b) the capability classes above; (c) a one-line rationale.
4. Agreement. Report Cohen's kappa on (a), and per-class agreement on (b). Kappa below ~0.6 means the label definition is too loose and this file needs tightening before any PR curve is published.
5. Resolution. Disagreements are resolved by discussion or a third rater; the resolved tags become the sample's capability ground truth.
6. Record. Ratings land in corpus/adjudication.csv (id, rater, concur, capabilities, note) — gitignored if it is ever to hold anything beyond public slugs, resolved by the same rule as the cache.

The drawn sample

Seed 20261102. Reproduce with:

.venv/bin/python -c "import yaml,random; \
ids=sorted(s['id'] for s in yaml.safe_load(open('corpus/manifest.yaml'))['samples'] if s.get('expected')=='flag'); \
print('\n'.join(sorted(random.Random(20261102).sample(ids,50))))"

msb-ASB04_000034, msb-ASB04_000181, msb-ASB04_000597, msb-ASB04_000757, msb-ASB04_000954, msb-ASB04_000979, msb-ASB04_001042, msb-ASB04_001046, msb-ASB04_001156, msb-ASB04_001502, msb-ASB04_001614, msb-ASB04_001630, msb-ASB04_001790, msb-ASB04_002341, msb-ASB04_002563, msb-ASB04_002647, msb-ASB04_002754, msb-ASB04_002842, msb-ASB04_003145, msb-ASB04_003177, msb-ASB04_003206, msb-ASB04_003439, msb-ASB04_003489, msb-ASB04_003622, msb-ASB04_003855, msb-ASB04_004213, msb-ASB04_004290, msb-ASB04_004303, msb-ASB04_004314, msb-ASB04_004520, msb-ASB04_004548, msb-ASB04_004586, msb-ASB04_004834, msb-ASB04_005070, msb-ASB04_005083, msb-ASB04_005117, msb-ASB04_005148, msb-ASB04_005231, msb-ASB04_005259, msb-ASB04_005448, msb-ASB04_005517, msb-ASB04_005545, msb-ASB04_005838, msb-ASB04_005883, msb-ASB04_006140, msb-ASB04_006185, msb-ASB04_006503, msb-ASB04_006745, msb-ASB04_007158, msb-ASB04_007327

What this plan does not do

• It does not relabel the full corpus. The 148 keep the inherited binary label; adjudication validates a sample of it and adds capability tags to that sample. Recall by capability is reported on the sample, with its sampling frame stated.
• It does not execute any sample. Rating is reading, consistent with the no-sandbox, static-only rule in corpus-sources.md. Capability tags are what a rater observes in the files, not a dynamic trace.
• It is a plan, not a result. The ratings are a human judgement Martin assigns and runs; this tooling draws the sample and fixes the schema, and does not generate the capability determinations itself — that is the point of an independent inter-rater check.