Notebook interview transcript¶

Generated by python -m msr_pipeline elicit. Do not edit by hand: the team's interpretation and verification of each answer are in elicitation/notebook-interview.md.

Item	Value
Generated	2026-09-18 18:47 UTC
Source pack	built 2026-09-18T18:47:17+00:00; repository commit `6b40b7e909e7`; 24 files indexed; A 336 passages, G 926 passages, P 493 passages
Tier P included	yes
Method	BM25 (k1 1.5, b 0.75); passages of at most 120 words; top 3 per question, at most 2 per file
Status	"match" means at least 50% of the question's terms occur in the returned passages. It does not mean the passages answer the question; the team decides that in the interview record

NB-Q01 (Meaning and scope; scope: gitskills)¶

Question. What exactly does one row of the GitSkills artifacts table represent, and how should a study count distinct skills separately from copies? Which column marks the representative copy, and is the representative the original source of the content?

Status. match. Question terms found in the passages: 13 of 18. Terms no returned passage contains: table, represent, study, separately, mark.

Sources cited. A6, A9, A2

[1] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > Dataset Structure > artifacts · line 248 (BM25 21.4)

One row per discovered file: repository, path, exact basename, location class, and content hash. For the representative of each distinct content: full text, parsed front matter (name, description), body size, and — for 458,548 files in standard locations plus a size-stratified sample of the rest — the first and last commit of the file, with anonymised author accounts and commit count.

[2] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Schema > artifacts — one row per file occurrence · line 75 (BM25 20.0)

Column Meaning

repo_full_name, path where the file lives (primary key)

filename exact basename as returned by code search (case preserved)

location_class canonical (.claude/skills/<name>/SKILL.md), skills-dir (under a skills/ directory), or other

file_sha git blob hash of the content; identical hash = identical bytes

discovered_at when the collection run recorded the file

dedup_primary 1 for the one representative of each distinct content

content full text (representatives only)

content_fetched, content_sha_ok retrieval bookkeeping; content_sha_ok = 1 when the downloaded bytes reproduce file_sha, 2 when repaired via the blob API

[3] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 3 · line 252 (BM25 20.0)

files ultimately retrieved. We therefore partitioned the search space by file size, splitting any range with more than 1,000 results until every range could be retrieved completely. Deduplication.We group files by content hash and select one representative per group for enrichment, preferring a file in the .claude/skills/ directory, with a deterministic rule to break ties. We do not assume that the representative is the original source. All copies remain in the dataset with their repository, path, and location class, so the spread of each content can be measured. Enrichment.For each representative, the tool downloads the SKILL.md file, parses its front matter, and records the bundled scripts and reference documents in the skill folder,

NB-Q02 (Meaning and scope; scope: gitskills)¶

Question. What population does GitSkills cover, and what does GitHub code search leave out? Does the dataset include files that are not Agent Skills in the October 2025 sense?

Status. match. Question terms found in the passages: 12 of 15. Terms no returned passage contains: leave, include, sense.

Sources cited. A6, A4

[1] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > Collection Methodology · line 290 (BM25 30.9)

The dataset covers public repositories only and should be read as a lower bound on the population: GitHub code search indexes only default branches, files under 384 KB, recently active repositories with fewer than 500,000 files, and forks only when they have more stars than the parent.

[2] A4 · A4 GitSkills Zenodo record.md · Description (from the record) · line 28 (BM25 28.4)

An agent skill is a folder containing a SKILL.md file with instructions for a language-model agent, optionally accompanied by scripts and reference files. The agent loads the skill when it judges that a task matches the skill description. Anthropic introduced the format in October 2025 as an open specification. Nine months later, skill files in the millions sit in public GitHub repositories — yet no dataset recorded this population before GitSkills.

[3] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub · line 179 (BM25 26.6)

Paper (arXiv:2608.10906) · Sample repository · Zenodo DOI: 10.5281/zenodo.21875637 An agent skill is a folder containing a SKILL.md file with instructions for a language-model agent, optionally accompanied by scripts and reference files. The agent loads the skill when it judges that a task matches the skill description. Anthropic introduced the format in October 2025 as an open specification. Nine months later, skill files in the millions sit in public GitHub repositories — yet no dataset recorded this population before GitSkills.

NB-Q03 (Meaning and scope; scope: gitskills)¶

Question. How is the official GitSkills sample drawn, how does it relate to the full release, and which exact identifiers should a paper cite for the sample and for the full dataset?

Status. match. Question terms found in the passages: 7 of 12. Terms no returned passage contains: official, relate, release, exact, identifier.

Sources cited. A9, A4, A6

[1] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Sample vs full dataset · line 59 (BM25 17.8)

The sample is drawn deterministically: the 13,000 distinct contents with the lowest content hashes (uniform in practice, reproducible at any time), together with every row the full dataset holds about them — all copies across repositories, the metadata of every repository involved, all bundled files, and all history rows. The sample therefore has the same schema and the same invariants as the full dataset: every content group is complete and has exactly one representative carrying the text. Queries written against the sample run unchanged on the full dataset.

[2] A4 · A4 GitSkills Zenodo record.md · Description (from the record) · line 48 (BM25 16.7)

This dataset accompanies a paper accepted at the MSR 2027 Mining Challenge, also available as a preprint at arXiv:2608.10906. If you use the GitSkills dataset, please cite:

[3] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > Citation · line 314 (BM25 16.3)

This dataset accompanies a paper accepted at the MSR 2027 Mining Challenge, also available as a preprint at arXiv:2608.10906. If you use the GitSkills dataset, please cite:

NB-Q04 (Schema and relationships; scope: gitskills)¶

Question. How do artifact_siblings rows join to artifacts, which sibling rows carry text, and for which copies of a skill is folder composition recorded?

Status. match. Question terms found in the passages: 9 of 12. Terms no returned passage contains: carry, copy, composition.

Sources cited. A6, A9, A4

[1] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > Dataset Structure > artifact_siblings · line 257 (BM25 22.7)

Files stored alongside a representative skill: path, entry type (file or dir), size, and the text of files under a size cap. Whether a skill bundles scripts or reference material is recorded per skill in artifacts (has_scripts, has_references), not per individual sibling file.

[2] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Schema > artifact_siblings — files bundled with a representative skill · line 102 (BM25 21.8)

repo_full_name, artifact_path (joins to artifacts), entry_name, entry_type (file/dir), entry_size, entry_sha, content (text files up to 100 KB), content_fetched, skipped_reason (binary, oversize, or folder above the listing cap).

[3] A4 · A4 GitSkills Zenodo record.md · Description (from the record) · line 36 (BM25 20.4)

artifact_siblings (7,264,865 rows) — scripts and reference files stored alongside a representative skill.

NB-Q05 (Schema and relationships; scope: gitskills)¶

Question. What do content_sha_ok, composition_truncated, and history_fetched mean, and which values mark missing or unrecoverable data?

Status. match. Question terms found in the passages: 9 of 16. Terms no returned passage contains: history_fetched, mean, value, mark, missing, unrecoverable, data.

Sources cited. A9, A6

[1] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Schema > artifacts — one row per file occurrence · line 75 (BM25 22.9)

Column Meaning

repo_full_name, path where the file lives (primary key)

filename exact basename as returned by code search (case preserved)

location_class canonical (.claude/skills/<name>/SKILL.md), skills-dir (under a skills/ directory), or other

file_sha git blob hash of the content; identical hash = identical bytes

discovered_at when the collection run recorded the file

dedup_primary 1 for the one representative of each distinct content

content full text (representatives only)

content_fetched, content_sha_ok retrieval bookkeeping; content_sha_ok = 1 when the downloaded bytes reproduce file_sha, 2 when repaired via the blob API

[2] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Schema > artifacts — one row per file occurrence · line 85 (BM25 21.2)

| frontmatter_valid, name, description, body_chars | parsed YAML front matter and body length | | sibling_count, sibling_bytes, has_scripts, has_references | folder composition summary (representatives only) | | composition_fetched, composition_truncated | bookkeeping; composition_truncated = 1 flags folders larger than the listing cap (some repositories hold tens of thousands of skills) | | first_commit_at, last_commit_at, commit_count | commit history of the file at its current path (sampled rows only) | | first_commit_author, last_commit_author | anonymised author codes (see below); bots keep their login | | first_commit_author_type, last_commit_author_type | User, Bot, Organization, or empty when the commit has no linked GitHub account | | first_commit_message, last_commit_message | commit messages with emails and personal names masked |

[3] A6 · A6 GitSkills dataset card.md · (top of file) · line 63 (BM25 20.7)

dtype: int64
name: sibling_bytes dtype: int64

name: has_scripts dtype: int64

name: has_references dtype: int64

name: content_sha_ok dtype: int64

name: composition_truncated dtype: int64

name: first_commit_author dtype: string

name: first_commit_author_type dtype: string

name: first_commit_message dtype: string

name: last_commit_author dtype: string

name: last_commit_author_type dtype: string

name: last_commit_message dtype: string splits:

name: train num_examples: 3797117

config_name: artifact_siblings features:

name: repo_full_name dtype: string

name: artifact_path dtype: string

name: entry_name dtype: string

name: entry_type dtype: string

name: entry_size dtype: int64

name: entry_sha dtype: string

name: content dtype: string …

NB-Q05a (Schema and relationships; scope: gitskills; follow-up to NB-Q05)¶

Question. Which value of content_sha_ok records a representative whose file changed on GitHub before it was fetched and whose original content could not be recovered?

Status. match. Question terms found in the passages: 12 of 14. Terms no returned passage contains: value, whose.

Sources cited. A9, A2

[1] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Schema > artifacts — one row per file occurrence · line 75 (BM25 28.8)

Column Meaning

repo_full_name, path where the file lives (primary key)

filename exact basename as returned by code search (case preserved)

location_class canonical (.claude/skills/<name>/SKILL.md), skills-dir (under a skills/ directory), or other

file_sha git blob hash of the content; identical hash = identical bytes

discovered_at when the collection run recorded the file

dedup_primary 1 for the one representative of each distinct content

content full text (representatives only)

content_fetched, content_sha_ok retrieval bookkeeping; content_sha_ok = 1 when the downloaded bytes reproduce file_sha, 2 when repaired via the blob API

[2] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 3 · line 303 (BM25 22.3)

For 42 representatives the file changed on GitHub between discovery and fetch and the original could not be recovered; content_sha_okrecords this. Anonymization.We masked email addresses in commit messages, including GitHub noreply addresses, with a fixed marker, and a scan of the released file confirmed that none remain; the first and last commit messages are included in this redacted form. We replaced commit author accounts and personal names in trailer lines with codes from a keyed one- way function; the codes cannot be reversed and stay stable across the dataset. The search matches the basename case-insensitively and also returns case variants such as skill.md (54,289 files, 1.4%), including lowercase files predating the format. We retained

[3] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 3 · line 252 (BM25 16.4)

files ultimately retrieved. We therefore partitioned the search space by file size, splitting any range with more than 1,000 results until every range could be retrieved completely. Deduplication.We group files by content hash and select one representative per group for enrichment, preferring a file in the .claude/skills/ directory, with a deterministic rule to break ties. We do not assume that the representative is the original source. All copies remain in the dataset with their repository, path, and location class, so the spread of each content can be measured. Enrichment.For each representative, the tool downloads the SKILL.md file, parses its front matter, and records the bundled scripts and reference documents in the skill folder,

NB-Q06 (Schema and relationships; scope: gitskills)¶

Question. What does first_commit_at represent for a renamed file, and which rows have commit history at all?

Status. match. Question terms found in the passages: 7 of 8. Terms no returned passage contains: represent.

Sources cited. A9, A1

[1] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Known limitations · line 178 (BM25 20.9)

GitHub code search indexes default branches only, files under 384 KB, recently active repositories with fewer than 500,000 files, and forks only when more starred than the parent. The dataset is a lower bound on the population.

The dataset is a point-in-time snapshot (July 2026).

Commit history follows the file's current path: for a renamed file, first_commit_at dates the rename and commit_count covers the current path only.

History is collected for every skill in a standard location plus a size-stratified sample of the rest; history_fetched marks which rows.

Folder composition and history describe the representative's repository only; other copies of the same content may differ.

[2] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Schema > artifacts — one row per file occurrence · line 85 (BM25 15.2)

| frontmatter_valid, name, description, body_chars | parsed YAML front matter and body length | | sibling_count, sibling_bytes, has_scripts, has_references | folder composition summary (representatives only) | | composition_fetched, composition_truncated | bookkeeping; composition_truncated = 1 flags folders larger than the listing cap (some repositories hold tens of thousands of skills) | | first_commit_at, last_commit_at, commit_count | commit history of the file at its current path (sampled rows only) | | first_commit_author, last_commit_author | anonymised author codes (see below); bots keep their login | | first_commit_author_type, last_commit_author_type | User, Bot, Organization, or empty when the commit has no linked GitHub account | | first_commit_message, last_commit_message | commit messages with emails and personal names masked |

[3] A1 · A1 MSR 2027 Mining Challenge page.md · Call for Mining Challenge Papers > GitSkills · line 29 (BM25 11.8)

Depth: 7,264,865 bundled files (scripts and reference material) alongside representative skills, and 458,548 sampled commit histories with anonymized first- and last-commit author accounts

NB-Q07 (Feasibility; scope: gitskills)¶

Question. Our question asks whether skills with risk-relevant capabilities reach more repositories through verbatim copying, and whether modified variants add or remove those capabilities. Can the GitSkills sample support each part, and what in the sampling or history coverage limits it?

Status. weak match. Question terms found in the passages: 11 of 23. Terms no returned passage contains: ask, risk, relevant, capability, reach, variant, add, remove, sample, part, sampling, limit.

Sources cited. A6, A1, A2

[1] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > Intended Uses · line 219 (BM25 22.8)

Adoption and linguistic evolution — how the format spreads across languages, project types, and monthly cohorts; how many skills are themselves agent-authored.

Development of a shared format — vendor-neutral vs. tool-specific skill locations, and whether skill text targets one tool or the open spec.

Reuse without a package manager — 50.5% of collected files are verbatim copies; the dataset supports studying concentration of reuse and clone-genealogy patterns.

Software metrics for natural-language instructions — size, churn, age, clone coverage, and readability equivalents for skills, and whether description properties predict reuse and maintenance.

Maintenance and trust — staleness relative to described tools, and whether modified copies introduce command execution or network access

[2] A1 · A1 MSR 2027 Mining Challenge page.md · Call for Mining Challenge Papers > Challenges · line 65 (BM25 14.2)

3) Reuse without a package manager. Skills have no central registry; reuse happens by copying folders, and 50.5% of the collected files are verbatim copies.

[3] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 1 · line 27 (BM25 14.0)

time, and no compiler or type checker verifies the selection. Skills also have no central registry or package manager; developers reuse them by copying folders between repositories. How developers write, reuse, and maintain skills is therefore an empirical question, and no existing dataset records this population. We present GitSkills, a dataset of 3,797,117 SKILL.md files collected from 282,200 public repositories in July 2026. The dataset retains every file occurrence with its repository, path, and content hash. We group identical files into 1,877,981 distinct contents and enrich one representative per group with the full text, parsed front matter, folder contents, repository metadata, and, for a subset, the commit history of the file. A single self-contained SQLite file

NB-Q07a (Feasibility; scope: gitskills; follow-up to NB-Q07)¶

Question. When a distinct content is in the sample, does the sample keep every copy of it across repositories, and are variants of the same skill kept together?

Status. match. Question terms found in the passages: 8 of 11. Terms no returned passage contains: keep, variant, kept.

Sources cited. A9, A2

[1] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Sample vs full dataset · line 59 (BM25 21.7)

The sample is drawn deterministically: the 13,000 distinct contents with the lowest content hashes (uniform in practice, reproducible at any time), together with every row the full dataset holds about them — all copies across repositories, the metadata of every repository involved, all bundled files, and all history rows. The sample therefore has the same schema and the same invariants as the full dataset: every content group is complete and has exactly one representative carrying the text. Queries written against the sample run unchanged on the full dataset.

[2] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Two views of the full dataset · line 169 (BM25 17.0)

Body length of the 1.88M distinct skills (log scale): most skills are between one and ten thousand characters of instructions.

[3] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 1 · line 66 (BM25 14.2)

GitSkills records 3,797,117 SKILL.md files from 282,200 public GitHub repositories owned by 195,841 accounts, col- lected in July 2026. We group the files by content hash into 1,877,981 distinct contents and enrich one representative file per group with its full text, parsed front matter, folder contents, and repository metadata; commit history, with first- and last- commit author accounts, covers skills in standard locations (recognized skill directories such as .claude/skills/) and a size-stratified sample of the rest. Every occurrence is retained with its repository and path, so researchers can study both unique contents and their copies. The dataset covers the early adoption of the format, while conventions and tooling are still developing, so researchers can

NB-Q08 (Feasibility; scope: gitskills)¶

Question. Do the sources describe any way to link modified (near-duplicate) copies of the same skill, or only byte-identical copies?

Status. match. Question terms found in the passages: 8 of 11. Terms no returned passage contains: source, near, duplicate.

Sources cited. A2, A9

[1] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 1 · line 93 (BM25 16.0)

the dataset stores the text once and links all copies to it through the content hash. Identical hashes mean that the files contain exactly the same bytes. We collect folder contents and commit history for only one copy of each distinct skill. This information applies only to that specific repository. Other copies may have different scripts, reference files, or commit histories. III. POTENTIALRESEARCHQUESTIONS Agent skills influence how coding agents operate within software projects, so their content, structure, reuse, and maintenance are empirical questions, which researchers can now study at the level of the population. 1) Adoption and linguistic evolution. a) How quickly does the format spread, and which projects adopt it first, in terms of programming language, popular-

[2] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 2 · line 203 (BM25 14.0)

5) Maintenance and trust.Skills can instruct agents to run commands, access external resources, and execute bundled scripts, and they are copied between repositories without formal review. a) How often do skills become outdated relative to the projects and tools they describe? b) Do modified copies of widely reused skills introduce command execution or network access absent from the original, the analog of a supply-chain attack in an ecosystem without a registry? c) How often do skills bundle executable files, and how widely are these skills copied?

[3] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Anonymisation · line 129 (BM25 12.8)

Commit author accounts are replaced by keyed one-way codes, identical for the same account throughout, so authorship can be traced across skills and repositories without identifying anyone. The key is not distributed. Accounts of type Bot keep their login.

Email addresses (including GitHub noreply addresses) and personal names in commit messages are masked with a fixed marker; names of AI assistants in Co-authored-by trailers are kept. A scan of the released file confirmed no email addresses remain in commit messages or repository descriptions.

File contents are reproduced byte-for-byte as published on GitHub, so every content row still verifies against its git blob hash.

NB-Q09 (Implementation; scope: gitskills)¶

Question. What loading approach do the sources recommend for the sample and for the full dataset, and what example queries do they give?

Status. match. Question terms found in the passages: 6 of 10. Terms no returned passage contains: loading, source, recommend, give.

Sources cited. A9, A1

[1] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Sample vs full dataset · line 59 (BM25 14.5)

The sample is drawn deterministically: the 13,000 distinct contents with the lowest content hashes (uniform in practice, reproducible at any time), together with every row the full dataset holds about them — all copies across repositories, the metadata of every repository involved, all bundled files, and all history rows. The sample therefore has the same schema and the same invariants as the full dataset: every content group is complete and has exactly one representative carrying the text. Queries written against the sample run unchanged on the full dataset.

[2] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Example queries · line 225 (BM25 13.5)

Skill count by repository language:

SELECT r.language, COUNT(*) AS skills
FROM artifacts a JOIN repos r ON r.full_name = a.repo_full_name
GROUP BY r.language
ORDER BY skills DESC LIMIT 10;

[3] A1 · A1 MSR 2027 Mining Challenge page.md · Call for Mining Challenge Proposals · line 273 (BM25 8.7)

The International Conference on Mining Software Repositories (MSR) has hosted a mining challenge since 2006. With this challenge, we call upon everyone interested to apply their tools to a common dataset. The challenge is for researchers and practitioners to bravely use their mining tools and approaches on a dare.

NB-Q10 (Validation; scope: group)¶

Question. Under Group 9's validation protocol, how is a rule match labelled, who rates it, which agreement measure is reported, and what is the blindness rule?

Status. match. Question terms found in the passages: 10 of 11. Terms no returned passage contains: match.

Sources cited. G6, G1

[1] G6 · G6 Project management and decisions.md · Team Charter and Working Agreement > 9. Validation independence · line 306 (BM25 19.3)

Jerad Dunne (rater id jd) is the primary rater. At least one teammate labels round 1 independently as a second rater: Leticia la, Allie ah, Hina hk. The team decides at the kickoff who second-rates which kind (signals, lineage, drift); a split is allowed.

Blindness rule. The primary rater's label files are not committed until the second rater's labels for the same kind are committed. A second rater never opens the primary rater's labels.

Inter-rater Cohen's kappa is the primary reliability measure; intra-rater re-labelling is optional. An LLM rater is only allowed under an llm- id, is disclosed, and never counts as human agreement.

Protocol and dates: docs/validation/README.md.

[2] G1 · G1 Research design P-01.md · Research Question > 8. Required implementation component and minimum evidence > Validation design (group) · line 285 (BM25 15.8)

Optional intra-rater check. A random 30% of the sample may be re-labelled at least seven days later, blind to the first labels; that kappa is reported separately.

LLM raters. An LLM rater is allowed only under an llm- rater id, is disclosed, and never counts as human agreement. Its labels are never merged into the primary labels.

[3] G6 · G6 Project management and decisions.md · Work sign-up > 2. Workstreams and P-01 deliverables · line 657 (BM25 15.1)

| W5 | Validation and annotation | Annotation protocol, labelling, agreement, error analysis | Careful reading, domain knowledge | Sprint 2 (error analysis in Sprint 3) | Primary and second-rater labels with the annotation kit; blindness rule; precision per rule; inter-rater Cohen's kappa; adjudication of disagreements | #23, #29 | | W6 | Testing, CI, and reproducibility | Unit tests, CI, fresh-clone runs, release | pytest, GitHub Actions | Every sprint | Rule regression tests; gemba walk each review; release tags | #22, #33, #37 |

NB-Q11 (Implementation (process); scope: process)¶

Question. According to Group 9's process documents, what has to happen before a change to a detection rule or to the research question reaches main?

Status. match. Question terms found in the passages: 6 of 12. Terms no returned passage contains: according, group, process, document, happen, change.

Sources cited. G1, G4

[1] G1 · G1 Research design P-01.md · Research Question > 8. Required implementation component and minimum evidence · line 262 (BM25 13.8)

Minimum evidence: "Threat model, detection rules, annotated examples, false-positive discussion, and a safe reporting artifact. Students must not execute untrusted scripts from the dataset."

[2] G1 · G1 Research design P-01.md · P-01: Risky capabilities in copied agent skills: prevalence, reach, and drift > 2. Research question · line 317 (BM25 13.8)

Main question. In the GitSkills July 2026 sample, how prevalent are skill instructions and bundled scripts that enable risk-relevant capabilities, do skills carrying them reach more repositories through verbatim copying, and do modified variants of the same skill add or remove those capabilities?

[3] G4 · G4 Threats risks and draft report.md · Research question and hypotheses or expectations · line 223 (BM25 13.5)

Main question. In the GitSkills July 2026 sample, how prevalent are skill instructions and bundled scripts that enable risk-relevant capabilities, do skills carrying them reach more repositories through verbatim copying, and do modified variants of the same skill add or remove those capabilities?

NB-Q11a (Implementation (process); scope: process; follow-up to NB-Q11)¶

Question. What are the change management rules for research changes and process changes, and what review does every pull request need before it merges?

Status. match. Question terms found in the passages: 8 of 10. Terms no returned passage contains: rule, need.

Sources cited. G6, G7

[1] G6 · G6 Project management and decisions.md · Project Plan > 11. Change management · line 184 (BM25 19.6)

Backlog changes: any member may add issues; the Product Owner orders them; the sprint commitment changes only at a check-in, with a note in the sprint planning file.

Research changes: a decision issue, discussion, an ADR, and a line in the retrospective.

Process changes: a kaizen issue with a measurable expected effect, verified at the next retrospective.

Charter changes: a pull request approved by all other members and a row in the decision log.

[2] G7 · G7 Git documentation and work intake.md · Contributing > Branch, commit, pull request · line 34 (BM25 19.5)

Every change starts from an issue. Create one with the issue forms if none exists, and assign yourself.

Branch: git switch -c <type>/<issue>-<slug>, where type is one of research, pipeline, data, docs, test, process, report, presentation, fix.

Commit in imperative mood with the issue number: Add variant linking (#21).

Push and open a pull request using the template. Fill every section.

Request a review from a member who did not write the change. Rotate reviewers. Reviews are due within 48 hours.

Squash-merge after one approval and green CI. Delete the branch.

[3] G7 · G7 Git documentation and work intake.md · Getting started > 1. Onboarding path for a new member > Step 5. Your first pull request · line 130 (BM25 16.8)

Branch: git switch -c process/<issue>-onboarding-<firstname>.

Fill in your profile in docs/team/<firstname>-<lastname>.md (a stub with your GitHub handle already exists).

In TEAM_CHARTER.md section 14, add today's date and this pull request's number to your row.

Commit, push, and open a pull request with Closes #<your onboarding issue>.

Request a review from another member. You cannot merge until someone other than you approves.

Review one other member's onboarding pull request, so everyone practices both sides before real work starts.

NB-Q12 (Ethics and privacy; scope: gitskills)¶

Question. What anonymisation was applied to commit authors and commit messages, what was left unanonymised, and what must a study never do with author codes or personal data?

Status. match. Question terms found in the passages: 6 of 12. Terms no returned passage contains: applied, left, unanonymised, study, never, data.

Sources cited. A6, A9, A4

[1] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > Anonymisation · line 297 (BM25 22.3)

Commit author accounts are replaced by keyed one-way codes, identical for the same account throughout the dataset, so authorship can be traced without identifying anyone; the key used to generate the codes is not distributed. Bot accounts keep their login. Email addresses (including GitHub noreply addresses) and personal names in commit messages are redacted with a fixed marker; AI assistant names in Co-authored-by trailers are kept.

[2] A9 · A9 GitSkills sample repository README.md · GitSkills — sample for the MSR 2027 Mining Challenge proposal > Anonymisation · line 129 (BM25 19.3)

Commit author accounts are replaced by keyed one-way codes, identical for the same account throughout, so authorship can be traced across skills and repositories without identifying anyone. The key is not distributed. Accounts of type Bot keep their login.

Email addresses (including GitHub noreply addresses) and personal names in commit messages are masked with a fixed marker; names of AI assistants in Co-authored-by trailers are kept. A scan of the released file confirmed no email addresses remain in commit messages or repository descriptions.

File contents are reproduced byte-for-byte as published on GitHub, so every content row still verifies against its git blob hash.

[3] A4 · A4 GitSkills Zenodo record.md · Description (from the record) · line 44 (BM25 18.7)

Commit author accounts are replaced by keyed one-way codes, identical for the same account throughout the dataset, so authorship can be traced without identifying anyone; the key used to generate the codes is not distributed. Bot accounts keep their login. Email addresses and personal names in commit messages are redacted.

NB-Q13 (Ethics and security; scope: all)¶

Question. What license covers the GitSkills metadata and the skill content text, what does that mean for quoting skill text in a report, and is it safe to run the scripts bundled with skills?

Status. match. Question terms found in the passages: 10 of 14. Terms no returned passage contains: cover, mean, safe, bundled.

Sources cited. A6, G4

[1] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > License · line 306 (BM25 22.4)

The dataset's collected metadata and aggregation are released under CC-BY-4.0. The content fields reproduce file text from public GitHub repositories and remain subject to the license of their origin repository; consult repos.license (and the origin repository itself) before reusing specific file contents.

[2] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > Quick Look · line 237 (BM25 21.0)

Config Rows Content

artifacts 3,797,117 One row per discovered SKILL.md occurrence; full text, front matter, and commit history for representatives

artifact_siblings 7,264,865 Scripts and reference files stored alongside a representative skill

repos 282,200 Repository metadata: stars, language, fork status, license, dates

mining_runs 7 Provenance log of each collection run

[3] G4 · G4 Threats risks and draft report.md · Threats to validity > Ethics and responsible use · line 89 (BM25 19.1)

ID Threat Mitigation Status

X1 Deanonymization of GitSkills author codes Never attempt re-identification; aggregate reporting; no repository or account names in results/ Active: P-01, mitigated

X2 License compliance when quoting skill content in the report Quote minimally and only as needed for false-positive discussion; no runnable payloads; respect the repository license Active: P-01, mitigation planned

X3 AI-generated analysis or text presented without verification Every AI-assisted artifact is logged in ai-use-log.md and reviewed by a member other than the author before merge Active: P-01, mitigation planned

NB-Q14 (Not established; scope: gitskills)¶

Question. Do the sources define what counts as a "script" for has_scripts, or what makes a skill instruction risky or malicious?

Status. match. Question terms found in the passages: 6 of 10. Terms no returned passage contains: define, make, risky, malicious.

Sources cited. A6, A2

[1] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > Dataset Structure > artifact_siblings · line 257 (BM25 12.1)

Files stored alongside a representative skill: path, entry type (file or dir), size, and the text of files under a size cap. Whether a skill bundles scripts or reference material is recorded per skill in artifacts (has_scripts, has_references), not per individual sibling file.

[2] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 3 · line 286 (BM25 10.2)

Initialize the frontend repo using ‘scripts/init- artifact.sh‘

Develop your artifact by editing the generated code

Bundle all code into a single HTML file using ‘scripts/ bundle-artifact.sh‘

Display artifact to user

(Optional) Test the artifact [...] Figure 2. Opening of the SKILL.md of Anthropic’s web-artifacts-builder skill, reproduced verbatim and truncated. Source: https://github.com/anthropics/skills. The YAML front matter carries the name and the description that the agent matches against the task when deciding whether to load the skill; the Markdown body holds the instructions, here referencing bundled executable scripts. sample of the others; the file’s own history dates the skill’s addition. Folder listings are missing for 1,544 representatives and commit history for 1,166 of those in standard locations.

[3] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 2 · line 116 (BM25 10.1)

Table I OVERVIEW OF THEGITSKILLS DATASET. Table # Records Content Coreartifacts3,797,117 One row per discovered file: repository, path, exact basename, location class, content hash, and representative flag; for representatives, also the full text, parsed front matter, and body size. repos282,200 Repository metadata: owner, star count, primary language, fork status, creation date, and last-push date. Compositionartifact_siblings7,264,865 Files stored alongside a representative skill: path, entry type (file or directory), size, and the text of files under a size cap. Whether a skill bundles scripts or reference material is recorded per skill in artifacts (has_scripts, has_references). History (columns inartifacts) 458,548 First and last commit dates of the SKILL.md file, their author accounts (anonymized; user or

NB-Q15 (Not established; scope: gitskills)¶

Question. Do the sources establish that a copy of a skill in a repository is actually loaded or run by an agent there, so that a copy count measures use?

Status. match. Question terms found in the passages: 7 of 11. Terms no returned passage contains: establish, loaded, measure, use.

Sources cited. A2, A6

[1] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 2 · line 186 (BM25 12.4)

b) Through which mechanisms do skills move between re- positories, such as direct addition, catalogs, or scaffolding tools? c) Do skill copies follow the genealogy patterns known from code clones, such as consistent and inconsistent propagation of changes? 4) Software metrics for natural-language instructions. a) Which established metrics, such as size, churn, age, clone coverage, and readability, have meaningful equivalents for skills, and how do their distributions compare with those of source code? b) Can observable indicators of skill quality be defined and compared with proxies such as copy count and subsequent edits? c) Which properties of the description, the text the agent matches against when deciding whether to load the skill, are associated with reuse and maintenance?

[2] A6 · A6 GitSkills dataset card.md · GitSkills: A Dataset of Agent Skills on GitHub > Dataset Structure > mining_runs · line 269 (BM25 12.2)

Query, start/end timestamps, and result count for each collection run.

[3] A2 · A2 GitSkills preprint arXiv 2608.10906v3.md · Page 2 · line 127 (BM25 11.7)

bot), and commit count, for standard locations and a size-stratified sample of the rest. Collectionmining_runs7 Query, start and end timestamps, and search-result count for each collection run; four runs that crashed were closed retroactively with a zero count. GitHub public repositories 1. Discovery search for files matching SKILL.md; partition by file size to pass the 1,000-result limit; classify each result by location retain every result 2. Deduplication group files by content hash; select one representative per group, preferring a standard location retain all copies 3. Enrichment for representatives:content and front matter; folder con- tents; repository metadata; sampled SKILL.md commit history Dataset (SQLite) artifacts artifact_siblings repos mining_runs

Column	Meaning
`repo_full_name`, `path`	where the file lives (primary key)
`filename`	exact basename as returned by code search (case preserved)
`location_class`	`canonical` (`.claude/skills/<name>/SKILL.md`), `skills-dir` (under a `skills/` directory), or `other`
`file_sha`	git blob hash of the content; identical hash = identical bytes
`discovered_at`	when the collection run recorded the file
`dedup_primary`	1 for the one representative of each distinct content
`content`	full text (representatives only)
`content_fetched`, `content_sha_ok`	retrieval bookkeeping; `content_sha_ok` = 1 when the downloaded bytes reproduce `file_sha`, 2 when repaired via the blob API

Config	Rows	Content
`artifacts`	3,797,117	One row per discovered `SKILL.md` occurrence; full text, front matter, and commit history for representatives
`artifact_siblings`	7,264,865	Scripts and reference files stored alongside a representative skill
`repos`	282,200	Repository metadata: stars, language, fork status, license, dates
`mining_runs`	7	Provenance log of each collection run

ID	Threat	Mitigation	Status
X1	Deanonymization of GitSkills author codes	Never attempt re-identification; aggregate reporting; no repository or account names in `results/`	Active: P-01, mitigated
X2	License compliance when quoting skill content in the report	Quote minimally and only as needed for false-positive discussion; no runnable payloads; respect the repository license	Active: P-01, mitigation planned
X3	AI-generated analysis or text presented without verification	Every AI-assisted artifact is logged in `ai-use-log.md` and reviewed by a member other than the author before merge	Active: P-01, mitigation planned