Skip to content

Facilitated elicitation: repository search notebook interview

Assignment Phase B, cross-check. The same 18 questions that were put to the Google Notebook (elicitation/google-notebook-interview.md, the primary record) were also put to the repository's own search notebook, which answers only by quoting the passages that best match the question, each with a locator. It is a bounded domain-information source: not an oracle, and not a substitute for the dataset.

1. Session record

Item Value
Notebook Search notebook over the pinned source pack: notebooks/01_elicitation_notebook.ipynb, code in src/msr_pipeline/elicitation.py, tests in tests/test_elicitation.py. It quotes passages; it writes no text of its own
Source pack Built 2026-09-18 18:47 UTC by scripts/build_notebook_sources.py from repository commit 6b40b7e (this branch's uncommitted changes touched only pyproject.toml among tier-G files). 24 files, 1,755 passages: A 336, G 926, P 493
Tier A versions MSR 2027 challenge page retrieved 2026-09-18; GitSkills preprint arXiv 2608.10906v3; SpecMine preprint 2608.25202v3; GitSkills Zenodo record 21875637 (v1.0.0); SpecMine Zenodo record 22102780 (v1.0, concept DOI 22102779); dataset cards at Hugging Face revisions ebab174 and ca682c6; SpecMine mirror commit 9fbb615; GitSkills sample commit fff3df9
Tier P Included: one member's personal build method (Jerad Dunne's PPF framework). Used for process questions only
Interview date 2026-09-18
Full transcript elicitation/notebook-transcript.md, regenerated with python -m msr_pipeline elicit
Interviewers Jerad Dunne, with Claude Code (Claude Opus 5) running the notebook and drafting the interpretations. Disclosed in ai-use-log.md
Review Team interpretations are drafts until a second member reviews this file in the pull request

2. Protocol

  1. Scope before asking. Each question names the tiers it may be answered from. A GitSkills question uses scope gitskills (A1, A2, A4, A6, A9), so it cannot be answered from the SpecMine data dictionary or the group's own notes. Process questions use group or process.
  2. Word for word, in order. Questions are asked exactly as written in elicitation/questions.yaml. When an answer was incomplete, the team asked a follow-up with its own ID (NB-Q05a, NB-Q07a, NB-Q11a). Three follow-ups were added after the first pass.
  3. What the notebook returns. The three best-matching passages (at most two per file), quoted verbatim with a locator source · file · section or page · line, plus a mechanical status: "match" means at least half the question's terms occur in the passages. It does not mean the passages answer the question. The terms no passage contains are listed, and they often point at what the sources do not establish.
  4. Verify. Open the cited source at the locator, confirm the passage is there and says what the answer says, decide whether it answers the question, look for sources that disagree, and for claims about the data run the read-only checks V01 to V18 (python scripts/check_interview_claims.py). Section 4 carries the result for every answer.

3. Interview record

The columns the assignment requires. "Notebook answer" quotes the returned passages; "…" marks a cut. Full passages are in elicitation/notebook-transcript.md.

Question Notebook answer Source cited by notebook Team interpretation Follow-up or uncertainty
NB-Q01 What exactly does one row of the GitSkills artifacts table represent, and how should a study count distinct skills separately from copies? Which column marks the representative copy, and is the representative the original source of the content? Match, 13 of 18 terms. [1] "One row per discovered file: repository, path, exact basename, location class, and content hash. For the representative of each distinct content: full text, parsed front matter …" [2] "file_sha | git blob hash of the content; identical hash = identical bytes … dedup_primary | 1 for the one representative of each distinct content" [3] "We do not assume that the representative is the original source. All copies remain in the dataset …" A6 · Dataset Structure > artifacts · line 248; A9 · Schema > artifacts · line 75; A2 · Page 3 · line 252 The unit of analysis is one distinct content (file_sha), represented by its dedup_primary = 1 row; copies are the outcome (RR-01, DR-02). The representative is chosen by location, not age, so it is not the origin. None. 16 non-representative rows also carry text (V03), which does not affect counting by dedup_primary.
NB-Q02 What population does GitSkills cover, and what does GitHub code search leave out? Does the dataset include files that are not Agent Skills in the October 2025 sense? Match, 12 of 15 terms. [1] "The dataset covers public repositories only and should be read as a lower bound on the population: GitHub code search indexes only default branches, files under 384 KB, recently active repositories with fewer than 500,000 files, and forks only when they have more stars than the parent." A6 · Collection Methodology · line 290; A4 · Description · line 28; A6 · introduction · line 179 The population is SKILL.md files discoverable by code search on public default branches in July 2026, a lower bound (DR-03; threats E1, P8). The second half was not answered. The team found it at A2 page 3: case variants such as skill.md (54,289 files, 1.4%) and pre-format files are retained, which is why the front-matter-valid subset exists.
NB-Q03 How is the official GitSkills sample drawn, how does it relate to the full release, and which exact identifiers should a paper cite for the sample and for the full dataset? Match, 7 of 12 terms. [1] "The sample is drawn deterministically: the 13,000 distinct contents with the lowest content hashes … together with every row the full dataset holds about them … Queries written against the sample run unchanged on the full dataset." A9 · Sample vs full dataset · line 59; A4 · Description · line 48; A6 · Citation · line 314 The sampling rule is checkable, and DR-01 takes the identifiers from A1, A4, and A9: Zenodo version 1.0.0 (doi:10.5281/zenodo.21875637), sample commit fff3df9, database SHA-256 in data/samples/MANIFEST.json. The identifiers were not in the returned passages. Sources also give the file as 44 GB (A1) and 41 GB (A9); it is 44,388,249,600 bytes, so cite bytes. data/README.md mislabelled the SpecMine release; corrected in this change.
NB-Q04 How do artifact_siblings rows join to artifacts, which sibling rows carry text, and for which copies of a skill is folder composition recorded? Match, 9 of 12 terms. [1] "Files stored alongside a representative skill: path, entry type (file or dir), size, and the text of files under a size cap. Whether a skill bundles scripts or reference material is recorded per skill in artifacts (has_scripts, has_references)…" [2] "repo_full_name, artifact_path (joins to artifacts), … content (text files up to 100 KB), content_fetched, skipped_reason (binary, oversize, or folder above the listing cap)." A6 · Dataset Structure > artifact_siblings · line 257; A9 · Schema > artifact_siblings · line 102; A4 · Description · line 36 Join on (repo_full_name, artifact_path); only the representative's folder is recorded, so script signals describe representatives (FR-04). content_fetched = 2 marks 16,697 of the 16,768 text-less rows and is undocumented; skipped_reason is never set (V07); 13 rows join a copy (V08). The scanner must count unread files (DR-04).
NB-Q05 What do content_sha_ok, composition_truncated, and history_fetched mean, and which values mark missing or unrecoverable data? Match, 9 of 16 terms. [1] "content_fetched, content_sha_ok | retrieval bookkeeping; content_sha_ok = 1 when the downloaded bytes reproduce file_sha, 2 when repaired via the blob API" [2] "composition_fetched, composition_truncated | bookkeeping; composition_truncated = 1 flags folders larger than the listing cap". Terms not found: history_fetched, missing, unrecoverable. A9 · Schema > artifacts · lines 75 and 85; A6 · YAML header · line 63 Two of three columns are defined here; history_fetched is defined in A9 Known limitations. 1,755 representatives (13.5%) have truncated listings and hold far more files (mean 8.5 against 2.9), so script signals are a lower bound for them (DR-04, FR-04). No passage names a value for unrecoverable content, which prompted NB-Q05a.
NB-Q05a Which value of content_sha_ok records a representative whose file changed on GitHub before it was fetched and whose original content could not be recovered? Match, 12 of 14 terms. [2] "For 42 representatives the file changed on GitHub between discovery and fetch and the original could not be recovered; content_sha_ok records this." A9 · Schema > artifacts · line 75; A2 · Page 3 · line 303 (Appendix A); A2 · Page 3 · line 252 The preprint says the column records the case, but no source names the value, and only 1 and 2 occur in the sample (V09), so P-01 is unaffected. A full-dataset pass must exclude and count anything else (DR-04). Not established: the value itself. Inference only: 0 or null.
NB-Q06 What does first_commit_at represent for a renamed file, and which rows have commit history at all? Match, 7 of 8 terms. [1] "Commit history follows the file's current path: for a renamed file, first_commit_at dates the rename and commit_count covers the current path only. History is collected for every skill in a standard location plus a size-stratified sample of the rest; history_fetched marks which rows." A9 · Known limitations · line 178; A9 · Schema > artifacts · line 85; A1 · GitSkills · line 29 first_commit_at is a lower bound on age. Only 3,010 of 13,000 representatives (23%) have history, nearly every canonical skill but about 11% of the others (V11), so ordered pairs over-represent canonical skills (FR-06; threats P5, I3, I4). Report ordered-pair counts by location class. Inference to flag: "standard location" appears to mean the canonical .claude/skills/ path.
NB-Q07 Our question asks whether skills with risk-relevant capabilities reach more repositories through verbatim copying, and whether modified variants add or remove those capabilities. Can the GitSkills sample support each part, and what in the sampling or history coverage limits it? Weak match, 11 of 23 terms. [1] "Reuse without a package manager — 50.5% of collected files are verbatim copies… Maintenance and trust — … whether modified copies introduce command execution or network access…" Terms not found: risk, capability, variant, sample, sampling, limit. A6 · Intended Uses · line 219; A1 · Challenges · line 65; A2 · Page 1 · line 27 The sources confirm the topic is one the dataset authors invite (A1 challenge 5; A2 III.5b), but they do not say whether the sample can answer it. That argument is the team's. Answered through NB-Q07a and the data checks V12 and V13.
NB-Q07a When a distinct content is in the sample, does the sample keep every copy of it across repositories, and are variants of the same skill kept together? Match, 8 of 11 terms. [1] "… together with every row the full dataset holds about them — all copies across repositories … every content group is complete and has exactly one representative carrying the text." [3] "Every occurrence is retained with its repository and path…" Term not found: variant. A9 · Sample vs full dataset · line 59; A9 · Two views of the full dataset · line 169; A2 · Page 1 · line 66 Copies: yes, so RQ2 is fully supported (RR-02). Variants: not established by any source, and by construction no, since selection is by content hash; RQ3 stays descriptive (RR-03). Complete variant families need the Sprint 1 decision on a streamed full-dataset pass (S1-13).
NB-Q08 Do the sources describe any way to link modified (near-duplicate) copies of the same skill, or only byte-identical copies? Match, 8 of 11 terms. [1] "… the dataset stores the text once and links all copies to it through the content hash. Identical hashes mean that the files contain exactly the same bytes." [2] "b) Do modified copies of widely reused skills introduce command execution or network access absent from the original…?" A2 · Page 1 · line 93 (Section II); A2 · Page 2 · line 203 (Section III.5); A9 · Anonymisation · line 129 (not relevant) Only byte-identical copies are linked; near-duplicate linking appears only as an open research question. P-01's lineage construct is therefore the team's own and must be swept and validated (FR-06, VR-03). None.
NB-Q09 What loading approach do the sources recommend for the sample and for the full dataset, and what example queries do they give? Match, 6 of 10 terms. [2] an example query: "SELECT r.language, COUNT(*) AS skills FROM artifacts a JOIN repos r ON r.full_name = a.repo_full_name GROUP BY r.language …" Terms not found: loading, recommend. A9 · Sample vs full dataset · line 59; A9 · Example queries · line 225; A1 · Call for Mining Challenge Proposals · line 273 (not relevant) The team found the loading instructions directly: one SQLite file, "No server, no credentials; any SQLite client works" (A9 Quick start); the full dataset as SQLite or the Parquet mirror (A6, A1). Any full pass should stream Parquet (NFR-04). The returned passages were correct but incomplete.
NB-Q10 Under Group 9's validation protocol, how is a rule match labelled, who rates it, which agreement measure is reported, and what is the blindness rule? Match, 10 of 11 terms. [1] "Jerad Dunne (rater id jd) is the primary rater. At least one teammate labels round 1 independently … Blindness rule. The primary rater's label files are not committed until the second rater's labels for the same kind are committed … Inter-rater Cohen's kappa is the primary reliability measure". Term not found: match. G6 · Team Charter > 9. Validation independence · line 306; G1 · Validation design (group) · line 285; G6 · Work sign-up · line 657 Raters, measure, and blindness rule confirmed; these become VR-01 to VR-03. The label values were not returned; the team read them in the guideline: risky in context, benign in context, not present. Second rater unassigned (issue #53).
NB-Q11 According to Group 9's process documents, what has to happen before a change to a detection rule or to the research question reaches main? Match, 6 of 12 terms, but off target. [1] "Minimum evidence: 'Threat model, detection rules, annotated examples, false-positive discussion, and a safe reporting artifact …'" [2] and [3] the statement of the main research question. Terms not found: change, process, document. G1 · section 8 · line 262; G1 · P-01 proposal section 2 · line 317; G4 · report draft · line 223 The notebook failed: common words ("research question", "detection rule") pulled passages that mention them but say nothing about change control. This is the weakness of a search notebook, and the reason the protocol allows follow-ups. Answered through NB-Q11a and, independently, PROJECT_PLAN.md section 11 and CONTRIBUTING.md.
NB-Q11a What are the change management rules for research changes and process changes, and what review does every pull request need before it merges? Match, 8 of 10 terms. [1] "Research changes: a decision issue, discussion, an ADR, and a line in the retrospective. Process changes: a kaizen issue with a measurable expected effect… Charter changes: a pull request approved by all other members…" [2] "5. Request a review from a member who did not write the change… 6. Squash-merge after one approval and green CI." G6 · Project Plan > 11. Change management · line 184; G7 · Contributing > Branch, commit, pull request · line 34; G7 · Getting started > Step 5 · line 130 A change to a rule or to the research question is a research change: decision issue, discussion, ADR, retrospective line, then a reviewed pull request with green CI. The same governs RESEARCH_SPEC.md (NFR-05). The process sources also record one deviation: pull request #49 merged under an admin bypass with zero reviews (docs/sprints/sprint-1/planning.md).
NB-Q12 What anonymisation was applied to commit authors and commit messages, what was left unanonymised, and what must a study never do with author codes or personal data? Match, 6 of 12 terms. [1] "Commit author accounts are replaced by keyed one-way codes … Bot accounts keep their login. Email addresses … and personal names in commit messages are redacted with a fixed marker; AI assistant names in Co-authored-by trailers are kept." [2] the same from A9, plus "A scan of the released file confirmed no email addresses remain…". Terms not found: never, data. A6 · Anonymisation · line 297; A9 · Anonymisation · line 129; A4 · Description · line 44 Left unanonymised: bot logins, AI-assistant names, repository and owner names, and the skill text itself. ER-02: never reverse or link author codes; never write codes, logins, or repository names to an output. The prohibition was not returned; it is in A1 FAQ Q1 ("PII is strictly prohibited"). Source conflict: A2 page 3 says names in trailer lines became keyed codes, while A6 and A9 say a fixed marker. The data settles it (V15: 130 markers, no codes); the report's data section should say so.
NB-Q13 What license covers the GitSkills metadata and the skill content text, what does that mean for quoting skill text in a report, and is it safe to run the scripts bundled with skills? Match, 10 of 14 terms. [1] "The dataset's collected metadata and aggregation are released under CC-BY-4.0. The content fields reproduce file text from public GitHub repositories and remain subject to the license of their origin repository; consult repos.license…" [3] "X2 | License compliance when quoting skill content in the report | Quote minimally…". Terms not found: safe, bundled. A6 · License · line 306; A6 · Quick Look · line 237; G4 · Ethics and responsible use · line 89 Metadata is reusable with attribution; skill text is not assumed reusable, and 59.5% of representatives have no usable license (V17), so the report quotes sparingly and describes the rest (ER-01). The safety half is not established by tier A; the rule is the course's and the group's: "Students must not execute untrusted scripts" (RESEARCH_QUESTION.md section 8) and data/README.md (ER-03).
NB-Q14 Do the sources define what counts as a "script" for has_scripts, or what makes a skill instruction risky or malicious? Match, 6 of 10 terms. [1] "Whether a skill bundles scripts or reference material is recorded per skill in artifacts (has_scripts, has_references), not per individual sibling file." [2] the preprint's Figure 2 caption: "…the Markdown body holds the instructions, here referencing bundled executable scripts." Terms not found: define, risky, malicious. A6 · Dataset Structure > artifact_siblings · line 257; A2 · Page 3 · line 286; A2 · Page 2 · line 116 Not established. Both "script" and "risky" are the team's constructs (skill_risk.SCRIPT_EXTENSIONS; docs/research/THREAT_MODEL.md; the rule file), so they must be stated as ours and validated (FR-02, RR-01, VR-01). V18: 57 representatives with has_scripts = 0 contain a file with a P-01 script extension, so the two definitions are close but not identical.
NB-Q15 Do the sources establish that a copy of a skill in a repository is actually loaded or run by an agent there, so that a copy count measures use? Match, 7 of 11 terms. [1] "b) Can observable indicators of skill quality be defined and compared with proxies such as copy count and subsequent edits? c) Which properties of the description, the text the agent matches against when deciding whether to load the skill…" Terms not found: establish, loaded, measure, use. A2 · Page 2 · line 186 (Section III.4); A6 · mining_runs · line 269 (not relevant); A2 · Page 2 · line 127 Not established. The sources describe how an agent would load a skill and treat copy count as a proxy to study, not as a measure of use. "Reach" is the spread of identical content (RR-02); the report must not say risky skills are "used more". None.

4. Verification

Question Result Checked against
NB-Q01 Verified A2 page 3 (matches the PDF), A9 Schema, A6; data V01, V02, V03
NB-Q02 Partly verified Part 1 at A6 and A2 App. A; part 2 unanswered, found at A2 page 3 and A9; data V04, V05
NB-Q03 Partly verified Sampling rule at A9 and data V06; identifiers found independently in A1, A4, A9
NB-Q04 Partly verified, and contradicted by the data Join key and text cap at A9 and A6; contradicted by V07 and V08
NB-Q05 Partly verified Definitions at A9 lines 75 and 85; data V09, V10
NB-Q05a Verified, value not established A2 page 3 matches the PDF; A9 documents only 1 and 2; data V09
NB-Q06 Verified A9 Known limitations, A2 Table I and App. A; data V11
NB-Q07 Partly verified Research-direction fit at A1 Challenges and A2 page 2; feasibility not answered, investigated via NB-Q07a, V12, V13
NB-Q07a Partly verified Copies at A9 and data V02, V12; variants not established by any source, V13
NB-Q08 Verified A2 pages 1 and 2 against the PDF; no tier-A source describes near-duplicate linking
NB-Q09 Partly verified [1] and [2] at A9; loading approach found independently at A9 Quick start, A6 Overview, A1
NB-Q10 Verified TEAM_CHARTER.md section 9, RESEARCH_QUESTION.md section 8; label values in the guideline
NB-Q11 Unverified The cited passages exist but do not bear on the question; investigated via NB-Q11a, PROJECT_PLAN.md section 11, CONTRIBUTING.md
NB-Q11a Verified PROJECT_PLAN.md section 11 and CONTRIBUTING.md at commit 6b40b7e
NB-Q12 Verified, with a source conflict resolved by data A6, A9, A4; conflict with A2 App. A settled by V14, V15, V16
NB-Q13 Partly verified License at A6 and A4; safety not established by tier A; rule in group sources; data V17
NB-Q14 Verified as not established No definition in A1, A2, A4, A6, A9; data V18
NB-Q15 Verified as not established A2 abstract, Section I, Section III.4; A1 GitSkills section

Totals: 9 verified, 8 partly verified, 1 unverified (NB-Q11, where the notebook returned irrelevant passages and the team investigated independently).

Lessons about this notebook. It cannot invent facts, and every answer came back with a locator, so no answer needed an "unverified" label for lack of a citation. Its weakness is relevance: common words can pull passages that mention the terms without answering (NB-Q11), and it misses answers phrased differently from the question (NB-Q02 part 2, NB-Q09). Scoping every question to a tier, reading the "terms not found" line, and asking follow-ups handled all of these cases.

Contradictions found across both interviews: the documented skipped_reason is never set in the sample (V07); 13 bundled-file rows join a copy rather than a representative (V08); 16 copies carry text (V03); and the preprint's description of co-author anonymisation does not match the data or the other sources (V15).

Follow-ups. Item 1 was done in the same change as this record; items 2 to 4 are proposed issues, not yet opened:

  1. Done: data/README.md now labels the SpecMine Zenodo release version 1.0 and the GitHub sample v1.1, and names the arXiv v3 preprints (RESEARCH_SPEC.md DR-01). report/references.bib still needs the arXiv version.
  2. Document artifact_siblings.content_fetched = 2, the empty skipped_reason, and composition_truncated in DATA_DICTIONARY.md, and add the truncated-listing subset to results/scripts_summary.csv.
  3. Report the preprint's anonymisation description against V15 in the report's data section.
  4. Decide whether to add an exact-basename (SKILL.md) sensitivity filter alongside front-matter validity.

What the interview changed in the requirements (RESEARCH_SPEC.md): DR-01 version identity; DR-04 and FR-04 unread inputs; RR-01 and RR-02 the representative is not the origin and reach is not use; RR-03 and FR-06 split variant families and ordered pairs; FR-02, VR-01 and VR-03 the team's own constructs; ER-01 and ER-02 licensing and personal data.