Facilitated elicitation: Google Notebook interview¶
Assignment Phase B. Group 9 interviewed a Google Notebook (Gemini Notebook, formerly NotebookLM) built for this study, treating it as a bounded domain-information source: not an oracle, and not a substitute for the dataset. Every important answer was checked against the source the notebook cited and, where it claims something about the data, against the GitSkills sample.
This is the primary Phase B record. The same questions were also put to the repository's own search notebook (elicitation/notebook-interview.md); section 5 compares the two.
1. Session record¶
| Item | Value |
|---|---|
| Notebook | "Group 9 · P-01 elicitation notebook (CSC 580 RE)", Google Gemini Notebook, account jeraddu@umich.edu |
| Link | https://notebook.google.com/notebook/f6dac9ee-331e-498a-afe6-0b7b824602ef |
| Sources | 24 files uploaded from the pinned pack: 11 authoritative (A1 to A9, preprints as original PDFs), 8 group (G1 to G8), 5 practitioner (P1 to P5). Register and hashes: elicitation/notebook/SOURCE_REGISTER.md, SOURCES_MANIFEST.json; rebuild with python scripts/build_notebook_sources.py |
| Chat configuration | Custom instructions set in Configure chat, kept in elicitation/google-notebook/notebook-instructions.txt: answer only from the sources, cite every factual sentence, prefer tier A for dataset facts, report disagreements, label inferences, say "Not established by the sources", never advise running dataset scripts or deanonymising authors |
| Interview date | 2026-09-19 |
| Questions | The 15 prepared questions in elicitation/questions.yaml plus the 3 follow-ups, asked word for word, in order, in one chat |
| Raw answers | elicitation/google-notebook/answers.json, each with a SHA-256 prefix taken from the page, so the text here is provably the notebook's own |
| Interviewers | Jerad Dunne, with Claude Code (Claude Opus 5) driving the browser and drafting the interpretations. Disclosed in ai-use-log.md |
| Review | Team interpretations are drafts until a second member reviews this file in the pull request |
2. Protocol¶
- One chat, in order, word for word. Questions were never reworded. Where an answer was incomplete, the prepared follow-up (
NB-Q05a,NB-Q07a,NB-Q11a) was asked as a new question. - Capture. Each answer was taken from the page with its citations. The notebook cites in two ways, and the difference matters:
- Grounded citations (numbered chips that link to a source passage): only NB-Q01 and NB-Q02.
- Text citations (the model writes "[A2, A9]" and a Sources list): every other answer. A text citation names a source but does not point at a passage, so the team had to locate the passage itself.
- Verify. For each answer: open each cited source, find the passage, and decide whether it supports the claim; look for a source that disagrees; and for claims about the data, run the read-only checks in
elicitation/verification-checks.md(V01 to V18,python scripts/check_interview_claims.py). Results are Verified, Partly verified, Contradicted, or Unverified (no citation, or the citation does not bear on the claim). Section 4 carries the result for every answer. - Interpret. Say what the answer means for P-01 and which
RESEARCH_SPEC.mdrequirement it informs.
3. Interview record¶
The columns the assignment requires. Answers are quoted from answers.json; long answers are excerpted at the sentences that carry the claim, and "…" marks a cut. Two conventions inside a quotation: the notebook's inline citation markers ([8], [A2, Section II]) are left out, because the next column lists them; and the notebook's own double quotes appear as single quotes. Nothing else is changed. The exact text of every answer, with its markers, is in elicitation/google-notebook/answers.json.
| Question | Notebook answer | Source cited by notebook | Team interpretation | Follow-up or uncertainty |
|---|---|---|---|---|
NB-Q01 What exactly does one row of the GitSkills artifacts table represent, and how should a study count distinct skills separately from copies? Which column marks the representative copy, and is the representative the original source of the content? |
"one row of the artifacts table represents a single discovered SKILL.md file occurrence at a specific repository and file path… To count distinct skills separately from verbatim copies, a study can count unique content hashes using the file_sha column or filter for representative rows where dedup_primary = 1… The representative is not assumed to be the original source of the content, as the dataset authors explicitly select one representative per content hash for enrichment using deterministic tie-breaking rules (preferring standard paths like .claude/skills/)." | Grounded: A2 (Abstract, Section II, Table I, Appendix A), A6 (artifacts config features), A1 (Section "GitSkills"), A9 (Descriptive statistics, Example queries), G2 (Part A.1, data/README.md, FINDINGS_LOG.md) | Fixes the unit of analysis in RESEARCH_QUESTION.md section 5: one distinct content (file_sha), represented by its dedup_primary = 1 row, with copies as an outcome (RR-01, DR-02). Because the representative is not the origin, P-01 must never present the representative's repository as the source of a copied skill. |
None. The 16 copies that also carry text (V03) are a documentation deviation that does not affect counting. |
| NB-Q02 What population does GitSkills cover, and what does GitHub code search leave out? Does the dataset include files that are not Agent Skills in the October 2025 sense? | "…it leaves out private repositories, non-default branches, files 384 KB or larger, recently active repositories containing 500,000 or more files, and forks unless they have more stars than their parent repository… The dataset does include files that are not agent skills in the October 2025 open specification sense… case variants such as skill.md (54,289 files, 1.4% of GitSkills Zenodo 1.0.0), including files with first commits dating back to 2014… only 1,625,701 distinct contents (86.6%) carry valid YAML front matter". | Grounded: A1 (Section "GitSkills"), A2 (Sections I and II, Appendix A), A4 (Description), A9 (Descriptive statistics, Known limitations, Before the format), A6 | The population is a lower bound (DR-03; threats E1, P8), and pre-format files sit inside it, which is why P-01 keeps a front-matter-valid sensitivity subset (RR-04). | One clause is wrong: the sources say code search indexes "recently active repositories with fewer than 500,000 files", so repositories that are not recently active are excluded too. Consider an exact-basename (SKILL.md) sensitivity filter as well. |
| NB-Q03 How is the official GitSkills sample drawn, how does it relate to the full release, and which exact identifiers should a paper cite for the sample and for the full dataset? | "…drawn deterministically by selecting the 13,000 distinct contents with the lowest content hashes along with all their associated occurrences… allowing SQL queries written for the sample to run unchanged on the full dataset… cite… the GitSkills Zenodo 1.0.0 release (Version DOI 10.5281/zenodo.21875637, Concept DOI 10.5281/zenodo.21875636), and the Hugging Face Parquet mirror… To cite the official sample… specify commit fff3df9 alongside the database file agent_skills_sample.db". | Text only: A1 (Section "GitSkills", "Submission"), A2 (Section IV, Appendix A), A4 (Metadata, Description), A6 (Header, Citation), A9 (Header, Descriptive statistics), G1, G2 | Exactly what DR-01 requires: the Zenodo version DOI, the sample commit, and the database hash. It also separates version DOI from concept DOI, the distinction that made data/README.md mislabel the SpecMine release. |
It did not mention that sources give the full file as both 44 GB (A1) and 41 GB (A9); the Zenodo file is 44,388,249,600 bytes, which is both. Cite bytes. |
NB-Q04 How do artifact_siblings rows join to artifacts, which sibling rows carry text, and for which copies of a skill is folder composition recorded? |
"Rows in artifact_siblings join to artifacts using a composite key matching repository name and path… Folder composition is recorded exclusively for the single primary representative copy of each distinct skill (dedup_primary = 1)… full text is carried in the content column only for text files that fall under a size cap of up to 100 KB… Sibling rows for binary files, files exceeding the 100 KB size cap, or entries in folders above the listing cap do not carry text and record a skipped_reason instead". | Text only: A2 (Section II, Table I, Appendix A), A6 (artifacts and artifact_siblings config features), A9 (artifact_siblings, Example queries), G2 (Part A.1) | The join key and the representative-only rule are what scan_siblings implements (FR-04), so script signals describe representatives, not every copy. |
The last sentence repeats documentation the data contradicts: skipped_reason is never set, and unread files carry an undocumented content_fetched = 2 (V07). The scanner must count files it could not read (DR-04). |
NB-Q05 What do content_sha_ok, composition_truncated, and history_fetched mean, and which values mark missing or unrecoverable data? |
"…a value of 1 indicates that the downloaded file bytes reproduce file_sha and 2 indicates the content was repaired via the GitHub blob API… composition_truncated… 1 flags that a representative skill folder exceeded the folder listing cap… history_fetched… covering 458,548 commit histories in the full GitSkills Zenodo 1.0.0 release and 3,010 in sample commit fff3df9… The specific numerical code for content_sha_ok representing failed or unrecoverable content retrieval is not established by the sources." | Text only: A2 (Table I), A4 (Description), A6 (artifacts config features), A9 (How the data was collected), G2 (Part A.1, FINDINGS_LOG.md) | The three columns are defined, and the notebook volunteered the gap about unrecoverable content. For P-01 the truncation flag is the consequential one: bundled-script signals are a lower bound for those skills (DR-04, FR-04). | It did not quantify truncation. The data does: 1,755 representatives, 13.5%, averaging 8.5 bundled files against 2.9 (V10). It also missed A2 Appendix A on the 42 unrecoverable representatives, which prompted NB-Q05a. |
NB-Q05a Which value of content_sha_ok records a representative whose file changed on GitHub before it was fetched and whose original content could not be recovered? |
"The sources do not define a value of content_sha_ok for a representative whose file changed on GitHub prior to retrieval and could not be recovered… files deleted from GitHub between discovery and content retrieval—comprising 0.9% of distinct contents… were excluded from the dataset entirely rather than retained as incomplete records." Not established: "which value of content_sha_ok (if any exists) records a representative whose file changed on GitHub before it was fetched". | Text only: A6 (artifacts config features), A9 (How the data was collected), G2 (Part A.1) | The value is genuinely undocumented, so any full-dataset pass must exclude and count representatives whose value is neither 1 nor 2 (DR-04). The sample is unaffected. | It answered "no such value is defined" without finding A2 Appendix A, which says the column does record the case but names no value. |
NB-Q06 What does first_commit_at represent for a renamed file, and which rows have commit history at all? |
"…first_commit_at records the timestamp when the file was renamed to its current path rather than its original creation date… Inference: Consequently, first_commit_at provides a lower bound on the true age of renamed skill files… Commit history is collected only for representative skill rows (dedup_primary = 1), specifically covering standard locations such as .claude/skills/ along with a size-stratified sample of other locations… Rows that possess commit history are marked… by history_fetched = 1." | Text only: A2 (Section II, Table I, Appendix A), A4, A6, A9 (Known limitations), G2, G4 (Threats I3, I4) | Dates are a lower bound and exist for only 23% of representatives, so RQ3's ordered pairs over-represent canonical skills (FR-06, RR-03; threats P5, I3, I4). | Report ordered-pair counts by location class (the RR-03 change). The notebook labelled its inference, as the instructions require. |
| NB-Q07 Our question asks whether skills with risk-relevant capabilities reach more repositories through verbatim copying, and whether modified variants add or remove those capabilities. Can the GitSkills sample support each part, and what in the sampling or history coverage limits it? | "The GitSkills sample (commit fff3df9) fully supports evaluating whether skills with risk-relevant capabilities reach more repositories through verbatim copying (RQ2), as it retains all 29,786 file occurrences across 11,841 repositories… For modified variants (RQ3), the sample can identify near-duplicate variant pairs (245 pairs across 141 names…), but it is severely limited… leaving only 10 near-duplicate variant pairs with commit dates for both variants… supports statistical regression and reach models for RQ2, but restricts RQ3 to descriptive case studies." | Text only: A2 (Sections I, II), A9, G1 (Sections 4, 5.3, 6, 7.3, 7.4), G2 (Part A.1, FINDINGS_LOG F-007, F-008, F-011, F-012), G4 (Threats E2, I3, I4, P5) | The feasibility answer P-01 needs, correctly sourced: tier A establishes the sample's structure, and the judgement about RQ2 and RQ3 comes from the group's own analysis (RR-02, RR-03). | Lifting the RQ3 limit needs the Sprint 1 decision on one streamed full-dataset pass (S1-13). |
| NB-Q07a When a distinct content is in the sample, does the sample keep every copy of it across repositories, and are variants of the same skill kept together? | "Yes… the sample retains every verbatim copy of that content across all repositories… However, modified variants of the same skill—which carry different content hashes (file_sha) despite sharing a front-matter name—are not kept together in the sample… variant families are split across the sampling cutoff." | Text only: A2 (Section I, Appendix A), A9 (Descriptive statistics, How the data was collected), G1 (Section 4), G2, G4 (Threats E2, P5) | RQ2 is fully supported on the sample because copy groups are complete; RQ3 stays descriptive (RR-02, RR-03). | None. |
| NB-Q08 Do the sources describe any way to link modified (near-duplicate) copies of the same skill, or only byte-identical copies? | "The official GitSkills dataset natively links only byte-identical copies using the git blob content hash (file_sha)… The authoritative dataset sources do not contain a pre-computed column or schema link for modified near-duplicate files… Group 9's research design introduces a similarity heuristic that pairs distinct contents sharing a front-matter name… and measures word 5-shingle Jaccard similarity… Group 9 explicitly notes that this name-and-similarity pairing is a derived research heuristic rather than a dataset ground truth". | Text only: A2 (Section II), A6, A9 (Example queries), G1 (Sections 5.3, 6, 11), G2 (Part B.3, FINDINGS_LOG F-011), G4 (Threat P4), G8 (Module skill_risk) | Lineage is the team's construct, so it must be defined, swept across thresholds, and validated (FR-06, VR-03), and never attributed to the dataset. | None. |
| NB-Q09 What loading approach do the sources recommend for the sample and for the full dataset, and what example queries do they give? | "For the official GitSkills sample (commit fff3df9, 277 MB SQLite database…), sources recommend executing direct SQL queries, whereas for the full GitSkills Zenodo 1.0.0 dataset (44,388,249,600 bytes… or Hugging Face Parquet mirror), sources recommend using filtered SQL queries or reading Parquet files with pandas, polars, DuckDB, or the Hugging Face datasets library… For SpecMine… without requiring a full load of the 14.7 GB uncompressed MySQL dump". | Text only: A2 (Section IV), A3 (Sections III, V), A4, A6, A7 (Usage), A8a (Quick start), A8c (load.py), A9 (Header, Example queries), G2 | Our loader opens the sample read-only through SQLite (FR-01); any full-dataset pass should stream the Parquet mirror rather than load 44 GB (NFR-04). | "pandas, polars" for GitSkills is not supported: neither word appears in A1, A2, A4, A6, or A9. Polars appears only in the SpecMine preprint. Treat that part as the notebook's extrapolation. |
| NB-Q10 Under Group 9's validation protocol, how is a rule match labelled, who rates it, which agreement measure is reported, and what is the blindness rule? | "…each sampled high-risk rule match is labelled as RISKY…, BENIGN_CONTEXT…, or NOT_PRESENT… performed independently by Jerad Dunne (jd) as the primary rater and at least one teammate—Leticia Aderhold (la), Allie Hodges (ah), or Hina Kramer (hk)… The primary agreement measure… is inter-rater Cohen's kappa alongside percent agreement… second raters never open another rater's label files, and the primary rater's filled label files… are committed… only after the second rater's label file for that kind has been committed." | Text only: G1 (Section 7.5), G3 (validation README, ANNOTATION_GUIDELINE Sections 2.1 and 5), G4 (Threat P6), G6 (ADR-0006, TEAM_CHARTER Section 9), G7 | VR-01 to VR-03 restated correctly, including the three label values. | The second rater is still unassigned (issue #53), so inter-rater kappa cannot be computed yet. |
NB-Q11 According to Group 9's process documents, what has to happen before a change to a detection rule or to the research question reaches main? |
"For any change to reach main, Group 9 requires creating a pull request that receives at least one approving review from a team member other than the author and passes all automated CI checks… To change a detection rule, the author must update rules/skill_risk_rules.yaml, add match and no-match test examples… To change the research question, the team must reach agreement at a ceremony, record a decision row in docs/workspace/DECISION_LOG.md, and write an Architectural Decision Record (ADR)". | Text only: G1, G3, G5, G6 (TEAM_CHARTER Sections 6 and 12, DECISION_LOG), G7 (CONTRIBUTING, decision issue form) | Changes to RESEARCH_SPEC.md requirements follow the research-change rule: decision issue, ADR for scope changes, reviewed pull request with green CI (NFR-05). |
It did not mention the recorded deviation: pull request #49 merged under an admin bypass with zero reviews (docs/sprints/sprint-1/planning.md). |
| NB-Q11a What are the change management rules for research changes and process changes, and what review does every pull request need before it merges? | "…research changes… team agreement recorded in docs/workspace/DECISION_LOG.md, an Architectural Decision Record (ADR)… and an explicit entry in that sprint's retrospective.md… Process changes are tracked as kaizen issues… logged in docs/lean-six-sigma/KAIZEN_BACKLOG.md… team charter modifications require approval from all other team members in a pull request… Before any pull request can be merged into main, it requires a green automated CI build and exactly one approving review from a team member other than the author." | Text only: G1, G3, G5 (Sections 1, 2, 4, 6, 7, KAIZEN_BACKLOG), G6 (TEAM_CHARTER Sections 6, 12, 13, DECISION_LOG, ADR-0006), G7 | Confirms NFR-05 and adds the charter rule: a charter change needs every other member's approval. | "Exactly one" is the minimum, not a cap (project.yml: min_reviewers: 1). Same omission of the admin bypass. |
| NB-Q12 What anonymisation was applied to commit authors and commit messages, what was left unanonymised, and what must a study never do with author codes or personal data? | "…commit author accounts were anonymised by replacing them with keyed one-way codes… In commit messages, personal names and email addresses were redacted… bot accounts retained their logins, AI assistant names in Co-authored-by trailers were preserved, and repository owner logins remained unanonymised… Studies are strictly prohibited from attempting to deanonymise author codes… researchers must never publish personal data, include individual account/repository names in published outputs, or paste dataset content containing personal data into external AI tools". | Text only: A2 (Section IV, Appendix A), A4 (Description), A6 (repos config features), G2 (data/README.md, Part A.1), G4 (Threat X1), G6 (TEAM_CHARTER Section 11), G7 | ER-02: never reverse or link author codes, and never write repository names, account names, author codes, or bot logins to an output. Its point that repository owner logins are not anonymised is a useful addition: outputs must aggregate. | It attributes the prohibition to A4, which contains none; the prohibition is in A1's FAQ and data/README.md. It also did not report the disagreement between A2 (keyed codes for names in trailers) and A6/A9 (a fixed marker), although its instructions require reporting disagreements. The data settles it (V15). |
NB-Q13 What license covers the GitSkills metadata and the skill content text, what does that mean for quoting skill text in a report, and is it safe to run the scripts bundled with skills? |
"The GitSkills metadata and dataset aggregation are released under the Creative Commons Attribution 4.0 International (CC-BY-4.0) license… the skill content text… remains governed by the individual origin repository's license recorded in repos.license… researchers must consult repos.license and quote text minimally only when necessary, excluding runnable payloads and personal data… It is strictly unsafe to run scripts bundled with skills… Group 9 rules therefore mandate treating all skill content purely as static text". | Text only: A1 (Section "GitSkills"), A4 (License), A6 (License), G1 (Dataset acquisition), G2 (Licensing and ethics), G4 (Threats R6, R09, X2), G7 (Data rules) | ER-01 and ER-03: quote short excerpts only, from permissively licensed repositories, never runnable payloads; never execute anything from the dataset. | "Strictly unsafe" is an inference presented as sourced; no tier-A source says it. And the licensing consequence is larger than stated: 59.5% of representatives have no usable license (V17), so most skill text cannot be quoted at all. |
NB-Q14 Do the sources define what counts as a "script" for has_scripts, or what makes a skill instruction risky or malicious? |
"…they do not state the specific file extensions or parsing criteria used to classify a file as a script… Group 9 operationalizes 'risk-relevant capabilities'… through a static scanner of 19 regular expression rules across 12 capability categories… classifying rules with severity 6 or higher… as high-risk signals… a rule match is merely a text signal… not a verdict that the instruction is malicious." Not established: "the exact file extensions or criteria used by the GitSkills dataset authors… and any definition or criteria for determining whether an instruction is intentionally malicious." | Text only: A2 (Table I, Section II), A6 (artifacts config features), G1 (Sections 6, 7, 8), G2 (Part A.1), G3 (rules file), G4 (Threat P1) | Both constructs are the team's own (FR-02, FR-04, RR-01), so they must be validated by annotation (VR-01) and never attributed to the dataset. | The two definitions nearly agree but not exactly: 57 representatives with has_scripts = 0 still contain a file with a P-01 script extension (V18). The report must say which definition each number uses. |
| NB-Q15 Do the sources establish that a copy of a skill in a repository is actually loaded or run by an agent there, so that a copy count measures use? | "The sources do not establish that a copy of a skill in a repository is actually loaded or run by an agent, as the GitSkills dataset is a static repository corpus that does not track dynamic agent execution logs… an agent loads a skill probabilistically at run time when it compares a task against the skill's description… Inference: A high copy count demonstrates widespread folder copying or template distribution across repositories, but does not prove that an agent has ever executed the skill in practice." | Text only: A1, A2 (Abstract, Section I), A4, A9, G1, G2, G4 (Threat C1) | RR-02's interpretation boundary: reach is the spread of identical content, not use or exposure. The report must not say risky skills are "used more". | Its citation of threat C1 is loose (C1 is marked "Not applicable" for P-01), but the substance is right. |
4. Verification¶
One row per answer. "Checked against" names the source passage the team opened and the data checks it ran (elicitation/verification-checks.md).
| Question | Result | Checked against | How the repository notebook compared |
|---|---|---|---|
| NB-Q01 | Verified | A2 App. A ("We do not assume that the representative is the original source"), A9 Schema, A6; data V01, V02 | Same conclusion, same three passages |
| NB-Q02 | Partly verified | A2 App. A and A9 confirm the counts and case variants; the code-search clause is misstated; data V04, V05 | Returned the limitation passage but never reached the second half of the question |
| NB-Q03 | Verified | A4 (record 21875637, version 1.0.0, concept DOI …636), A9 sampling rule, A1 Submission; data V06 | Found the sampling rule but not the identifiers; the team supplied them |
| NB-Q04 | Partly verified, and contradicted by the data | Join key and 100 KB cap confirmed at A9 and A6; contradicted by V07 (skipped_reason never set; 16,697 rows carry content_fetched = 2) and V08 (13 rows join a copy) |
Returned the same passages and repeated the same documented-but-false claim |
| NB-Q05 | Partly verified | Definitions at A9 lines 75 and 85; counts at A2 Table I and V11; missed A2 App. A (42 representatives); data V09, V10 | Also stopped at values 1 and 2 |
| NB-Q05a | Verified as not established | A9 documents only 1 and 2; A2 App. A says the column records the case but names no value; data V09 | Better: its second passage was exactly that A2 sentence |
| NB-Q06 | Verified | A9 Known limitations; A2 Table I and App. A; data V11 | Same passage, same conclusion |
| NB-Q07 | Verified | F-011 (245 pairs across 141 names), F-012 (10 dated pairs), F-007 (3,010 histories); A9; data V12 | Weak match: returned research directions, no feasibility answer |
| NB-Q07a | Verified | A9 "Sample vs full dataset"; data V02, V12, V13 | Same on copies; on variants the team drew the conclusion |
| NB-Q08 | Verified | A2 Section II; no tier-A source describes near-duplicate linking (checked A1, A2, A4, A6, A9); F-011 | Same conclusion from the same A2 passage |
| NB-Q09 | Partly verified | 277 MB (A9), 44,388,249,600 bytes (A4), example queries (A9), 14.7 GB dump (A3), DuckDB query matches A8a word for word; "pandas, polars" unsupported for GitSkills | Also incomplete, in a different way: missed the loading instructions |
| NB-Q10 | Verified | ANNOTATION_GUIDELINE.md (label values), TEAM_CHARTER.md section 9, docs/validation/README.md ("Cohen's kappa and percent agreement") |
Got raters, kappa, and blindness, but not the label values |
| NB-Q11 | Verified | CONTRIBUTING.md; PROJECT_PLAN.md sections 2 and 11 |
Failed: returned the research question itself; the team asked a follow-up |
| NB-Q11a | Verified | PROJECT_PLAN.md section 11, CONTRIBUTING.md, TEAM_CHARTER.md sections 12 and 13, KAIZEN_BACKLOG.md |
Also correct; the two agree |
| NB-Q12 | Partly verified | Anonymisation confirmed at A2 Section IV, A6, A9, A4; attribution error and unreported conflict; data V14, V15, V16 | Same passages; also did not flag the conflict |
| NB-Q13 | Partly verified | CC-BY-4.0 at A6 and A4; group rules at FMEA R09, threats R6 and X2; "strictly unsafe" unsupported by tier A; data V17 | Same A6 passage; also found no tier-A safety statement, because none exists |
| NB-Q14 | Verified as not established | No definition in A1, A2, A4, A6, A9; 19 rules and 12 categories confirmed in the rule file and ANALYSIS_MANIFEST.json; data V18 |
Same conclusion |
| NB-Q15 | Verified as not established | A2 Abstract and Section I; no source or column records loading | Same conclusion from the same passages |
Totals: 12 verified, 6 partly verified, 0 unverified (every answer carried at least a text citation). One partly verified answer, NB-Q04, is also contradicted by the data.
Where the notebook fell short of its own instructions:
- Grounded citations stopped after two answers. NB-Q01 and NB-Q02 carry citation chips that link to passages; from NB-Q03 on, the notebook wrote source lists as plain text, so the team had to locate each passage.
- One wrong restatement (NB-Q02, the code-search exclusion condition).
- One unsupported recommendation (NB-Q09, pandas and polars for GitSkills).
- One inference presented as a sourced fact (NB-Q13, "strictly unsafe").
- One attribution error and one missed conflict (NB-Q12).
What it did well: it never invented a number. Every figure checked out, including ones from the group's own findings log (245 pairs across 141 names; 10 dated pairs; 3,010 commit histories). It volunteered gaps, answered "not established" three times, and labelled its inferences.
What the interview changed in the requirements (RESEARCH_SPEC.md):
| Finding | Requirement |
|---|---|
| Cite the Zenodo version DOI, concept DOI, sample commit, and file hash | DR-01 |
content_fetched = 2, empty skipped_reason, and truncated listings leave bundled files unread |
DR-04, FR-04 |
| The representative is not the origin; copies measure spread, not use | RR-01, RR-02 |
| The sample splits variant families; dated pairs are mostly canonical skills | RR-03, FR-06 |
| "Script", "risky", and "lineage" are the team's constructs | FR-02, FR-06, VR-01, VR-03 |
| Skill text keeps its origin repository's license; owner logins are not anonymised | ER-01, ER-02 |
5. The two notebooks compared¶
Both notebooks were given the same 24 sources and the same 18 questions.
| Google Notebook | Repository search notebook | |
|---|---|---|
| How it answers | Generates prose from the sources | Quotes the passages that match, verbatim |
| Citations | Grounded chips for 2 answers, text-only for 16 | A locator for every passage, always |
| Reproducible | No: the same question can produce different wording | Yes: same pack, same question, same answer |
| Answered the question asked | 17 of 18 | 15 of 18 (NB-Q07 weak, NB-Q09 incomplete, NB-Q11 off target) |
| Errors found in verification | 5 (listed in section 4) | 0 of fact; its failures are misses, not mistakes |
| Best moments | NB-Q11 change control, NB-Q07 feasibility with exact numbers, NB-Q10 label values | NB-Q05a: found the A2 passage on the 42 unrecoverable representatives that Google reasoned past |
The two are complementary, and the difference is the point of Phase B: a generative notebook answers the question you asked but can quietly overstate a source, while a search notebook cannot overstate anything but will sometimes hand you the wrong passage. Neither replaces reading the source, and neither replaces the dataset: four of the findings that changed requirements came from the data checks (V07 to V18), which no notebook could have produced.