Proof, and what Traceome is not
Every result here comes from a development Run, one Run each, examined in a post-Run
review. None is a formal eval. A small evaluation shows the first patterns; a bigger one is planned.
What Traceome is not
- Not a clinical tool. It is research decision support. It makes no diagnosis and claims no function or causality, and "no record" never becomes "not carried" R9.
- Not autonomous, by design. The agents investigate, but they can't settle every question alone. When evidence conflicts, it takes a knowledgeable researcher to drive the answer, so the researcher approves at the key points and nothing consequential runs without that approval R2.
- A prototype, not a full analysis suite. It answers one kind of question well: does this sample carry these variants? We tested it with one Parabricks workflow, a single paper at a time, and one open-access literature source.
- Not fully evaluated yet. A small evaluation shows the first patterns. A bigger evaluation, with more hand-picked scenarios, is planned E1–E4.
Validation
| What we measured | Result | What it shows | Claim |
|---|---|---|---|
| Run H: 29 Work items on HG001 | 17 observed · 6 not carried · 6 no VCF record | Every item ends in exactly one bounded outcome | R3 |
| Two planted bad identifiers | 2 of 2 unresolved, 0 invented | The agent doesn't paper over what it can't place | R4 |
| Same inputs, Nemotron vs Qwen3.5 | 29/29 findings identical; VCF identical record for record (260,016) | The evidence decides the answer, not the model | R5 |
| Parabricks on one H100 | about 9.5 min; about 61 min from first contact to the explained report | A genome-scale Run fits in an hour | R6 R7 |
| Nemotron on Token Factory, Run H | 145 requests, all HTTP 200, 0 re-sends | Every agent call went through Nebius | R8 |
| Journey integrity, Run H | 337 of 337 file addresses re-verified | The record is checkable, not just stored | R14 |
| Forks | 0 contacts; parent unchanged; about 38 min skipped in #437 | Corrections and new branches are cheap | R10 |
| Formal evals (repeated Runs, held-out cases) | pending | E1–E4 |
Planted failures
We put inputs into Run H that should not resolve, to check that Traceome says so instead of inventing an answer.
| Planted input | Why it is bad | What Traceome did | Claim |
|---|---|---|---|
rs9999999999 | planted; source lookup does not find it | reported as unresolved (not found); no Work item, no finding | R4 |
rs1569937886 | planted; source lookup marks it incompatible | reported as unresolved (incompatible); no Work item, no finding | R4 |
The 6 "no VCF record" outcomes are not failures. Traceome reports that no record exists. It never reports that the variant is absent R9.
Anchor Runs
| Key | Run ID | Date | Commit | Model (Token Factory) | Inputs |
|---|---|---|---|---|---|
| H | 387-human-a2-20261022T204321Z-7be9123c-a1 | 2026-10-22 | 7be9123c | nvidia/Nemotron-3.5-Super-VL | HG001 paired FASTQ (1.91 GB + 1.95 GB), GRCh38, 24 rsIDs |
| C | 387-celegans-a2-20261022T193856Z-59c33dda-a1 | 2026-10-22 | 59c33dda | nvidia/Nemotron-3.5-Super-VL | SRR7145496 paired FASTQ, WBcel235, 4 position targets |
| Q | 141-full-20261018T095210Z-5b06c210-a2 | 2026-10-18 | 5b06c210 | Qwen/Qwen3.5-397B-A17B (comparison baseline) | same as H |
| 457 (demo) | 457-human-20261029T084406Z-d3f3c340-a1 | 2026-10-29 | d3f3c340 (earlier build) | nvidia/Nemotron-3.5-Super-VL | HG001 paired FASTQ, GRCh38; a request of 24 rsIDs, BRAF V600E, a GRCh37 position and other targets |
| M | 4-mlive-mullins-20261030T095056Z-a3cd32a2-a1 | 2026-10-30 | a3cd32a2 | nvidia/Nemotron-3.5-Super-VL | a bare citation of Mullins 2020 (PMC7599709); HG002 paired FASTQ (about 1×), GRCh38.p14 |
| S | 4-msmoke-celegans-20261030T132733Z-028a063a-a1 | 2026-10-30 | 028a063a | nvidia/Nemotron-3.5-Super-VL | SRR7145496 staged in Object Storage, WBcel235, Galaxy prebuilt index |
| D | 457-celegans-20261025T195807Z-0d3c1b08-a1 | 2026-10-25 | 0d3c1b08 | Nemotron 3.5 Super VL (build defaults) | same as C |
The GUI screenshots and the current demo replay show Run Q.
Claims
Dev-verified seen in a retained development Run and examined in review; one Run, not repeated. Demonstrated shown working, not measured. Planned / Unresolved no evidence yet; never written as a result.
| ID | Claim | Status | Evidence |
|---|---|---|---|
| R1 | One fresh Run went from raw FASTQ through planning, an approved Parabricks job on one H100, Investigation and Finalization to a published Grounded report and explanation. No step was redone. | Dev-verified | Run H, combined review |
| R2 | Three human approvals gate the consequential steps: Execution, Investigation (per Work item) and Finalization. Each approval binds an exact proposal or basis. | Dev-verified (H, C) | Run H answers |
| R3 | 24 requested rsIDs became 29 Work items: 17 exact allele observed, 6 target allele not carried, 6 no VCF record. | Dev-verified | Run H, combined review |
| R4 | Two planted bad inputs, rs9999999999 (not found) and rs1569937886 (incompatible), were reported as unresolved. No Work item or finding was invented for them. | Dev-verified | Run H, combined review |
| R5 | Changing the model did not change the science: all 29/29 human Work items, and 4/4 for C. elegans, equal the Qwen Run's item for item. The VCF is identical record for record (260,016 records); only the run-date header differs. | Dev-verified (one pair) | Run H vs Q; C vs #141 |
| R6 | Parabricks pbrun germline (4.7.0-1) on one H100 took about 9.5 min. The FASTQ upload took about 18.5 min. | Dev-verified (one Run) | Run H, execution review |
| R7 | About 61 min from first contact to the published explanation. The C. elegans Run took 28 min 22 s. | Dev-verified (one Run each; the host clock stepped back about 104 s during H) | Runs H and C |
| R8 | 145 Nemotron requests, all HTTP 200, with 0 re-sends and 0 Agent work stops. 1.79 M input tokens (442 k cached) and 327 k output tokens, of which the explainer used 248 k. | Dev-verified (one Run) | Run H |
| R9 | Every report statement is bounded by its evidence. Beyond the measured Run health (R19), the report claims no callability, absence, function, causality or diagnosis. "No VCF record" is not reported as "reference genotype". | Dev-verified | Run H report limitations |
| R10 | A finished Run can be forked. fork writes one RunForked fact (no new authority) whose artifact is the parent's exact prefix, makes 0 contacts and leaves the parent byte-unchanged; the child needs fresh approvals. #369 amended two items in a Fork and both outcomes changed to observed; all 186 parent files were hash-identical before and after. #437 forked after Execution: 29/29 outcomes equal, about 38 min of planning, approval and H100 skipped. | Dev-verified (several Forks) | 369-fork-0f89a89e-a1, 437-fork-8d30b415-a1, 141-celegans-fork-d23e4135-a1 |
| R11 | The GUI replays a retained Journey: results, the Journey path, decisions, evidence and the report. | Demonstrated (replay of Q only) | GUI receipt |
| R12 | A Run reads a paper. From a bare title citation, the fresh Mullins 2020 + HG002 Run on the final build found and retrieved PMC7599709 (4 paper GETs). The paper reader judged 3 findings from cited spans (rs429358 and rs7412 from the text and Table 3; rs1801133 from the text and Table 2), the reference organism was resolved from the paper, and Investigation placed all three on GRCh38.p14. The report's "What papers found" section names the paper, each finding, its quote and its passages, and credits the paper as the asserter. Open on the page: a paper target is labelled by its whole paragraph, and table passages carry the wrong section. | Dev-verified (one Run) | 4-mlive-mullins-20261030T095056Z-a3cd32a2-a1, M live a3cd32a23 post-mortem |
| R13 | C. elegans biological correctness. The sample's identity is in question: metadata, run alias and reads point to different strains. The four expected sites show no record in every live Run. | Unresolved | #141 completion audit |
| R14 | The Journey is append-only typed facts plus write-once, content-addressed artifacts. Run H: 1,834 events (30.7 MB), 65 event types, 337 artifacts, all 337 addresses re-verified with 0 mismatches. | Dev-verified | Run H, combined review |
| R15 | Researchers read the Journey while the Run happens and afterwards. observe, journey-feed and run-artifacts read it without contact and without changing its bytes. | Dev-verified | #457 a3, #437 |
| R16 | Agents read the same record: planning and Investigation agents call search_journey. Run H made 12 such reads. | Dev-verified | Run H |
| R17 | Correction at the junctions (live): subset approval with other items pending (#266); revise (#206); amend an item's target (#369); stop Investigation and still report (#300); reject an extra target (#389); clarify (#156, #380). | Dev-verified; Finalization correct/stop: offline only | #266, #300, #369 reviews |
| R18 | The GUI is a researcher workspace over the CLI: it replays recorded Runs, starts and follows live Runs, and shows report.md. | Built (#494, tested offline); a reviewed live Run driven from the GUI is next | GUI #494 |
| R21 | The demo Run (#457, HG001): 29 targets became 33 Work items; 27 targets searched in the VCF, 20 with the target allele called, 2 not searched. Outcomes: 18 observed, 8 not carried, 7 no VCF record. Merged IDs were followed (rs60037639 → rs4149056), BRAF V600E was placed from the gene change, a GRCh37 APOE position was placed on GRCh38, and rs9999999999 stayed unresolved. 2,398 Journey facts, 184 Nemotron requests, about an hour end to end. | Dev-verified (one Run, earlier build) | #457 human a1 post-mortem |
| R19 | A Run measures its own health and each Work item's coverage, and the report has a "Run health" section. C. elegans smoke Run on the final build: 19,528,076 reads, 97.61% mapped, mean depth 17.1, 97.5% of bases covered, Ti/Tv 0.95; each of the 4 Work items has its depth (13, 20, 17, 16), all above the minimum of 10. | Dev-verified (one Run) | 4-msmoke-celegans-20261030T132733Z-028a063a-a1, M smoke 028a063a7 post-mortem |
| R20 | A Run can take its FASTQ from staged Object Storage files: copied server-side, verified against the staging manifest, never uploaded again. The same Run used a prebuilt Galaxy Project reference index, with 0 CPU jobs and 1 H100 job. | Dev-verified (one Run) | M smoke 028a063a7 post-mortem |
Formal evals
These slots are filled only from reviewed formal evals, which have not reported yet.
| Slot | Metric | Cases | Value |
|---|---|---|---|
| E1 | Core gates accepted, usable and questions B-* | planning cases s1, s2, s5, s7, s12, s16, s20 | pending |
| E2 | Extended: target recall, precision, F1, exact set match | paper cores (not built in the scorer yet) | planned |
| E3 | Run time and tokens over repeated Runs (median and range) | human + C. elegans | pending |
| E4 | Cost per Run (Token Factory + H100) | not recorded yet |
Limits and what went wrong
- Silent truncation. Super VL replies were cut at the provider's default 8,192 tokens. We caught it when planning replies ended at exactly 8,192, held that Run at the Execution approval and set the reply limit explicitly.
- A model that wouldn't stop calling. Nemotron 3 Ultra worked in simple cases but looped as
complexity grew: thousands of copies of one tool call, retries and gateway timeouts. We took control of the turn
in code and moved every role to Super VL, with
tool_choice: auto. - The explainer is the weak part. An earlier explainer wrote prose that contradicted its correct findings. Now it may cite only approved facts, each genotype is checked against the VCF row it cites, and a malformed output gets a bounded number of repairs before it is refused. Run H still needed 20 repairs in 45 outputs.
- Our answer key was wrong, not the agent. For the C. elegans sample our expected answer said two sites were carried; the reads showed only the reference base R13.
- Even the clock lied. The host's clock stepped back about 104 s during Run H. The Journey's ordering exposed it, so readiness now checks the clock first R7.
- Coverage is measured, but "no record" still never means "not carried". Each Work item's depth is reported R19; the outcome stays "no VCF record" R9.
- The paper page needs polish. In the paper Run a target is labelled by its whole paragraph, and table passages name the wrong section R12.
- Every result comes from one Run. There are no repeated measurements until the evals E3.
Public data
- Human: HG001, the Genome in a Bottle reference sample (NA12878). It is public, consented reference data, not patient data. Runs H and Q use the same HG001 paired FASTQ against GRCh38.
- C. elegans: SRR7145496, a public sequencing run. Its identity is unresolved: metadata, run alias and reads point to different strains. We report no biological conclusion from it R13.
- Background, not a Traceome finding: rs4149056 in SLCO1B1 is widely studied for statin side effects.