adamAI-written ADaM derivations, checked
A model’s derivation counts only when it reproduces the reference, row for row.
adam asks a model to write the {admiral} program for one ADaM variable at a time, runs it in isolation on public SDTM data, and accepts the column only if it equals the published reference exactly. Every program and every verdict is recorded with its reason, and any verdict can be re-run in your browser.
Accepted only on an exact match
| gpt-5 | qwen2.5-coder 7B | |
|---|---|---|
| All variables | 118 of 159 | 35 of 159 |
| Copied from an input | 89 of 109 | 32 of 109 |
| Derived | 29 of 50 | 3 of 50 |
Public data: the CDISC pilot study as packaged by pharmaverse. Reference: the published pharmaverseadam datasets.
The demo, in two minutes
An accepted derivation, a column gpt-5 filled where the reference is empty, and that verdict re-run in the browser: recorded from the live demo.
What was measured
159 variables: every column of pharmaverseadam’s ADSL and ADAE except the keys, 109 copied unchanged from one input column and 50 derived. One answer per variable and model; intervals are Wilson 95%. The protocol was committed before either model was asked anything.
Copied variables 89 of 109 (73–88%); derived 29 of 50 (44–71%); derived with no other form of the variable in context 13 of 30.
gpt-5-2025-08-07 through the OpenAI API. Rejections: 38 wrong values, 3 errors.
Copied variables 32 of 109 (22–38%); derived 3 of 50 (2–16%); derived with no other form of the variable in context 2 of 30.
qwen2.5-coder:7b on a local laptop GPU. Rejections: 71 errors, 32 wrong values, 15 contract breaks, 6 wrong class.
On the same 159 variables, 92 were accepted only for gpt-5 and 9 only for qwen2.5-coder 7B.
9 of gpt-5’s 38 wrong-value rejections are columns the reference leaves empty: gpt-5 filled them anyway, with invented dictionary codes, consent dates, and an action taken for every one of the 1,191 adverse events, inferred from the dosing records. For RFSTDTC it filled in start dates for the 52 screen-failed subjects, whose reference value is empty.
Read in the demo: filter on gpt-5’s rejections and open the difference from the reference.
Before any result counted, the check accepted 2 known-correct programs and rejected all 8 planted faults for the planted reason: a date one day late, a population flag missing placebo, partial dates left unimputed, lost rows, a wrong class, an error, a network call and an endless loop.
Every recorded program was re-run in headless Chrome with webR, through the demo’s own button, on the same data and the same comparison code.
317 reproduced in one continuous page session; 1 more only in a fresh page, after the session’s WebAssembly memory ran out on a program that joins tables many-to-many.
How the check works
The same check a statistical programming team runs between a first-line program and its second-line QC, applied to a model as the first line.
Inputs the reference was built from
pharmaversesdtm 1.4.0 and admiral 1.4.1, the exact set that produced pharmaverseadam 1.3.0. Running admiral’s own templates on them reproduces both reference datasets with zero differences.
One variable at a time
The model gets the variable’s name, label and required class, and every input column’s name, class, label and a few example values. Never the reference values of the variable, never the template code.
An isolated run
The program runs in a fresh R process with no network, a two-minute limit and a memory limit. The rest of the reference dataset is the context, so one wrong variable cannot spoil the next.
An exact comparison
{diffdf}compares the produced column with the reference on the dataset keys. Any differing row, a wrong class, lost rows or an error is a rejection, recorded with its reason.Re-run anywhere
The demo carries the same data files and the same R comparison code, and runs them in your browser with webR, so a verdict does not depend on trusting the machine that recorded it.
What this does not show
- The reference is the output of the public admiral templates. In production it would be an independent second-line QC program.
- The templates are public, so either model may have seen them. Standards knowledge and memorization cannot be separated here.
- The check is agreement with a reference, not correctness against a protocol or analysis plan. It is not validation in the GxP sense.
- One answer per variable and model, each variable derived alone with every other reference column available.
Talk about it
If you run statistical programming or a computing platform for it and want to discuss how a check like this fits a first-line and QC workflow, send me a message on LinkedIn.
Who is behind it
adam is built by Alexander Chernov, whose work is on governed AI systems: deciding, with evidence, what an automated system may do.
Every figure on this page comes from the committed run records.
From the same work
Grafferene, a response cache for LLM endpoints that measures before it serves, and Agentic Crawl, goal-driven web crawling that decides every request before it is sent.