Over the past few weeks we have tested our AI agents on the R Consortium Pilot 3, a public ADaM submission package that the FDA has reviewed. Given only the SAP, the SDTM data and ADaM specifications, our agents derived all five ADaM datasets that were validated by expert review using the Verisian Validator. Specification, implementation and validation were completed in a single working day, and no human wrote a line of code.

There are four main outcomes: 

1. More than half the custom derivation specifications were not AI-ready: they were incomplete, inconsistent, or ambiguous. Our planning agents, under expert review, improved specifications to make them AI-implementable, increasing first-shot accuracy by 60 percentage points.
2. By iterating through implementation-validation cycles under expert guidance and review, the Verisian Builder and Validator agents completed the entire ADaM implementation without a single line written or corrected manually.
3. Using the Verisian Validator, expert review revealed that the reference implementation itself deviated from its own specifications, and that this deviation was carried into the final submission package through the accompanying Define-XML.
4. Frontier models reproduced the pilot’s code from the repository in their training data instead of following the specifications and explicit instructions. While the problem was identified easily using the Verisian Validator, the incident highlights how naive AI deployments are prone to guardrail and alignment failures creating incalculable risks. Safe AI deployment requires a strong harness and optimized expert validation.

The full white paper will be published next week. Here is a summary of what we did and what we found.

Why validation is the bottleneck, not code generation

Despite the immense progress in AI over the past few years, its adoption in biostatistics and statistical programming has been slow. The obstacle is not code generation, which is now a commodity, but validation, which remains the rate-limiting step. This is compounded by industry-standard specifications that are ambiguous, hindering automated implementation and, more importantly, their validation.

AI-written code often looks correct while containing compounding errors. Beyond validation, experts must maintain their understanding of the codebase, whether it was written by humans or by AI. At Verisian, we combine a deterministic code-traceability harness with an agentic automation framework governed by experts at defined quality-control gates. Three agents operate on this harness: a Planning agent ensures the specifications are AI implementation ready; a Builder agent writes and executes the code; and a Validation agent reviews the output and guides further iterations. Experts then review, guide additional iterations, and give final sign-off on validity using the Verisian platform based on the same harness.

Case Study on Pilot 3

We assessed the process, agents and platform using the R Consortium Pilot 3 submission package. It is composed of five ADaM datasets (ADSL, ADAE, ADADAS, ADLBC, ADTTE) and contains 216 analysis variables, 76 of which require custom derivations. The SAP, SDTM data and ADaM specifications served as inputs, while the reference ADaM data and Define-XML were withheld from the agents and used for evaluation only.

Our integrity analysis found that 39 of the 76 custom variable specifications (51%) were of insufficient quality for AI implementation. Planning agent iterations and expert feedback produced improved, AI-ready specifications, which raised first-shot cell accuracy from 8.7% (original specifications) to 68.6% (improved specifications). In total, the agents completed all five datasets in 21 review cycles within a single working day: 10 cycles to improve specifications and 11 cycles to reach a validated implementation. Humans wrote no code.

Because the Pilot 3 code is publicly available, it appears in the training data of frontier general-purpose models. Despite explicit instructions and guardrails to follow the provided specifications, the models copied the code from the repository in their training data instead. We caught this using the Verisian Validator as the implementation was clearly not according to specifications but the output matched the reference data perfectly. This is crucial evidence that AI must be both contained within a deterministic harness and kept under expert control for safe deployment and use.

Expert review through the Verisian Validator also revealed discrepancies between the reference implementation and both its specifications and Define-XML. The discrepancy was discovered in the calculation of the cumulative dose, data crucial for safety and efficacy evaluation. This serves as a reminder of how difficult it is to create consistent, integral, and valid regulatory submissions outside Verisian, even in small and simple studies.

What’s next

Because the harness supplies AI and experts with only the information they require for any given task, the context per task does not grow with study size. We therefore expect the approach to scale without difficulty to much larger studies, which is our next focus.

Our results show that pairing Verisian AI agents with deterministic traceability and targeted expert oversight delivers both speed and validity, and can keep studies continuously validated and submission-ready.

About the author

Henning Kuich is the CTO and Co-Founder of Verisian. Henning is a biomedical and computational scientist, software architect and developer. Upon completing his MSc at the University of Virginia, he went on to earn his PhD on novel methods for blood diagnostics from the Max Delbrück Center for Molecular Medicine/Berlin Institute of Health at the Freie Universität Berlin, graduating summa cum laude. He then joined Bayer where his teams established infrastructure-as-code, IT security architectures, and built data processing systems, among them an infrastructure for biomedical literature and patent research that won the 2019 EMEA Elasticsearch Award. The experiments described in this article were run by the Verisian engineering team.