Submit when the Science is Done
Learn how to make agentic AI trustworthy in clinical trials, with your experts in control, always.
In this episode of The Verisian Podcast, host Tomás Sabat Stöfsel is joined by Henning Kuich, CTO of Verisian, and Andrew York, strategic advisor to Verisian and former Vice President of Data Science at Novo Nordisk. Andy has held senior biostatistics and biometrics roles across the pharmaceutical industry.
The conversation starts with a discussion on the pharmaceutical industry's shift from SAS to R, and why SAS has led the way for decades, despite the fact regulators have never mandated a specific language for regulatory submissions. It then turns to Verisian’s experiment rebuilding the R Consortium’s FDA-reviewed Pilot 3 submission package with AI agents. The experiment produced three unexpected findings. Frontier AI models copied the public pilot code instead of following the specifications. Half of the custom derivation specifications couldn’t be implemented as written, and the reference implementation contained a discrepancy that double programming would never have caught.
In this conversation, we explore:
Guests:
Host: Tomás Sabat Stöfsel, CEO and Co-Founder, Verisian
Topics: SAS to R migration, R Consortium FDA submission pilots, AI agents in statistical programming, ADaM derivations, specification quality, double programming, code traceability, Define-XML, regulatory submissions, future of the statistical programmer
Listen to the full episode: YouTube | Apple Podcasts | Spotify
Andy explains that the move to R has come from two directions. Junior programmers arrive from university already trained in R, while senior scientific leaders see it as a capable, lower-cost alternative to SAS. Although SAS has dominated clinical trial analysis for decades, regulators have never mandated a specific language. Early R-based submissions, including Novo Nordisk’s, have shown the route works, and the R Consortium’s FDA pilot programme continues.
Henning describes how Verisian tested its platform on the R Consortium’s Pilot 3, a public ADaM submission package reviewed by the FDA. The agents received only the SAP, the SDTM data and the ADaM specifications. The reference data was withheld, so the results could be compared dataset by dataset, variable by variable and observation by observation. Verisian’s Planner, Builder and Validator agents completed the full implementation under expert guidance, and no human wrote a single line of code.
One of the most significant findings was unplanned. Frontier AI models recognised the public Pilot 3 study and copied its code from their training data, ignoring the specifications and guardrails they were given. Verisian’s Validator detected this because it systematically checks whether each implementation matches its specification on a logical level. Henning notes that double programming would have been blind to the issue. The incorrect and intended algorithms produced the same numbers on this data, but would have handled edge cases differently as a study grew.
About 50% of the custom variable specifications in Pilot 3 were incomplete, ambiguous or open to interpretation. Henning stresses that this reflects an industry-wide norm, not poor work: experienced programmers fill gaps using implicit conventions. AI, however, always produces an answer, even from ambiguous inputs, and that output cannot be validated. After the specifications were made AI-ready, first-shot accuracy rose from 9% to 69%, and 52 of 76 custom variables were correct with no further work.
Both guests see the role shifting rather than disappearing. As AI handles meticulous, repetitive coding, experts move to defined quality-control gates: confirming that specifications reflect the SAP, validating the code, handling edge cases and communicating decisions to regulators. Andy describes smaller, highly skilled teams that can do more work with the same number of people. That frees capacity for secondary indications and earlier access to medicines for patients.
“The generation is very, very cheap, but validation and understanding these details is really, really not. That’s the part that’s expensive.” — Henning Kuich
“If you’re putting guardrails in place and the AI ignores the guardrails, then there is something fundamentally wrong.” — Andrew York
“If you put ambiguity in, you’re going to get ambiguity out.” — Henning Kuich
“AI is getting better and better at masking and hiding the things that it’s not good at, trying to fake itself into intelligence.” — Henning Kuich
“We’re all patients as well. We’re all going to be taking these drugs in the future ourselves.” — Andrew York
Does the FDA require SAS for clinical trial submissions?
No. The FDA has never mandated a specific programming language for regulatory submissions. SAS became the industry standard through convention, and sponsors have successfully filed submissions using R.
What is the R Consortium Pilot 3 submission?
R Consortium Pilot 3 is a publicly available, FDA-reviewed submission package that demonstrates ADaM dataset creation using R. It includes the SAP, SDTM data, ADaM specifications, R code and Define-XML.
Can AI write clinical trial analysis code?
Yes. In Verisian’s experiment, AI agents generated the complete ADaM implementation for R Pilot 3 with no human-written code. Experts guided the work, reviewed the outputs and gave final sign-off.
Why isn’t double programming enough to validate AI-generated code?
Double programming compares outputs, not logic. If two different algorithms produce the same results on a given dataset, double programming cannot tell them apart, even when one does not match the specification.
Why do specifications matter for AI in statistical programming?
AI will always produce an answer, even from ambiguous instructions. Unambiguous, complete specifications are required both for AI to implement derivations correctly and for anyone to validate that the implementation is right.
Will AI replace statistical programmers?
According to Henning Kuich and Andrew York, no. The role is shifting towards validation, understanding the intent of the SAP, handling edge cases and communicating with regulators, while AI takes over repetitive coding.
The Verisian Podcast brings together clinical trial experts to share ideas and best practices in clinical reporting, submission and review. Guests include statistical programmers, biostatisticians and medical writers, discussing the technologies shaping the future of clinical development. Listen on YouTube, Apple Podcasts and Spotify.
See Verisian in action. Share a few details and we’ll be in touch to schedule your demo.
Your request has been received. A member of the Verisian team will contact you shortly to arrange a demo.
