ADaM, AI and Code-Level Traceability

In this article, we review the Analysis Data Model’s (ADaM) limitations in communicating study analysis decisions, primarily due to its reliance on metadata traceability, and how AI adds to these challenges. As a response, we present the Verisian Validator as a solution by introducing code-level traceability, thereby achieving ADaM’s goal of unambiguous communication of analysis decisions between sponsor and reviewer. 

Challenges with ADaM's Focus on Metadata

The Clinical Data Interchange Standards Consortium (CDISC) was established to standardize collected clinical trial data, leading to the development of the now ubiquitous Study Data Tabulation Model (SDTM). When it became apparent that SDTM was unsuitable to represent data derived through complex statistical analyses, the Analysis Data Model (ADaM) was introduced to specifically cater for the intricacies of clinical trial analysis data. 

Besides standardizing the delivery of clinical trial analysis data to regulatory authorities, ADaM’s foundational principles were to ensure unambiguous communication between the sponsor and regulatory reviewers regarding the analytical choices made during the creation of analysis datasets and results. This would make it easier for reviewers to understand and validate study analyses. For this, ADaM mandates traceability with comprehensive metadata documentation at dataset, variable and analysis results levels. The importance of traceability is emphasized by the FDA in the Study Data Technical Conformance Guide: ‍

Establishing traceability is one of the most problematic issues associated with any data conversion. If the reviewer is unable to trace study data from the data collection of subjects participating in a study to the analysis of the overall study data, then the regulatory review of a submission may be compromised.

However, instead of fulfilling its mission of clear analysis dataset communication, ADaM has now become more associated with adherence to standard data structures. This is largely due to its reliance on define-xml to represent complex metadata. 

Define-xml’s focus on individual variable metadata struggles to capture adequately the complexity and interdependencies of variables derived from other variables or source data. The complexity and nuances of derived variables and their analytical pathways often remain obscured, making it difficult for reviewers to fully grasp the derivation logic and dependencies within a dataset. Moreover, ADaM does not mandate consistency in how metadata is formatted. Derivation descriptions can be written in plain English, sometimes in code, or a mix of both. Sponsors may opt for a two-level naming convention for each variable, such as “ADTTE.TTEVENT”, whereas others do not. 

The example below illustrates this issue. Here we look into how the Time to Event variable is described in an ADaM Time-to-Event dataset. The column description is meant to explain how the variable is derived, but the complexity of the description makes it difficult for reviewers to fully grasp the logic behind the derivation. It also allows too much room for interpretation which can lead to different programmers interpreting it differently.

This brings up a crucial question: how can reviewers be sure that the documentation actually describes what is in the code and follows the SAP?

In response to some of these challenges, the Analysis Data Reviewer’s Guide (ADRG) was introduced, where sponsors provide information that is supplementary to the define.xml, for example, to describe in more detail how ADaM datasets differ from source SDTM. However, rather than simplify, these efforts have added more complexity. For example, the definition of a concept like treatment emergent must now be documented in not just one, but across various different documents (ADRG, SAP, define.xml, etc). This increases the risk of inconsistencies and a potential loss of integrity in the analysis. 

While standards like ADaM play a crucial role in the submission process, their capacity to communicate what actually happened in a study is limited. With the rise of AI, there’s now an even greater need for a more effective method to guarantee the integrity of analysis datasets.

ADaM in the Age of AI

The rise of AI to generate analysis datasets and outputs shifts the focus from code creation, a task Large Language Models (LLMs) are poised to perform, to validation tasks that ensure the generated analysis adheres to the SAP. 

The nature of LLMs, which operate on probabilistic principles rather than deterministic rules, means their outputs are generated from likelihoods inferred from their training data. This results in outputs that are inherently not reproducible, presenting a unique challenge: ensuring that AI-generated analyses adhere to their SAP. 

To conduct this level of validation effectively, it is crucial to understand exactly what happened in a study, especially the logic behind variable derivations. As we have outlined, ADaM’s focus on metadata makes it ill-equipped to give this deep understanding to reviewers. After all, analysis results are not generated by predefined metadata, but code. Should the implemented analysis code deviate from the prescribed SAP, it is self-evident that even flawless metadata will not salvage the validity of trial results.

Validation with Code-Level Traceability

To address the gaps in ADaM’s ability to unambiguously communicate the analysis decisions in a study, a new approach is necessary. Instead of focusing solely on metadata traceability at variable or dataset level, we need to focus on code-level traceability.  

To verify something as fundamental as whether the definition of a treatment-emergent time-frame is accurately implemented, a reviewer at present has no other choice but to go into the study’s code base and examine it line by line. This is a daunting task, made even more challenging by macro invocations which act as black boxes with state-dependent executions. 

Instead, the solution lies in representing a study’s code as a sequence of independent derivations, akin to ETL pipelines, which take in raw data, transform them according to predefined analysis steps, and organize them into tables, listings and figures. These derivations should be de-macrofied and designed to function autonomously. This would enable anyone to trace how a particular variable or outcome was derived by simply accessing the exact code used, regardless of the statistical analysis code’s complexity. 

LLMs can then be used to generate natural language summaries for each derivation, transforming complex code into digestible explanations. These summaries can be shared with both team members and reviewers to facilitate the validation process. 

LLMs are particularly useful when distilling complex information into accessible insights. Given a dense prompt - a well-defined, detail-rich input, in this case, the specific derivation - LLMs excel at producing accurate answers. This detailed prompting guarantees that the summaries are both accurate and informative, offering reviewers insight into the analysis decisions without the need to manually go through the original code. This saves time, but more importantly, ensures quality. 

This is exactly what we are building at Verisian. We offer the Verisian Validator, which safeguards quality through automated quality control. The Validator is backed by a study knowledge graph, which leverages graph theory to reproducibly create a  dependency graph of everything that happens in a study at code-level. This includes every variable assignment, applied format, procedure step and data manipulation. Right now it only supports studies built in SAS, but we will soon also add R and Python. 

With the Validator, you can select any dataset (and soon also any variable) and it provides the exact code necessary to generate that variable or dataset, free of any macros or superfluous code. This can be exported for review or future re-use, and its upstream dependencies and downstream effects can be visualized. The Verisian AI co-pilot can then be used to generate accurate study documentation as described above. 

Besides ensuring SAP compliance and analysis integrity, Verisian also helps to build and submit studies by enabling code reuse across studies, providing AI co-pilot support, automating define-xml generation, and more. You can read more about it on our website.

Fulfilling ADaM’s Promise

The Verisian Validator seeks to achieve what ADaM initially set out to do, and more: unambiguous communication of analysis decisions between sponsors and reviewers. It does this by going beyond metadata by introducing full code-level traceability empowered by AI. This approach guarantees the integrity of a study,facilitates adherence to the SAP, and streamlines its review, regardless of whether the analysis was done by humans or AI. If you’d like to learn more, feel free to get in touch with us.