Artificial Intelligence for Drug & Biological Products – Credibility of AI Models for Context of Use – The “Model Risk” Paradigm.
The FDA has recently issued draft guidance (non-binding recommendations) for drug and biological product sponsors and computer programmers regarding the use of Machine Learning (ML) and artificial intelligence (AI) to support regulatory decision-making about the safety, effectiveness, or quality of drugs[1]. ML generally refers to techniques used to train algorithms to improve performance based on data. The draft guidance aims to provide a risk-based credibility assessment framework for establishing and evaluating the credibility of an AI model for a particular context of use (COU). Credibility evidence includes any evidence that supports the credibility of an AI model output for a specific COU, which defines the role and scope of the AI model used to address a question of interest.
ML is typically utilized throughout the drug product life cycle. Certain AI models may be considered devices under FD&C Act Section 201(h)(1). According to the guidance, activities such as the level of oversight by FDA, the sponsor, or other responsible parties, the stringency of credibility assessments and performance acceptance criteria, risk mitigation strategies, and documentation requirements should align with the AI model risk and be tailored to the specific COU.
In summary, the guidance
- AI in Drug Development: The use of AI in the drug product life cycle has increased, offering potential benefits in accelerating drug development and enhancing patient care.
- Examples of AI Applications: AI is used for various purposes, including reducing animal-based studies, predictive modeling, integrating data from various sources, and processing large datasets for clinical trials.
- Challenges of AI Use: Challenges include variability in data quality, potential biases, difficulty in understanding AI models, and changes in model performance over time.
- Risk-Based Credibility Assessment Framework: A 7-step process is proposed to assess the credibility of AI model outputs, including defining the question of interest, context of use, assessing model risk, and developing a credibility assessment plan.
- Examples of Credibility Assessment: Two examples illustrate the process: one involving AI in clinical development and the other in manufacturing, demonstrating how model risk might be assessed.
- Importance of Life Cycle Maintenance: Life cycle maintenance ensures AI models remain fit for use, addressing changes in model inputs and performance over time.
- Risk-Based Approach for Maintenance: A risk-based approach for life cycle maintenance helps assess the impact of changes to AI model performance and ensures compliance with regulatory requirements.
- Early Engagement with FDA: Early engagement with the FDA is encouraged to set expectations for credibility assessment activities and address potential challenges.
The risk-based credibility assessment’s most compelling area entails the concept of “model risk”, which the FDA defines to be a combination of two sub-factors “(a) model influence, which is the contribution of the evidence derived from the AI model relative to other contributing evidence used to inform the question of interest and (b) decision consequence, which describes the significance of an adverse outcome resulting from an incorrect decision concerning the question of interest[2]”.
The draft guidance utilizes a “risk matrix” to evaluate the weight to give decision consequence (i.e., to answer the clinical question of which patients can be considered “low risk” and may not need inpatient monitoring?). While drug and biologics sponsors, manufacturers and clinicians have seemingly opened the door to ML with respect to developing therapeutics and diagnostic products, the model risk danger is real and should be carefully evaluated when relying on ML and AI for data purposes. In other words, the guidance illustrates the dangers of blind reliance on ML and its inherent shortcomings. For instance, if a software program determines which patients or participants should be monitored or not, there could be serious consequences to the patient’s health. This also creates multiple sets of ethical dilemmas for clinicians and sponsors alike – for instance, to what degree should AI be entrusted with making such decisions? And what is the review process for the decisions it makes?
While AI has revolutionized and accelerated both drug and biologics development and has been welcomed with open arms by the industry. The FDAs draft guidance is a welcome step in the direction of carefully evaluating the risks associated with ML. One thing is for certain – human intervention will be required for the foreseeable future at least.
[1] Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products Guidance for Industry and Other Interested Parties DRAFT GUIDANCE U.S.
Department of Health and Human Services
Food and Drug Administration
Center for Drug Evaluation and Research (CDER)
Center for Biologics Evaluation and Research (CBER)
Center for Devices and Radiological Health (CDRH)
Center for Veterinary Medicine (CVM)
Oncology Center of Excellence (OCE)
Office of Combination Products (OCP)
Office of Inspections and Investigations (OII)
January 2025
Artificial Intelligence
[1] Id. At page 8
