Wednesday 27 January 2027 | 11:30am - 13:00pm | Genome Dome

As AI adoption accelerates across genomics, pathology, imaging, and clinical diagnostics, ensuring trust in model outputs has become a critical challenge.

This interactive session explores the most common causes of AI failure in research. Participants will discuss how seemingly high-performing models can produce misleading or unreliable results when deployed in clinical settings and the practical approaches needed to build AI systems that are robust, reproducible, and clinically trustworthy.

While attention often focuses on model architecture and performance metrics, many AI failures originate much earlier in the development process through data leakage, hidden biases, poor dataset design, confounding variables, and inappropriate validation strategies, resulting in models that perform well during development but fail when deployed in real-world clinical settings. 

Trust in AI

Learning outcomes

  • Understand why strong AI performance in research does not always translate into clinical utility.  

  • Learn how hidden bias, confounding variables and poor validation can undermine medical AI.  

  • Explore practical strategies for developing robust AI systems that perform reliably across different healthcare settings.  

Who should attend

This workshop is designed for:

  • Researchers, data scientists, bioinformaticians, clinicians, laboratory leaders, digital health professionals and AI practitioners involved in developing, evaluating, or deploying AI within healthcare and life sciences. 

Session outline

  • Explore why strong model performance doesn’t always translate to clinical impact, including data leakage, validation challenges and poor generalisability.
  • Better understand how data leakage can inflate results and learn practical approaches to robust study design, data partitioning and external validation.
  • Examine how AI can learn unintended signals rather than clinically meaningful features, with examples from cancer imaging and real-world deployment.
  • Explore best practices for validation, governance, explainability and independent testing, alongside human-in-the-loop approaches that support clinical expertise.

Roundtable Hosts

Loading
Speaker profile image for Fazeleh Hoseini

Fazeleh Hoseini

AI Privacy Researcher & Project Lead
AI Sweden
Speaker profile image for David Leslie

David Leslie

Director of Ethics and Responsible Innovation Research
The Alan Turing Institute
Speaker profile image for Ian Simpson

Ian Simpson

Professor of Biomedical Informatics & Director UKRI Artificial Intelligence CDT in Biomedical Innovation
University of Edinburgh
Speaker profile image for Inga Prokopenko

Inga Prokopenko

Professor e-One Health and Head of Statistical Multi-Omics
University of Surrey
Speaker profile image for Shaheer Saeed

Shaheer Saeed

Lecturer in Computational Biomedical Engineering
Queen Mary University of London

Session 1 — Why AI can look better than it really is (20 minutes) 

  • Why high model accuracy does not always translate into clinical impact.  

  • Understanding the gap between benchmark performance and real-world deployment.  

  • Common misconceptions around model validation and generalisability.  

  • How data leakage can artificially inflate model performance and create a false sense of clinical readiness.  

  • Recognising when AI results may be "too good to be true."  

Session 2 — Data Leakage: The hidden cause of overoptimistic AI Performance (20 minutes)

  • What data leakage is and why it is one of the most common causes of inflated AI performance.  

  • Common sources of leakage in healthcare AI. 

  • Practical strategies for preventing data leakage through robust study design, appropriate data partitioning, external validation, and reproducible evaluation pipelines.  

Session 3 — Hidden Bias in Medical AI (20 minutes) 

  • How AI models can learn unintended signals instead of clinically meaningful features.  

  • Examples from cancer imaging, including scanner characteristics, acquisition protocols, and site-specific artefacts.  

  • Lessons learned from deploying AI.  

Session 4 — Building trustworthy AI in Biomedicine and Healthcare (20 minutes) 

  • Best practices for validation, governance, and independent testing.  

  • The role of external validation and leakage-aware evaluation before clinical deployment.  

  • The importance of explainability, transparency, and clinically meaningful outputs.  

  • Human-in-the-loop approaches that support, rather than replace, clinical expertise.