The session aims to facilitate engagement across GA4GH groups, specifically the Clinical & Phenotypic Data Capture (Clin/Pheno) Work Stream and the Rare Disease Community of Interest, through active participation in a benchmarking exercise focused on evaluating the use of large language models (LLMs) for rare disease diagnostics. The primary goal is to assess how effectively LLMs can organise complex knowledge and support diagnostic processes, ultimately contributing to the development of advanced AI tools for this field.  

A suggested proposal is to design benchmarking data for systems that query a knowledge base (or graph) with an AI component, helping us assess how well LLMs use structured data to answer specific questions. This could include tasks such as mapping phenotypes to potential rare disease names. Exploration of this topic began at the joint Clin/Pheno and Rare Disease Community meeting in January 2025 (see here).