Comparing AI language models with tumour boards on cancer treatment plans
Compares AI language models with expert tumour boards on treatment plans for synthetic cancer cases.
Plain-language title generated from the registry record
Registry title: Benchmarking Large Language Models Against Tumour Boards for Oncology Treatment Recommendations
Key facts
- Where
- Boulogne-Billancourt and Paris, FranceSee the 5 sites
Where in France
- Hôpital Ambroise Paré AP-HPBoulogne-Billancourt, FranceNot yet recruiting
- Hôpital Cochin AP-HPParis, FranceNot yet recruiting
- Hôpital Européen Georges PompidouParis, FranceTaking people
- Hôpital Saint Antoine AP-HPParis, FranceNot yet recruiting
- Hôpital TenonParis, FranceNot yet recruiting
Questions to ask the study team
- How often do I need to come in, and for how long each time?
- Are travel costs paid?
- Can I keep my own doctor?
- What happens if I want to stop?
Trial details
- Who runs it
- Assistance Publique - Hôpitaux de Paris
- Trial code
- NCT07739121
- Record updated
- 7 October 2026
- Original description
BEACON (Benchmarking AI for Clinical Oncology decisioNmaking) is a prospective, multicentre, comparative, blinded, non-interventional benchmark comparing the treatment recommendations of six frontier large language models (LLMs) with those of multidisciplinary tumour boards (RCP) in oncology treatment planning. One hundred standardised synthetic cases (20 per localisation, across breast, lung, urological, digestive and gynaecological cancers) are submitted as identical structured input to two independent tumour boards per localisation and to six frontier LLMs. Every recommendation, human or model, is decomposed into five predefined decision domains (intent, surgery, radiotherapy, systemic therapy with its line, work-up and biomarkers). For each domain, concordance between two recommendations is scored 2 when the same options are selected, 1 when at least one option is shared, and 0 when none is shared; the score is computed automatically from the recommendation forms, without human scoring. The primary endpoint is the mean domain score between each LLM and each tumour board, read against the same score computed between the two tumour boards of the same case, so that humans and models are measured by an identical rule and no respondent contributes to its own reference. A co-primary safety endpoint captures the proportion of recommendations carrying serious harm potential, because concordance alone can conceal dangerous errors. Where the two boards select different options, the case-domain is classified EQUIPOISE and, like every discordance, is adjudicated by an independent panel blinded to the source of the recommendation. Tumour boards record their recommendations during the meeting itself, on a dedicated platform (https://etude-beacon.fr) that reveals a case only once it is presented and time-stamps that reveal, so that no board can read a case in advance. BEACON is designed as reusable, openly licensed, pre-registered infrastructure: all synthetic cases, evaluation rubrics, scoring algorithms, verbatim prompts and raw model outputs are released for full reproducibility.Read full original descriptionShow less
Benchmarking AI for Clinical Oncology decisioNmaking (BEACON): A Prospective, Multicentre, Blinded Evaluation of Frontier Large Language Models Against Multidisciplinary Tumour Board Recommendations in Oncology Treatment Planning
BEACON (Benchmarking AI for Clinical Oncology decisioNmaking) is a prospective, multicentre, comparative, blinded, non-interventional benchmark comparing the treatment recommendations of six frontier large language models (LLMs) with those of multidisciplinary tumour boards (RCP) in oncology treatment planning. One hundred standardised synthetic cases (20 per localisation, across breast, lung, urological, digestive and gynaecological cancers) are submitted as identical structured input to two independent tumour boards per localisation and to six frontier LLMs. Every recommendation, human or model, is decomposed into five predefined decision domains (intent, surgery, radiotherapy, systemic therapy with its line, work-up and biomarkers). For each domain, concordance between two recommendations is scored 2 when the same options are selected, 1 when at least one option is shared, and 0 when none is shared; the score is computed automatically from the recommendation forms, without human scoring. The primary endpoint is the mean domain score between each LLM and each tumour board, read against the same score computed between the two tumour boards of the same case, so that humans and models are measured by an identical rule and no respondent contributes to its own reference. A co-primary safety endpoint captures the proportion of recommendations carrying serious harm potential, because concordance alone can conceal dangerous errors. Where the two boards select different options, the case-domain is classified EQUIPOISE and, like every discordance, is adjudicated by an independent panel blinded to the source of the recommendation. Tumour boards record their recommendations during the meeting itself, on a dedicated platform (https://etude-beacon.fr) that reveals a case only once it is presented and time-stamps that reveal, so that no board can read a case in advance. BEACON is designed as reusable, openly licensed, pre-registered infrastructure: all synthetic cases, evaluation rubrics, scoring algorithms, verbatim prompts and raw model outputs are released for full reproducibility.
Inclusion: Synthetic oncology case within one of the five predefined localisations (breast, lung, urological, digestive, gynaecological); Complete structured schema: UICC 8th-edition stage, biomarkers, ECOG performance status, comorbidities and a standardised clinical question; A clinically answerable treatment-planning question that is mappable to the locked guideline matrix.
Exclusion: Case outside the five predefined localisations; Incomplete, internally inconsistent or ambiguous schema; Duplicate or near-duplicate of an existing case in the set; Question not resolvable by current guidelines.