Comparing AI language models with tumour boards on cancer treatment plans

AI summary

From the trial's eligibility criteria. Machine-read from the registry text on this page, so it can be wrong. The registry text below and the trial team decide who can take part.

This study uses made-up (synthetic) cancer cases, not real patients. It compares treatment plans from AI language models with plans from tumour boards. The cases cover breast, lung, urological, digestive and gynaecological cancers.

Age

Age 18 years or older.

Sex

All sexes.

Disease stage

Cases must use UICC 8th-edition stage and cover one of five cancer groups: breast, lung, urological, digestive or gynaecological.

Biomarker or test result

Each case must list its biomarkers, ECOG performance status and other illnesses in a standard format.

Main reasons for not taking part

• Case outside the five cancer groups
• Incomplete, inconsistent or unclear case description
• Duplicate or near-duplicate of another case
• Question that current guidelines cannot answer

The registry criteria, word for word

What the trial wants to find out

BEACON (Benchmarking AI for Clinical Oncology decisioNmaking) is a prospective, multicentre, comparative, blinded, non-interventional benchmark comparing the treatment recommendations of six frontier large language models (LLMs) with those of multidisciplinary tumour boards (RCP) in oncology treatment planning. One hundred standardised synthetic cases (20 per localisation, across breast, lung, urological, digestive and gynaecological cancers) are submitted as identical structured input to two independent tumour boards per localisation and to six frontier LLMs. Every recommendation, human or model, is decomposed into five predefined decision domains (intent, surgery, radiotherapy, systemic therapy with its line, work-up and biomarkers). For each domain, concordance between two recommendations is scored 2 when the same options are selected, 1 when at least one option is shared, and 0 when none is shared; the score is computed automatically from the recommendation forms, without human scoring. The primary endpoint is the mean domain score between each LLM and each tumour board, read against the same score computed between the two tumour boards of the same case, so that humans and models are measured by an identical rule and no respondent contributes to its own reference. A co-primary safety endpoint captures the proportion of recommendations carrying serious harm potential, because concordance alone can conceal dangerous errors. Where the two boards select different options, the case-domain is classified EQUIPOISE and, like every discordance, is adjudicated by an independent panel blinded to the source of the recommendation. Tumour boards record their recommendations during the meeting itself, on a dedicated platform (https://etude-beacon.fr) that reveals a case only once it is presented and time-stamps that reveal, so that no board can read a case in advance. BEACON is designed as reusable, openly licensed, pre-registered infrastructure: all synthetic cases, evaluation rubrics, scoring algorithms, verbatim prompts and raw model outputs are released for full reproducibility.

The 5 groups in the trial

• Breast cancers

  Treatment: Multidisciplinary tumour boards and Frontier large language models

• Lung cancers

  Treatment: Multidisciplinary tumour boards and Frontier large language models

• Urological cancers

  Treatment: Multidisciplinary tumour boards and Frontier large language models

• Digestive cancers

  Treatment: Multidisciplinary tumour boards and Frontier large language models

• Gynaecological cancers

  Treatment: Multidisciplinary tumour boards and Frontier large language models