Patient inclusion/exclusion criteria, reproduced from [34].

Patient inclusion/exclusion criteria, reproduced from [34].</p

Appendix A: Think aloud protocol.

Machine learning tools are increasingly used to improve the quality of care and the soundness of a treatment plan. Explainable AI (XAI) helps users in understanding the inner mechanisms of opaque machine learning models and is a driver of trust and adoption. Explanation methods for black-box models exist, but there is a lack of user studies on the interpretability of the provided explanations. We used a Think Aloud Protocol (TAP) to explore oncologists’ assessment of a lung cancer relapse prediction system with the aim of refining the purpose-built explanation model for better credibility and utility. Novel to this context, TAP is used as a neutral methodology to elicit experts’ thought processes and judgements of the AI system, without explicit prompts. TAP aims to elicit the factors which influenced clinicians’ perception of credibility and usefulness of the system. Ten oncologists took part in the study. We conducted a thematic analysis of their verbalized responses, generating five themes that help us to understand the context within which oncologists’ may (or may not) integrate an explainable AI system into their working day.</div

Full features list obtained including training features, identification, label, and features with missing values filtered out before training.

Full features list obtained including training features, identification, label, and features with missing values filtered out before training.</p

Results of a literature search on PubMed, on the use of the think aloud protocol to assess clinical decision making.

Results of a literature search on PubMed, on the use of the think aloud protocol to assess clinical decision making.</p

Variables significantly associated with survival at 10 years.

<p>Variables significantly associated with survival at 10 years.</p

Overall survival (in months) according to performance status ECOG (B) and FLIPI score at diagnosis (C) for all patients (n = 1074).

<p>Overall survival (in months) according to performance status ECOG (B) and FLIPI score at diagnosis (C) for all patients (n = 1074).</p

Patient characteristics.

<p>Patient characteristics.</p

Results of univariate analysis of different prognostic factors in the whole population of 1074 patients with follicular lymphoma (FL).

<p>The table shows the statistically significant prognostic factors.</p

Correlation between group of age and HR value, obtained by Cox model.

<p>HR indicates hazard risk and R²explains the great correlation between HR and age, with less than a 5% error.</p

Multivariate analysis: Characteristics associated with overall survival.

<p>Multivariate analysis: Characteristics associated with overall survival.</p