Imperfectly interoperable (IIO) data sets.

From the 3,192-patient CDSS-derived data set, we create two training sets with three levels of imperfect feature overlap (60, 80 and 90%) compared with perfect interoperability (100%). In our experiments, the owner of a small ‘target’ data set (fewer patients) wants to benefit from a larger ‘source’ data set without having access to this data. The ‘source’ may lack several features that are available in the ‘target’, yielding several levels of ‘imperfect interoperability’. We construct validation sets with and without these missing features, as well as a held-out test set. The F1 scores we report in this paper are averages over five randomized folds of this data-splitting procedure.</p

MoDN calibration curve of the predictions on the test set after having encoded all available features.

MoDN calibration curve of the predictions on the test set after having encoded all available features.</p

MoDN’s feature-wise predictive evolution in two random patients.

Each graph represents a single patient randomly selected from the test set. The y–axis lists the eight possible diagnoses predicted by our model. The true diagnosis of the patient is in bold and marked by an ‘*’. The x–axis is a sequential list of questions asked during the consultation (the response of that specific patient is also listed). In each case the model predicts the true label correctly. The heatmap represents a scale of predictive certainty from red (positive, has diagnosis) to blue (negative, does not have diagnosis), where white is uncertain. (a) Patient with the true diagnosis of pneumonia and anaemia. Here, predictive confidence accumulates slowly throughout the consultation. (b) Patient with a true diagnosis of FWS. Here, a confident prediction is achieved early after a highly determinant question of “fever only”. *: True diagnosis, URTI: Upper Respiratory Tract Infection, FWS: Fever Without Source, Threshold: probability at which the model categorises the patient with a diagnosis(50%).</p

Implementation details, feature decoding, idempotence and calibration.

Implementation details, feature decoding, idempotence and calibration.</p

Experimental set up for porting MoDN modules in IIO settings.

MoDN is tested in two “model porting experiments” (grey), where modules are ported from a larger source data set (A) for fine-tuning or updating on a smaller, imperfectly interoperable target data set (B). The two experiments represent either a scenario where a new user with different resources starts using a CDSS or where an existing user gains new resources and would like to merge training. Three baselines are proposed. Static (blue) where modules trained in A are directly tested in B, thus not considering additional IIO features. Local (green) where modules are only trained on the target data set B, thus without insights from the larger source data set. Global (purple) is the ideal but unlikely, scenario of when all data can be shared between A and B and the modules are trained on the union of data (A ∪ B). The modularised fine-tuning experiment, pre-trains on A and then fine-tunes all modules (for all features) on B (thus personalising the modules trained on A). The modularised update experiment, pre-trains the blue modules on A and then adds modules specific to the new IIO features (in green) which have been independently trained on B (thus preserving the validity of the modules trained on A). The colors of the MoDN modules illustrate their training on distinct data sets and their potential re-combination in the porting experiments. In particular, the modules trained on A (blue) and fine-tuned on B (green) are thus depicted in teal.</p

AUC-ROC curves for diagnosis.

Mean and standard deviation of the AUC and ROC curve for diagnosis prediction computed on the test set, using bootstrapping. (TIFF)</p

Comparison between the ported models and the baselines.

Performance metric is the mean macro F1 scores with 95% CIs. Modularised fine-tuning or updating on additional local features (gray) consistently increases the model’s performance compared to statically using a source model that only uses shared features (teal). The modularised update scenario achieves this without changing the model’s behaviour on patients in the source dataset. The fine-tuning approaches perform almost as well as the global baseline (purple) that trains on the union of shared data. When the percentage of shared features is 80 or 100%, fine-tuning is significantly better than training only locally on the small ‘target’ dataset (green).</p

Macro F1 scores for the disease prediction on test set, when the model is additionally trained to perform feature decoding.

The baselines of MLP and logistic regression with L2 penalty were tuned to achieve maximal performance. MoDN outperforms the baselines significantly for the overall disease prediction. Furthermore, it outperforms the performance of at least one of the baselines for each of the individual diseases, except for pneumonia. (TIFF)</p

Summary statistics of the features of the e-POCT data set.

Summary statistics of the features of the e-POCT data set.</p

Two dimensional t-SNE decomposition of the <i>state</i> vector for the patients of the training set.

The projection for each data point is overlaid with a color representing the true diagnosis/diagnoses of the patients. URTI: Upper Respiratory Tract Infection, FWS: Fever Without Source.</p