How good Neural Networks interpretation methods really are? A quantitative benchmark

Saliency Maps (SMs) have been extensively used to interpret deep learning models decision by highlighting the features deemed relevant by the model. They are used on highly nonlinear problems, where linear feature selection (FS) methods fail at highlighting relevant explanatory variables. However, the reliability of gradient-based feature attribution methods such as SM has mostly been only qualitatively (visually) assessed, and quantitative benchmarks are currently missing, partially due to the lack of a definite ground truth on image data. Concerned about the apophenic biases introduced by visual assessment of these methods, in this paper we propose a synthetic quantitative benchmark for Neural Networks (NNs) interpretation methods. For this purpose, we built synthetic datasets with nonlinearly separable classes and increasing number of decoy (random) features, illustrating the challenge of FS in high-dimensional settings. We also compare these methods to conventional approaches such as mRMR or Random Forests. Our results show that our simple synthetic datasets are sufficient to challenge most of the benchmarked methods. TreeShap, mRMR and LassoNet are the best performing FS methods. We also show that, when quantifying the relevance of a few non linearly-entangled predictive features diluted in a large number of irrelevant noisy variables, neural network-based FS and interpretation methods are still far from being reliable

Le document iconographique dans son contexte : le hors-champ des images du spectacle

International audienc

Le document iconographique dans son contexte : le hors-champ des images du spectacle

International audienc

Le document iconographique dans son contexte : le hors-champ des images du spectacle

International audienc

Dynamics of the Full Length and Mutated Heat Shock Factor 1 in Human Cells

Heat shock factor 1 is the key transcription factor of the heat shock response. Its function is to protect the cell against the deleterious effects of stress. Upon stress, HSF1 binds to and transcribes hsp genes and repeated satellite III (sat III) sequences present at the 9q12 locus. HSF1 binding to pericentric sat III sequences forms structures known as nuclear stress bodies (nSBs). nSBs represent a natural amplification of RNA pol II dependent transcription sites. Dynamics of HSF1 and of deletion mutants were studied in living cells using multi-confocal Fluorescence Correlation Spectroscopy (mFCS) and Fluorescence Recovery After Photobleaching (FRAP). In this paper, we show that HSF1 dynamics modifications upon heat shock result from both formation of high molecular weight complexes and increased HSF1 interactions with chromatin. These interactions involve both DNA binding with Heat Shock Element (HSE) and sat III sequences and a more transient sequence-independent binding likely corresponding to a search for more specific targets. We find that the trimerization domain is required for low affinity interactions with chromatin while the DNA binding domain is required for site-specifi

Sizes of HSF1 complex in unstressed and heat shocked HeLa cells using glycerol gradient fractionation.

<p>HSF1-eGFP and HSF1-K80Q-eGFP were estimated to 150 kD in NHS and superior to 669 kDa in HS. The complex size of HSF1- ΔTRIM-eGFP in NHS was estimated to 66–150 kDa in NHS and about 200 kDa in HS. The complex size of HSF1-ΔDBD-eGFP in NHS was estimated to 66–150 kDa in NHS and about 450 kDa in HS. TE: total extract before ultra-centrifugation. The fraction corresponding to standard proteins of masses 66, 200 and 669 kDa are indicated with arrows.</p

FRAP analysis of the WT HSF1 and mutants in unstressed and stressed cells.

<p>Fluorescence recovery curves after photobleaching of HSF1 full length-eGFP before (NHS) and after heat shock (HS) in the nucleus. WT HSF1-eGFP (red), HSF1-ΔTRIM-eGFP (green), HSF1-K80Q-eGFP (blue) and HSF1-ΔDBD-eGFP (purple).</p

Hsp70 or sat III RNA expression following heat shock.

<p>Inductions of hsp70 (A) and sat III transcripts (B) were quantified by RT-QPCR before (-) and after a one hour HS followed by 4 hour recovery (+), in the absence of HSF1 (Ctl) or in the presence of the different HSF1 constructions. The results are expressed as fold induction in comparison to NHS condition, after normalization with GAPDH mRNA. The values correspond to the mean values obtained out of 6 (HSF1, HSF1-eGFP), 5 (HSF1-K80Q) or 3 (HSF1-ΔTRIM) experiments. Error bars correspond to S.E.M. * Significant difference (P<0.05).</p

FRAP analysis of the WT HSF1 in and out nuclear stress bodies.

<p>Fluorescence recovery curves after photobleaching of HSF1 full length-eGFP before (NHS) and after heat shock (HS) in the nucleus outside nSBs (green and red) and in nSBs (orange).</p

Diffusion and interaction parameters determined by FRAP.

<p>Comparison between data obtained for HSF1 (WT) and HSF1 mutated in the trimerisation (ΔTRIM), DNA binding (ΔDBD) domain, punctual mutation in the DNA binding domain in presence (K80Q) or absence of endogenous HSF1(K80Q, siRNA).</p