Deriving Temporal Prototypes from Saliency Map Clusters for the Analysis of Deep-Learning-based Facial Action Unit Classification

Reliably determining the emotional state of a person is a difficult task for both humans as well as machines. Automatic detection and evaluation of facial expressions is particularly important if people are unable to express their emotional state themselves, for example due to cognitive impairments. Identifying the presence of Action Units in a human’s face is a psychologically validated approach of quantifying which emotion is expressed. To automate the detection process of Action Units Neural Networks have been trained. However, the black-box nature of Deep Neural Networks provides no insight on the relevant features identified during the decision process. Approaches of Explainable Artificial Intelligence have to be applied to provide an explanation why the network came to a certain conclusion. In this work "Layer-Wise Relevance Propagation" (LRP) in combination with the meta analysis approach "Spectral Relevance Analysis" (SpRAy) is used to derive temporal prototypes from predictions in video sequences. Temporal prototypes provide an aggregated view on the prediction of the network by grouping together similar frames by considering relevance. Additionally, a specific visualization method for temporal prototypes is presented that highlights the most relevant areas for a prediction of an Action Unit. A quantitative evaluation of our approach shows that temporal prototypes aggregate temporal information well. The proposed method can be used to generate concise visual explanations for a sequence of interpretable saliency maps. Based on the above, this work shall provide the foundation for a new temporal analysis method as well as an explanation approach that is supposed to help researchers and experts to gain a deeper understanding of how the underlying network decides which Action Units are active in a particular emotional state

Verifying Deep Learning-based Decisions for Facial Expression Recognition

Neural networks with high performance can still be biased towards non-relevant features. However, reliability and robustness is especially important for high-risk fields such as clinical pain treatment. We therefore propose a verification pipeline, which consists of three steps. First, we classify facial expressions with a neural network. Next, we apply layer-wise relevance propagation to create pixel-based explanations. Finally, we quantify these visual explanations based on a bounding-box method with respect to facial regions. Although our results show that the neural network achieves state-of-the-art results, the evaluation of the visual explanations reveals that relevant facial regions may not be considered.Comment: accepted at ESANN 202

A multicenter randomized-controlled trial of hypothermic oxygenated perfusion (HOPE) for human liver grafts before transplantation

Background &amp; Aims: Machine perfusion is a novel method intended to optimize livers before transplantation. However, its effect on morbidity within a 1-year period after transplantation has remained unclear. Methods: In this multicenter controlled trial, we randomly assigned livers donated after brain death (DBD) for liver transplantation (LT). Livers were either conventionally cold stored (control group), or cold stored and subsequently treated by 1-2 h hypothermic oxygenated perfusion (HOPE) before implantation (HOPE group). The primary endpoint was the occurrence of at least one post-transplant complication per patient, graded by the Clavien score of ≥III, within 1-year after LT. The comprehensive complication index (CCI), laboratory parameters, as well as duration of hospital and intensive care unit stay, graft survival, patient survival, and biliary complications served as secondary endpoints. Results: Between April 2015 and August 2019, we randomized 177 livers, resulting in 170 liver transplantations (85 in the HOPE group and 85 in the control group). The number of patients with at least one Clavien ≥III complication was 46/85 (54.1%) in the control group and 44/85 (51.8%) in the HOPE group (odds ratio 0.91; 95% CI 0.50-1.66; p = 0.76). Secondary endpoints were also not significantly different between groups. A post hoc analysis revealed that liver-related Clavien ≥IIIb complications occurred less frequently in the HOPE group compared to the control group (risk ratio 0.26; 95% CI 0.07-0.77; p = 0.027). Likewise, graft failure due to liver-related complications did not occur in the HOPE group, but occurred in 7% (6 of 85) of the control group (log-rank test, p = 0.004, Gray test, p = 0.015). Conclusions: HOPE after cold storage of DBD livers resulted in similar proportions of patients with at least one Clavien ≥III complication compared to controls. Exploratory findings suggest that HOPE decreases the risk of severe liver graft-related events. Impact and implications: This randomized controlled phase III trial is the first to investigate the impact of hypothermic oxygenated perfusion (HOPE) on cumulative complications within a 12-month period after liver transplantation. Compared to conventional cold storage, HOPE did not have a significant effect on the number of patients with at least one Clavien ≥III complication. However, we believe that HOPE may have a beneficial effect on the quantity of complications per patient, based on its application leading to fewer severe liver graft-related complications, and to a lower risk of liver-related graft loss. The HOPE approach can be applied easily after organ transport during recipient hepatectomy. This appears fundamental for wide acceptance since concurring perfusion technologies need either perfusion at donor sites or continuous perfusion during organ transport, which are much costlier and more laborious. We conclude therefore that the post hoc findings of this trial should be further validated in future studies.</p

A multicenter randomized-controlled trial of hypothermic oxygenated perfusion (HOPE) for human liver grafts before transplantation

Background &amp; Aims: Machine perfusion is a novel method intended to optimize livers before transplantation. However, its effect on morbidity within a 1-year period after transplantation has remained unclear. Methods: In this multicenter controlled trial, we randomly assigned livers donated after brain death (DBD) for liver transplantation (LT). Livers were either conventionally cold stored (control group), or cold stored and subsequently treated by 1-2 h hypothermic oxygenated perfusion (HOPE) before implantation (HOPE group). The primary endpoint was the occurrence of at least one post-transplant complication per patient, graded by the Clavien score of ≥III, within 1-year after LT. The comprehensive complication index (CCI), laboratory parameters, as well as duration of hospital and intensive care unit stay, graft survival, patient survival, and biliary complications served as secondary endpoints. Results: Between April 2015 and August 2019, we randomized 177 livers, resulting in 170 liver transplantations (85 in the HOPE group and 85 in the control group). The number of patients with at least one Clavien ≥III complication was 46/85 (54.1%) in the control group and 44/85 (51.8%) in the HOPE group (odds ratio 0.91; 95% CI 0.50-1.66; p = 0.76). Secondary endpoints were also not significantly different between groups. A post hoc analysis revealed that liver-related Clavien ≥IIIb complications occurred less frequently in the HOPE group compared to the control group (risk ratio 0.26; 95% CI 0.07-0.77; p = 0.027). Likewise, graft failure due to liver-related complications did not occur in the HOPE group, but occurred in 7% (6 of 85) of the control group (log-rank test, p = 0.004, Gray test, p = 0.015). Conclusions: HOPE after cold storage of DBD livers resulted in similar proportions of patients with at least one Clavien ≥III complication compared to controls. Exploratory findings suggest that HOPE decreases the risk of severe liver graft-related events. Impact and implications: This randomized controlled phase III trial is the first to investigate the impact of hypothermic oxygenated perfusion (HOPE) on cumulative complications within a 12-month period after liver transplantation. Compared to conventional cold storage, HOPE did not have a significant effect on the number of patients with at least one Clavien ≥III complication. However, we believe that HOPE may have a beneficial effect on the quantity of complications per patient, based on its application leading to fewer severe liver graft-related complications, and to a lower risk of liver-related graft loss. The HOPE approach can be applied easily after organ transport during recipient hepatectomy. This appears fundamental for wide acceptance since concurring perfusion technologies need either perfusion at donor sites or continuous perfusion during organ transport, which are much costlier and more laborious. We conclude therefore that the post hoc findings of this trial should be further validated in future studies.</p

Biological effects of sirodesmin pl, a phytotoxin produced by Leptosphaeria maculans

International audienc

Deriving Temporal Prototypes from Saliency Map Clusters for the Analysis of Deep-Learning-based Facial Action Unit Classification

Reliably determining the emotional state of a person is a difficult task for both humans as well as machines. Automatic detection and evaluation of facial expressions is particularly important if people are unable to express their emotional state themselves, for example due to cognitive impairments. Identifying the presence of Action Units in a human’s face is a psychologically validated approach of quantifying which emotion is expressed. To automate the detection process of Action Units Neural Networks have been trained. However, the black-box nature of Deep Neural Networks provides no insight on the relevant features identified during the decision process. Approaches of Explainable Artificial Intelligence have to be applied to provide an explanation why the network came to a certain conclusion. In this work "Layer-Wise Relevance Propagation" (LRP) in combination with the meta analysis approach "Spectral Relevance Analysis" (SpRAy) is used to derive temporal prototypes from predictions in video sequences. Temporal prototypes provide an aggregated view on the prediction of the network by grouping together similar frames by considering relevance. Additionally, a specific visualization method for temporal prototypes is presented that highlights the most relevant areas for a prediction of an Action Unit. A quantitative evaluation of our approach shows that temporal prototypes aggregate temporal information well. The proposed method can be used to generate concise visual explanations for a sequence of interpretable saliency maps. Based on the above, this work shall provide the foundation for a new temporal analysis method as well as an explanation approach that is supposed to help researchers and experts to gain a deeper understanding of how the underlying network decides which Action Units are active in a particular emotional state

A multicenter randomized-controlled trial of hypothermic oxygenated perfusion (HOPE) for human liver grafts before transplantation

BACKGROUND & AIMS Machine perfusion is a novel method intended to optimize livers before transplantation. However, its effect on morbidity within a 1-year period after transplantation has remained unclear. METHODS In this multicenter controlled trial, we randomly assigned livers donated after brain death (DBD) for liver transplantation (LT). Livers were either conventionally cold stored (control group), or cold stored and subsequently treated by 1-2 h hypothermic oxygenated perfusion (HOPE) before implantation (HOPE group). The primary endpoint was the occurrence of at least one post-transplant complication per patient, graded by the Clavien score of ≥III, within 1-year after LT. The comprehensive complication index (CCI), laboratory parameters, as well as duration of hospital and intensive care unit stay, graft survival, patient survival, and biliary complications served as secondary endpoints. RESULTS Between April 2015 and August 2019, we randomized 177 livers, resulting in 170 liver transplantations (85 in the HOPE group and 85 in the control group). The number of patients with at least one Clavien ≥III complication was 46/85 (54.1%) in the control group and 44/85 (51.8%) in the HOPE group (odds ratio 0.91; 95% CI 0.50-1.66; p = 0.76). Secondary endpoints were also not significantly different between groups. A post hoc analysis revealed that liver-related Clavien ≥IIIb complications occurred less frequently in the HOPE group compared to the control group (risk ratio 0.26; 95% CI 0.07-0.77; p = 0.027). Likewise, graft failure due to liver-related complications did not occur in the HOPE group, but occurred in 7% (6 of 85) of the control group (log-rank test, p = 0.004, Gray test, p = 0.015). CONCLUSIONS HOPE after cold storage of DBD livers resulted in similar proportions of patients with at least one Clavien ≥III complication compared to controls. Exploratory findings suggest that HOPE decreases the risk of severe liver graft-related events. IMPACT AND IMPLICATIONS This randomized controlled phase III trial is the first to investigate the impact of hypothermic oxygenated perfusion (HOPE) on cumulative complications within a 12-month period after liver transplantation. Compared to conventional cold storage, HOPE did not have a significant effect on the number of patients with at least one Clavien ≥III complication. However, we believe that HOPE may have a beneficial effect on the quantity of complications per patient, based on its application leading to fewer severe liver graft-related complications, and to a lower risk of liver-related graft loss. The HOPE approach can be applied easily after organ transport during recipient hepatectomy. This appears fundamental for wide acceptance since concurring perfusion technologies need either perfusion at donor sites or continuous perfusion during organ transport, which are much costlier and more laborious. We conclude therefore that the post hoc findings of this trial should be further validated in future studies

A multicenter randomized-controlled trial of hypothermic oxygenated perfusion (HOPE) for human liver grafts before transplantation

Background &amp; Aims: Machine perfusion is a novel method intended to optimize livers before transplantation. However, its effect on morbidity within a 1-year period after transplantation has remained unclear. Methods: In this multicenter controlled trial, we randomly assigned livers donated after brain death (DBD) for liver transplantation (LT). Livers were either conventionally cold stored (control group), or cold stored and subsequently treated by 1-2 h hypothermic oxygenated perfusion (HOPE) before implantation (HOPE group). The primary endpoint was the occurrence of at least one post-transplant complication per patient, graded by the Clavien score of ≥III, within 1-year after LT. The comprehensive complication index (CCI), laboratory parameters, as well as duration of hospital and intensive care unit stay, graft survival, patient survival, and biliary complications served as secondary endpoints. Results: Between April 2015 and August 2019, we randomized 177 livers, resulting in 170 liver transplantations (85 in the HOPE group and 85 in the control group). The number of patients with at least one Clavien ≥III complication was 46/85 (54.1%) in the control group and 44/85 (51.8%) in the HOPE group (odds ratio 0.91; 95% CI 0.50-1.66; p = 0.76). Secondary endpoints were also not significantly different between groups. A post hoc analysis revealed that liver-related Clavien ≥IIIb complications occurred less frequently in the HOPE group compared to the control group (risk ratio 0.26; 95% CI 0.07-0.77; p = 0.027). Likewise, graft failure due to liver-related complications did not occur in the HOPE group, but occurred in 7% (6 of 85) of the control group (log-rank test, p = 0.004, Gray test, p = 0.015). Conclusions: HOPE after cold storage of DBD livers resulted in similar proportions of patients with at least one Clavien ≥III complication compared to controls. Exploratory findings suggest that HOPE decreases the risk of severe liver graft-related events. Impact and implications: This randomized controlled phase III trial is the first to investigate the impact of hypothermic oxygenated perfusion (HOPE) on cumulative complications within a 12-month period after liver transplantation. Compared to conventional cold storage, HOPE did not have a significant effect on the number of patients with at least one Clavien ≥III complication. However, we believe that HOPE may have a beneficial effect on the quantity of complications per patient, based on its application leading to fewer severe liver graft-related complications, and to a lower risk of liver-related graft loss. The HOPE approach can be applied easily after organ transport during recipient hepatectomy. This appears fundamental for wide acceptance since concurring perfusion technologies need either perfusion at donor sites or continuous perfusion during organ transport, which are much costlier and more laborious. We conclude therefore that the post hoc findings of this trial should be further validated in future studies.</p