A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification

Reliable application of machine learning-based decision systems in the wild is one of the major challenges currently investigated by the field. A large portion of established approaches aims to detect erroneous predictions by means of assigning confidence scores. This confidence may be obtained by either quantifying the model's predictive uncertainty, learning explicit scoring functions, or assessing whether the input is in line with the training distribution. Curiously, while these approaches all state to address the same eventual goal of detecting failures of a classifier upon real-life application, they currently constitute largely separated research fields with individual evaluation protocols, which either exclude a substantial part of relevant methods or ignore large parts of relevant failure sources. In this work, we systematically reveal current pitfalls caused by these inconsistencies and derive requirements for a holistic and realistic evaluation of failure detection. To demonstrate the relevance of this unified perspective, we present a large-scale empirical study for the first time enabling benchmarking confidence scoring functions w.r.t all relevant methods and failure sources. The revelation of a simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation in the abundance of publicized research on confidence scoring. Code and trained models are at https://github.com/IML-DKFZ/fd-shifts

Quantitative tandem affinity purification, an effective tool to investigate protein complex composition in plant hormone signaling : strigolactones in the spotlight

Phytohormones tightly regulate plant growth by integrating changing environmental and developmental cues. Although the key players have been identified in many plant hormonal pathways, the molecular mechanisms and mode of action of perception and signaling remain incompletely resolved. Characterization of protein partners of known signaling components provides insight into the formed protein complexes, but, unless quantification is involved, does not deliver much, if any, information about the dynamics of the induced or disrupted protein complexes. Therefore, in proteomics research, the discovery of what actually triggers, regulates or interrupts the composition of protein complexes is gaining importance. Here, tandem affinity purification coupled to mass spectrometry (TAP-MS) is combined with label-free quantification (LFQ) to a highly valuable tool to detect physiologically relevant, dynamic protein-protein interactions in Arabidopsis thaliana cell cultures. To demonstrate its potential, we focus on the signaling pathway of one of the most recently discovered phytohormones, strigolactones

Effect of Error Augmentation on Brain Activation and Motor Learning of a Complex Locomotor Task

Up to date, the functional gains obtained after robot-aided gait rehabilitation training are limited. Error augmenting strategies have a great potential to enhance motor learning of simple motor tasks. However, little is known about the effect of these error modulating strategies on complex tasks, such as relearning to walk after a neurologic accident. Additionally, neuroimaging evaluation of brain regions involved in learning processes could provide valuable information on behavioral outcomes. We investigated the effect of robotic training strategies that augment errors—error amplification and random force disturbance—and training without perturbations on brain activation and motor learning of a complex locomotor task. Thirty-four healthy subjects performed the experiment with a robotic stepper (MARCOS) in a 1.5 T MR scanner. The task consisted in tracking a Lissajous figure presented on a display by coordinating the legs in a gait-like movement pattern. Behavioral results showed that training without perturbations enhanced motor learning in initially less skilled subjects, while error amplification benefited better-skilled subjects. Training with error amplification, however, hampered transfer of learning. Randomly disturbing forces induced learning and promoted transfer in all subjects, probably because the unexpected forces increased subjects' attention. Functional MRI revealed main effects of training strategy and skill level during training. A main effect of training strategy was seen in brain regions typically associated with motor control and learning, such as, the basal ganglia, cerebellum, intraparietal sulcus, and angular gyrus. Especially, random disturbance and no perturbation lead to stronger brain activation in similar brain regions than error amplification. Skill-level related effects were observed in the IPS, in parts of the superior parietal lobe (SPL), i.e., precuneus, and temporal cortex. These neuroimaging findings indicate that gait-like motor learning depends on interplay between subcortical, cerebellar, and fronto-parietal brain regions. An interesting observation was the low activation observed in the brain's reward system after training with error amplification compared to training without perturbations. Our results suggest that to enhance learning of a locomotor task, errors should be augmented based on subjects' skill level. The impacts of these strategies on motor learning, brain activation, and motivation in neurological patients need further investigation

Distribution pattern following systemic mesenchymal stem cell injection depends on the age of the recipient and neuronal health

BACKGROUND: Mesenchymal stem cells (MSCs) show therapeutic efficacy in many different age-related degenerative diseases, including Alzheimer's disease. Very little is currently known about whether or not aging impacts the transplantation efficiency of MSCs. METHODS: In this study, we investigated the distribution of intravenously transplanted syngeneic MSCs derived from young and aged mice into young, aged, and transgenic APP/PS1 Alzheimer's disease mice. MSCs from male donors were transplanted into female mice and their distribution pattern was monitored by PCR using Y-chromosome specific probes. Biodistribution of transplanted MSCs in the brains of APP/PS1 mice was additionally confirmed by immunofluorescence and confocal microscopy. RESULTS: Four weeks after transplantation into young mice, young MSCs were found in the lung, axillary lymph nodes, blood, kidney, bone marrow, spleen, liver, heart, and brain cortex. In contrast, young MSCs that were transplanted into aged mice were only found in the brain cortex. In both young and aged mouse recipients, transplantation of aged MSCs showed biodistribution only in the blood and spleen. Although young transplanted MSCs only showed neuronal distribution in the brain cortex in young mice, they exhibited a wide neuronal distribution pattern in the brains of APP/PS1 mice and were found in the cortex, cerebellum, hippocampus, olfactory bulb, and brainstem. The immunofluorescent signal of both transplanted MSCs and resident microglia was robust in the brains of APP/PS1 mice. Monocyte chemoattractant-1 levels were lowest in the brain cortex of young mice and were significantly increased in APP/PS1 mice. Within the hippocampus, monocyte chemoattractant-1 levels were significantly higher in aged mice compared with younger and APP/PS1 mice. CONCLUSIONS: We demonstrate in vivo that MSC biodistribution post transplantation is detrimentally affected by aging and neuronal health. Aging of both the recipient and the donor MSCs used attenuates transplantation efficiency. Clinically, our data would suggest that aged MSCs should not be used for transplantation and that transplantation of MSCs into aged patients will be less efficacious

Unraveling the MAX2 protein network in Arabidopsis thaliana : identification of the protein phosphatase PAPP5 as a novel MAX2 interactor

The F-box protein MORE AXILLARY GROWTH 2 (MAX2) is a central component in the signaling cascade of strigolactones (SLs) as well as of the smoke-derived karrikins (KARs) and the so far unknown endogenous KAI2 ligand (KL). The two groups of molecules are involved in overlapping and unique developmental processes, and signal-specific outcomes are attributed to perception by the paralogous α/β-hydrolases DWARF14 (D14) for SL and KARRIKIN INSENSITIVE 2/HYPOSENSITIVE TO LIGHT (KAI2/HTL) for KAR/KL. In addition, depending on which receptor is activated, specific members of the SUPPRESSOR OF MAX2 1 (SMAX1)-LIKE (SMXL) family control KAR/KL and SL responses. As proteins that function in the same signal transduction pathway often occur in large protein complexes, we aimed at discovering new players of the MAX2, D14, and KAI2 protein network by tandem affinity purification in Arabidopsis cell cultures. When using MAX2 as a bait, various proteins were copurified, among which were general components of the Skp1-Cullin-F-box complex and members of the CONSTITUTIVE PHOTOMORPHOGENIC 9 signalosome. Here, we report the identification of a novel interactor of MAX2, a type 5 serine/threonine protein phosphatase, designated PHYTOCHROME-ASSOCIATED PROTEIN PHOSPHATASE 5 (PAPP5). Quantitative affinity purification pointed at PAPP5 as being more present in KAI2 rather than in D14 protein complexes. In agreement, mutant analysis suggests that PAPP5 modulates KAR/KL-dependent seed germination under suboptimal conditions and seedling development. In addition, a phosphopeptide enrichment experiment revealed that PAPP5 might dephosphorylate MAX2 in vivo independently of the synthetic SL analog, rac-GR24. Together, by analyzing the protein complexes to which MAX2, D14, and KAI2 belong, we revealed a new MAX2 interactor, PAPP5, that might act through dephosphorylation of MAX2 to control mainly KAR/KL-related phenotypes and, hence, provide another link with the light pathway

Health & Demographic Surveillance System Profile: The Taabo Health and Demographic Surveillance System, Côte d'Ivoire

The Taabo Health and Demographic Surveillance System (HDSS) is located in south-central Côte d'Ivoire, approximately 150 km north-west of Abidjan. The Taabo HDSS started surveillance activities in early 2009 and the man-made Lake Taabo is a key eco-epidemiological feature. Since inception, there has been a strong interest in research and integrated control of water-associated diseases such as schistosomiasis and malaria. The Taabo HDSS has generated setting-specific evidence on the impact of targeted interventions against malaria, schistosomiasis and other neglected tropical diseases. The Taabo HDSS consists of a small town, 13 villages and over 100 hamlets. At the end of 2013, a total population of 42 480 inhabitants drawn from 6707 households was under surveillance. Verbal autopsies have been conducted to determine causes of death. Repeated cross-sectional epidemiological surveys on approximately 5-7% of the population and specific, layered-on haematological, parasitological and questionnaire surveys have been conducted. The Taabo HDSS provides a database for surveys, facilitates interdisciplinary research, as well as surveillance, and provides a platform for the evaluation of health interventions. Requests to collaborate and to access data are welcome and should be addressed to the secretariat of the Centre Suisse de Recherches Scientifiques en Côte d'Ivoire: [[email protected]

Polarization-independent mode-evolution-based coupler for the silicon-on-insulator platform

We demonstrate a polarization-independent mode-evolution-based coupler for the silicon-on-insulator platform. The measured coupler has negligible insertion loss over a bandwidth of about 100 nm, i.e., from 1500 to 1600 nm. The measured maximum power imbalances for the polarization-independent coupler are 1.2 and 0.2 dB for the fundamental transverse electric (TE00) mode and the fundamental transverse magnetic (TM00) mode, respectively. Our coupler also has a compact design footprint with mode-evolution region not more than 75−μm long

65 GOPS/neuron Photonic Tensor Core with Thin-film Lithium Niobate Photonics

Photonics offers a transformative approach to artificial intelligence (AI) and neuromorphic computing by providing low latency, high bandwidth, and energy-efficient computations. Here, we introduce a photonic tensor core processor enabled by time-multiplexed inputs and charge-integrated outputs. This fully integrated processor, comprising only two thin-film lithium niobate (TFLN) modulators, a III-V laser, and a charge-integration photoreceiver, can implement an entire layer of a neural network. It can execute 65 billion operations per second (GOPS) per neuron, including simultaneous weight updates-a hitherto unachieved speed. Our processor stands out from conventional photonic processors, which have static weights set during training, as it supports fast "hardware-in-the-loop" training, and can dynamically adjust the inputs (fan-in) and outputs (fan-out) within a layer, thereby enhancing its versatility. Our processor can perform large-scale dot-product operations with vector dimensions up to 131,072. Furthermore, it successfully classifies (supervised learning) and clusters (unsupervised learning) 112*112-pixel images after "hardware-in-the-loop" training. To handle "hardware-in-the-loop" training for clustering AI tasks, we provide a solution for multiplications involving two negative numbers based on our processor.Comment: 19 pages, 6 figure