Memory degradation induced by attention in recurrent neural architectures

This paper studies the memory mechanisms in recurrent neural architectures when attention models are included. Pure-attention models like Transformers are more and more popular as they tend to outperform models with recurrent connections in many different tasks. Our conjecture is that attention prevents the recurrent connections from transferring information properly between consecutive next steps. This conjecture is empirically tested using five different models, namely, a model without attention, a standard Loung attention model, a standard Bahdanau attention model, and our proposal to add attention to the inputs in order to fill the gap between recurrent and parallel architectures (for both Luong and Bahdanau attention models). Eight different problems are considered to assess the five models: a sequence-reverse copy problem, a sequence-reverse copy problem with repetitions, a filter sequence problem, a sequence-reverse copy problem with bigrams and four translation problems (English to Spanish, English to French, English to German and English to Italian). The achieved results reinforce our conjecture on the interaction between attention and recurrence

Neural Network-Based Calculator for Rat Glomerular Filtration Rate

Glomerular filtration is a pivotal process of renal physiology, and its alterations are a central pathological event in acute kidney injury and chronic kidney disease. Creatinine clearance (ClCr), a standard method for glomerular filtration rate (GFR) measurement, requires a long and tedious procedure of timed (usually 24 h) urine collection. We have developed a neural network (NN)-based calculator of rat ClCr from plasma creatinine (pCr) and body weight. For this purpose, matched pCr, weight, and ClCr trios from our historical records on male Wistar rats were used. When evaluated on the training (1165 trios), validation (389), and test sets (660), the model committed an average prediction error of 0.196, 0.178, and 0.203 mL/min and had a correlation coefficient of 0.863, 0.902, and 0.856, respectively. More importantly, for all datasets, the NN seemed especially effective at comparing ClCr among groups within individual experiments, providing results that were often more congruent than those measured experimentally. ACLARA, a friendly interface for this calculator, has been made publicly available to ease and expedite experimental procedures and to enhance animal welfare in alignment with the 3Rs principles by avoiding unnecessary stressing metabolic caging for individual urine collection