Hierarchical Context enabled Recurrent Neural Network for Recommendation

A long user history inevitably reflects the transitions of personal interests over time. The analyses on the user history require the robust sequential model to anticipate the transitions and the decays of user interests. The user history is often modeled by various RNN structures, but the RNN structures in the recommendation system still suffer from the long-term dependency and the interest drifts. To resolve these challenges, we suggest HCRNN with three hierarchical contexts of the global, the local, and the temporary interests. This structure is designed to withhold the global long-term interest of users, to reflect the local sub-sequence interests, and to attend the temporary interests of each transition. Besides, we propose a hierarchical context-based gate structure to incorporate our \textit{interest drift assumption}. As we suggest a new RNN structure, we support HCRNN with a complementary \textit{bi-channel attention} structure to utilize hierarchical context. We experimented the suggested structure on the sequential recommendation tasks with CiteULike, MovieLens, and LastFM, and our model showed the best performances in the sequential recommendations

Adversarial Dropout for Recurrent Neural Networks

Successful application processing sequential data, such as text and speech, requires an improved generalization performance of recurrent neural networks (RNNs). Dropout techniques for RNNs were introduced to respond to these demands, but we conjecture that the dropout on RNNs could have been improved by adopting the adversarial concept. This paper investigates ways to improve the dropout for RNNs by utilizing intentionally generated dropout masks. Specifically, the guided dropout used in this research is called as adversarial dropout, which adversarially disconnects neurons that are dominantly used to predict correct targets over time. Our analysis showed that our regularizer, which consists of a gap between the original and the reconfigured RNNs, was the upper bound of the gap between the training and the inference phases of the random dropout. We demonstrated that minimizing our regularizer improved the effectiveness of the dropout for RNNs on sequential MNIST tasks, semi-supervised text classification tasks, and language modeling tasks.Comment: published in AAAI1

Sequential Recommendation with Relation-Aware Kernelized Self-Attention

Recent studies identified that sequential Recommendation is improved by the attention mechanism. By following this development, we propose Relation-Aware Kernelized Self-Attention (RKSA) adopting a self-attention mechanism of the Transformer with augmentation of a probabilistic model. The original self-attention of Transformer is a deterministic measure without relation-awareness. Therefore, we introduce a latent space to the self-attention, and the latent space models the recommendation context from relation as a multivariate skew-normal distribution with a kernelized covariance matrix from co-occurrences, item characteristics, and user information. This work merges the self-attention of the Transformer and the sequential recommendation by adding a probabilistic model of the recommendation task specifics. We experimented RKSA over the benchmark datasets, and RKSA shows significant improvements compared to the recent baseline models. Also, RKSA were able to produce a latent space model that answers the reasons for recommendation.Comment: 8 pages, 5 figures, AAA

Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching

Knowledge distillation extracts general knowledge from a pretrained teacher network and provides guidance to a target student network. Most studies manually tie intermediate features of the teacher and student, and transfer knowledge through predefined links. However, manual selection often constructs ineffective links that limit the improvement from the distillation. There has been an attempt to address the problem, but it is still challenging to identify effective links under practical scenarios. In this paper, we introduce an effective and efficient feature distillation method utilizing all the feature levels of the teacher without manually selecting the links. Specifically, our method utilizes an attention based meta network that learns relative similarities between features, and applies identified similarities to control distillation intensities of all possible pairs. As a result, our method determines competent links more efficiently than the previous approach and provides better performance on model compression and transfer learning tasks. Further qualitative analyses and ablative studies describe how our method contributes to better distillation

In Vivo Expression of Reprogramming Factors Increases Hippocampal Neurogenesis and Synaptic Plasticity in Chronic Hypoxic-Ischemic Brain Injury

Neurogenesis and synaptic plasticity can be stimulated in vivo in the brain. In this study, we hypothesized that in vivo expression of reprogramming factors such as Klf4, Sox2, Oct4, and c-Myc would facilitate endogenous neurogenesis and functional recovery. CD-1® mice were induced at 1 week of age by unilaterally carotid artery ligation and exposure to hypoxia. At 6 weeks of age, mice were injected GFP only or both four reprogramming factors and GFP into lateral ventricle. Passive avoidance task and open field test were performed to evaluate neurobehavioral function. Neurogenesis and synaptic activity in the hippocampus were evaluated using immunohistochemistry, qRT-PCR, and/or western blot analyses. Whereas BrdU+GFAP+ cells in the subgranular zone of the hippocampus were not significantly different, the numbers of BrdU+βIII-tubulin+ and BrdU+NeuN+ cells were significantly higher in treatment group than control group. Expressions of synaptophysin and PSD-95 were also higher in treatment group than control group. Importantly, passive avoidance task and open field test showed improvement in long-term memory and decreased anxiety in treatment group. In conclusion, in vivo expression of reprogramming factors improved behavioral functions in chronic hypoxic-ischemic brain injury. The mechanisms underlying these repair processes included endogenous neurogenesis and synaptic plasticity in the hippocampus

BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents

Key information extraction (KIE) from document images requires understanding the contextual and spatial semantics of texts in two-dimensional (2D) space. Many recent studies try to solve the task by developing pre-trained language models focusing on combining visual features from document images with texts and their layout. On the other hand, this paper tackles the problem by going back to the basic: effective combination of text and layout. Specifically, we propose a pre-trained language model, named BROS (BERT Relying On Spatiality), that encodes relative positions of texts in 2D space and learns from unlabeled documents with area-masking strategy. With this optimized training scheme for understanding texts in 2D space, BROS shows comparable or better performance compared to previous methods on four KIE benchmarks (FUNSD, SROIE*, CORD, and SciTSR) without relying on visual features. This paper also reveals two real-world challenges in KIE tasks--(1) minimizing the error from incorrect text ordering and (2) efficient learning from fewer downstream examples--and demonstrates the superiority of BROS over previous methods

Suppression of Cation Segregation in (La,Sr)CoO3−δ by Elastic Energy Minimization

Strontium segregation at perovskite surfaces deteriorates the oxygen reduction reaction kinetics of cathodes and therefore the long-term stability of solid oxide fuel cells (SOFCs). For the systematic and quantitative assessment of the elastic energy in perovskite oxides, which is known to be one of the main origins for dopant segregation, we report the fractional free volume as a new descriptor for the elastic energy in the perovskite oxide system. To verify the fractional free volume model, three samples were prepared with different A-site dopants: La0.6Sr0.4CoO3-delta, La0.6Sr0.2Ca0.2CoO3-delta, and La0.6Ca0.4CoO3-delta. A combination of the theoretical calculations of the segregation energy and oxide formation energy and experimental measurements of the structural, chemical, and electrochemical degradation substantiated the validity of using the fractional free volume to predict the dopant segregation. Furthermore, the dopant segregation could be significantly suppressed by increasing the fractional free volume in the perovskite oxides with dopant substitution. Our results provide insight into dopant segregation from the elastic energy perspective and offer a design guideline for SOFC cathodes with enhanced stability at elevated temperatures.113Nsciescopu

Utilizing a Siloxane-Modified Organic Semiconductor for Photoelectrochemical Water Splitting

Weexplore the potential of employing diketopyrrolopyrrole (DPP)based pi-conjugated OSs as a hole transport layer material inheteroatom-doped hematite (Ti-Fe2O3/Ge-Fe2O3) photoanodes for efficient photoelectrochemicalwater splitting. The siloxane-modified pi-conjugated polymer(P-Si) with a high carrier mobility and crystallinity revealedgreat potential to extract holes by forming a built-in potential withhematite photoanodes while showing high stability in an alkaline electrolytefor photoelectrochemical water oxidation. Because of the easy holeextraction and subsequent fast hole transport property of the P-Si interlayer between NiFe-(OH)( x ) and Ge-doped porous Fe2O3(Ge-PH), NiFe-(OH)( x )/P-Si/Ge-PH showed a 1.8-fold increasein photocurrent density (4.57 mA cm(-2) at 1.23 V-RHE) with a cathodic shift of the onset potential (0.735 V-RHE) and good stability for 65 h compared to Ge-PH. This studydemonstrates the successful use of inherently unstable pi-conjugatedOSs as a hole extracting/transport medium in a photoanode, addressingthe intrinsic recombination issues of hematite for efficient and stablewater splitting