Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation

We propose a hierarchically structured reinforcement learning approach to address the challenges of planning for generating coherent multi-sentence stories for the visual storytelling task. Within our framework, the task of generating a story given a sequence of images is divided across a two-level hierarchical decoder. The high-level decoder constructs a plan by generating a semantic concept (i.e., topic) for each image in sequence. The low-level decoder generates a sentence for each image using a semantic compositional network, which effectively grounds the sentence generation conditioned on the topic. The two decoders are jointly trained end-to-end using reinforcement learning. We evaluate our model on the visual storytelling (VIST) dataset. Empirical results from both automatic and human evaluations demonstrate that the proposed hierarchically structured reinforced training achieves significantly better performance compared to a strong flat deep reinforcement learning baseline.Comment: Accepted to AAAI 201

Improving Pneumonia Classification and Lesion Detection Using Spatial Attention Superposition and Multilayer Feature Fusion

Pneumonia is a severe inflammation of the lung that could cause serious complications. Chest X-rays (CXRs) are commonly used to make a diagnosis of pneumonia. In this paper, we propose a deep-learning-based method with spatial attention superposition (SAS) and multilayer feature fusion (MFF) to facilitate pneumonia diagnosis based on CXRs. Specifically, an SAS module, which takes advantage of the channel and spatial attention mechanisms, was designed to identify intrinsic imaging features of pneumonia-related lesions and their locations, and an MFF module was designed to harmonize disparate features from different channels and emphasize important information. These two modules were concatenated to extract critical image features serving as the basis for pneumonia diagnosis. We further embedded the proposed modules into a baseline neural network and developed a model called SAS-MFF-YOLO to diagnose pneumonia. To validate the effectiveness of our model, extensive experiments were conducted on two CXR datasets provided by the Radiological Society of North America (RSNA) and the AI Research Institute. SAS-MFF-YOLO achieved a precision of 88.1%, a recall of 98.2% for pneumonia classification and an AP50 of 99% for lesion detection on the AI Research Institute dataset. The visualization of intermediate feature maps showed that our method could facilitate uncovering pneumonia-related lesions in CXRs. Our results demonstrated that our approach could be used to enhance the performance of the overall pneumonia detection on CXR imaging

Case report: Reoperative parathyroidectomy for large ectopic hyperplastic parathyroid in the mediastinum of a patient with recurrent secondary hyperparathyroidism

IntroductionSecondary hyperparathyroidism (SHPT) is a common complication in hemodialysis patients with chronic renal failure uremia. For severe SHPT, parathyroidectomy is effective. Owing to the variability in parathyroid anatomy, surgical parathyroidectomy can be complex and many patients experience recurrent SHPT, which may require repeated surgery. These cases pose significant challenges to surgeons.Case descriptionAn elderly woman with recurrent severe SHPT was admitted to our hospital. Preoperative methoxyisobutylisonitrile (MIBI) examination found a large ectopic parathyroid gland in the superior mediastinum, and she underwent reoperative parathyroidectomy. A large parathyroid gland in the right anterior mediastinum and another parathyroid gland in the left lingual lobe of the thymus were removed. The patient had postoperative hypocalcemia that was successfully corrected with calcium supplementation via femoral vein catheterization. During the 1-year postoperative follow-up, the patient's iPTH was well controlled and her blood calcium was within the normal range.ConclusionWe report a case of parathyroidectomy to remove multifocal ectopic hyperplastic parathyroid tissue in the mediastinum. Preoperative MIBI accurately detected the lesions. Calcium supplementation via femoral vein catheterization successfully corrected postoperative hypocalcemia. Postoperative follow-up for 1 year indicated that the surgery was successful

KAT: A Knowledge Augmented Transformer for Vision-and-Language

The primary focus of recent work with largescale transformers has been on optimizing the amount of information packed into the model's parameters. In this work, we ask a different question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these challenges, we propose a novel model - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result (+6 points absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an end to end encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. An additional benefit of explicit knowledge integration is seen in improved interpretability of model predictions in our analysis.Comment: Accepted by NAACL 202

Triplet-radical spin entanglement: potential of molecular materials for high-temperature quantum information processing

Recently, spin-bearing molecules have been experimentally demonstrated to have great potential as building blocks for quantum information processing due to their substantial advantages including tunability, portability, and scalability. Here, we propose a theoretical model based on the theory of open quantum systems for spin dynamics in a molecule containing one radical, which can interact with the triplet state arising from another part of the molecule owing to optical excitation and intersystem crossing. With the initial state being a classical mixture of a radical 1/2-spin, the exchange interaction between the radical and the triplet produces a spin coherent state, which could potentially be used for a qubit-qutrit quantum entangling gate. Our calculations for the time-resolved electron paramagnetic resonance spectra showed good qualitative agreement with the related experimental results for radical-bearing molecules at high temperature (~77 K, the boiling point of liquid nitrogen). This work therefore lays a solid theoretical cornerstone for optically driven quantum gate operations in radical-bearing molecular materials, aiming toward high-temperature quantum information processing