What is the Role of Recurrent Neural Networks (RNNs) in an Image Caption Generator?

In neural image captioning systems, a recurrent neural network (RNN) is typically viewed as the primary `generation' component. This view suggests that the image features should be `injected' into the RNN. This is in fact the dominant view in the literature. Alternatively, the RNN can instead be viewed as only encoding the previously generated words. This view suggests that the RNN should only be used to encode linguistic features and that only the final representation should be `merged' with the image features at a later stage. This paper compares these two architectures. We find that, in general, late merging outperforms injection, suggesting that RNNs are better viewed as encoders, rather than generators.Comment: Appears in: Proceedings of the 10th International Conference on Natural Language Generation (INLG'17

Where to put the image in an image caption generator

When a neural language model is used for caption generation, the image information can be fed to the neural network either by directly in- corporating it in a recurrent neural network { conditioning the language model by injecting image features { or in a layer following the recurrent neural network { conditioning the language model by merging the image features. While merging implies that visual features are bound at the end of the caption generation process, injecting can bind the visual features at a variety stages. In this paper we empirically show that late binding is superior to early binding in terms of di erent evaluation metrics. This suggests that the di erent modalities (visual and linguistic) for caption generation should not be jointly encoded by the RNN; rather, the multi- modal integration should be delayed to a subsequent stage. Furthermore, this suggests that recurrent neural networks should not be viewed as actu- ally generating text, but only as encoding it for prediction in a subsequent layer.peer-reviewe

Investigating the role for IL-21 in rabies virus vaccine-induced immunity.

Over two-thirds of the world\u27s population lives in regions where rabies is endemic, resulting in over 15 million people receiving multi-dose post-exposure prophylaxis (PEP) and over 55,000 deaths per year globally. A major goal in rabies virus (RABV) research is to develop a single-dose PEP that would simplify vaccination protocols, reduce costs associated with RABV prevention, and save lives. Protection against RABV infections requires virus neutralizing antibodies; however, factors influencing the development of protective RABV-specific B cell responses remain to be elucidated. Here we used a mouse model of IL-21 receptor-deficiency (IL-21R-/-) to characterize the role for IL-21 in RABV vaccine-induced immunity. IL-21R-/- mice immunized with a low dose of a live recombinant RABV-based vaccine (rRABV) produced only low levels of primary or secondary anti-RABV antibody response while wild-type mice developed potent anti-RABV antibodies. Furthermore, IL-21R-/- mice immunized with low-dose rRABV were only minimally protected against pathogenic RABV challenge, while all wild-type mice survived challenge, indicating that IL-21R signaling is required for antibody production in response to low-dose RABV-based vaccination. IL-21R-/- mice immunized with a higher dose of vaccine produced suboptimal anti-RABV primary antibody responses, but showed potent secondary antibodies and protection similar to wild-type mice upon challenge with pathogenic RABV, indicating that IL-21 is dispensable for secondary antibody responses to live RABV-based vaccines when a primary response develops. Furthermore, we show that IL-21 is dispensable for the generation of Tfh cells and memory B cells in the draining lymph nodes of immunized mice but is required for the detection of optimal GC B cells or plasma cells in the lymph node or bone marrow, respectively, in a vaccine dose-dependent manner. Collectively, our preliminary data show that IL-21 is critical for the development of optimal vaccine-induced primary but not secondary antibody responses against RABV infections

Assessing the long-term growth response and age estimation precision for Arctic whitefishes in a rapidly changing nearshore environment

Thesis (M.S.) University of Alaska Fairbanks, 2021Accurate monitoring of population-level health and productivity is essential for assessing the status and availability of subsistence harvested species at the forefront of climate change. This study used otolith biochronology to assess the long-term growth response of Arctic Cisco Coregonus autumnalis during a period of rapid environmental change in the Beaufort Sea region and to identify drivers of growth. A biochronology spanning 22 years (1996-2018) revealed significant interannual variation, with faster growth rates in years with warmer (R² = 0.31) and more saline (R² = 0.47) waters during the ice-free summer feeding period (July-September). These results suggested that warming may benefit Arctic Cisco. This study also compared age estimates made using fin rays, scales, and otoliths of four subsistence whitefishes (Arctic Cisco, Least Cisco Coregonus sardinella, Broad Whitefish Coregonus nasus, and Humpback Whitefish Coregonus pidschian) from the Beaufort Sea to compare the aging precision of non-lethal structures (fin rays and scales) to otoliths. Fin rays and scales provided similar age estimates as otoliths until the age of sexual maturity and underestimated otolith age for mature individuals. Scales underestimated age more often and were more difficult to which to assign age than the other two structures. Among Arctic Cisco in Alaska, fin rays and scales provided similar age estimates as otoliths for all age and size classes examined because most individuals in the study area were immature fish. These results suggested that dorsal fin rays may be used to estimate age in Least Cisco <300 mm, Broad Whitefish <450 mm, and Humpback Whitefish <350 mm, and that otoliths should remain the primary aging structure for the largest whitefishes. Overall, this research complements existing monitoring by providing evidence of an Arctic subsistence species that may benefit in part from warming and highlights non-lethal alternatives for monitoring the age structure of juvenile whitefishes.Hilcorp Alask

The role of hnRNPs in frontotemporal dementia and amyotrophic lateral sclerosis

Dysregulated RNA metabolism is emerging as a crucially important mechanism underpinning the pathogenesis of frontotemporal dementia (FTD) and the clinically, genetically and pathologically overlapping disorder of amyotrophic lateral sclerosis (ALS). Heterogeneous nuclear ribonucleoproteins (hnRNPs) comprise a family of RNA-binding proteins with diverse, multi-functional roles across all aspects of mRNA processing. The role of these proteins in neurodegeneration is far from understood. Here, we review some of the unifying mechanisms by which hnRNPs have been directly or indirectly linked with FTD/ALS pathogenesis, including their incorporation into pathological inclusions and their best-known roles in pre-mRNA splicing regulation. We also discuss the broader functionalities of hnRNPs including their roles in cryptic exon repression, stress granule assembly and in co-ordinating the DNA damage response, which are all emerging pathogenic themes in both diseases. We then present an integrated model that depicts how a broad-ranging network of pathogenic events can arise from declining levels of functional hnRNPs that are inadequately compensated for by autoregulatory means. Finally, we provide a comprehensive overview of the most functionally relevant cellular roles, in the context of FTD/ALS pathogenesis, for hnRNPs A1-U

Reference Production as Search:The Impact of Domain Size on the Production of Distinguishing Descriptions

When producing a description of a target referent in a visual context, speakers need to choose a set of properties that distinguish it from its distractors. Computational models of language production/generation usually model this as a search process and predict that the time taken will increase both with the number of distractors in a scene and with the number of properties required to distinguish the target. These predictions are reminiscent of classic ndings in visual search; however, unlike models of reference production, visual search models also predict that search can become very e cient under certain conditions, something that reference production models do not consider. This paper investigates the predictions of these models empirically. In two experiments, we show that the time taken to plan a referring expression { as re ected by speech onset latencies { is in uenced by distractor set size and by the number of properties required, but this crucially depends on the discriminability of the properties under consideration. We discuss the implications for current models of reference production and recent work on the role of salience in visual search.peer-reviewe