Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling

Recurrent neural networks have shown remarkable success in modeling sequences. However low resource situations still adversely affect the generalizability of these models. We introduce a new family of models, called Lattice Recurrent Units (LRU), to address the challenge of learning deep multi-layer recurrent models with limited resources. LRU models achieve this goal by creating distinct (but coupled) flow of information inside the units: a first flow along time dimension and a second flow along depth dimension. It also offers a symmetry in how information can flow horizontally and vertically. We analyze the effects of decoupling three different components of our LRU model: Reset Gate, Update Gate and Projected State. We evaluate this family on new LRU models on computational convergence rates and statistical efficiency. Our experiments are performed on four publicly-available datasets, comparing with Grid-LSTM and Recurrent Highway networks. Our results show that LRU has better empirical computational convergence rates and statistical efficiency values, along with learning more accurate language models.Comment: 8 pages, 7 figure

Using Intelligent Prefetching to Reduce the Energy Consumption of a Large-scale Storage System

Many high performance large-scale storage systems will experience significant workload increases as their user base and content availability grow over time. The U.S. Geological Survey (USGS) Earth Resources Observation and Science (EROS) center hosts one such system that has recently undergone a period of rapid growth as its user population grew nearly 400% in just about three years. When administrators of these massive storage systems face the challenge of meeting the demands of an ever increasing number of requests, the easiest solution is to integrate more advanced hardware to existing systems. However, additional investment in hardware may significantly increase the system cost as well as daily power consumption. In this paper, we present evidence that well-selected software level optimization is capable of achieving comparable levels of performance without the cost and power consumption overhead caused by physically expanding the system. Specifically, we develop intelligent prefetching algorithms that are suitable for the unique workloads and user behaviors of the world\u27s largest satellite images distribution system managed by USGS EROS. Our experimental results, derived from real-world traces with over five million requests sent by users around the globe, show that the EROS hybrid storage system could maintain the same performance with over 30% of energy savings by utilizing our proposed prefetching algorithms, compared to the alternative solution of doubling the size of the current FTP server farm

LERC: Coordinated Cache Management for Data-Parallel Systems

Memory caches are being aggressively used in today's data-parallel frameworks such as Spark, Tez and Storm. By caching input and intermediate data in memory, compute tasks can witness speedup by orders of magnitude. To maximize the chance of in-memory data access, existing cache algorithms, be it recency- or frequency-based, settle on cache hit ratio as the optimization objective. However, unlike the conventional belief, we show in this paper that simply pursuing a higher cache hit ratio of individual data blocks does not necessarily translate into faster task completion in data-parallel environments. A data-parallel task typically depends on multiple input data blocks. Unless all of these blocks are cached in memory, no speedup will result. To capture this all-or-nothing property, we propose a more relevant metric, called effective cache hit ratio. Specifically, a cache hit of a data block is said to be effective if it can speed up a compute task. In order to optimize the effective cache hit ratio, we propose the Least Effective Reference Count (LERC) policy that persists the dependent blocks of a compute task as a whole in memory. We have implemented the LERC policy as a memory manager in Spark and evaluated its performance through Amazon EC2 deployment. Evaluation results demonstrate that LERC helps speed up data-parallel jobs by up to 37% compared with the widely employed least-recently-used (LRU) policy

Distributed Selfish Coaching

Although cooperation generally increases the amount of resources available to a community of nodes, thus improving individual and collective performance, it also allows for the appearance of potential mistreatment problems through the exposition of one node's resources to others. We study such concerns by considering a group of independent, rational, self-aware nodes that cooperate using on-line caching algorithms, where the exposed resource is the storage at each node. Motivated by content networking applications -- including web caching, CDNs, and P2P -- this paper extends our previous work on the on-line version of the problem, which was conducted under a game-theoretic framework, and limited to object replication. We identify and investigate two causes of mistreatment: (1) cache state interactions (due to the cooperative servicing of requests) and (2) the adoption of a common scheme for cache management policies. Using analytic models, numerical solutions of these models, as well as simulation experiments, we show that on-line cooperation schemes using caching are fairly robust to mistreatment caused by state interactions. To appear in a substantial manner, the interaction through the exchange of miss-streams has to be very intense, making it feasible for the mistreated nodes to detect and react to exploitation. This robustness ceases to exist when nodes fetch and store objects in response to remote requests, i.e., when they operate as Level-2 caches (or proxies) for other nodes. Regarding mistreatment due to a common scheme, we show that this can easily take place when the "outlier" characteristics of some of the nodes get overlooked. This finding underscores the importance of allowing cooperative caching nodes the flexibility of choosing from a diverse set of schemes to fit the peculiarities of individual nodes. To that end, we outline an emulation-based framework for the development of mistreatment-resilient distributed selfish caching schemes. Our framework utilizes a simple control-theoretic approach to dynamically parameterize the cache management scheme. We show performance evaluation results that quantify the benefits from instantiating such a framework, which could be substantial under skewed demand profiles.National Science Foundation (CNS Cybertrust 0524477, CNS NeTS 0520166, CNS ITR 0205294, EIA RI 0202067); EU IST (CASCADAS and E-NEXT); Marie Curie Outgoing International Fellowship of the EU (MOIF-CT-2005-007230

Stomatal control of leaf fluxes of carbonyl sulfide and CO₂ in a <i>Typha</i> freshwater marsh

Carbonyl sulfide (COS) is an emerging tracer to constrain land photosynthesis at canopy to global scales, because leaf COS and CO2 uptake processes are linked through stomatal diffusion. The COS tracer approach requires knowledge of the concentration normalized ratio of COS uptake to photosynthesis, commonly known as the leaf relative uptake (LRU). LRU is known to increase under low light, but the environmental controls over LRU variability in the field are poorly understood due to scant leaf scale observations.Here we present the first direct observations of LRU responses to environmental variables in the field. We measured leaf COS and CO2 fluxes at a freshwater marsh in summer 2013. Daytime leaf COS and CO2 uptake showed similar peaks in the mid-morning and late afternoon separated by a prolonged midday depression, highlighting the common stomatal control on diffusion. At night, in contrast to CO2, COS uptake continued, indicating partially open stomata. LRU ratios showed a clear relationship with photosynthetically active radiation (PAR), converging to 1.0 at high PAR, while increasing sharply at low PAR. Daytime integrated LRU (calculated from daytime mean COS and CO2 uptake) ranged from 1 to 1.5, with a mean of 1.2 across the campaign, significantly lower than the previously reported laboratory mean value (∼ 1.6). Our results indicate two major determinants of LRU – light and vapor deficit. Light is the primary driver of LRU because CO2 assimilation capacity increases with light, while COS consumption capacity does not. Superimposed upon the light response is a secondary effect that high vapor deficit further reduces LRU, causing LRU minima to occur in the afternoon, not at noon. The partial stomatal closure induced by high vapor deficit suppresses COS uptake more strongly than CO2 uptake because stomatal resistance is a more dominant component in the total resistance of COS. Using stomatal conductance estimates, we show that LRU variability can be explained in terms of different patterns of stomatal vs. internal limitations on COS and CO2 uptake. Our findings illustrate the stomata-driven coupling of COS and CO2 uptake during the most photosynthetically active period in the field and provide an in situ characterization of LRU – a key parameter required for the use of COS as a photosynthetic tracer