Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles, the inference formulas for the LSTM network and its parent, RNN, are stated axiomatically, while the training formulas are omitted altogether. In addition, the technique of "unrolling" an RNN is routinely presented without justification throughout the literature. The goal of this paper is to explain the essential RNN and LSTM fundamentals in a single document. Drawing from concepts in signal processing, we formally derive the canonical RNN formulation from differential equations. We then propose and prove a precise statement, which yields the RNN unrolling technique. We also review the difficulties with training the standard RNN and address them by transforming the RNN into the "Vanilla LSTM" network through a series of logical arguments. We provide all equations pertaining to the LSTM system together with detailed descriptions of its constituent entities. Albeit unconventional, our choice of notation and the method for presenting the LSTM system emphasizes ease of understanding. As part of the analysis, we identify new opportunities to enrich the LSTM system and incorporate these extensions into the Vanilla LSTM network, producing the most general LSTM variant to date. The target reader has already been exposed to RNNs and LSTM networks through numerous available resources and is open to an alternative pedagogical approach. A Machine Learning practitioner seeking guidance for implementing our new augmented LSTM model in software for experimentation and research will find the insights and derivations in this tutorial valuable as well.Comment: 43 pages, 10 figures, 78 reference

Fractured reservoir discrete feature network technologies. Annual report, March 7, 1996--February 28, 1997

This report describes progress on the project, {open_quotes}Fractured Reservoir Discrete Feature Network Technologies{close_quotes} during the period March 7, 1996 to February 28, 1997. The report presents summaries of technology development for the following research areas: (1) development of hierarchical fracture models, (2) fractured reservoir compartmentalization and tributary volume, (3) fractured reservoir data analysis, and (4) integration of fractured reservoir data and production technologies. In addition, the report provides information on project status, publications submitted, data collection activities, and technology transfer through the world wide web (WWW). Research on hierarchical fracture models included geological, mathematical, and computer code development. The project built a foundation of quantitative, geological and geometrical information about the regional geology of the Permian Basin, including detailed information on the lithology, stratigraphy, and fracturing of Permian rocks in the project study area (Tracts 17 and 49 in the Yates field). Based on the accumulated knowledge of regional and local geology, project team members started the interpretation of fracture genesis mechanisms and the conceptual modeling of the fracture system in the study area. Research on fractured reservoir compartmentalization included basic research, technology development, and application of compartmentalized reservoir analyses for the project study site. Procedures were developed to analyze compartmentalization, tributary drainage volume, and reservoir matrix block size. These algorithms were implemented as a Windows 95 compartmentalization code, FraCluster

Fractals in the Nervous System: conceptual Implications for Theoretical Neuroscience

This essay is presented with two principal objectives in mind: first, to document the prevalence of fractals at all levels of the nervous system, giving credence to the notion of their functional relevance; and second, to draw attention to the as yet still unresolved issues of the detailed relationships among power law scaling, self-similarity, and self-organized criticality. As regards criticality, I will document that it has become a pivotal reference point in Neurodynamics. Furthermore, I will emphasize the not yet fully appreciated significance of allometric control processes. For dynamic fractals, I will assemble reasons for attributing to them the capacity to adapt task execution to contextual changes across a range of scales. The final Section consists of general reflections on the implications of the reviewed data, and identifies what appear to be issues of fundamental importance for future research in the rapidly evolving topic of this review

The quality of data and the accuracy of energy generation forecast by artificial neural networks

The paper presents the issues related to predicting the amount of energy generation, in a particular wind power plant comprising five generators located in south-eastern Poland. The location of wind power plant, the distribution and type of applied generators, and topographical conditions were given and the correlation between selected weather parameters and the volume of energy generation was discussed. The primary objective of the paper was to select learning data and perform forecasts using artificial neural networks. For comparison, conservative forecasts were also presented. Forecasts results obtained shaw that Artificial Neural Networks are more universal than conservative method. However their forecast accuracy of forecasts strongly depends on the selection of explanatory dat