Entropy-based Guidance of Deep Neural Networks for Accelerated Convergence and Improved Performance

Neural networks have dramatically increased our capacity to learn from large, high-dimensional datasets across innumerable disciplines. However, their decisions are not easily interpretable, their computational costs are high, and building and training them are uncertain processes. To add structure to these efforts, we derive new mathematical results to efficiently measure the changes in entropy as fully-connected and convolutional neural networks process data, and introduce entropy-based loss terms. Experiments in image compression and image classification on benchmark datasets demonstrate these losses guide neural networks to learn rich latent data representations in fewer dimensions, converge in fewer training epochs, and achieve better test metrics.Comment: 13 pages, 4 figure

Taking a PEEK into YOLOv5 for Satellite Component Recognition via Entropy-based Visual Explanations

The escalating risk of collisions and the accumulation of space debris in Low Earth Orbit (LEO) has reached critical concern due to the ever increasing number of spacecraft. Addressing this crisis, especially in dealing with non-cooperative and unidentified space debris, is of paramount importance. This paper contributes to efforts in enabling autonomous swarms of small chaser satellites for target geometry determination and safe flight trajectory planning for proximity operations in LEO. Our research explores on-orbit use of the You Only Look Once v5 (YOLOv5) object detection model trained to detect satellite components. While this model has shown promise, its inherent lack of interpretability hinders human understanding, a critical aspect of validating algorithms for use in safety-critical missions. To analyze the decision processes, we introduce Probabilistic Explanations for Entropic Knowledge extraction (PEEK), a method that utilizes information theoretic analysis of the latent representations within the hidden layers of the model. Through both synthetic in hardware-in-the-loop experiments, PEEK illuminates the decision-making processes of the model, helping identify its strengths, limitations and biases

Multiscale Modeling of Information Conveyed by Gene-Regulatory Signaling

Cells leverage signaling molecules to carry information about the cellular state to receptors that regulate protein synthesis in order to suit the cell's dynamically evolving needs. This regulation remains efficient and robust, despite that substantial stochasticity pervades the sub-cellular environment. In electronic and wireless signaling systems, the mutual information quantifies the extent to which information in a signal can be received across a communications channel. Applying this same metric to gene-regulatory interactions can better clarify how these biological signaling systems mitigate environmental noise. In this paper we study the information-transmission characteristics of a single gene-regulatory interaction by employing an exactly solvable master equation model for the production and degradation of individual proteins. This molecular-scale description is then coupled to a mass-action kinetics model of dynamic protein concentrations in a macroscopic sample of cells, enabling parameter values to be obtained by experiments performed using cell-based assays. We find that the mutual information depends monotonically on two parameters: one which characterizes stochastic variations in the concentration of signaling molecules, and the other the ratio of kinetic production to degradation rates of the regulated protein

Devil in the details: Mechanistic variations impact information transfer across models of transcriptional cascades.

The transcriptional network determines a cell's internal state by regulating protein expression in response to changes in the local environment. Due to the interconnected nature of this network, information encoded in the abundance of various proteins will often propagate across chains of noisy intermediate signaling events. The data-processing inequality (DPI) leads us to expect that this intracellular game of "telephone" should degrade this type of signal, with longer chains losing successively more information to noise. However, a previous modeling effort predicted that because the steps of these signaling cascades do not truly represent independent stages of data processing, the limits of the DPI could seemingly be surpassed, and the amount of transmitted information could actually increase with chain length. What that work did not examine was whether this regime of growing information transmission was attainable by a signaling system constrained by the mechanistic details of more complex protein-binding kinetics. Here we address this knowledge gap through the lens of information theory by examining a model that explicitly accounts for the binding of each transcription factor to DNA. We analyze this model by comparing stochastic simulations of the fully nonlinear kinetics to simulations constrained by the linear response approximations that displayed a regime of growing information. Our simulations show that even when molecular binding is considered, there remains a regime wherein the transmitted information can grow with cascade length, but ends after a critical number of links determined by the kinetic parameter values. This inflection point marks where correlations decay in response to an oversaturation of binding sites, screening informative transcription factor fluctuations from further propagation down the chain where they eventually become indistinguishable from the surrounding levels of noise

Benchmarking the communication fidelity of biomolecular signaling cascades featuring pseudo-one-dimensional transport

Synthetic biologists endeavor to predict how the increasing complexity of multi-step signaling cascades impacts the fidelity of molecular signaling, whereby information about the cellular state is often transmitted with proteins that diffuse by a pseudo-one-dimensional stochastic process. This begs the question of how the cell leverages passive transport mechanisms to distinguish informative signals from the intrinsic noise of diffusion. We address this problem by using a one-dimensional drift-diffusion model to derive an approximate lower bound on the degree of facilitation needed to achieve single-bit informational efficiency in signaling cascades as a function of their length. Within the assumptions of our model, we find that a universal curve of the Shannon-Hartley form describes the information transmitted by a signaling chain of arbitrary length and depends upon only a small number of physically measurable parameters. This enables our model to be used in conjunction with experimental measurements to aid in the selective design of biomolecular systems that can overcome noise to function reliably, even at the single-cell level

Capacity estimates of additive inverse Gaussian molecular channels with relay characteristics

Molecular communications is an emergent field that seeks to develop nanoscale communication devices using design principles gleaned from studies of the topology and dynamic properties of biological signaling networks. To understand how these networks function, we must first characterize the functional building blocks that compose them, and the best candidates for those are the topologically distinct subnetworks, or motifs, that appear in a statistically improbable abundance. In transcriptional networks, one of the most prevalent motifs is the feed-forward loop, a three node motif wherein one top-level protein regulates the expression of a target gene either directly or indirectly through an intermediate regulator protein. Currently, no systematic effort has been made to treat an isolated feed-forward loop as a stand-alone signal amplifying/attenuating device and understand its communication capacity in terms of the diffusion of individual molecules. To address this issue, we derive a theorem that estimates the upper and lower bounds of the channel capacity for a relay channel, which structurally corresponds to a feed-forward loop, by using an additive inverse Gaussian noise channel model of protein-ligand binding. Our results are just a first step towards assessing the performance bounds of simplified biological circuits in order to guide the development and optimization of synthetic, bio-inspired devices that can be used as information processing and forwarding units