Local Intrinsic Dimensional Entropy

Most entropy measures depend on the spread of the probability distribution over the sample space X, and the maximum entropy achievable scales proportionately with the sample space cardinality |X|. For a finite |X|, this yields robust entropy measures which satisfy many important properties, such as invariance to bijections, while the same is not true for continuous spaces (where |X|=infinity). Furthermore, since R and R^d (d in Z+) have the same cardinality (from Cantor's correspondence argument), cardinality-dependent entropy measures cannot encode the data dimensionality. In this work, we question the role of cardinality and distribution spread in defining entropy measures for continuous spaces, which can undergo multiple rounds of transformations and distortions, e.g., in neural networks. We find that the average value of the local intrinsic dimension of a distribution, denoted as ID-Entropy, can serve as a robust entropy measure for continuous spaces, while capturing the data dimensionality. We find that ID-Entropy satisfies many desirable properties and can be extended to conditional entropy, joint entropy and mutual-information variants. ID-Entropy also yields new information bottleneck principles and also links to causality. In the context of deep learning, for feedforward architectures, we show, theoretically and empirically, that the ID-Entropy of a hidden layer directly controls the generalization gap for both classifiers and auto-encoders, when the target function is Lipschitz continuous. Our work primarily shows that, for continuous spaces, taking a structural rather than a statistical approach yields entropy measures which preserve intrinsic data dimensionality, while being relevant for studying various architectures.Comment: Proceedings of the AAAI Conference on Artificial Intelligence 202

Towards Better Long-range Time Series Forecasting using Generative Forecasting

Long-range time series forecasting is usually based on one of two existing forecasting strategies: Direct Forecasting and Iterative Forecasting, where the former provides low bias, high variance forecasts and the latter leads to low variance, high bias forecasts. In this paper, we propose a new forecasting strategy called Generative Forecasting (GenF), which generates synthetic data for the next few time steps and then makes long-range forecasts based on generated and observed data. We theoretically prove that GenF is able to better balance the forecasting variance and bias, leading to a much smaller forecasting error. We implement GenF via three components: (i) a novel conditional Wasserstein Generative Adversarial Network (GAN) based generator for synthetic time series data generation, called CWGAN-TS. (ii) a transformer based predictor, which makes long-range predictions using both generated and observed data. (iii) an information theoretic clustering algorithm to improve the training of both the CWGAN-TS and the transformer based predictor. The experimental results on five public datasets demonstrate that GenF significantly outperforms a diverse range of state-of-the-art benchmarks and classical approaches. Specifically, we find a 5% - 11% improvement in predictive performance (mean absolute error) while having a 15% - 50% reduction in parameters compared to the benchmarks. Lastly, we conduct an ablation study to further explore and demonstrate the effectiveness of the components comprising GenF.Comment: 14 pages. arXiv admin note: substantial text overlap with arXiv:2110.0877

AP: Selective Activation for De-sparsifying Pruned Neural Networks

The rectified linear unit (ReLU) is a highly successful activation function in neural networks as it allows networks to easily obtain sparse representations, which reduces overfitting in overparameterized networks. However, in network pruning, we find that the sparsity introduced by ReLU, which we quantify by a term called dynamic dead neuron rate (DNR), is not beneficial for the pruned network. Interestingly, the more the network is pruned, the smaller the dynamic DNR becomes during optimization. This motivates us to propose a method to explicitly reduce the dynamic DNR for the pruned network, i.e., de-sparsify the network. We refer to our method as Activating-while-Pruning (AP). We note that AP does not function as a stand-alone method, as it does not evaluate the importance of weights. Instead, it works in tandem with existing pruning methods and aims to improve their performance by selective activation of nodes to reduce the dynamic DNR. We conduct extensive experiments using popular networks (e.g., ResNet, VGG) via two classical and three state-of-the-art pruning methods. The experimental results on public datasets (e.g., CIFAR-10/100) suggest that AP works well with existing pruning methods and improves the performance by 3% - 4%. For larger scale datasets (e.g., ImageNet) and state-of-the-art networks (e.g., vision transformer), we observe an improvement of 2% - 3% with AP as opposed to without. Lastly, we conduct an ablation study to examine the effectiveness of the components comprising AP.Comment: 16 Page

Solute Concentration Effects on Microstructure and the Compressive Strength of Ice-Templated Sintered Lithium Titanate

This work investigated the role of sucrose and cationic dispersant (1‐hexadecyl)trimethylammonium bromide concentration on ice‐templated sintered lithium titanate microstructure and compressive strength, to enable a comprehensive understanding of composition selection and elucidate processing–microstructure–mechanical property relationships. Sucrose and dispersant concentrations were varied to change total solute concentration in suspensions and viscosity. Dispersant was more effective in reducing viscosity than sucrose; however, their combination had an even greater impact on reducing viscosity. Based on viscosity measurements, a total of 12 suspension compositions were developed, and materials were fabricated at two different freezing front velocity (FFV) regimes. Solute concentration greatly influenced ice‐templated microstructure and microstructure development improved with solute concentration. Depending on solute concentration, type of solute, viscosity, and FFV, a wide variety of microstructures were observed ranging from lamellar to dendritic morphologies. Solute concentration effect was rationalized based on solid–liquid planar interface instability. For suspensions with comparable viscosity, solute concentration can be varied to tune microstructure, whereas for suspensions with comparable solute concentration, viscosity variation can tune microstructure. Compressive strength of sintered materials generally increased with total solute concentration, sucrose concentration, viscosity, and FFV. Due to the wide variety of microstructure, strength also varied over a wide range, 23–128 MPa

Characterization Of Probiotic Lactic Acid Bacteria From Honey And Assessment Of Their Effects On Consumption By Type-2- Diabetes Using Wistar Rat

The relevance of probiotic, especially lactic acid bacteria cannot be over emphasized. In this present study three honey sources were serially diluted and cultured on De Man, Rogosa and Sharpe (MRS) agar among which only one of this sources grew on MRS agar. The pure Lactobacilli isolate were subjected to gram staining, biochemical tests, physiological test, molecular analysis using Polymerase Chain Reaction (PCR) techniques and Deoxyribonucleotide (DNA) sequencing. Only one isolate was obtained known as Enterococcus fecalis. The isolate was subjected to probiotic selection and was found fit for consumption, however their effect when consumed by type-2 diabetic are alarming and based on the outcome of this study, diabetic patient are advised not consume honey

A Neurally Inspired Robotic Control Algorithm for Gait Rehabilitation in Hemiplegic Stroke Patients

Abstract-Cerebrovascular accident or stroke is one of the major brain impairments that affects numerous people globally. After a unilateral stroke, sensory motor damages contralateral to the brain lesion occur in many patients. As a result, gait remains impaired and asymmetric. This paper describes and simulates a novel closed loop algorithm designed for the control of a lower limb exoskeleton for post-stroke rehabilitation. The algorithm has been developed to control a lower limb exoskeleton including actuators for the hip and knee joints, and feedback sensors for the measure of joint angular excursions. It has been designed to control and correct the gait cycle of the affected leg using kinematics information from the unaffected one. In particular, a probabilistic particle filter like algorithm has been used at the top-level control to modulate gait velocity and the joint angular excursions. Simulation results show that the algorithm is able to correct and control velocity of the affected side restoring phase synchronization between the legs

Pore Microstructure Impacts on Lithium Ion Transport and Rate Capability of Thick Sintered Electrodes

Increasing electrode thickness is one route to improve the energy density of lithium-ion battery cells. However, restricted Li+ transport in the electrolyte phase through the porous microstructure of thick electrodes limits the ability to achieve high current densities and rates of charge/discharge with these high energy cells. In this work, processing routes to mitigate transport restrictions were pursued. The electrodes used were comprised of only active material sintered together into a porous pellet. For one of the electrodes, comparisons were done between using ice-templating to provide directional porosity and using sacrificial particles during processing to match the geometric density without pore alignment. The ice-templated electrodes retained much greater discharge capacity at higher rates of cycling, which was attributed to improved transport properties provided by the processing. The electrodes were further characterized using an electrochemical model of the cells evaluated and neutron imaging of a cell containing the ice-templated pellet. The results indicate that significant improvements can be made to electrochemical cell properties via templating the electrode microstructure for situations where the rate limiting step includes ion transport limitations in the cell