How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks

Deep neural networks typically outperform more traditional machine learning models in their ability to classify complex data, and yet is not clear how the individual hidden layers of a deep network contribute to the overall classification performance. We thus introduce a Generalized Discrimination Value (GDV) that measures, in a non-invasive manner, how well different data classes separate in each given network layer. The GDV can be used for the automatic tuning of hyper-parameters, such as the width profile and the total depth of a network. Moreover, the layer-dependent GDV(L) provides new insights into the data transformations that self-organize during training: In the case of multi-layer perceptrons trained with error backpropagation, we find that classification of highly complex data sets requires a temporal {\em reduction} of class separability, marked by a characteristic 'energy barrier' in the initial part of the GDV(L) curve. Even more surprisingly, for a given data set, the GDV(L) is running through a fixed 'master curve', independently from the total number of network layers. Furthermore, applying the GDV to Deep Belief Networks reveals that also unsupervised training with the Contrastive Divergence method can systematically increase class separability over tens of layers, even though the system does not 'know' the desired class labels. These results indicate that the GDV may become a useful tool to open the black box of deep learning

Integration of Leaky-Integrate-and-Fire-Neurons in Deep Learning Architectures

Up to now, modern Machine Learning is mainly based on fitting high dimensional functions to enormous data sets, taking advantage of huge hardware resources. We show that biologically inspired neuron models such as the Leaky-Integrate-and-Fire (LIF) neurons provide novel and efficient ways of information encoding. They can be integrated in Machine Learning models, and are a potential target to improve Machine Learning performance. Thus, we derived simple update-rules for the LIF units from the differential equations, which are easy to numerically integrate. We apply a novel approach to train the LIF units supervisedly via backpropagation, by assigning a constant value to the derivative of the neuron activation function exclusively for the backpropagation step. This simple mathematical trick helps to distribute the error between the neurons of the pre-connected layer. We apply our method to the IRIS blossoms image data set and show that the training technique can be used to train LIF neurons on image classification tasks. Furthermore, we show how to integrate our method in the KERAS (tensorflow) framework and efficiently run it on GPUs. To generate a deeper understanding of the mechanisms during training we developed interactive illustrations, which we provide online. With this study we want to contribute to the current efforts to enhance Machine Intelligence by integrating principles from biology

Impact of a Dietary Supplement Containing 1,3-Dimethylamylamine on Blood Pressure and Bloodborne Markers of Health: a 10-Week Intervention Study

Background 1,3-dimethylamylamine is a commonly used ingredient within dietary supplements. Our prior work with this agent indicates a transient increase in blood pressure (systolic in particular) following oral ingestion of a single dosage, but no significant increase in resting blood pressure following chronic ingestion. Moreover, intervention studies involving both two and eight weeks of treatment with finished products containing 1,3-dimethylamylamine indicate minimal or no change in bloodborne markers of health. The present study sought to extend these findings by using a 10 -week intervention trial to determine the change in selected markers of health in a sample of men. Methods 25 healthy men were randomly assigned to either a placebo (n = 13) or to a supplement containing 1,3-dimethylamylamine (n = 12) for a period of 10 weeks. Before and after the intervention, resting blood pressure and heart rate were measured, and blood samples were collected for determination of complete blood count, metabolic panel, and lipid panel. Results No significant differences were noted between conditions for blood pressure ( P > 0.05), although systolic blood pressure increased approximately 6 mmHg with the supplement (diastolic blood pressure decreased approximately 4 mmHg). A main effect for time was noted for heart rate ( P = 0.016), with values decreasing from pre to post intervention. There were significant main effects for time for creatinine (increased from pre to post intervention; P = 0.043) and alkaline phosphatase (decreased from pre to post intervention; P = 0.009), with no condition differences noted ( P > 0.05). There was a significant interaction noted for low density lipoprotein cholesterol (LDL-C) ( P = 0.043), with values decreasing in the supplement group from pre to post intervention approximately 7 mg · dL -1 ( P = 0.034). No other effects of significance were noted for bloodborne variables. Conclusion These data indicate that a dietary supplement containing 1,3-dimethylamylamine does not result in a statistically significant increase in resting heart rate or blood pressure (although systolic blood pressure is increased ~6 mmHg with supplement use). The supplement does not negatively impact bloodborne markers of health. Further study is needed involving a longer intervention period, a larger sample size, and additional measures of health and safety

Digital Livestock Technologies as boundary objects: Investigating impacts on farm management and animal welfare

Digital Livestock Technologies (DLTs) can assist farmer decision-making and promise benefits to animal health and welfare. However, the extent to which they can help improve animal welfare is unclear. This study explores how DLTs may impact farm management and animal welfare by promoting learning, using the concept of boundary objects. Boundary objects may be interpreted differently by different social worlds but are robust enough to share a common identity across them. They facilitate communication around a common issue, allowing stakeholders to collaborate and co-learn. The type of learning generated may impact management and welfare differently. For example, it may help improve existing strategies (single-loop learning), or initiate reflection on how these strategies were framed initially (double-loop learning). This study focuses on two case studies, during which two DLTs were developed and tested on farms. In-depth, semi-structured interviews were conducted with stakeholders involved in the case studies (n = 31), and the results of a separate survey were used to complement our findings. Findings support the important potential of DLTs to help enhance animal welfare, although the impacts vary between technologies. In both case studies, DLTs facilitated discussions between stakeholders, and whilst both promoted improved management strategies, one also promoted deeper reflection on the importance of animal emotional well-being and on providing opportunities for positive animal welfare. If DLTs are to make significant improvements to animal welfare, greater priority should be given to DLTs that promote a greater understanding of the dimensions of animal welfare and a reframing of values and beliefs with respect to the importance of animals’ well-being

Sparsity through evolutionary pruning prevents neuronal networks from overfitting

Modern Machine learning techniques take advantage of the exponentially rising calculation power in new generation processor units. Thus, the number of parameters which are trained to resolve complex tasks was highly increased over the last decades. However, still the networks fail - in contrast to our brain - to develop general intelligence in the sense of being able to solve several complex tasks with only one network architecture. This could be the case because the brain is not a randomly initialized neural network, which has to be trained by simply investing a lot of calculation power, but has from birth some fixed hierarchical structure. To make progress in decoding the structural basis of biological neural networks we here chose a bottom-up approach, where we evolutionarily trained small neural networks in performing a maze task. This simple maze task requires dynamical decision making with delayed rewards. We were able to show that during the evolutionary optimization random severance of connections lead to better generalization performance of the networks compared to fully connected networks. We conclude that sparsity is a central property of neural networks and should be considered for modern Machine learning approaches

Physiological and Pharmacokinetic Effects of Oral 1,3-Dimethylamylamine Administration in Men

BACKGROUND: 1,3-dimethylamylamine (DMAA) has been a component of dietary supplements and is also used within "party pills," often in conjunction with alcohol and other drugs. Ingestion of higher than recommended doses results in untoward effects including cerebral hemorrhage. To our knowledge, no studies have been conducted to determine both the pharmacokinetic profile and physiologic responses of DMAA. METHODS: Eight men reported to the lab in the morning following an overnight fast and received a single 25 mg oral dose of DMAA. Blood samples were collected before and through 24 hours post-DMAA ingestion and analyzed for plasma DMAA concentration using high-performance liquid chromatography–mass spectrometry. Resting heart rate, blood pressure, and body temperature was also measured. RESULTS: One subject was excluded from the data analysis due to abnormal DMAA levels. Analysis of the remaining seven participants showed DMAA had an oral clearance of 20.02 ± 5 L∙hr(-1), an oral volume of distribution of 236 ± 38 L, and terminal half-life of 8.45 ± 1.9 hr. Lag time, the delay in appearance of DMAA in the circulation following extravascular administration, varied among participants but averaged approximately 8 minutes (0.14 ± 0.13 hr). The peak DMAA concentration for all subjects was observed within 3–5 hours following ingestion and was very similar across subjects, with a mean of ~70 ng∙mL(-1). Heart rate, blood pressure, and body temperature were largely unaffected by DMAA treatment. CONCLUSIONS: These are the first data to characterize the oral pharmacokinetic profile of DMAA. These findings indicate a consistent pattern of increase across subjects with regards to peak DMAA concentration, with peak values approximately 15–30 times lower than those reported in case studies linking DMAA intake with adverse events. Finally, a single 25 mg dose of DMAA does not meaningfully impact resting heart rate, blood pressure, or body temperature. TRIAL REGISTRATION: NCT0176593

Streambank Erosion Rates and Loads within a Single Watershed: Bridging the Gap between Temporal and Spatial Scales

The importance of streambank erosion to watershed-scale sediment export is being increasingly recognized. However few studies have quantified bank erosion and watershed sediment flux at the basin scale across temporal and spatial scales. In this study we evaluated the spatial distribution, extent, and temporal frequency of bank erosion in the 5218 ha Walnut Creek watershed in Iowa across a seven year period. We inventoried severely eroding streambanks along over 10 km of stream and monitored erosion pins at 20 sites within the watershed. Annual streambank recession rates ranged from 0.6 cm/yr during years of hydrological inactivity to 28.2 cm/yr during seasons with high discharge rates, with an overall average of 18.8 cm/yr. The percentage of total basin export attributed to streambank erosion along the main stem of Walnut Creek ranged from 23 to 53%. Large variations in individual site, annual rates and percentage of annual load suggested that developing direct relationships between streambank erosion rates and total sediment discharge may be confounded by the timing and magnitude of discharge events, storage of sediments within channel system and the remobilization of eroded material