Dynamic Memory Based Adaptive Optimization

Define an optimizer as having memory $k$ if it stores $k$ dynamically changing vectors in the parameter space. Classical SGD has memory $0$, momentum SGD optimizer has $1$ and Adam optimizer has $2$. We address the following questions: How can optimizers make use of more memory units? What information should be stored in them? How to use them for the learning steps? As an approach to the last question, we introduce a general method called "Retrospective Learning Law Correction" or shortly RLLC. This method is designed to calculate a dynamically varying linear combination (called learning law) of memory units, which themselves may evolve arbitrarily. We demonstrate RLLC on optimizers whose memory units have linear update rules and small memory ($\leq 4$ memory units). Our experiments show that in a variety of standard problems, these optimizers outperform the above mentioned three classical optimizers. We conclude that RLLC is a promising framework for boosting the performance of known optimizers by adding more memory units and by making them more adaptive

Mode Combinability: Exploring Convex Combinations of Permutation Aligned Models

We explore element-wise convex combinations of two permutation-aligned neural network parameter vectors $\Theta_A$ and $\Theta_B$ of size $d$. We conduct extensive experiments by examining various distributions of such model combinations parametrized by elements of the hypercube $[0,1]^{d}$ and its vicinity. Our findings reveal that broad regions of the hypercube form surfaces of low loss values, indicating that the notion of linear mode connectivity extends to a more general phenomenon which we call mode combinability. We also make several novel observations regarding linear mode connectivity and model re-basin. We demonstrate a transitivity property: two models re-based to a common third model are also linear mode connected, and a robustness property: even with significant perturbations of the neuron matchings the resulting combinations continue to form a working model. Moreover, we analyze the functional and weight similarity of model combinations and show that such combinations are non-vacuous in the sense that there are significant functional differences between the resulting models

Isolation and Characterisation of Electrogenic Bacteria from Mud Samples

To develop efficient microbial fuel cell systems for green energy production using different waste products, establishing characterised bacterial consortia is necessary. In this study, bacteria with electrogenic potentials were isolated from mud samples and examined to determine biofilm-formation capacities and macromolecule degradation. Matrix-assisted laser desorption/ionization time-of-flight mass spectrometry identifications have revealed that isolates represented 18 known and 4 unknown genuses. They all had the capacities to reduce the Reactive Black 5 stain in the agar medium, and 48 of them were positive in the wolfram nanorod reduction assay. The isolates formed biofilm to different extents on the surfaces of both adhesive and non-adhesive 96-well polystyrene plates and glass. Scanning electron microscopy images revealed the different adhesion potentials of isolates to the surface of carbon tissue fibres. Eight of them (15%) were able to form massive amounts of biofilm in three days at 23 °C. A total of 70% of the isolates produced proteases, while lipase and amylase production was lower, at 38% and 27% respectively. All of the macromolecule-degrading enzymes were produced by 11 isolates, and two isolates of them had the capacity to form a strong biofilm on the carbon tissue one of the most used anodic materials in MFC systems. This study discusses the potential of the isolates for future MFC development applications

Similarity and Matching of Neural Network Representations

We employ a toolset -- dubbed Dr. Frankenstein -- to analyse the similarity of representations in deep neural networks. With this toolset, we aim to match the activations on given layers of two trained neural networks by joining them with a stitching layer. We demonstrate that the inner representations emerging in deep convolutional neural networks with the same architecture but different initializations can be matched with a surprisingly high degree of accuracy even with a single, affine stitching layer. We choose the stitching layer from several possible classes of linear transformations and investigate their performance and properties. The task of matching representations is closely related to notions of similarity. Using this toolset, we also provide a novel viewpoint on the current line of research regarding similarity indices of neural network representations: the perspective of the performance on a task.Comment: To appear in the Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021

Comparative Study on Grape Berry Anthocyanins of Various Teinturier Varieties

The red-fleshed grape cultivars, called teinturier or dyer grapes, contain anthocyanins in both the skin and flesh. These phenolic compounds exhibit excellent coloring ability, and as antioxidants, they are important bioactive compounds in food crops. In this work, anthocyanin patterns of grape berries of fifteen teinturier varieties collected from the gene bank located at Pécs in the southwest of Hungary were compared. Anthocyanin profiles of numerous varieties originating from Hungary such as ‘Bíborkadarka’, ‘Kármin’, ‘Kurucvér’, and ‘Turán’ are reported for the first time. Anthocyanins extracted separately from the skin and juice were analyzed using high-performance liquid chromatography coupled with a photodiode array detector. For the identification of compounds, high-resolution orbitrap mass spectrometry was used. All in all, twenty-one anthocyanins were identified and quantified. We found that anthocyanin patterns differed significantly in the skin and juice for all investigated cultivars. For Vitis vinifera varieties, the predominant anthocyanin in the skin was malvidin-3-O-glucoside, while the main pigment in the juice was peonidin-3-O-glucoside. For the first time, a significant amount of diglucosides was detected in two Vitis Vinifera cultivars with a direct relationship. In general, the pigment composition of the skin was much more complex than that of the juice. The comparative study with presented patterns gives valuable and beneficial information from a chemotaxonomical point of view. Our results also help to choose the appropriate teinturier varieties with the desired anthocyanins for food coloring or winemaking purposes