Regularizing Neural Networks with Meta-Learning Generative Models

This paper investigates methods for improving generative data augmentation for deep learning. Generative data augmentation leverages the synthetic samples produced by generative models as an additional dataset for classification with small dataset settings. A key challenge of generative data augmentation is that the synthetic data contain uninformative samples that degrade accuracy. This is because the synthetic samples do not perfectly represent class categories in real data and uniform sampling does not necessarily provide useful samples for tasks. In this paper, we present a novel strategy for generative data augmentation called meta generative regularization (MGR). To avoid the degradation of generative data augmentation, MGR utilizes synthetic samples in the regularization term for feature extractors instead of in the loss function, e.g., cross-entropy. These synthetic samples are dynamically determined to minimize the validation losses through meta-learning. We observed that MGR can avoid the performance degradation of na\"ive generative data augmentation and boost the baselines. Experiments on six datasets showed that MGR is effective particularly when datasets are smaller and stably outperforms baselines.Comment: Accepted to NeurIPS 202

Transfer Learning with Pre-trained Conditional Generative Models

Transfer learning is crucial in training deep neural networks on new target tasks. Current transfer learning methods always assume at least one of (i) source and target task label spaces overlap, (ii) source datasets are available, and (iii) target network architectures are consistent with source ones. However, holding these assumptions is difficult in practical settings because the target task rarely has the same labels as the source task, the source dataset access is restricted due to storage costs and privacy, and the target architecture is often specialized to each task. To transfer source knowledge without these assumptions, we propose a transfer learning method that uses deep generative models and is composed of the following two stages: pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). PP trains a target architecture with an artificial dataset synthesized by using conditional source generative models. P-SSL applies SSL algorithms to labeled target data and unlabeled pseudo samples, which are generated by cascading the source classifier and generative models to condition them with target samples. Our experimental results indicate that our method can outperform the baselines of scratch training and knowledge distillation.Comment: 24 pages, 6 figure

On the origin and evolution of the asteroid Ryugu: A comprehensive geochemical perspective

Presented here are the observations and interpretations from a comprehensive analysis of 16 representative particles returned from the C-type asteroid Ryugu by the Hayabusa2 mission. On average Ryugu particles consist of 50% phyllosilicate matrix, 41% porosity and 9% minor phases, including organic matter. The abundances of 70 elements from the particles are in close agreement with those of CI chondrites. Bulk Ryugu particles show higher δ18O, Δ17O, and ε54Cr values than CI chondrites. As such, Ryugu sampled the most primitive and least-thermally processed protosolar nebula reservoirs. Such a finding is consistent with multi-scale H-C-N isotopic compositions that are compatible with an origin for Ryugu organic matter within both the protosolar nebula and the interstellar medium. The analytical data obtained here, suggests that complex soluble organic matter formed during aqueous alteration on the Ryugu progenitor planetesimal (several 10’s of km), <2.6 Myr after CAI formation. Subsequently, the Ryugu progenitor planetesimal was fragmented and evolved into the current asteroid Ryugu through sublimation

A pristine record of outer Solar System materials from asteroid Ryugu’s returned sample

Volatile and organic-rich C-type asteroids may have been one of the main sources of Earth’s water. Our best insight into their chemistry is currently provided by carbonaceous chondritic meteorites, but the meteorite record is biased: only the strongest types survive atmospheric entry and are then modified by interaction with the terrestrial environment. Here we present the results of a detailed bulk and microanalytical study of pristine Ryugu particles, brought to Earth by the Hayabusa2 spacecraft. Ryugu particles display a close compositional match with the chemically unfractionated, but aqueously altered, CI (Ivuna-type) chondrites, which are widely used as a proxy for the bulk Solar System composition. The sample shows an intricate spatial relationship between aliphatic-rich organics and phyllosilicates and indicates maximum temperatures of ~30 °C during aqueous alteration. We find that heavy hydrogen and nitrogen abundances are consistent with an outer Solar System origin. Ryugu particles are the most uncontaminated and unfractionated extraterrestrial materials studied so far, and provide the best available match to the bulk Solar System composition

A dehydrated space-weathered skin cloaking the hydrated interior of Ryugu

Without a protective atmosphere, space-exposed surfaces of airless Solar System bodies gradually experience an alteration in composition, structure and optical properties through a collective process called space weathering. The return of samples from near-Earth asteroid (162173) Ryugu by Hayabusa2 provides the first opportunity for laboratory study of space-weathering signatures on the most abundant type of inner solar system body: a C-type asteroid, composed of materials largely unchanged since the formation of the Solar System. Weathered Ryugu grains show areas of surface amorphization and partial melting of phyllosilicates, in which reduction from Fe3+ to Fe2+ and dehydration developed. Space weathering probably contributed to dehydration by dehydroxylation of Ryugu surface phyllosilicates that had already lost interlayer water molecules and to weakening of the 2.7 µm hydroxyl (–OH) band in reflectance spectra. For C-type asteroids in general, this indicates that a weak 2.7 µm band can signify space-weathering-induced surface dehydration, rather than bulk volatile loss

Observation of Interface Deformation in Sodium Polytungstate Solution–Silicone Oil System due to Single Rising Bubble

The interfacial behavior between sodium polytungstate solution (SPTS) and silicone oil (SO) due to a single rising bubble was directly observed to investigate the influence of the Eötvös number on the flow characteristics. We found that the transient behavior of the jet under the bubble strongly depended on the SPTS density in the range of 1000–3000 kg/m3. Although the SPTS film generated in the SO influenced the detention time of the jet under the bubble, the lifetime of the film did not depend on the SPTS density

Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks

Few-shot learning for neural networks (NNs) is an important problem that aims to train NNs with a few data. The main challenge is how to avoid overfitting since over-parameterized NNs can easily overfit to such small dataset. Previous work (e.g. MAML by Finn et al. 2017) tackles this challenge by meta-learning, which learns how to learn from a few data by using various tasks. On the other hand, one conventional approach to avoid overfitting is restricting hypothesis spaces by endowing sparse NN structures like convolution layers in computer vision. However, although such manually-designed sparse structures are sample-efficient for sufficiently large datasets, they are still insufficient for few-shot learning. Then the following questions naturally arise: (1) Can we find sparse structures effective for few-shot learning by meta-learning? (2) What benefits will it bring in terms of meta-generalization? In this work, we propose a novel meta-learning approach, called Meta-ticket, to find optimal sparse subnetworks for few-shot learning within randomly initialized NNs. We empirically validated that Meta-ticket successfully discover sparse subnetworks that can learn specialized features for each given task. Due to this task-wise adaptation ability, Meta-ticket achieves superior meta-generalization compared to MAML-based methods especially with large NNs.Comment: Code will be available at https://github.com/dchiji-ntt/meta-ticke