TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning

The increasing of pre-trained models has significantly facilitated the performance on limited data tasks with transfer learning. However, progress on transfer learning mainly focuses on optimizing the weights of pre-trained models, which ignores the structure mismatch between the model and the target task. This paper aims to improve the transfer performance from another angle - in addition to tuning the weights, we tune the structure of pre-trained models, in order to better match the target task. To this end, we propose TransTailor, targeting at pruning the pre-trained model for improved transfer learning. Different from traditional pruning pipelines, we prune and fine-tune the pre-trained model according to the target-aware weight importance, generating an optimal sub-model tailored for a specific target task. In this way, we transfer a more suitable sub-structure that can be applied during fine-tuning to benefit the final performance. Extensive experiments on multiple pre-trained models and datasets demonstrate that TransTailor outperforms the traditional pruning methods and achieves competitive or even better performance than other state-of-the-art transfer learning methods while using a smaller model. Notably, on the Stanford Dogs dataset, TransTailor can achieve 2.7% accuracy improvement over other transfer methods with 20% fewer FLOPs.Comment: This paper has been accepted by AAAI202

Constraining the denudation process in the eastern Sichuan Basin, China using low-temperature thermochronology and vitrinite reflectance data

The temperature history of samples and maximum palaeogeothermal profiles of boreholes were reconstructed based on low‐temperature thermochronology and vitrinite reflectance data, and the results provide limits for the timescale and amount of uplift–denudation of the eastern Sichuan Basin. The thermal history showed that the uplifting and cooling of eastern Sichuan Basin began around the Late Cretaceous (approximately 100–80 Ma). The region had experienced a continuous cooling process from the Late Cretaceous until the present, with the geothermal gradient decreasing from 32–36 °C/km to 20–23 °C/km. The amount of denudation at the Puguang region in north‐eastern Sichuan was approximately 2.3 km, whereas that at south‐eastern Sichuan was 1.9 km, and the erosion thickness in the eastern Sichuan fold belt that was revealed via the field samples is 2.3 ± 0.3–2.6 ± 0.3 km. The north‐eastern Sichuan experienced sustained cooling with inconspicuous fluctuations, whereas the thrust belt and the south‐eastern Sichuan Basin presented 2–4 stages with different cooling rates. It may indicate that the eastern Sichuan fold belt experienced a complex structural evolution, characterized by episodic upliftings and deformations since Late Cretaceous, while a different and gentle deformation took place in the northeastern Sichuan Basin

Probabilistic Constellation Shaping for OFDM-Based ISAC Signaling

Integrated Sensing and Communications (ISAC) has garnered significant attention as a promising technology for the upcoming sixth-generation wireless communication systems (6G). In pursuit of this goal, a common strategy is that a unified waveform, such as Orthogonal Frequency Division Multiplexing (OFDM), should serve dual-functional roles by enabling simultaneous sensing and communications (S&C) operations. However, the sensing performance of an OFDM communication signal is substantially affected by the randomness of the data symbols mapped from bit streams. Therefore, achieving a balance between preserving communication capability (i.e., the randomness) while improving sensing performance remains a challenging task. To cope with this issue, in this paper we analyze the ambiguity function of the OFDM communication signal modulated by random data. Subsequently, a probabilistic constellation shaping (PCS) method is proposed to devise the probability distributions of constellation points, which is able to strike a scalable S&C tradeoff of the random transmitted signal. Finally, the superiority of the proposed PCS method over conventional uniformly distributed constellations is validated through numerical simulations

Pollen morphology of selected tundra plants from the high Arctic of Ny-Ålesund, Svalbard

Documenting morphological features of modern pollen is fundamental for the identification of fossil pollen, which will assist researchers to reconstruct the vegetation and climate of a particular geologic period. This paper presents the pollen morphology of 20 species of tundra plants from the high Arctic of Ny-Ålesund, Svalbard, using light and scanning electron microscopy. The plants used in this study belong to 12 families: Brassicaceae, Caryophyllaceae, Cyperaceae, Ericaceae, Juncaceae, Papaveraceae, Poaceae, Polygonaceae, Ranunculaceae, Rosaceae, Salicaceae, and Scrophulariaceae. Pollen grain shapes included: spheroidal, subprolate, and prolate. Variable apertural patterns ranged from 2-syncolpate, 3-colpate, 3-(-4)-colpate, 3-(-5)-colpate, 3-colporate, 5-poroid, ulcerate, ulcus to pantoporate. Exine ornamentations comprised psilate, striate-perforate, reticulate, microechinate, microechinate-perforate, scabrate, granulate, and granulate-perforate. This study provided a useful reference for comparative studies of fossil pollen and for the reconstruction of paleovegetation and paleoclimate in Svalbard region of Arctic

Optimal Region Search with Submodular Maximization

Region search is an important problem in location based services due to its wide applications. In this paper, we study the problem of optimal region search with submodular maximization (ORS-SM). This problem considers a region as a connected subgraph. We compute an objective value over the locations in the region using a submodular function and a budget value by summing up the costs of edges in the region, and aim to search the region with the largest objective score under a budget value constraint. ORS-SM supports many applications such as the most diversified region search. We prove that the problem is NP-hard and develop two approximation algorithms with guaranteed error bounds. We conduct experiments on two applications using three real-world datasets. The results demonstrate that our algorithms can achieve high quality solutions and are faster than a state-of-the art method by orders of magnitude

Symbolic Discovery of Optimization Algorithms

We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ (\textit{Evo\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}tum}). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. The implementation of Lion is publicly available.Comment: 30 pages, update the tuning instruction

ATPT: Automate Typhoon Contingency Plan Generation from Text

Artificial intelligence (AI) planning models play an important role in decision support systems for disaster management e.g. typhoon contingency plan development. However, constructing an AI planning model always requires significant amount of manual effort, which becomes a bottleneck to emergency response in a time-critical situation. In this demonstration, we present a framework of automating a domain model of planning domain definition language from natural language input through deep learning techniques. We implement this framework in a typhoon response system and demonstrate automatic generation of typhoon contingency plan from official typhoon plan documents