Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective

We present a new dataset condensation framework termed Squeeze, Recover and Relabel (SRe$^2$L) that decouples the bilevel optimization of model and synthetic data during training, to handle varying scales of datasets, model architectures and image resolutions for effective dataset condensation. The proposed method demonstrates flexibility across diverse dataset scales and exhibits multiple advantages in terms of arbitrary resolutions of synthesized images, low training cost and memory consumption with high-resolution training, and the ability to scale up to arbitrary evaluation network architectures. Extensive experiments are conducted on Tiny-ImageNet and full ImageNet-1K datasets. Under 50 IPC, our approach achieves the highest 42.5% and 60.8% validation accuracy on Tiny-ImageNet and ImageNet-1K, outperforming all previous state-of-the-art methods by margins of 14.5% and 32.9%, respectively. Our approach also outperforms MTT by approximately 52$\times$ (ConvNet-4) and 16$\times$ (ResNet-18) faster in speed with less memory consumption of 11.6$\times$ and 6.4$\times$ during data synthesis. Our code and condensed datasets of 50, 200 IPC with 4K recovery budget are available at https://zeyuanyin.github.io/projects/SRe2L/.Comment: Technical repor

One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, layer-wise structure search that learns individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adapts to new tasks through not only weights but also additional dimensions like activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured vision benchmarks, achieving superior accuracy with fewer parameters and computations. The proposed method on LLaMA-1 and LLaMA-2 also show considerable enhancements compared to the original LoRA in the language domain. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications. Code and models are available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.Comment: Technical report. v2: Add LLaMA-1&2 results. Code and models at https://github.com/Arnav0400/ViT-Slim/tree/master/GLoR

Generalized-Equiangular Geometry CT: Concept and Shift-Invariant FBP Algorithms

With advanced X-ray source and detector technologies being continuously developed, non-traditional CT geometries have been widely explored. Generalized-Equiangular Geometry CT (GEGCT) architecture, in which an X-ray source might be positioned radially far away from the focus of arced detector array that is equiangularly spaced, is of importance in many novel CT systems and designs. GEGCT, unfortunately, has no theoretically exact and shift-invariant analytical image reconstruction algorithm in general. In this study, to obtain fast and accurate reconstruction from GEGCT and to promote its system design and optimization, an in-depth investigation on a group of approximate Filtered BackProjection (FBP) algorithms with a variety of weighting strategies has been conducted. The architecture of GEGCT is first presented and characterized by using a normalized-radial-offset distance (NROD). Next, shift-invariant weighted FBP-type algorithms are derived in a unified framework, with pre-filtering, filtering, and post-filtering weights. Three viable weighting strategies are then presented including a classic one developed by Besson in the literature and two new ones generated from a curvature fitting and from an empirical formula, where all of the three weights can be expressed as certain functions of NROD. After that, an analysis of reconstruction accuracy is conducted with a wide range of NROD. We further stretch the weighted FBP-type algorithms to GEGCT with dynamic NROD. Finally, the weighted FBP algorithm for GEGCT is extended to a three-dimensional form in the case of cone-beam scan with a cylindrical detector array.Comment: 31 pages, 13 figure

Data-Free Neural Architecture Search via Recursive Label Calibration

This paper aims to explore the feasibility of neural architecture search (NAS) given only a pre-trained model without using any original training data. This is an important circumstance for privacy protection, bias avoidance, etc., in real-world scenarios. To achieve this, we start by synthesizing usable data through recovering the knowledge from a pre-trained deep neural network. Then we use the synthesized data and their predicted soft-labels to guide neural architecture search. We identify that the NAS task requires the synthesized data (we target at image domain here) with enough semantics, diversity, and a minimal domain gap from the natural images. For semantics, we propose recursive label calibration to produce more informative outputs. For diversity, we propose a regional update strategy to generate more diverse and semantically-enriched synthetic data. For minimal domain gap, we use input and feature-level regularization to mimic the original data distribution in latent space. We instantiate our proposed framework with three popular NAS algorithms: DARTS, ProxylessNAS and SPOS. Surprisingly, our results demonstrate that the architectures discovered by searching with our synthetic data achieve accuracy that is comparable to, or even higher than, architectures discovered by searching from the original ones, for the first time, deriving the conclusion that NAS can be done effectively with no need of access to the original or called natural data if the synthesis method is well designed.Comment: ECCV 202