Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representations

Joint-embedding based learning (e.g., SimCLR, MoCo, DINO) and reconstruction-based learning (e.g., BEiT, SimMIM, MAE) are the two leading paradigms for self-supervised learning of vision transformers, but they differ substantially in their transfer performance. Here, we aim to explain these differences by analyzing the impact of these objectives on the structure and transferability of the learned representations. Our analysis reveals that reconstruction-based learning features are significantly dissimilar to joint-embedding based learning features and that models trained with similar objectives learn similar features even across architectures. These differences arise early in the network and are primarily driven by attention and normalization layers. We find that joint-embedding features yield better linear probe transfer for classification because the different objectives drive different distributions of information and invariances in the learned representation. These differences explain opposite trends in transfer performance for downstream tasks that require spatial specificity in features. Finally, we address how fine-tuning changes reconstructive representations to enable better transfer, showing that fine-tuning re-organizes the information to be more similar to pre-trained joint embedding models

Do SSL Models Have D\'ej\`a Vu? A Case of Unintended Memorization in Self-supervised Learning

Self-supervised learning (SSL) algorithms can produce useful image representations by learning to associate different parts of natural images with one another. However, when taken to the extreme, SSL models can unintendedly memorize specific parts in individual training samples rather than learning semantically meaningful associations. In this work, we perform a systematic study of the unintended memorization of image-specific information in SSL models -- which we refer to as d\'ej\`a vu memorization. Concretely, we show that given the trained model and a crop of a training image containing only the background (e.g., water, sky, grass), it is possible to infer the foreground object with high accuracy or even visually reconstruct it. Furthermore, we show that d\'ej\`a vu memorization is common to different SSL algorithms, is exacerbated by certain design choices, and cannot be detected by conventional techniques for evaluating representation quality. Our study of d\'ej\`a vu memorization reveals previously unknown privacy risks in SSL models, as well as suggests potential practical mitigation strategies. Code is available at https://github.com/facebookresearch/DejaVu

A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation

Self-Supervised Learning (SSL) models rely on a pretext task to learn representations. Because this pretext task differs from the downstream tasks used to evaluate the performance of these models, there is an inherent misalignment or pretraining bias. A commonly used trick in SSL, shown to make deep networks more robust to such bias, is the addition of a small projector (usually a 2 or 3 layer multi-layer perceptron) on top of a backbone network during training. In contrast to previous work that studied the impact of the projector architecture, we here focus on a simpler, yet overlooked lever to control the information in the backbone representation. We show that merely changing its dimensionality -- by changing only the size of the backbone's very last block -- is a remarkably effective technique to mitigate the pretraining bias. It significantly improves downstream transfer performance for both Self-Supervised and Supervised pretrained models

Seismoelectric wave propagation numerical modelling in partially saturated materials

International audienceTo better understand and interpret seismoelectric measurements acquired over vadose environments, both the existing theory and the wave propagation modelling programmes, available for saturated materials, should be extended to partial saturation conditions. We propose here an extension of Pride's equations aiming to take into account partially saturated materials, in the case of a water-air mixture. This new set of equations was incorporated into an existing seismoelectric wave propagation modelling code, originally designed for stratified saturated media. This extension concerns both the mechanical part, using a generalization of the Biot-Gassmann theory, and the electromagnetic part, for which dielectric permittivity and electrical conductivity were expressed against water saturation. The dynamic seismoelectric coupling was written as a function of the streaming potential coefficient, which depends on saturation, using four different relations derived from recent laboratory or theoretical studies. In a second part, this extended programme was used to synthesize the seismoelectric response for a layered medium consisting of a partially saturated sand overburden on top of a saturated sandstone half-space. Subsequent analysis of the modelled amplitudes suggests that the typically very weak interface response (IR) may be best recovered when the shallow layer exhibits low saturation. We also use our programme to compute the seismoelectric response of a capillary fringe between a vadose sand overburden and a saturated sand half-space. Our first modelling results suggest that the study of the seismoelectric IR may help to detect a sharp saturation contrast better than a smooth saturation transition. In our example, a saturation contrast of 50 per cent between a fully saturated sand half-space and a partially saturated shallow sand layer yields a stronger IR than a stepwise decrease in saturation

PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning

Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible to (i) render as many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation. Despite such promise, the use of synthetic image data is still limited -- and often played down -- mainly due to their lack of realism. Most works therefore rely on datasets of real images, which have often been scraped from public images on the internet, and may have issues with regards to privacy, bias, and copyright, while offering little control over how objects precisely appear. In this work, we present a path to democratize the use of photorealistic synthetic data: we develop a new generation of interactive environments for representation learning research, that offer both controllability and realism. We use the Unreal Engine, a powerful game engine well known in the entertainment industry, to produce PUG (Photorealistic Unreal Graphics) environments and datasets for representation learning. In this paper, we demonstrate the potential of PUG to enable more rigorous evaluations of vision models

Antibacterial Properties of Polyphenols: Characterization and QSAR (Quantitative Structure–Activity Relationship) Models

Besides their established antioxidant activity, many phenolic compounds may exhibit significant antibacterial activity. Here, the effect of a large dataset of 35 polyphenols on the growth of 6 foodborne pathogenic or food-spoiling bacterial strains, three Gram-positive ones (Staphylococcus aureus, Bacillus subtilis, and Listeria monocytogenes) and three Gram-negative ones (Escherichia coli, Pseudomonas aeruginosa, and Salmonella Enteritidis), have been characterized. As expected, the effects of phenolic compounds were highly heterogeneous ranging from bacterial growth stimulation to antibacterial activity and depended on bacterial strains. The effect on bacterial growth of each of the polyphenols was expressed as relative Bacterial Load Difference (BLD) between a culture with and without (control) polyphenols at a 1 g L−1 concentration after 24 h incubation at 37°C. Reliable Quantitative Structure-Activity Relationship (QSAR) models were developed (regardless of polyphenol class or the mechanism of action involved) to predict BLD for E. coli, S. Enteritidis, S. aureus, and B. subtilis, unlike for L. monocytogenes and P. aeruginosa. L. monocytogenes was generally sensitive to polyphenols whereas P. aeruginosa was not. No satisfactory models predicting the BLD of P. aeruginosa and L. monocytogenes were obtained due to their specific and quite constant behavior toward polyphenols. The main descriptors involved in reliable QSAR models were the lipophilicity and the electronic and charge properties of the polyphenols. The models developed for the two Gram-negative bacteria (E. coli, S. Enteritidis) were comparable suggesting similar mechanisms of toxic action. This was not clearly observed for the two Gram-positive bacteria (S. aureus and B. subtilis). Interestingly, a preliminary evaluation by Microbial Adhesion To Solvents (MATS) measurements of surface properties of the two Gram-negative bacteria for which QSAR models were based on similar physico-chemical descriptors, revealed that MATS results were also quite similar. Moreover, the MATS results of the two Gram-positive bacterial strains S. aureus and B. subtilis for which QSARs were not based on similar physico-chemical descriptors also strongly differed. These observations suggest that the antibacterial activity of most of polyphenols likely depends on interactions between polyphenols and bacterial cells surface, although the surface properties of the bacterial strains should be further investigated with other techniques than MATS

Effect of interactions of plant phenolics with bovine meat proteins on their antibacterial activity

The activity of 5 phenolics totally inhibiting the growth of Staphylococcus aureus CNRZ3 at a 1 g L−1 concentration in Mueller-Hinton broth for 24 h incubation at 37 °C was reevaluated at 37 °C for 24 h, 15 °C for 6 days, or 6 °C for 8 days in the presence of up to 20% (w/w) bovine meat proteins to mimic the temperature of refrigerated storage of bovine meat and its protein content, respectively. These changes affected in a different way the antibacterial activity of the 5 phenolics. Isobutyl-4-hydroxybenzoate kept its bactericidal activity, while naphthazarin was bactericidal at 6 °C and 15 °C but not at 37 °C in the presence of bovine meat proteins. Gallocyanin was bactericidal at 37 °C up to a 5% (w/w) protein content in the medium but not at 15 °C or 6 °C. Resveratrol and chrysin always lost their bacteriostatic activity when bovine meat proteins were added. The partition coefficient at 6 °C of each phenolic between a 20% (w/w) bovine meat extract suspension with and without proteins was determined. The antibacterial activity reduction of phenolics in the presence of bovine meat proteins was correlated with their affinity for bovine meat protein

Molecular Adaptation of Photoprotection: Triplet States in Light-Harvesting Proteins

The photosynthetic light-harvesting systems of purple bacteria and plants both utilize specific carotenoids as quenchers of the harmful (bacterio)chlorophyll triplet states via triplet-triplet energy transfer. Here, we explore how the binding of carotenoids to the different types of light-harvesting proteins found in plants and purple bacteria provides adaptation in this vital photoprotective function. We show that the creation of the carotenoid triplet states in the light-harvesting complexes may occur without detectable conformational changes, in contrast to that found for carotenoids in solution. However, in plant light-harvesting complexes, the triplet wavefunction is shared between the carotenoids and their adjacent chlorophylls. This is not observed for the antenna proteins of purple bacteria, where the triplet is virtually fully located on the carotenoid molecule. These results explain the faster triplet-triplet transfer times in plant light-harvesting complexes. We show that this molecular mechanism, which spreads the location of the triplet wavefunction through the pigments of plant light-harvesting complexes, results in the absence of any detectable chlorophyll triplet in these complexes upon excitation, and we propose that it emerged as a photoprotective adaptation during the evolution of oxygenic photosynthesis

La praxéologie motrice face aux alternatives paradigmatiques : de l'identité des STAPS

International audienc