Random Teachers are Good Teachers

In this work, we investigate the implicit regularization induced by teacher-student learning dynamics in self-distillation. To isolate its effect, we describe a simple experiment where we consider teachers at random initialization instead of trained teachers. Surprisingly, when distilling a student into such a random teacher, we observe that the resulting model and its representations already possess very interesting characteristics; (1) we observe a strong improvement of the distilled student over its teacher in terms of probing accuracy. (2) The learned representations are data-dependent and transferable between different tasks but deteriorate strongly if trained on random inputs. (3) The student checkpoint contains sparse subnetworks, so-called lottery tickets, and lies on the border of linear basins in the supervised loss landscape. These observations have interesting consequences for several important areas in machine learning: (1) Self-distillation can work solely based on the implicit regularization present in the gradient dynamics without relying on any dark knowledge, (2) self-supervised learning can learn features even in the absence of data augmentation and (3) training dynamics during the early phase of supervised training do not necessarily require label information. Finally, we shed light on an intriguing local property of the loss landscape: the process of feature learning is strongly amplified if the student is initialized closely to the teacher. These results raise interesting questions about the nature of the landscape that have remained unexplored so far. Code is available at https://github.com/safelix/dinopl

Dune ages in the sand deserts of the southern Sahara and Sahel

In this paper we aim to document the history of aeolian processes within the southern Sahara as part of the INQUA Dune Atlas. We review available luminescence ages for sand dunes across the southern Sahara to develop an improved understanding of the dune chronology on a regional basis and attempt to correlate periods of sand accumulation. This was achieved by analysing dune age by country, as well as by latitude and longitude. The results show a very patchy spatial distribution of dune ages with large gaps that encompass some of the largest sand seas. Despite these gaps, some related patterns in dune morphology and stratigraphy appear to be consistent between northern Nigeria and southern Mali where older linear dunes are distinct from younger Late Holocene transverse and barchanoid dunes. Elsewhere in Mauretania linear dunes with different orientations appear to have accumulated at different times, most likely in response to changes in atmospheric circulation. Regional climatic changes are identified where dunes are transgressed by lake deposits within endorehic basins. We identify four locations where dune accumulation is terminated by lacustrine transgressions, two of which, in Lake Chad and the Bodélé Depression, occur shortly after the last glacial maximum (LGM). The third example at Gobiero in Niger occurred later, in the early Holocene, around 8.4 ka and a fourth marks a later transgression of Palaeolake MegaChad after 4.7 ka. Larger-scale latitudinal and longitudinal distributions in dune ages across the southern Sahara do not show any consistent patterns, though this may be due to the small sample size relative to the study area. In addition, local variations in external controls such as wind regime, rainfall, vegetation and sand supply need to be considered, sometimes on a site by site basis. Limiting the analysis to dune ages determined using the single-aliquot regenerative-dose (SAR) protocol indicates a lack of dune preservation during the LGM and the Younger Dryas, times associated with increased dust input to the oceans which is assumed to indicate increased aeolian activity. The SAR dune dates suggest that preservation of dunes at the onset of succeeding humid intervals is an important component of the dune record. The most striking examples of this phenomenon occur where dunes are preserved within endorehic basins by lacustrine transgressions

Human Intracranial High Frequency Oscillations (HFOs) Detected by Automatic Time-Frequency Analysis

Objectives High frequency oscillations (HFOs) have been proposed as a new biomarker for epileptogenic tissue. The exact characteristics of clinically relevant HFOs and their detection are still to be defined. Methods We propose a new method for HFO detection, which we have applied to six patient iEEGs. In a first stage, events of interest (EoIs) in the iEEG were defined by thresholds of energy and duration. To recognize HFOs among the EoIs, in a second stage the iEEG was Stockwell-transformed into the time-frequency domain, and the instantaneous power spectrum was parameterized. The parameters were optimized for HFO detection in patient 1 and tested in patients 2–5. Channels were ranked by HFO rate and those with rate above half maximum constituted the HFO area. The seizure onset zone (SOZ) served as gold standard. Results The detector distinguished HFOs from artifacts and other EEG activity such as interictal epileptiform spikes. Computation took few minutes. We found HFOs with relevant power at frequencies also below the 80–500 Hz band, which is conventionally associated with HFOs. The HFO area overlapped with the SOZ with good specificity > 90% for five patients and one patient was re-operated. The performance of the detector was compared to two well-known detectors. Conclusions Compared to methods detecting energy changes in filtered signals, our second stage - analysis in the time-frequency domain - discards spurious detections caused by artifacts or sharp epileptic activity and improves the detection of HFOs. The fast computation and reasonable accuracy hold promise for the diagnostic value of the detector.ISSN:1932-620

Human intracranial high frequency oscillations (HFOs) detected by automatic time-frequency analysis

OBJECTIVES: High frequency oscillations (HFOs) have been proposed as a new biomarker for epileptogenic tissue. The exact characteristics of clinically relevant HFOs and their detection are still to be defined. METHODS: We propose a new method for HFO detection, which we have applied to six patient iEEGs. In a first stage, events of interest (EoIs) in the iEEG were defined by thresholds of energy and duration. To recognize HFOs among the EoIs, in a second stage the iEEG was Stockwell-transformed into the time-frequency domain, and the instantaneous power spectrum was parameterized. The parameters were optimized for HFO detection in patient 1 and tested in patients 2-5. Channels were ranked by HFO rate and those with rate above half maximum constituted the HFO area. The seizure onset zone (SOZ) served as gold standard. RESULTS: The detector distinguished HFOs from artifacts and other EEG activity such as interictal epileptiform spikes. Computation took few minutes. We found HFOs with relevant power at frequencies also below the 80-500 Hz band, which is conventionally associated with HFOs. The HFO area overlapped with the SOZ with good specificity > 90% for five patients and one patient was re-operated. The performance of the detector was compared to two well-known detectors. CONCLUSIONS: Compared to methods detecting energy changes in filtered signals, our second stage - analysis in the time-frequency domain - discards spurious detections caused by artifacts or sharp epileptic activity and improves the detection of HFOs. The fast computation and reasonable accuracy hold promise for the diagnostic value of the detector

Temporal and spectral characteristics of HFOs in individual patients.

<p>For each patient we present the total number of EoIs detected in Stage 1 and the number of HFOs accepted in Stage 2. Based on their peak frequency, HFOs were then classified into ripples (80–200 Hz) and FRs (200–500 Hz).</p

HFO in a neocortical recording.

<p>(<b>A</b>) Raw iEEG data 10 s epoch recorded from a frontal channel in patient 6. (<b>B</b>) Raw iEEG at extended time scale of 500 ms. (<b>C</b>) Filtered data with envelope (red line). The envelope satisfies the criteria for an EoI. The peak of the envelope is marked by dashed vertical lines in panels A, B and C. (<b>D</b>) Time frequency representation of the iEEG of panel B. The circle marks the peak of the envelope of the EoI. (<b>E</b>) Power spectral density (PSD, unit: 10log₁₀μV²Hz⁻¹) at the peak of the envelope. The high frequency peak is at 82 Hz, the trough at 40 Hz and the low frequency peak at 30 Hz for this event, which was accepted as HFO in Stage 2 of the detection.</p

Sharp artifact.

<p>(<b>A</b>) Raw iEEG data 10 s epoch from a frontal channel in patient 6. (<b>B</b>) Raw iEEG at extended time scale. (<b>C</b>) Filtered data (blue line) with envelope (red line). The envelope satisfies the criteria for an EoI (Stage 1 of detection). While the high-frequency activity is separated by a trough (<b>D, E</b>), it is excluded from acceptance as HFO because the peak of the spectral power appears at frequencies above 500 Hz.</p

Clinical data and implantation sites.

<p>Pathologies: FCD focal cortical dysplasia; HS hippocampal sclerosis. Procedures: DBS deep brain stimulation; LE extended lesionectomy; sAHE selective amygdala-hippocampectomy. Implantation sites: AL amygdala left; AR amygdala right; EL entorhinal cortex left; ER entorhinal cortex right; FAR frontal anterior right; FL frontal lobe; FPR frontal posterior right; HL hippocampus left; HR hippocampus right; MTL mesial temporal lobe; PL perirhinal cortex left; PR perirhinal cortex right; TBAL temporal basal anterior left; TBAR temporal basal anterior right; TBPL temporal basal posterior left; TBPR temporal basal posterior right; TR depth frontal right; TLL temporal lateral left.</p

HFO peak frequency distribution.

<p>(<b>A</b>) Total of all HFOs recorded from the four patients with temporomesial electrodes (N  =  1179). The sharp edge to low frequencies stems from the frequency threshold of 60 Hz of the detector. (<b>B</b>) Total of all HFOs recorded from the two patients with neocortical electrodes (N  =  3561). The distributions are fitted with a lognormal function (red line). The frequency peaks of HFOs from temporomesial recordings are more widely distributed than those from neocortical recordings.</p