Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture

Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts and better performance. However, existing architectures such as Transformers scale quadratically along both these axes. We ask: are there performant architectures that can scale sub-quadratically along sequence length and model dimension? We introduce Monarch Mixer (M2), a new architecture that uses the same sub-quadratic primitive along both sequence length and model dimension: Monarch matrices, a simple class of expressive structured matrices that captures many linear transforms, achieves high hardware efficiency on GPUs, and scales sub-quadratically. As a proof of concept, we explore the performance of M2 in three domains: non-causal BERT-style language modeling, ViT-style image classification, and causal GPT-style language modeling. For non-causal BERT-style modeling, M2 matches BERT-base and BERT-large in downstream GLUE quality with up to 27% fewer parameters, and achieves up to 9.1$\times$ higher throughput at sequence length 4K. On ImageNet, M2 outperforms ViT-b by 1% in accuracy, with only half the parameters. Causal GPT-style models introduce a technical challenge: enforcing causality via masking introduces a quadratic bottleneck. To alleviate this bottleneck, we develop a novel theoretical view of Monarch matrices based on multivariate polynomial evaluation and interpolation, which lets us parameterize M2 to be causal while remaining sub-quadratic. Using this parameterization, M2 matches GPT-style Transformers at 360M parameters in pretraining perplexity on The PILE--showing for the first time that it may be possible to match Transformer quality without attention or MLPs.Comment: NeurIPS 2023 (Oral

Factor graph based detection approach for high-mobility OFDM systems with large FFT modes

In this article, a novel detector design is proposed for orthogonal frequency division multiplexing (OFDM) systems over frequency selective and time varying channels. Namely, we focus on systems with large OFDM symbol lengths where design and complexity constraints have to be taken into account and many of the existing ICI reduction techniques can not be applied. We propose a factor graph (FG) based approach for maximum a posteriori (MAP) symbol detection which exploits the frequency diversity introduced by the ICI in the OFDM symbol. The proposed algorithm provides high diversity orders allowing to outperform the free-ICI performance in high-mobility scenarios with an inherent parallel structure suitable for large OFDM block sizes. The performance of the mentioned near-optimal detection strategy is analyzed over a general bit-interleaved coded modulation (BICM) system applying low-density parity-check (LDPC) codes. The inclusion of pilot symbols is also considered in order to analyze how they assist the detection process

Discovery of Miocene adakitic dacite from the Eastern Pontides Belt (NE Turkey) and a revised geodynamic model for the late Cenozoic evolution of the Eastern Mediterranean region

The Cenozoic magmatic record within the ca. 500 km long eastern Pontides orogen, located within the Alpine metallogenic belt, is critical to evaluate the tectonic history and geodynamic evolution of the eastern Mediterranean region. In this paper we report for the first time late Miocene adakitic rocks from the southeastern part of the eastern Pontides belt and present results from geochemical and Sr–Nd isotopic studies as well as zircon U–Pb geochronology. The Tavdagi dacite that we investigate in this study is exposed as round or ellipsoidal shaped bodies, sills, and dikes in the southeastern part of the belt. Zircons in the dacite show euhedral crystal morphology with oscillatory zoning and high Th/U values (up to 1.69) typical of magmatic origin. Zircon LA–ICPMS analysis yielded a weighted mean 206Pb/238U age of 7.86 ± 0.15 Ma. SHRIMP analyses of zircons with typical magmatic zoning from another sample yielded a weighted mean 206Pb/238U age of 8.79 ± 0.19 Ma. Both ages are identical and constrain the timing of dacitic magmatism as late Miocene. The Miocene Tavdagi dacite shows adakitic affinity with high SiO2 (68.95–71.41 wt.%), Al2O3 (14.88–16.02 wt.%), Na2O (3.27–4.12 wt.%), Sr (331.4–462.1 ppm), Sr/Y (85–103.7), LaN/YbN (34.3–50.9) and low Y (3.2–5 ppm) values. Their initial 143Nd/144Nd (0.512723–0.512736) and 87Sr/86Sr (0.70484–0.70494) ratios are, respectively, lower and higher than those of normal oceanic crust. The geological, geochemical and isotopic data suggest that the adakitic magmatism was generated by partial melting of the mafic lower crust in the southeastern part of the eastern Pontide belt during the late Miocene. Based on the results presented in this study and a synthesis of the geological and tectonic information on the region, we propose that the entire northern edge of the eastern Pontides–Lesser Caucasus–Elbruz magmatic arc was an active continental margin during the Cenozoic. We identify a migration of the Cenozoic magmatism towards north over time resulting from the roll-back of the southward subducted Tethys oceanic lithosphere. Slab break-off during Pliocene is proposed to have triggered asthenospheric upwelling and partial melting of the subduction-modified mantle wedge which generated the alkaline magmatic rocks exposed in the northern part of the magmatic arc.Yener Eyuboglu, M.Santosh, Keewook Yi, Osman Bektaş, Sanghoon Kwo

Petrogenesis and U-Pb zircon chronology of adakitic porphyries within the Kop ultramafic massif (eastern Pontides orogenic belt, NE Turkey)

Abstract not availableYener Eyuboglu, Francis O. Dudas, M. Santosh, Keewook Yi, Sanghoon Kwon, Enver Akaryal

Where are the remnants of a Jurassic ocean in the eastern Mediterranean region?

Abstract not availableYener Eyuboglu, Francis O. Dudas, M. Santosh, Yilin Xiao, Keewook Yi, Nilanjan Chatterjee, Fu-Yuan Wu, Osman Bekta

Late Miocene K-rich volcanism in the Eslamieh Peninsula (Saray), NW Iran: Implications for geodynamic evolution of the Turkish–Iranian High Plateau

Post-collisional volcanism in northwestern Iran is represented by the Saray high-K rocks including leucite-bearing under-saturated and leucite-free silica saturated rocks. We report Ar–Ar age data which constrain the age as ca. 11Ma (lateMiocene).Most of clinopyroxene phenocrysts fromthe volcanic rocks have complex oscillatory zoning, with high Ti and Al cores, low Ti and high Almantled clinopyroxenes, grading into lowTi and Al outer rims. All the rocks are highly enriched in incompatible trace elements and have identical Sr–Nd–Pb isotopes. Enrichment in incompatible elements and other geochemical features for the Saray lavas suggest a metasomatized subcontinental lithospheric mantle (SCLM) as the magma source. The negative Nb–Ta–Ti anomalies for the Saray lavas compare with the features of subduction-related magmatism with negligible contamination with ancient crustal components. The highly radiogenic 87Sr/86Sr and 207Pb/204Pb isotopic values of the Saray lavas imply the involvement of slab terrigenous sediments and/or a continental lithosphere. Isotopically, the volcanic rocks define a binary trend, representing 5–8% mixing between the primary mantle and sediment melts. Our melting models suggest residual garnet in the source and are incompatible with partial melting of amphibole and/or phlogopite bearing lherzolites, although the complex geochemical features might indicate the result of mixing between melts produced by different sources or a homogenous melt passing through a compositionally-zoned mantle during multiple stages of partial melting and melt migration. The geochronological, geochemical and isotopic data for the Saray rocks suggest that these Late Miocene magmas were derived from a small degree of partial melting of subduction-metasomatized (subcontinental) lithospheric mantle source in a post-collisional setting

Cenozoic forearc gabbros from the northern zone of the Eastern Pontides Orogenic Belt, NE Turkey: implications for slab window magmatism and convergent margin tectonics

Abstract not availableYener Eyuboglu, Francis O. Dudas, M. Santosh, Di-Cheng Zhu, Keewook Yi, Nilanjan Chatterjee, Youn-Joong Jeong, Enver Akaryalı, Ze Li