MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks

Recently, there has been a rapid advancement in research on Large Language Models (LLMs), resulting in significant progress in several Natural Language Processing (NLP) tasks. Consequently, there has been a surge in LLM evaluation research to comprehend the models' capabilities and limitations. However, much of this research has been confined to the English language, leaving LLM building and evaluation for non-English languages relatively unexplored. There has been an introduction of several new LLMs, necessitating their evaluation on non-English languages. This study aims to expand our MEGA benchmarking suite by including six new datasets to form the MEGAVERSE benchmark. The benchmark comprises 22 datasets covering 81 languages, including low-resource African languages. We evaluate several state-of-the-art LLMs like GPT-3.5-Turbo, GPT4, PaLM2, and Llama2 on the MEGAVERSE datasets. Additionally, we include two multimodal datasets in the benchmark and assess the performance of the LLaVa-v1.5 model. Our experiments suggest that GPT4 and PaLM2 outperform the Llama models on various tasks, notably on low-resource languages, with GPT4 outperforming PaLM2 on more datasets than vice versa. However, issues such as data contamination must be addressed to obtain an accurate assessment of LLM performance on non-English languages.Comment: 23 pages, 30 figures and 1 tabl

Inclusive quarkonium production in pp collisions at √s = 5.02 TeV

This article reports on the inclusive production cross section of several quarkonium states, $\textrm{J}/\psi$, $\psi \mathrm{(2S)}$, $\Upsilon \mathrm (1S)$, $\Upsilon \mathrm{(2S)}$, and $\Upsilon \mathrm{(3S)}$, measured with the ALICE detector at the LHC, in pp collisions at $\sqrt{s} = 5.02$ TeV. The analysis is performed in the dimuon decay channel at forward rapidity ($2.5< y < 4$). The integrated cross sections and transverse-momentum ($p_{\textrm{T}}$) and rapidity ($y$) differential cross sections for $\textrm{J}/\psi$, $\psi \mathrm{(2S)}$, $\Upsilon \mathrm (1S)$, and the $\psi \mathrm{(2S)}$-to-$\textrm{J}/\psi$ cross section ratios are presented. The integrated cross sections, assuming unpolarized quarkonia, are: $\sigma _{\textrm{J}/\psi }$ ($p_{\textrm{T}} <20$ GeV/c) = 5.88 ± 0.03 ± 0.34$~\mu$b, $\sigma _{\psi \mathrm{(2S)}}$ ($p_{\textrm{T}} <12$ GeV/c) = 0.87 ± 0.06 ± 0.10$~\mu$b, $\sigma _{\Upsilon \mathrm (1S)}$ ($p_{\textrm{T}} <15$ GeV/c) = 45.5 ± 3.9 ± 3.5 nb, $\sigma _{\Upsilon \mathrm{(2S)}}$ ($p_{\textrm{T}} <15$ GeV/c) = 22.4 ± 3.2 ± 2.7 nb, and $\sigma _{\Upsilon \mathrm{(3S)}}$ ($p_{\textrm{T}} <15$ GeV/c) = 4.9 ± 2.2 ± 1.0 nb, where the first (second) uncertainty is the statistical (systematic) one. For the first time, the cross sections of the three $\Upsilon$ states, as well as the $\psi \mathrm{(2S)}$ one as a function of $p_{\textrm{T}}$ and $y$, are measured at $\sqrt{s} = 5.02$ TeV at forward rapidity. These measurements also significantly extend the $\textrm{J}/\psi$ $p_{\textrm{T}}$ reach and supersede previously published results. A comparison with ALICE measurements in pp collisions at $\sqrt{s} = 2.76$, 7, 8, and 13 TeV is presented and the energy dependence of quarkonium production cross sections is discussed. Finally, the results are compared with the predictions from several production models.publishedVersio

Inclusive J/ψ production at midrapidity in pp collisions at √s=13 TeV

We report on the inclusive $\text {J}/\psi$ production cross section measured at the CERN Large Hadron Collider in proton–proton collisions at a center-of-mass energy $\sqrt{s}~=~13$ TeV. The $\text {J}/\psi$ mesons are reconstructed in the $\text {e}^{+}\text {e}^{-}$ decay channel and the measurements are performed at midrapidity ($|y|<0.9$) in the transverse-momentum interval $0<p_{\mathrm{T}} <40$ GeV/$c$, using a minimum-bias data sample corresponding to an integrated luminosity $L_{\text {int}} = 32.2~\text {nb}^{-1}$ and an Electromagnetic Calorimeter triggered data sample with $L_{\text {int}} = 8.3~\mathrm {pb}^{-1}$. The $p_{\mathrm{T}}$-integrated $\text {J}/\psi$ production cross section at midrapidity, computed using the minimum-bias data sample, is $\text {d}\sigma /\text {d}y|_{y=0} = 8.97\pm 0.24~(\text {stat})\pm 0.48~(\text {syst})\pm 0.15~(\text {lumi})~\mu \text {b}$. An approximate logarithmic dependence with the collision energy is suggested by these results and available world data, in agreement with model predictions. The integrated and $p_{\mathrm{T}}$-differential measurements are compared with measurements in pp collisions at lower energies and with several recent phenomenological calculations based on the non-relativistic QCD and Color Evaporation models.publishedVersio

KS0 - and (anti-) Λ -hadron correlations in pp collisions at √s=13 TeV

Two-particle Azimuthal correlations are measured with the ALICE apparatus in pp collisions at s√=13s=13 TeV to explore strangeness- and multiplicity-related effects in the fragmentation of jets and the transition regime between bulk and hard production, probed with the condition that a strange meson (KS0KS0) or baryon (ΛΛ) with transverse momentum pT>3pT>3 GeV/cc is produced. Azimuthal correlations between kaons or ΛΛ hyperons with other hadrons are presented at midrapidity for a broad range of the trigger (3<ptriggT<203<pTtrigg<20 GeV/cc) and associated particle pTpT (1 GeV/cc <passocT<ptriggT<pTassoc<pTtrigg), for minimum-bias events and as a function of the event multiplicity. The near- and away-side peak yields are compared for the case of either KS0KS0 or ΛΛ(Λ¯¯¯¯Λ¯) being the trigger particle with that of inclusive hadrons (a sample dominated by pions). In addition, the measurements are compared with predictions from PYTHIA 8 and EPOS LHC event generators.publishedVersio

First measurements of N-subjettiness in central Pb-Pb collisions at √sNN = 2.76 TeV

The ALICE Collaboration reports the first fully-corrected measurements of the N-subjettiness observable for track-based jets in heavy-ion collisions. This study is performed using data recorded in pp and Pb-Pb collisions at centre-of-mass energies of s√ = 7 TeV and sNN−−−√ = 2.76 TeV, respectively. In particular the ratio of 2-subjettiness to 1-subjettiness, τ2/τ1, which is sensitive to the rate of two-pronged jet substructure, is presented. Energy loss of jets traversing the strongly interacting medium in heavy-ion collisions is expected to change the rate of two-pronged substructure relative to vacuum. The results are presented for jets with a resolution parameter of R = 0.4 and charged jet transverse momentum of 40 ≤ pT,jet ≤ 60 GeV/c, which constitute a larger jet resolution and lower jet transverse momentum interval than previous measurements in heavy-ion collisions. This has been achieved by utilising a semi-inclusive hadron-jet coincidence technique to suppress the larger jet combinatorial background in this kinematic region. No significant modification of the τ2/τ1 observable for track-based jets in Pb-Pb collisions is observed relative to vacuum PYTHIA6 and PYTHIA8 references at the same collision energy. The measurements of τ2/τ1, together with the splitting aperture angle ∆R, are also performed in pp collisions at s√ = 7 TeV for inclusive jets. These results are compared with PYTHIA calculations at s√ = 7 TeV, in order to validate the model as a vacuum reference for the Pb-Pb centre-of-mass energy. The PYTHIA references for τ2/τ1 are shifted to larger values compared to the measurement in pp collisions. This hints at a reduction in the rate of two-pronged jets in Pb-Pb collisions compared to pp collisions.publishedVersio

Anisotropic flow of identified hadrons in Xe-Xe collisions at √sNN = 5.44 TeV

Measurements of elliptic (v2) and triangular (v3) flow coefficients of π±, K±, p+$\overline{\mathrm{p}}$, ${\mathrm{K}}_{\mathrm{S}}^0$, and Λ+$\overline{\Lambda}$ obtained with the scalar product method in Xe-Xe collisions at $\sqrt{s_{\mathrm{NN}}}$ = 5.44 TeV are presented. The results are obtained in the rapidity range |y| < 0.5 and reported as a function of transverse momentum, pT, for several collision centrality classes. The flow coefficients exhibit a particle mass dependence for pT < 3 GeV/c, while a grouping according to particle type (i.e., meson and baryon) is found at intermediate transverse momenta (3 < pT < 8 GeV/c). The magnitude of the baryon v2 is larger than that of mesons up to pT = 6 GeV/c. The centrality dependence of the shape evolution of the pT-differential v2 is studied for the various hadron species. The v2 coefficients of π±, K±, and p+$\overline{\mathrm{p}}$ are reproduced by MUSIC hydrodynamic calculations coupled to a hadronic cascade model (UrQMD) for pT < 1 GeV/c. A comparison with vn measurements in the corresponding centrality intervals in Pb-Pb collisions at $\sqrt{s_{\mathrm{NN}}}$ = 5.02 TeV yields an enhanced v2 in central collisions and diminished value in semicentral collisions.publishedVersio

Experimental Evidence for an Attractive p-φ Interaction

This Letter presents the first experimental evidence of the attractive strong interaction between a proton and a ϕ meson. The result is obtained from two-particle correlations of combined p-ϕ⊕ p¯-ϕ pairs measured in high-multiplicity pp collisions at s√=13  TeV by the ALICE Collaboration. The spin-averaged scattering length and effective range of the p-ϕ interaction are extracted from the fully corrected correlation function employing the Lednický-Lyuboshits approach. In particular, the imaginary part of the scattering length vanishes within uncertainties, indicating that inelastic processes do not play a prominent role for the p-ϕ interaction. These data demonstrate that the interaction is dominated by elastic p-ϕ scattering. Furthermore, an analysis employing phenomenological Gaussian- and Yukawa-type potentials is conducted. Under the assumption of the latter, the N-ϕ coupling constant is found to be gN-ϕ=0.14±0.03(stat)±0.02(syst). This work provides valuable experimental input to accomplish a self-consistent description of the N-ϕ interaction, which is particularly relevant for the more fundamental studies on partial restoration of chiral symmetry in nuclear medium.publishedVersio

Measurement of anti-3He nuclei absorption in matter and impact on their propagation in the Galaxy

In our Galaxy, light antinuclei composed of antiprotons and antineutrons can be produced through high-energy cosmic-ray collisions with the interstellar medium or could also originate from the annihilation of dark-matter particles that have not yet been discovered. On Earth, the only way to produce and study antinuclei with high precision is to create them at high-energy particle accelerators. Although the properties of elementary antiparticles have been studied in detail, the knowledge of the interaction of light antinuclei with matter is limited. We determine the disappearance probability of ${}^{3}\overline{{{{\rm{He}}}}}$ when it encounters matter particles and annihilates or disintegrates within the ALICE detector at the Large Hadron Collider. We extract the inelastic interaction cross section, which is then used as an input to the calculations of the transparency of our Galaxy to the propagation of ${}^{3}\overline{{{{\rm{He}}}}}$ stemming from dark-matter annihilation and cosmic-ray interactions within the interstellar medium. For a specific dark-matter profile, we estimate a transparency of about 50%, whereas it varies with increasing ${}^{3}\overline{{{{\rm{He}}}}}$ momentum from 25% to 90% for cosmic-ray sources. The results indicate that ${}^{3}\overline{{{{\rm{He}}}}}$ nuclei can travel long distances in the Galaxy, and can be used to study cosmic-ray interactions and dark-matter annihilation.publishedVersio

First study of the two-body scattering involving charm hadrons

This article presents the first measurement of the interaction between charm hadrons and nucleons. The two-particle momentum correlations of pD− and ¯pD+ pairs are measured by the ALICE Collaboration in high-multiplicity pp collisions at √s=13  TeV. The data are compatible with the Coulomb-only interaction hypothesis within (1.1–1.5)σ. The level of agreement slightly improves if an attractive nucleon (N)¯D strong interaction is considered, in contrast to most model predictions which suggest an overall repulsive interaction. This measurement allows for the first time an estimation of the 68% confidence level interval for the isospin I=0 inverse scattering length of the N¯D state f−10,I=0∈[−0.4,0.9]  fm−1, assuming negligible interaction for the isospin I=1 channel.publishedVersio

Measurement of inclusive charged-particle b-jet production in pp and p-Pb collisions at √sNN = 5.02 TeV

A measurement of the inclusive b-jet production cross section is presented in pp and p-Pb collisions at √ sNN = 5.02 TeV, using data collected with the ALICE detector at the LHC. The jets were reconstructed in the central rapidity region |η| < 0.5 from charged particles using the anti-kT algorithm with resolution parameter R = 0.4. Identification of b jets exploits the long lifetime of b hadrons, using the properties of secondary vertices and impact parameter distributions. The pT-differential inclusive production cross section of b jets, as well as the corresponding inclusive b-jet fraction, are reported for pp and p-Pb collisions in the jet transverse momentum range 10 ≤ pT,ch jet ≤ 100 GeV/c, together with the nuclear modification factor, R b-jet pPb . The analysis thus extends the lower pT limit of b-jet measurements at the LHC. The nuclear modification factor is found to be consistent with unity, indicating that the production of b jets in p-Pb at √ sNN = 5.02 TeV is not affected by cold nuclear matter effects within the current precision. The measurements are well reproduced by POWHEG NLO pQCD calculations with PYTHIA fragmentation.publishedVersio