Effective Long-Context Scaling of Foundation Models

We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks. Alongside these results, we provide an in-depth analysis on the individual components of our method. We delve into Llama's position encodings and discuss its limitation in modeling long dependencies. We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences

Exploring state-of-the-art advances in targeted nanomedicines for managing acute and chronic inflammatory lung diseases

Diagnosis and treatment of lung diseases pose serious challenges. Currently, diagnostic as well as therapeutic methods show poor efficacy toward drug-resistant bacterial infections, while chemotherapy causes toxicity and nonspecific delivery of drugs. Advanced treatment methods that cure lung-related diseases, by enabling drug bioavailability via nasal passages during mucosal formation, which interferes with drug penetration to targeted sites, are in demand. Nanotechnology confers several advantages. Currently, different nanoparticles, or their combinations, are being used to enhance targeted drug delivery. Nanomedicine, a combination of nanoparticles and therapeutic agents, that delivers drugs to targeted sites increases the bioavailability of drugs at these sites. Thus, nanotechnology is superior to conventional chemotherapeutic strategies. Here, the authors review the latest advancements in nanomedicine-based drug-delivery methods for managing acute and chronic inflammatory lung diseases

Exploring state-of-the-art advances in targeted nanomedicines for managing acute and chronic inflammatory lung diseases

Diagnosis and treatment of lung diseases pose serious challenges. Currently, diagnostic as well as therapeutic methods show poor efficacy toward drug-resistant bacterial infections, while chemotherapy causes toxicity and nonspecific delivery of drugs. Advanced treatment methods that cure lung-related diseases, by enabling drug bioavailability via nasal passages during mucosal formation, which interferes with drug penetration to targeted sites, are in demand. Nanotechnology confers several advantages. Currently, different nanoparticles, or their combinations, are being used to enhance targeted drug delivery. Nanomedicine, a combination of nanoparticles and therapeutic agents, that delivers drugs to targeted sites increases the bioavailability of drugs at these sites. Thus, nanotechnology is superior to conventional chemotherapeutic strategies. Here, the authors review the latest advancements in nanomedicine-based drug-delivery methods for managing acute and chronic inflammatory lung diseases

Measurement of inclusive J/ψ\psi pair production cross section in pp collisions at s=13\sqrt{s} = 13 TeV

International audienceThe production cross section of inclusive J/$\psi$ pairs in pp collisions at a centre-of-mass energy $\sqrt{s} = 13$ TeV is measured with ALICE. The measurement is performed for J/$\psi$ in the rapidity interval $2.5 0$. The production cross section of inclusive J/$\psi$ pairs is reported to be $10.3 \pm 2.3 {\rm (stat.)} \pm 1.3 {\rm (syst.)}$ nb in this kinematic interval. The contribution from non-prompt J/$\psi$ (i.e. originated from beauty-hadron decays) to the inclusive sample is evaluated. The results are discussed and compared with data

Inclusive and multiplicity dependent production of electrons from heavy-flavour hadron decays in pp and p−-Pb collisions

International audienceMeasurements of the production of electrons from heavy-flavour hadron decays in pp collisions at $\sqrt{s} = 13$ TeV at midrapidity with the ALICE detector are presented down to a transverse momentum ($p_{\rm T}$) of 0.2 GeV$/c$ and up to $p_{\rm T} = 35$ GeV$/c$, which is the largest momentum range probed for inclusive electron measurements in ALICE. In p$-$Pb collisions, the production cross section and the nuclear modification factor of electrons from heavy-flavour hadron decays are measured in the $p_{\rm T}$ range $0.5 < p_{\rm T} < 26$ GeV$/c$ at $\sqrt{s_{\rm NN}} = 8.16$ TeV. The nuclear modification factor is found to be consistent with unity within the statistical and systematic uncertainties. In both collision systems, first measurements of the yields of electrons from heavy-flavour hadron decays in different multiplicity intervals normalised to the multiplicity-integrated yield (self-normalised yield) at midrapidity are reported as a function of the self-normalised charged-particle multiplicity estimated at midrapidity. The self-normalised yields in pp and p$-$Pb collisions grow faster than linear with the self-normalised multiplicity. A strong $p_{\rm T}$ dependence is observed in pp collisions, where the yield of high-$p_{\rm T}$ electrons increases faster as a function of multiplicity than the one of low-$p_{\rm T}$ electrons. The measurement in p$-$Pb collisions shows no $p_{\rm T}$ dependence within uncertainties. The self-normalised yields in pp and p$-$Pb collisions are compared with measurements of other heavy-flavour, light-flavour, and strange particles, and with Monte Carlo simulations

Observation of medium-induced yield enhancement and acoplanarity broadening of low-pTp_\mathrm{T} jets from measurements in pp and central Pb−-Pb collisions at sNN=5.02\sqrt{s_{\rm NN}}=5.02 TeV

International audienceThe ALICE Collaboration reports the measurement of semi-inclusive distributions of charged-particle jets recoiling from a high transverse momentum (high $p_{\rm T}$) hadron trigger in proton$-$proton and central Pb$-$Pb collisions at $\sqrt{s_{\rm NN}} = 5.02$ TeV. A data-driven statistical method is used to mitigate the large uncorrelated background in central Pb$-$Pb collisions. Recoil jet distributions are reported for jet resolution parameter $R=0.2$, 0.4, and 0.5 in the range $7 < p_{\rm T,jet} < 140$ GeV$/c$ and trigger$-$recoil jet azimuthal separation $\pi/2 < \Delta\varphi < \pi$. The measurements exhibit a marked medium-induced jet yield enhancement at low $p_{\rm T}$ and at large azimuthal deviation from $\Delta\varphi\sim\pi$. The enhancement is characterized by its dependence on $\Delta\varphi$, which has a slope that differs from zero by 4.7$\sigma$. Comparisons to model calculations incorporating different formulations of jet quenching are reported. These comparisons indicate that the observed yield enhancement arises from the response of the QGP medium to jet propagation

Probing the Chiral Magnetic Wave with charge-dependent flow measurements in Pb-Pb collisions at the LHC

International audienceThe Chiral Magnetic Wave (CMW) phenomenon is essential to provide insights into the strong interaction in QCD, the properties of the quark-gluon plasma, and the topological characteristics of the early universe, offering a deeper understanding of fundamental physics in high-energy collisions. Measurements of the charge-dependent anisotropic flow coefficients are studied in Pb-Pb collisions at center-of-mass energy per nucleon-nucleon collision $\sqrt{s_{\mathrm{NN}}}=$ 5.02 TeV to probe the CMW. In particular, the slope of the normalized difference in elliptic ($v_{2}$) and triangular ($v_{3}$) flow coefficients of positively and negatively charged particles as a function of their event-wise normalized number difference, is reported for inclusive and identified particles. The slope $r_{3}^{\rm Norm}$ is found to be larger than zero and to have a magnitude similar to $r_{2}^{\rm Norm}$, thus pointing to a large background contribution for these measurements. Furthermore, $r_{2}^{\rm Norm}$ can be described by a blast wave model calculation that incorporates local charge conservation. In addition, using the event shape engineering technique yields a fraction of CMW ($f_{\rm CMW}$) contribution to this measurement which is compatible with zero. This measurement provides the very first upper limit for $f_{\rm CMW}$, and in the 10-60% centrality interval it is found to be 26% (38%) at 95% (99.7%) confidence level

Charged-particle production as a function of the relative transverse activity classifier in pp, p−-Pb, and Pb−-Pb collisions at the LHC

International audienceMeasurements of charged-particle production in pp, p$-$Pb, and Pb$-$Pb collisions in the toward, away, and transverse regions with the ALICE detector are discussed. These regions are defined event-by-event relative to the azimuthal direction of the charged trigger particle, which is the reconstructed particle with the largest transverse momentum ($p_{\mathrm{T}}^{\rm trig}$) in the range $8<p_{\mathrm{T}}^{\rm trig}<15$ GeV$/c$. The toward and away regions contain the primary and recoil jets, respectively; both regions are accompanied by the underlying event (UE). In contrast, the transverse region perpendicular to the direction of the trigger particle is dominated by the so-called UE dynamics, and includes also contributions from initial- and final-state radiation. The relative transverse activity classifier, $R_{\mathrm{T}}=N_{\mathrm{ch}}^{\mathrm{T}}/\langle N_{\mathrm{ch}}^{\mathrm{T}}\rangle$, is used to group events according to their UE activity, where $N_{\mathrm{ch}}^{\mathrm{T}}$ is the charged-particle multiplicity per event in the transverse region and $\langle N_{\mathrm{ch}}^{\mathrm{T}}\rangle$ is the mean value over the whole analysed sample. The energy dependence of the $R_{\mathrm{T}}$ distributions in pp collisions at $\sqrt{s}=2.76$, 5.02, 7, and 13 TeV is reported, exploring the Koba-Nielsen-Olesen (KNO) scaling properties of the multiplicity distributions. The first measurements of charged-particle $p_{\rm T}$ spectra as a function of $R_{\mathrm{T}}$ in the three azimuthal regions in pp, p$-$Pb, and Pb$-$Pb collisions at $\sqrt{s_{\rm NN}}=5.02$ TeV are also reported. Data are compared with predictions obtained from the event generators PYTHIA 8 and EPOS LHC. This set of measurements is expected to contribute to the understanding of the origin of collective-like effects in small collision systems (pp and p$-$Pb)