Characterizing how 'distributional' NLP corpora distance metrics are

A corpus of vector-embedded text documents has some empirical distribution. Given two corpora, we want to calculate a single metric of distance (e.g., Mauve, Frechet Inception) between them. We describe an abstract quality, called `distributionality', of such metrics. A non-distributional metric tends to use very local measurements, or uses global measurements in a way that does not fully reflect the distributions' true distance. For example, if individual pairwise nearest-neighbor distances are low, it may judge the two corpora to have low distance, even if their two distributions are in fact far from each other. A more distributional metric will, in contrast, better capture the distributions' overall distance. We quantify this quality by constructing a Known-Similarity Corpora set from two paraphrase corpora and calculating the distance between paired corpora from it. The distances' trend shape as set element separation increases should quantify the distributionality of the metric. We propose that Average Hausdorff Distance and energy distance between corpora are representative examples of non-distributional and distributional distance metrics, to which other metrics can be compared, to evaluate how distributional they are.Comment: Published in the August 2023 Joint Statistical Meetings proceeding

Classifier Data Quality: A Geometric Complexity Based Method for Automated Baseline And Insights Generation

Testing Machine Learning (ML) models and AI-Infused Applications (AIIAs), or systems that contain ML models, is highly challenging. In addition to the challenges of testing classical software, it is acceptable and expected that statistical ML models sometimes output incorrect results. A major challenge is to determine when the level of incorrectness, e.g., model accuracy or F1 score for classifiers, is acceptable and when it is not. In addition to business requirements that should provide a threshold, it is a best practice to require any proposed ML solution to out-perform simple baseline models, such as a decision tree. We have developed complexity measures, which quantify how difficult given observations are to assign to their true class label; these measures can then be used to automatically determine a baseline performance threshold. These measures are superior to the best practice baseline in that, for a linear computation cost, they also quantify each observation' classification complexity in an explainable form, regardless of the classifier model used. Our experiments with both numeric synthetic data and real natural language chatbot data demonstrate that the complexity measures effectively highlight data regions and observations that are likely to be misclassified.Comment: Accepted to EDSMLS workshop at AAAI conferenc

Understanding the Properties of Generated Corpora

Models for text generation have become focal for many research tasks and especially for the generation of sentence corpora. However, understanding the properties of an automatically generated text corpus remains challenging. We propose a set of tools that examine the properties of generated text corpora. Applying these tools on various generated corpora allowed us to gain new insights into the properties of the generative models. As part of our characterization process, we found remarkable differences in the corpora generated by two leading generative technologies

Azimuthal anisotropy of charged jet production in root s(NN)=2.76 TeV Pb-Pb collisions

We present measurements of the azimuthal dependence of charged jet production in central and semi-central root s(NN) = 2.76 TeV Pb-Pb collisions with respect to the second harmonic event plane, quantified as nu(ch)(2) (jet). Jet finding is performed employing the anti-k(T) algorithm with a resolution parameter R = 0.2 using charged tracks from the ALICE tracking system. The contribution of the azimuthal anisotropy of the underlying event is taken into account event-by-event. The remaining (statistical) region-to-region fluctuations are removed on an ensemble basis by unfolding the jet spectra for different event plane orientations independently. Significant non-zero nu(ch)(2) (jet) is observed in semi-central collisions (30-50% centrality) for 20 <p(T)(ch) (jet) <90 GeV/c. The azimuthal dependence of the charged jet production is similar to the dependence observed for jets comprising both charged and neutral fragments, and compatible with measurements of the nu(2) of single charged particles at high p(T). Good agreement between the data and predictions from JEWEL, an event generator simulating parton shower evolution in the presence of a dense QCD medium, is found in semi-central collisions. (C) 2015 CERN for the benefit of the ALICE Collaboration. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).Peer reviewe

Forward-central two-particle correlations in p-Pb collisions at root s(NN)=5.02 TeV

Two-particle angular correlations between trigger particles in the forward pseudorapidity range (2.5 2GeV/c. (C) 2015 CERN for the benefit of the ALICE Collaboration. Published by Elsevier B. V.Peer reviewe

Event-shape engineering for inclusive spectra and elliptic flow in Pb-Pb collisions at root(NN)-N-S=2.76 TeV

Peer reviewe

Elliptic flow of muons from heavy-flavour hadron decays at forward rapidity in Pb-Pb collisions at root s(NN)=2.76TeV

The elliptic flow, v(2), of muons from heavy-flavour hadron decays at forward rapidity (2.5 <y <4) is measured in Pb-Pb collisions at root s(NN)= 2.76TeVwith the ALICE detector at the LHC. The scalar product, two- and four-particle Q cumulants and Lee-Yang zeros methods are used. The dependence of the v(2) of muons from heavy-flavour hadron decays on the collision centrality, in the range 0-40%, and on transverse momentum, p(T), is studied in the interval 3 <p(T)<10 GeV/c. A positive v(2) is observed with the scalar product and two-particle Q cumulants in semi-central collisions (10-20% and 20-40% centrality classes) for the p(T) interval from 3 to about 5GeV/c with a significance larger than 3 sigma, based on the combination of statistical and systematic uncertainties. The v(2) magnitude tends to decrease towards more central collisions and with increasing pT. It becomes compatible with zero in the interval 6 <p(T)<10 GeV/c. The results are compared to models describing the interaction of heavy quarks and open heavy-flavour hadrons with the high-density medium formed in high-energy heavy-ion collisions. (C) 2015 CERN for the benefit of the ALICE Collaboration. Published by Elsevier B.V.Peer reviewe