AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors

This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial Intelligence Applications to Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including ~1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files $\times$ 8 tools), a software framework is designed to choreographed the experiment into a completely automated, time-synced, and reproducible workflow with substantial parallelization. A cost-benefit model was configured to integrate the tools' recall, precision, time to detection, and resource requirements into a single comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens through which to reason about the varied statistical viewpoints of the results. These statistical and cost-model results provide insights on state of commercial malware detection

Beyond the Hype: A Real-World Evaluation of the Impact and Cost of Machine Learning-Based Malware Detection

There is a lack of scientific testing of commercially available malware detectors, especially those that boast accurate classification of never-before-seen (i.e., zero-day) files using machine learning (ML). The result is that the efficacy and gaps among the available approaches are opaque, inhibiting end users from making informed network security decisions and researchers from targeting gaps in current detectors. In this paper, we present a scientific evaluation of four market-leading malware detection tools to assist an organization with two primary questions: (Q1) To what extent do ML-based tools accurately classify never-before-seen files without sacrificing detection ability on known files? (Q2) Is it worth purchasing a network-level malware detector to complement host-based detection? We tested each tool against 3,536 total files (2,554 or 72% malicious, 982 or 28% benign) including over 400 zero-day malware, and tested with a variety of file types and protocols for delivery. We present statistical results on detection time and accuracy, consider complementary analysis (using multiple tools together), and provide two novel applications of a recent cost-benefit evaluation procedure by Iannaconne & Bridges that incorporates all the above metrics into a single quantifiable cost. While the ML-based tools are more effective at detecting zero-day files and executables, the signature-based tool may still be an overall better option. Both network-based tools provide substantial (simulated) savings when paired with either host tool, yet both show poor detection rates on protocols other than HTTP or SMTP. Our results show that all four tools have near-perfect precision but alarmingly low recall, especially on file types other than executables and office files -- 37% of malware tested, including all polyglot files, were undetected.Comment: Includes Actionable Takeaways for SOC

Diffractive Dijet Production at sqrt(s)=630 and 1800 GeV at the Fermilab Tevatron

We report a measurement of the diffractive structure function $F_{jj}^D$ of the antiproton obtained from a study of dijet events produced in association with a leading antiproton in $\bar pp$ collisions at $\sqrt s=630$ GeV at the Fermilab Tevatron. The ratio of $F_{jj}^D$ at $\sqrt s=630$ GeV to $F_{jj}^D$ obtained from a similar measurement at $\sqrt s=1800$ GeV is compared with expectations from QCD factorization and with theoretical predictions. We also report a measurement of the $\xi$ ($x$-Pomeron) and $\beta$ ($x$ of parton in Pomeron) dependence of $F_{jj}^D$ at $\sqrt s=1800$ GeV. In the region $0.035<\xi<0.095$, $|t|<1$ GeV$^2$ and $\beta<0.5$, $F_{jj}^D(\beta,\xi)$ is found to be of the form $\beta^{-1.0\pm 0.1} \xi^{-0.9\pm 0.1}$, which obeys $\beta$-$\xi$ factorization.Comment: LaTeX, 9 pages, Submitted to Phys. Rev. Letter

A Study of B0 -> J/psi K(*)0 pi+ pi- Decays with the Collider Detector at Fermilab

We report a study of the decays B0 -> J/psi K(*)0 pi+ pi-, which involve the creation of a u u-bar or d d-bar quark pair in addition to a b-bar -> c-bar(c s-bar) decay. The data sample consists of 110 1/pb of p p-bar collisions at sqrt{s} = 1.8 TeV collected by the CDF detector at the Fermilab Tevatron collider during 1992-1995. We measure the branching ratios to be BR(B0 -> J/psi K*0 pi+ pi-) = (8.0 +- 2.2 +- 1.5) * 10^{-4} and BR(B0 -> J/psi K0 pi+ pi-) = (1.1 +- 0.4 +- 0.2) * 10^{-3}. Contributions to these decays are seen from psi(2S) K(*)0, J/psi K0 rho0, J/psi K*+ pi-, and J/psi K1(1270)

LSST: from Science Drivers to Reference Design and Anticipated Data Products

(Abridged) We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). A vast array of science will be enabled by a single wide-deep-fast sky survey, and LSST will have unique survey capability in the faint time domain. The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the Solar System, exploring the transient optical sky, and mapping the Milky Way. LSST will be a wide-field ground-based system sited at Cerro Pach\'{o}n in northern Chile. The telescope will have an 8.4 m (6.5 m effective) primary mirror, a 9.6 deg$^2$ field of view, and a 3.2 Gigapixel camera. The standard observing sequence will consist of pairs of 15-second exposures in a given field, with two such visits in each pointing in a given night. With these repeats, the LSST system is capable of imaging about 10,000 square degrees of sky in a single filter in three nights. The typical 5$\sigma$ point-source depth in a single visit in $r$ will be $\sim 24.5$ (AB). The project is in the construction phase and will begin regular survey operations by 2022. The survey area will be contained within 30,000 deg$^2$ with $\delta<+34.5^\circ$, and will be imaged multiple times in six bands, $ugrizy$, covering the wavelength range 320--1050 nm. About 90\% of the observing time will be devoted to a deep-wide-fast survey mode which will uniformly observe a 18,000 deg$^2$ region about 800 times (summed over all six bands) during the anticipated 10 years of operations, and yield a coadded map to $r\sim27.5$. The remaining 10\% of the observing time will be allocated to projects such as a Very Deep and Fast time domain survey. The goal is to make LSST data products, including a relational database of about 32 trillion observations of 40 billion objects, available to the public and scientists around the world.Comment: 57 pages, 32 color figures, version with high-resolution figures available from https://www.lsst.org/overvie

The Physics of the B Factories

This work is on the Physics of the B Factories. Part A of this book contains a brief description of the SLAC and KEK B Factories as well as their detectors, BaBar and Belle, and data taking related issues. Part B discusses tools and methods used by the experiments in order to obtain results. The results themselves can be found in Part C

The On-orbit Calibrations for the Fermi Large Area Telescope

The Large Area Telescope (LAT) on--board the Fermi Gamma ray Space Telescope began its on--orbit operations on June 23, 2008. Calibrations, defined in a generic sense, correspond to synchronization of trigger signals, optimization of delays for latching data, determination of detector thresholds, gains and responses, evaluation of the perimeter of the South Atlantic Anomaly (SAA), measurements of live time, of absolute time, and internal and spacecraft boresight alignments. Here we describe on orbit calibration results obtained using known astrophysical sources, galactic cosmic rays, and charge injection into the front-end electronics of each detector. Instrument response functions will be described in a separate publication. This paper demonstrates the stability of calibrations and describes minor changes observed since launch. These results have been used to calibrate the LAT datasets to be publicly released in August 2009.Comment: 60 pages, 34 figures, submitted to Astroparticle Physic

Search for Single-Top-Quark Production in p-pbar Collisions at sqrt(s)=1.8 TeV

We search for standard model single-top-quark production in the W-gluon fusion and W* channels using 106 pb^-1 of data from p-pbar collisions at sqrt(s)=1.8 TeV collected with the Collider Detector at Fermilab. We set an upper limit at 95% C.L. on the combined W-gluon fusion and W* single-top cross section of 14 pb, roughly six times larger than the standard model prediction. Separate 95% C.L. upper limits in the W-gluon fusion and W* channels are also determined and are found to be 13 and 18 pb, respectively.Comment: 6 pages, 2 figures; submitted to Phys. Rev. Let

Measurement of the Ratio of b Quark Production Cross Sections in Antiproton-Proton Collisions at 630 GeV and 1800 GeV

We report a measurement of the ratio of the bottom quark production cross section in antiproton-proton collisions at 630 GeV to 1800 GeV using bottom quarks with transverse momenta greater than 10.75 GeV identified through their semileptonic decays and long lifetimes. The measured ratio sigma(630)/sigma(1800) = 0.171 +/- .024 +/- .012 is in good agreement with next-to-leading order (NLO) quantum chromodynamics (QCD)