Automatic improvement of apache spark queries using semantics-preserving program reduction

© 2016 ACM. Apache Spark is a popular framework for large-scale data analytics. Unfortunately, Spark's performance can be difficult to optimise, since queries freely expressed in source code are not amenable to traditional optimisation techniques. This article describes Hylas, a tool for automatically optimising Spark queries embedded in source code via the application of semantics-preserving transformations. The transformation method is inspired by functional programming techniques of "deforestation", which eliminate intermediate data structures from a computation. This contrasts with approaches defined entirely within structured query formats such as Spark SQL. Hylas can identify certain computationally expensive operations and ensure that performing them creates no superfluous data structures. This optimisation leads to significant improvements in execution time, with over 10,000 times improvement observed in some cases

The Business of AI Startups

New machine learning techniques have led to an acceleration of “artificial intelligence” (AI). Numerous papers have projected substantial job losses based on assessments of technical feasibility. But what is the actual impact? This paper reports on a survey of commercial AI startups, documenting rich detail about their businesses and their impacts on their customers. These firms report benefits of AI that are more often about enhancing human capabilities than replacing them. Their applications more often increase professional, managerial, and marketing jobs and decrease manual, clerical, and frontline service jobs. These startups sell to firms of different sizes, in different industries and nations, but the distribution of activity is distinct from that of larger firms. Firms serving EU customers appear to use higher levels of data protection