Spreadsheet Tools for Data Analysts

Spreadsheets are a natural fit for data analysis, combining a simple data storage and presentation layer with a programming language and basic debugging tools. Because spreadsheets are accessible and flexible, they are used by both novices and experts. Consequently, spreadsheets are hugely popular, with more than 750 million copies of Microsoft Excel installed worldwide. This popularity means that spreadsheets are the most popular programming language on the planet and the de facto tool for data analysis. Nevertheless, spreadsheets do not address a number of important tasks in a typical analyst\u27s pipeline, and their design frequently complicates them. This thesis describes three key challenges for analysts using spreadsheets. 1) Data wrangling is the process of converting or mapping data from a raw form into another form suitable for use with automated tools. 2) Data cleaning is the process of locating and correcting omitted or erroneous data. 3) Formula auditing is the process of finding and correcting spreadsheet program errors. These three tasks combined are estimated to occupy more than three quarters of a data analyst\u27s time. Furthermore, errors not caught during these steps have led to catastrophically bad decisions resulting in billions of dollars in losses. Advances in automated techniques for these tasks may result in dramatic savings in both time and money. Three novel programming language-based techniques were created to address these key tasks. The first, automatic layout transformation using examples, is a program synthesis-based technique that lets spreadsheet users perform data wrangling tasks automatically, at scale, and without programming. The second, data debugging, is technique for data cleaning that combines program analysis and statistical analysis to automatically find likely data errors. The third, spatio-structural program analysis unifies positional and dependence information and finds spreadsheet errors using a kind of anomaly analysis. Each technique was implemented as an end-user tool---FlaskRelate, CheckCell, and ExceLint respectively---in the form of a point-and-click plugin for Microsoft Excel. Our evaluation demonstrates that these techniques substantially improve user efficiency. Finally, because these tools build on each other in a complementary fashion, data analysts can run data wrangling, cleaning, and formula auditing tasks together in a single analysis pipeline

BPMN task instance streaming for efficient micro-task crowdsourcing processes

The Business Process Model and Notation (BPMN) is a standard for modeling and executing business processes with human or machine tasks. The semantics of tasks is usually discrete: a task has exactly one start event and one end event; for multi-instance tasks, all instances must complete before an end event is emitted. We propose a new task type and streaming connector for crowdsourcing able to run hundreds or thousands of micro-task instances in parallel. The two constructs provide for task streaming semantics that is new to BPMN, enable the modeling and efficient enactment of complex crowdsourcing scenarios, and are applicable also beyond the special case of crowdsourcing. We implement the necessary design and runtime support on top of Crowd- Flower, demonstrate the viability of the approach via a case study, and report on a set of runtime performance experiments

Detecting semantic social engineering attacks with the weakest link: Implementation and empirical evaluation of a human-as-a-security-sensor framework

The notion that the human user is the weakest link in information security has been strongly, and, we argue, rightly contested in recent years. Here, we take a step further showing that the human user can in fact be the strongest link for detecting attacks that involve deception, such as application masquerading, spearphishing, WiFi evil twin and other types of semantic social engineering. Towards this direction, we have developed a human-as-a-security-sensor framework and a practical implementation in the form of Cogni-Sense, a Microsoft Windows prototype application, designed to allow and encourage users to actively detect and report semantic social engineering attacks against them. Experimental evaluation with 26 users of different profiles running Cogni-Sense on their personal computers for a period of 45 days has shown that human sensors can consistently outperform technical security systems. Making use of a machine learning based approach, we also show that the reliability of each report, and consequently the performance of each human sensor, can be predicted in a meaningful and practical manner. In an organisation that employs a human-as-a-security-sensor implementation, such as Cogni-Sense, an attack is considered to have been detected if at least one user has reported it. In our evaluation, a small organisation consisting only of the 26 participants of the experiment would have exhibited a missed detection rate below 10%, down from 81% if only technical security systems had been used. The results strongly point towards the need to actively involve the user not only in prevention through cyber hygiene and user-centric security design, but also in active cyber threat detection and reporting

INFRARED EMISSION OF ATOMIC AND MOLECULAR RYDBERG OXYGEN (TIME-RESOLVED, FOURIER-SPECTROSCOPY, GLOW-DISCHARGE)

Time-resolved Fourier spectroscopy has been applied to the observation of Rydberg O(I) and O(,2) infrared emission from a pulsed discharge at 0.1 torr. The emission of a 2.5 meter long column through the negative glow was observed with a newly designed discharge apparatus. A computer-controlled interferometer step-servo was developed to enable 100 microsecond time- and 0.12 cm(\u27-1) spectral-resolution capabilities. The Rydberg features were enhanced with respect to those of contaminates by signal processing methods that utilized their individual temporal behavior. Line position determinations of the 7h - 5g, 7g - 5f, 5g - 4f and 6g - 4f O(I) lines have been made to an accuracy of 0.015 cm(\u27-1). Polarization constants of the O(II) core have been deduced from this data. Nine newly observed and four previously observed line-like features have been attributed to Rydberg O(,2)

Heat and Smoke Transport in a Residential-Scale Live Fire Training Facility: Experiments and Modeling

Understanding fire behavior is critical to effective tactical decision making on the fireground, particularly since fireground operations significantly impact the growth and spread of the fire. Computer-based simulation is a flexible, low-cost training methodology with proven success in fields such as pilot training, space, and military applications. Computer-based simulation may enhance fire behavior training and promote effective fireground decision making. This study evaluates the potential of the NIST Fire Dynamics Simulator (FDS) and Smokeview to be utilized as a part of a computer-based fire fighter trainer. Laboratory compartment fire experiments and full-scale fire experiments in a live-fire training facility were both conducted as part of the NIST Multiphase Study on Fire Fighter Safety and the Deployment of Resources. The laboratory experiments characterized the burning behavior of wood pallets to design a repeatable fire for use in the field experiments. The field experiments observed the effects of varying fire fighter deployment configurations on the performance times of fire fighter actions at a live fire training facility. These actions included opening the front door and fire suppression. Because the field experiments simulated numerous fire department responses to a repeatable fire, data were available to evaluate FDS simulation of heat and smoke spread, and changes in the thermal environment after the front door is opened and fire suppressed. In simulating the field experiments, the laboratory-measured heat release rate was used as an input. Given this assumption, this study has two objectives: 1) to determine if simulations accurately spread heat and smoke through a multi-level, multi-compartment live fire training facility 2) to determine if the simulations properly reproduce changes in the thermal environment that result from two typical fire fighter actions: opening the front door and fire suppression. In simulation, heat and smoke spread to measurement locations throughout the test structure at times closely matching experimentally measured times. Predictions of peak temperatures near the ceiling were within approximately 20% for all measurement locations. Hot gas layer temperature and depth were both predicted within 10% of the floor to ceiling height. After the front door was opened, temperature changes near the door at the highest and lowest measurement locations matched with temperature changes in the experiments. After fire suppression, FDS simulated temperature decay at a rate within the range measured in the field experiments and approximated the total rise of the hot gas layer interface in the burn compartment 250 seconds after suppression

FlashRelate: Extracting relational data from semi-structured spreadsheets using examples.

Abstract Spreadsheets store a tremendous amount of important data. One reason spreadsheets are so successful is that they are both easy to use and allow users great expressiveness in storing and manipulating their data. This flexibility comes at a price, as presentation elements are often combined with the underlying data model. As a result, many spreadsheets contain data in ad-hoc formats. These formats complicate the use of traditional relational tools which require data in a normalized form. Normalizing data from these formats is often tedious or requires programming, and often, a user may prefer the original presentation. We describe an approach that allows users to easily extract structured data from spreadsheets without programming. We make two contributions. First, we describe a novel domain specific language called FLARE that extends traditional regular expressions with spatial constraints. Second, we describe an algorithm called FLASHRELATE that can synthesize FLARE programs from user-provided positive and negative examples. Using 43 benchmarks drawn both from a standard spreadsheet corpus and from Excel user-help forums, we demonstrate that correct extraction programs can be synthesized quickly from a small number of examples. Our approach generalizes to many data-cleaning tasks on semi-structured spreadsheets