Lessons Learned from Crowdsourcing Complex Engineering Tasks

<div><p>Crowdsourcing</p><p>Crowdsourcing is the practice of obtaining needed ideas, services, or content by requesting contributions from a large group of people. Amazon Mechanical Turk is a web marketplace for crowdsourcing microtasks, such as answering surveys and image tagging. We explored the limits of crowdsourcing by using Mechanical Turk for a more complicated task: analysis and creation of wind simulations.</p><p>Harnessing Crowdworkers for Engineering</p><p>Our investigation examined the feasibility of using crowdsourcing for complex, highly technical tasks. This was done to determine if the benefits of crowdsourcing could be harnessed to accurately and effectively contribute to solving complex real world engineering problems. Of course, untrained crowds cannot be used as a mere substitute for trained expertise. Rather, we sought to understand how crowd workers can be used as a large pool of labor for a preliminary analysis of complex data.</p><p>Virtual Wind Tunnel</p><p>We compared the skill of the anonymous crowd workers from Amazon Mechanical Turk with that of civil engineering graduate students, making a first pass at analyzing wind simulation data. For the first phase, we posted analysis questions to Amazon crowd workers and to two groups of civil engineering graduate students. A second phase of our experiment instructed crowd workers and students to create simulations on our Virtual Wind Tunnel website to solve a more complex task.</p><p>Conclusions</p><p>With a sufficiently comprehensive tutorial and compensation similar to typical crowd-sourcing wages, we were able to enlist crowd workers to effectively complete longer, more complex tasks with competence comparable to that of graduate students with more comprehensive, expert-level knowledge. Furthermore, more complex tasks require increased communication with the workers. As tasks become more complex, the employment relationship begins to become more akin to outsourcing than crowdsourcing. Through this investigation, we were able to stretch and explore the limits of crowdsourcing as a tool for solving complex problems.</p></div

A flow visualization view of a simulation containing a triangular cross-section.

<p>This figure shows a colored flow visualization of the fluid velocity. The colors vary from red through white to blue. The red represents fast air, blue represents slow air, and white represents median speed air. This graph is meant to allow the reader to see the varying wind speeds around a structure.</p

Mesh view of a simulation containing a triangular cross-section.

<p>This figure shows a mesh representing the area around a building's cross-section. The triangle in the center of the image represents the cross-section. The many smaller triangles represent “data points” or “pixels” in the image, each one will be a point at which velocity and pressure will be computed at each step of the simulation.</p

A. Phase 2 submission quality vs. the number of simulations run by participant. B. Phase 2 time logged vs. submission quality.

<p>In Fig 9A we show the number of simulations which a participant ran before submitting one, compared with the quality of their submission. The quality was subjectively determined as “good”, “fair” or “poor” according to the images and results in a manner similar to Phase 1. A look at the graph shows that better results seem to weakly coincide with more simulation attempts, but not without significant outliers. In Fig 9B we show the amount of time for which a user was logged-in, compared with the quality of their submission. The quality was subjectively determined as “good”, “fair” or “poor” according to the images and results in a manner similar to Phase 1. The time is measured in minutes. A look at the graph shows that better results seem to weakly coincide with more time spent, but not without significant outliers.</p

Four different graphs displaying wind tunnel data. A. Two flow view simulations contrasting a potentially useful, grainy simulation to a unhelpful, pristine simulation. B. A Velocity in the x-component versus location graph. C. A lift coefficient as a function of time graph.

<p>Fig 4A shows two sample graphs which were in the tutorial of the task. This is a prime example of how some simulations may be more visually appealing but do not show useful data. The first simulation shows a hexagonal cross-section building with a non-oscillating wake. The second simulation shows a hexagonal cross-section building with an oscillating wake. The first image is stated to be not worth keeping whereas the second is said to be worth keeping.</p

Screenshot of the Virtual Wind Tunnel Website.

<p>A screenshot of a page of the Virtual Wind Tunnel website that contains the tools for creating the simulations and was used by the crowdworkers in phase two of the experiment. This page shows various figures which are generated in each simulation. This simulation result shows a cross-section, simulation mesh, flow visualization and a velocity plot. Users can click on these images to see larger versions and get access to more results and images. There is also a summary of the simulation provided below the figures followed by a listing of the parameters used.</p

Four different cross-sections of buildings.

<p>The four images in this figure are shapes of buildings when viewed from the above. The cross-sections included are four different building shapes: a triangle, a diamond, a square, and a hexagon. This was shown in the tutorial and is called the plan view.</p

Toward a Framework for Evaluating Software Success: A Proposed First Step

<p>Software is a particularly critical technology in many computational science and engineering (CSE) sectors. Consequently, software is increasingly becoming an important component in the evaluation of competitive grants and the execution of research projects. As a result, software can be viewed as a scholarly contribution and has been proposed as a new factor to consider in tenure and promotion processes. However, existing metrics for evaluating the capability, use, reusability, or success of software are sorely lacking. This lack of software metrics permits the development of software based on poor development practices, which in turn allows poorly written software to “fly under the radar” in the scientific community and persist undetected. The absence of evaluation by knowledgeable peers often leads to the establishment and adoption of tools based on aggressive promotion by developers, ease-of-use, and other peripheral factors, hindering the sustainability, usefulness, and uptake of software and even leading to unreliable scientific findings. All of these factors mean that addressing the current lack of software evaluation metrics and methods is not just a question of increasing scientific productivity, but also a matter of preventing poor science.</p> <p>As a first step toward creating a methodology and framework for developing and evolving software success metrics for the CSE community, we propose the creation of a software “peer-review group.” This group, comprised of grant recipients funded to develop sustainable software, would meet periodically to evaluate their own and each others’ software, developing and refining success metrics along the way. We envision the group as a pilot test for a potential larger-scale effort to establish a more formal framework for software success metrics and evaluation.</p