Changing the focus: worker-centric optimization in human-in-the-loop computations

A myriad of emerging applications from simple to complex ones involve human cognizance in the computation loop. Using the wisdom of human workers, researchers have solved a variety of problems, termed as “micro-tasks” such as, captcha recognition, sentiment analysis, image categorization, query processing, as well as “complex tasks” that are often collaborative, such as, classifying craters on planetary surfaces, discovering new galaxies (Galaxyzoo), performing text translation. The current view of “humans-in-the-loop” tends to see humans as machines, robots, or low-level agents used or exploited in the service of broader computation goals. This dissertation is developed to shift the focus back to humans, and study different data analytics problems, by recognizing characteristics of the human workers, and how to incorporate those in a principled fashion inside the computation loop. The first contribution of this dissertation is to propose an optimization framework and a real world system to personalize worker’s behavior by developing a worker model and using that to better understand and estimate task completion time. The framework judiciously frames questions and solicits worker feedback on those to update the worker model. Next, improving workers skills through peer interaction during collaborative task completion is studied. A suite of optimization problems are identified in that context considering collaborativeness between the members as it plays a major role in peer learning. Finally, “diversified” sequence of work sessions for human workers is designed to improve worker satisfaction and engagement while completing tasks

Crowd-sourcing with uncertain quality - an auction approach

This article addresses two important issues in crowd-sourcing: ex ante uncertainty about the quality and cost of different workers and strategic behaviour. We present a novel multi-dimensional auction that incentivises the workers to make partial enquiry into the task and to honestly report quality-cost estimates based on which the crowd-sourcer can choose the worker that offers the best value for money. The mechanism extends second score auction design to settings where the quality is uncertain and it provides incentives to both collect information and deliver desired qualities

Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence

Data in the form of pairwise comparisons arises in many domains, including preference elicitation, sporting competitions, and peer grading among others. We consider parametric ordinal models for such pairwise comparison data involving a latent vector $w^* \in \mathbb{R}^d$ that represents the "qualities" of the $d$ items being compared; this class of models includes the two most widely used parametric models--the Bradley-Terry-Luce (BTL) and the Thurstone models. Working within a standard minimax framework, we provide tight upper and lower bounds on the optimal error in estimating the quality score vector $w^*$ under this class of models. The bounds depend on the topology of the comparison graph induced by the subset of pairs being compared via its Laplacian spectrum. Thus, in settings where the subset of pairs may be chosen, our results provide principled guidelines for making this choice. Finally, we compare these error rates to those under cardinal measurement models and show that the error rates in the ordinal and cardinal settings have identical scalings apart from constant pre-factors.Comment: 39 pages, 5 figures. Significant extension of arXiv:1406.661

People with dementia and carer preferences for home support services in early-stage dementia

OBJECTIVES: To examine people with dementia and carer preferences for home support attributes in early-stage dementia, building on the paucity of evidence in this area. METHOD: Preferences from 44 people with dementia and 103 carers, recruited through memory clinics and an online questionnaire, were assessed with a Discrete Choice Experiment survey, with attributes informed by an evidence synthesis and lay consultation. A conditional logit model was used to estimate preference weights for the attributes within a home support ‘package’. RESULTS: The most preferred attributes were support with personal feelings and concerns, provided by a trained counsellor at home (coefficient 0.67, p = <0.001) and information on coping with dementia, provided by an experienced worker at home (coefficient 0.59, p = <0.001). However, for people with dementia, opportunities for social and recreational activities were considered the most important (coefficient 0.48, p = <0.001). CONCLUSIONS: These preferences concur with emerging evidence on psychosocial interventions in dementia. Support with personal feelings, information and social engagement are important components. Additionally, knowledge of preferences of people with dementia and their carers can identify other attributes that may be important to effectiveness in ‘living well’ but for which there remains limited evidence

Eliciting Preferences for Collectively Financed Health Programmes: the Willingness to Assign Approach

Improving public involvement in health system decision making stands as a primary goal in health systems reform. However, still limited evidence is found on how best to elicit preferences for health care programs. This paper examines a contingent choice technique to elicit preferences among health programs so called, willingness to assign (WTAS). Moreover, we elicited contingent rankings as well as the willingness to pay extra taxes for comparative purposes. We argue that WTAS reveals relative (monetary-based) values of a set of competing public programmes under a hypothetical healthcare budget assessment. Experimental evidence is reported from a deliberative empirical study valuing ten health programmes in the context of the Catalan Health Service. Evidence from a our experimental study reveals that preferences are internally more consistent and slightly less affected by preference reversals as compared to values revealed from the willingness to pay (WTP) extra taxes approach. Consistent with prior studies, we find that the deliberative approach helped to avoid possible misunderstandings. Interestingly, although programmes promoting health received the higher relative valuation, those promoting other health benefits also ranked highly.willingness to assign, willingness to pay, health system benefits

Incentives and Gender in a Multi-task Setting: an Experimental Study with Real-Effort Tasks

This paper investigates the behavioural effects of competitive, social-value and social-image incentives on men’s and women’s allocation of effort in a multi-task environment. Specifically, using two real-effort laboratory tasks, we investigate how competitive prizes, social-value generation and public awards affect effort allocation decisions between the tasks. We find that all three types of incentives significantly focus effort allocation towards the task they are applied in, but the effect varies significantly between men and women. The highest effort distortion lies with competitive incentives, which is due to the effort allocation decision of men. Women exert similar amount of effort across the three incentive conditions, with slightly lower effort levels in the social-image incentivized tasks. Our results inform how and why genders differences may persist in competitive workplaces

Maximising the value of knowledge workers

This article focuses on the role of knowledge workers and their contribution to the achievement of an enterprise's objectives. Knowledge workers do not have enough time to keep abreast of new knowledge and need more than motivation to assist with the capturing of tacit knowledge. The purpose of the empirical survey was to determine the role and contribution of knowledge workers to the objectives of a South African technology-oriented company. A high percentage of respondents indicated a positive relationship between a worker's position on the organisational hierarchy and the opportunities for the worker to make knowledge contributions. The metrics applied to measuring the contribution of knowledge workers need to be considered carefully