Evaluating Variable-Length Multiple-Option Lists in Chatbots and Mobile Search

In recent years, the proliferation of smart mobile devices has lead to the gradual integration of search functionality within mobile platforms. This has created an incentive to move away from the "ten blue links'' metaphor, as mobile users are less likely to click on them, expecting to get the answer directly from the snippets. In turn, this has revived the interest in Question Answering. Then, along came chatbots, conversational systems, and messaging platforms, where the user needs could be better served with the system asking follow-up questions in order to better understand the user's intent. While typically a user would expect a single response at any utterance, a system could also return multiple options for the user to select from, based on different system understandings of the user's intent. However, this possibility should not be overused, as this practice could confuse and/or annoy the user. How to produce good variable-length lists, given the conflicting objectives of staying short while maximizing the likelihood of having a correct answer included in the list, is an underexplored problem. It is also unclear how to evaluate a system that tries to do that. Here we aim to bridge this gap. In particular, we define some necessary and some optional properties that an evaluation measure fit for this purpose should have. We further show that existing evaluation measures from the IR tradition are not entirely suitable for this setup, and we propose novel evaluation measures that address it satisfactorily.Comment: 4 pages, in Proceeding of SIGIR 201

Bootstrapping Conversational Agents With Weak Supervision

Many conversational agents in the market today follow a standard bot development framework which requires training intent classifiers to recognize user input. The need to create a proper set of training examples is often the bottleneck in the development process. In many occasions agent developers have access to historical chat logs that can provide a good quantity as well as coverage of training examples. However, the cost of labeling them with tens to hundreds of intents often prohibits taking full advantage of these chat logs. In this paper, we present a framework called \textit{search, label, and propagate} (SLP) for bootstrapping intents from existing chat logs using weak supervision. The framework reduces hours to days of labeling effort down to minutes of work by using a search engine to find examples, then relies on a data programming approach to automatically expand the labels. We report on a user study that shows positive user feedback for this new approach to build conversational agents, and demonstrates the effectiveness of using data programming for auto-labeling. While the system is developed for training conversational agents, the framework has broader application in significantly reducing labeling effort for training text classifiers.Comment: 6 pages, 3 figures, 1 table, Accepted for publication in IAAI 201

Why People Search for Images using Web Search Engines

What are the intents or goals behind human interactions with image search engines? Knowing why people search for images is of major concern to Web image search engines because user satisfaction may vary as intent varies. Previous analyses of image search behavior have mostly been query-based, focusing on what images people search for, rather than intent-based, that is, why people search for images. To date, there is no thorough investigation of how different image search intents affect users' search behavior. In this paper, we address the following questions: (1)Why do people search for images in text-based Web image search systems? (2)How does image search behavior change with user intent? (3)Can we predict user intent effectively from interactions during the early stages of a search session? To this end, we conduct both a lab-based user study and a commercial search log analysis. We show that user intents in image search can be grouped into three classes: Explore/Learn, Entertain, and Locate/Acquire. Our lab-based user study reveals different user behavior patterns under these three intents, such as first click time, query reformulation, dwell time and mouse movement on the result page. Based on user interaction features during the early stages of an image search session, that is, before mouse scroll, we develop an intent classifier that is able to achieve promising results for classifying intents into our three intent classes. Given that all features can be obtained online and unobtrusively, the predicted intents can provide guidance for choosing ranking methods immediately after scrolling

Ranking with Submodular Valuations

We study the problem of ranking with submodular valuations. An instance of this problem consists of a ground set $[m]$, and a collection of $n$ monotone submodular set functions $f^1, \ldots, f^n$, where each $f^i: 2^{[m]} \to R_+$. An additional ingredient of the input is a weight vector $w \in R_+^n$. The objective is to find a linear ordering of the ground set elements that minimizes the weighted cover time of the functions. The cover time of a function is the minimal number of elements in the prefix of the linear ordering that form a set whose corresponding function value is greater than a unit threshold value. Our main contribution is an $O(\ln(1 / \epsilon))$-approximation algorithm for the problem, where $\epsilon$ is the smallest non-zero marginal value that any function may gain from some element. Our algorithm orders the elements using an adaptive residual updates scheme, which may be of independent interest. We also prove that the problem is $\Omega(\ln(1 / \epsilon))$-hard to approximate, unless P = NP. This implies that the outcome of our algorithm is optimal up to constant factors.Comment: 16 pages, 3 figure

Προσεγγιστικοί αλγόριθμοι για το Generalized Min-Sum Set Cover

Στην παρούσα εργασία, μελετάμε προσεγγιστικούς αλγόριθμους, βασισμένους στο γραμμικό προγραμματισμό, για το πρόβλημα Generalized Min-Sum Set Cover ή Multiple Intents Re-Ranking, Σε αυτό το NP-hard πρόβλημα, μας δίνεται μια συλλογή από σύνολα. Το κάθε σύνολο S έχει μια απαραίτητη προϋπόθεση κάλυψης K(S). Στόχος είναι να διατάξουμε τα στοιχεία των συνόλων σε τέτοια σειρά ώστε ο μέσος χρόνος κάλυψης των συνόλων να ελαχιστοποιηθεί. Χρόνος κάλυψης ενός συνόλου S είναι το ελάχιστο i για το οποίο K(S) στοιχεία του περιέχονται στις πρώτες i θέσεις της διάταξης. Μέσω της μελέτης των κυριότερων προσεγγιστικών αλγορίθμων για το πρόβλημα, αντιμετωπίζουμε την εξέλιξη και βελτίωση των τεχνικών που έχουν αναπτυχθεί και την επίπτωση αυτών στο ζητούμενο, την εγγύηση απόδοσης.In this paper, we study approximation algorithms, based on linear programming, for the Generalized Min-Sum Set Cover or Multiple Intents Re-Ranking problem. In this NP-hard problem, we are given a collection of sets. Each set S has a necessary cover condition K (S). The goal is to order all the elements of the sets so as to minimize the total cover time. The cover time of a set S is the least index i in the ordering such that the first i elements in the ordering contain K(S) elements in S. Through the study of the premier approximation algorithms for the problem, we become acquainted with the evolution and improvement of the techniques that have been developed and the their impact on the main issue, the performance guarantee