Survey on Evaluation Methods for Dialogue Systems

In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class

Designing library tools: the (un)importance of employee involvement

The growing trend of public institutions to open up data and information to citizens encouraged archives and libraries to enhance the disclosure of their content towards end-users. This implies technical challenges as more and more information is exchanged not only between people, but also between different databases and applications which are consulted by different user groups through different devices and entry points. For libraries, the challenge lies in constructing a properly functioning catalogue which is able to combine a huge amount of information from various sources and is consultable by a large group of end-users in a user friendly manner. Based on the User Centred Design paradigm and Kaulio’s (1998) degrees of user involvement in innovation, this paper wants to consider whether involving users during the creation of metadata tools can result in more motivated library co-workers and a more appreciated tool and (hopefully) in a permanent tagging tool

Proposal for encoding the Mandaic script in the BMP of the UCS

This is a proposal to encode the Mandaic script in the international character encoding standard Unicode. This set of characters was published in Unicode Standard version 6.0 in October 2010. Mandaic was developed by followers of the Mandaean gnostic religion to write a dialect of Eastern Aramaic. It is used to write the liturgical language, called Classical Mandaic, as well as a form of Mandaic used today

Technology assessment of advanced automation for space missions

Six general classes of technology requirements derived during the mission definition phase of the study were identified as having maximum importance and urgency, including autonomous world model based information systems, learning and hypothesis formation, natural language and other man-machine communication, space manufacturing, teleoperators and robot systems, and computer science and technology

CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Machine Learning for Interactive Systems: Challenges and Future Trends

National audienceMachine learning has been introduced more than 40 years ago in interactive systems through speech recognition or computer vision. Since that, machine learning gained in interest in the scientific community involved in human- machine interaction and raised in the abstraction scale. It moved from fundamental signal processing to language understanding and generation, emotion and mood recogni- tion and even dialogue management or robotics control. So far, existing machine learning techniques have often been considered as a solution to some problems raised by inter- active systems. Yet, interaction is also the source of new challenges for machine learning and offers new interesting practical but also theoretical problems to solve. In this paper, we address these challenges and describe why research in machine learning and interactive systems should converge in the future