Usefulness, localizability, humanness, and language-benefit: additional evaluation criteria for natural language dialogue systems

Human–computer dialogue systems interact with human users using natural language. We used the ALICE/AIML chatbot architecture as a platform to develop a range of chatbots covering different languages, genres, text-types, and user-groups, to illustrate qualitative aspects of natural language dialogue system evaluation. We present some of the different evaluation techniques used in natural language dialogue systems, including black box and glass box, comparative, quantitative, and qualitative evaluation. Four aspects of NLP dialogue system evaluation are often overlooked: “usefulness” in terms of a user’s qualitative needs, “localizability” to new genres and languages, “humanness” or “naturalness” compared to human–human dialogues, and “language benefit” compared to alternative interfaces. We illustrated these aspects with respect to our work on machine-learnt chatbot dialogue systems; we believe these aspects are worthwhile in impressing potential new users and customers

A customized semantic service retrieval methodology for the digital ecosystems environment

With the emergence of the Web and its pervasive intrusion on individuals, organizations, businesses etc., people now realize that they are living in a digital environment analogous to the ecological ecosystem. Consequently, no individual or organization can ignore the huge impact of the Web on social well-being, growth and prosperity, or the changes that it has brought about to the world economy, transforming it from a self-contained, isolated, and static environment to an open, connected, dynamic environment. Recently, the European Union initiated a research vision in relation to this ubiquitous digital environment, known as Digital (Business) Ecosystems. In the Digital Ecosystems environment, there exist ubiquitous and heterogeneous species, and ubiquitous, heterogeneous, context-dependent and dynamic services provided or requested by species. Nevertheless, existing commercial search engines lack sufficient semantic supports, which cannot be employed to disambiguate user queries and cannot provide trustworthy and reliable service retrieval. Furthermore, current semantic service retrieval research focuses on service retrieval in the Web service field, which cannot provide requested service retrieval functions that take into account the features of Digital Ecosystem services. Hence, in this thesis, we propose a customized semantic service retrieval methodology, enabling trustworthy and reliable service retrieval in the Digital Ecosystems environment, by considering the heterogeneous, context-dependent and dynamic nature of services and the heterogeneous and dynamic nature of service providers and service requesters in Digital Ecosystems.The customized semantic service retrieval methodology comprises: 1) a service information discovery, annotation and classification methodology; 2) a service retrieval methodology; 3) a service concept recommendation methodology; 4) a quality of service (QoS) evaluation and service ranking methodology; and 5) a service domain knowledge updating, and service-provider-based Service Description Entity (SDE) metadata publishing, maintenance and classification methodology.The service information discovery, annotation and classification methodology is designed for discovering ubiquitous service information from the Web, annotating the discovered service information with ontology mark-up languages, and classifying the annotated service information by means of specific service domain knowledge, taking into account the heterogeneous and context-dependent nature of Digital Ecosystem services and the heterogeneous nature of service providers. The methodology is realized by the prototype of a Semantic Crawler, the aim of which is to discover service advertisements and service provider profiles from webpages, and annotating the information with service domain ontologies.The service retrieval methodology enables service requesters to precisely retrieve the annotated service information, taking into account the heterogeneous nature of Digital Ecosystem service requesters. The methodology is presented by the prototype of a Service Search Engine. Since service requesters can be divided according to the group which has relevant knowledge with regard to their service requests, and the group which does not have relevant knowledge with regard to their service requests, we respectively provide two different service retrieval modules. The module for the first group enables service requesters to directly retrieve service information by querying its attributes. The module for the second group enables service requesters to interact with the search engine to denote their queries by means of service domain knowledge, and then retrieve service information based on the denoted queries.The service concept recommendation methodology concerns the issue of incomplete or incorrect queries. The methodology enables the search engine to recommend relevant concepts to service requesters, once they find that the service concepts eventually selected cannot be used to denote their service requests. We premise that there is some extent of overlap between the selected concepts and the concepts denoting service requests, as a result of the impact of service requesters’ understandings of service requests on the selected concepts by a series of human-computer interactions. Therefore, a semantic similarity model is designed that seeks semantically similar concepts based on selected concepts.The QoS evaluation and service ranking methodology is proposed to allow service requesters to evaluate the trustworthiness of a service advertisement and rank retrieved service advertisements based on their QoS values, taking into account the contextdependent nature of services in Digital Ecosystems. The core of this methodology is an extended CCCI (Correlation of Interaction, Correlation of Criterion, Clarity of Criterion, and Importance of Criterion) metrics, which allows a service requester to evaluate the performance of a service provider in a service transaction based on QoS evaluation criteria in a specific service domain. The evaluation result is then incorporated with the previous results to produce the eventual QoS value of the service advertisement in a service domain. Service requesters can rank service advertisements by considering their QoS values under each criterion in a service domain.The methodology for service domain knowledge updating, service-provider-based SDE metadata publishing, maintenance, and classification is initiated to allow: 1) knowledge users to update service domain ontologies employed in the service retrieval methodology, taking into account the dynamic nature of services in Digital Ecosystems; and 2) service providers to update their service profiles and manually annotate their published service advertisements by means of service domain knowledge, taking into account the dynamic nature of service providers in Digital Ecosystems. The methodology for service domain knowledge updating is realized by a voting system for any proposals for changes in service domain knowledge, and by assigning different weights to the votes of domain experts and normal users.In order to validate the customized semantic service retrieval methodology, we build a prototype – a Customized Semantic Service Search Engine. Based on the prototype, we test the mathematical algorithms involved in the methodology by a simulation approach and validate the proposed functions of the methodology by a functional testing approach

CHORUS Deliverable 2.2: Second report - identification of multi-disciplinary key issues for gap analysis toward EU multimedia search engines roadmap

After addressing the state-of-the-art during the first year of Chorus and establishing the existing landscape in multimedia search engines, we have identified and analyzed gaps within European research effort during our second year. In this period we focused on three directions, notably technological issues, user-centred issues and use-cases and socio- economic and legal aspects. These were assessed by two central studies: firstly, a concerted vision of functional breakdown of generic multimedia search engine, and secondly, a representative use-cases descriptions with the related discussion on requirement for technological challenges. Both studies have been carried out in cooperation and consultation with the community at large through EC concertation meetings (multimedia search engines cluster), several meetings with our Think-Tank, presentations in international conferences, and surveys addressed to EU projects coordinators as well as National initiatives coordinators. Based on the obtained feedback we identified two types of gaps, namely core technological gaps that involve research challenges, and “enablers”, which are not necessarily technical research challenges, but have impact on innovation progress. New socio-economic trends are presented as well as emerging legal challenges

CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Reputation versus reality: the impact of US News and World Report rankings and education branding on hiring decisions in the job market

Typically, leading brands provide benchmarks for constructing consumer preference in the marketplace. Reputation rankings have sustained and advanced the status of brand names in higher education with an implication that the degrees awarded by higher ranked schools have added prestige, a cachet with the potential of facilitating success in the job market. This implication makes reputation rankings a dependable tool for college and university marketing departments eager to increase student enrollment and retention by communicating its superiority among its peers. By examining the influence of reputation rankings on the pre-decision preferences of human resource hiring professionals in evaluating employment applicants, this study found that there is little if any relationship between a degree from a higher education institution in the top tier of a reputation ranking and employment acquisition. Work experience emerged as the major deciding factor in the assessment of an applicant\u27s qualifications. Degree field and employee referral appeared as important matters, while education program and academic record followed in playing a slightly diminished role. Academic record and non academic activities had a lesser degree of influence on hiring decisions. Future study into the subconscious and conscious effect of reputation rankings on the job attainment goal of a college student in relation to the student\u27s choice of HEI could provide new insights into student choice, college marketing strategy, and the value of rankings in education

Jobseekers’ Beliefs about Comparative Advantage and (Mis)Directed Search

Worker sorting into tasks and occupations has long been recognized as an important feature of labor markets. But this sorting may be inefficient if jobseekers have inaccurate beliefs about their skills and therefore apply to jobs that do not match their skills. To test this idea, we measure young South African jobseekers’ communication and numeracy skills and their beliefs about their skill levels. Many jobseekers believe they are better at the skill in which they score lower, relative to other jobseekers. These beliefs predict the skill requirements of jobs where they apply. In two field experiments, giving jobseekers their skill assessment results shifts their beliefs toward their assessment results. It also redirects their search toward jobs that value the skill in which they score relatively higher—using measures from administrative, incentivized task, and survey data—but does not increase total search effort. It also raises earnings and job quality, consistent with inefficient sorting due to limited information

Who’s Watching: The Accuracy of Forecasting Broadcast TV Audience Demand Using Advertising Prices

Broadcast and cable networks are struggling to keep up with the multitude of entertainment options available today, including but not limited to streaming services. However, these networks still play a role in the entertainment landscape. In order to maintain their role, they must first assess which shows deliver higher ratings and why. Ratings indicate audience demand for a particular show, which can be unpredictable. Regardless, networks sell commercial spots to advertisers at predetermined prices based on their expectations of future ratings, or demand. As such, this research paper focuses on broadcast networks and investigates two questions: Are broadcast networks able to accurately predict ratings, or audience demand, for their upcoming season of primetime shows, indicated by the predetermined prices for ad spots in the shows? If advertising prices do not reflect audience demand for the upcoming season, what is the reason for this? To address the research questions, a two-part mixed method design of both quantitative and qualitative research was used, showing that broadcast networks have been successful in predicting ratings for their upcoming primetime shows, but they should consider additional factors when creating shows to capture the most audience attention

Semantic Routing in Peer-to-Peer Systems

Currently search engines like Google, Yahoo and Excite are centralized, which means that all queries that users post are sent to some big servers (or server group) that handle them. In this way it is easy for the systems to relate IP-addresses with the queries posted from them. Clearly privacy is a problem here. Also censoring out certain information which is not 'appropriate' is simple, and shown in recent examples. To give more privacy to the users and make censoring information more difficult, Peer-to-Peer (P2P) systems are a good alternative to the centralized approach. In P2P systems the search functionality can be devided over a large group of autonomous computers (Peers), where each computer only has a very small piece of information instead of everything. Now the problem in such a distributed system is to make the search process efficient in terms of bandwith, storage, time and CPU usage. In this Ph.D. thesis, three approaches are described that try to reach goal of finding the short routes between seeker and providers with high efficiency. These routing algorithms are all applied on 'Semantic-Overlay-Networks' (SONs). In a SON, peers maintain pointers to semantically relevant peers based on content descriptions, which makes them able to choose the relevant peers for queries instead of, for example, choosing random peers. This work tries to show that decentralized search algorithms based on semantic routing are a good alternative to centralized approaches.Harmelen, F.A.H. van [Promotor