Evaluating embodied conversational agents in multimodal interfaces

Based on cross-disciplinary approaches to Embodied Conversational Agents, evaluation methods for such human-computer interfaces are structured and presented. An introductory systematisation of evaluation topics from a conversational perspective is followed by an explanation of social-psychological phenomena studied in interaction with Embodied Conversational Agents, and how these can be used for evaluation purposes. Major evaluation concepts and appropriate assessment instruments – established and new ones – are presented, including questionnaires, annotations and log-files. An exemplary evaluation and guidelines provide hands-on information on planning and preparing such endeavours

Measuring Learnability in Human-Computer Interaction

It is well accepted that learnability is a crucial attribute of usability that should be considered in almost every software system. A good learnability leads within a short time and with minimal effort to a high level of proficiency of the user. Therefore, expensive training time of complex systems is reduced. However, there is only few consensus on how to define and evaluate learnability. In addition, gathering detailed information on learnability is quite difficult. In todays books on usability evaluation, learnability gets only few attention, research publications are spread to several other fields and the term learnability is also used in other context. The objective of this thesis is to give an structured overview of learnability and methods for evaluation and additionally assist in the evaluator’s individual choice of an appropriate method. First of all, several definitions of learnability are discussed. For a deeper understanding psychological background knowledge is provided. Afterwards, methods to asses learnability are presented. This comprises nine methods that seem particularly appropriate to measure learnability. As this methods are very diverse, a framework based on analytical hierarchy process is provided. This framework aims to classify presented methods with respect to certain criteria and assess practitioners in selecting an appropriate method to measure learnability

Learnability makes things click : a grounded theory approach to the software product evaluation

The aim of this doctoral dissertation is to investigate the phenomenon of learnability more deeply in order to better understand the learnability process. Grounded theory was used to determine the ground concepts based on fifteen users’ (N=15) actions (N=1836) in the WebCT Campus Edition’s virtual environment. Based on this study, the phenomenon of learnability and the learnability process is understood in greater detail and defined from the human centric of view, where the human being is the key actor. This doctoral dissertation answers the following research problem: 1. How learnable is the WebCT Campus Edition’s virtual environment? 2. How can the phenomenon of learnability be defined in a new way? The WebCT Campus Edition virtual environment’s learnability was measured with performance time and directions of action. In addition, the traditional learnability metrics of performance time and direction of users’ actions was used to verify the theoretical model of learnability and its nonlinearity. The result of this study showed that the variety of the WebCT Campus Edition’s learnability was higher between the individual users than it was between the different tasks. Therefore, the variety of task difficulty, i.e. the complexity or easiness of the different part of the user interface, have less influence on learnability in the WebCT Campus Edition than do the individual users’characteristics. Thus, the research results confirm the results found in earlier studies, where two important issues for usability evaluation and therefore evaluation on learnability, are the tasks and users individual characteristics. The theoretical model of learnability with following phases of information search, data collection, knowledge management, knowledge form, knowledge build and the result of action were determined from data. The theoretical model of learnability and its main patterns of a) data collection-information search, b)knowledge build-knowledge form and c) information searchknowledge management proves that learnability is a non-linear process. Therefore, the phenomenon of learnability cannot purely be defined by the separate properties of learnability, i.e. the properties of a user interface and a progressively enhanced linear process illustrated with learning curves. The theoretical model of learnability is one of the rare models of learnability that is based on empirical data. The use of grounded theory methodology means that the phenomenon of learnability is studied through tacit knowledge, i.e. through users’ real actions and though explicit knowledge, the users cognitive processes during interaction i.e. the phenomenon of learnability is approached from the holistic point of view, where the phenomenon of learnability is seen as one holistic process. Thus triangulation, where the phenomenon is interpreted through several split case studies are therefore unnecessary in this research setting. In conclusion, too many studies are still conducted in a laboratory situation using traditional methodological paradigms. More learnability studies with new methodological approaches in the natural environments are needed were the human, learning and non-linear process of learnability are in focus. It is important to understand more deeply the process of learnability and investigate more in greater detail the key elements that enhance learnability and on the other hand, cause learnability problems for users. Finally, based on the theoretical models of learnability, we can develop tools for the commercial user interface world in order to measure and test the learnability process more precisely and better understand how skills are actually learnt and how “to click” learnability

Professional Search in Pharmaceutical Research

In the mid 90s, visiting libraries – as means of retrieving the latest literature – was still a common necessity among professionals. Nowadays, professionals simply access information by ‘googling’. Indeed, the name of the Web search engine market leader “Google” became a synonym for searching and retrieving information. Despite the increased popularity of search as a method for retrieving relevant information, at the workplace search engines still do not deliver satisfying results to professionals. Search engines for instance ignore that the relevance of answers (the satisfaction of a searcher’s needs) depends not only on the query (the information request) and the document corpus, but also on the working context (the user’s personal needs, education, etc.). In effect, an answer which might be appropriate to one user might not be appropriate to the other user, even though the query and the document corpus are the same for both. Personalization services addressing the context become therefore more and more popular and are an active field of research. This is only one of several challenges encountered in ‘professional search’: How can the working context of the searcher be incorporated in the ranking process; how can unstructured free-text documents be enriched with semantic information so that the information need can be expressed precisely at query time; how and to which extent can a company’s knowledge be exploited for search purposes; how should data from distributed sources be accessed from into one-single-entry-point. This thesis is devoted to ‘professional search’, i.e. search at the workplace, especially in industrial research and development. We contribute by compiling and developing several approaches for facing the challenges mentioned above. The approaches are implemented into the prototype YASA (Your Adaptive Search Agent) which provides meta-search, adaptive ranking of search results, guided navigation, and which uses domain knowledge to drive the search processes. YASA is deployed in the pharmaceutical research department of Roche in Penzberg – a major pharmaceutical company – in which the applied methods were empirically evaluated. Being confronted with mostly unstructured free-text documents and having barely explicit metadata at hand, we faced a serious challenge. Incorporating semantics (i.e. formal knowledge representation) into the search process can only be as good as the underlying data. Nonetheless, we are able to demonstrate that this issue can be largely compensated by incorporating automatic metadata extraction techniques. The metadata we were able to extract automatically was not perfectly accurate, nor did the ontology we applied contain considerably “rich semantics”. Nonetheless, our results show that already the little semantics incorporated into the search process, suffices to achieve a significant improvement in search and retrieval. We thus contribute to the research field of context-based search by incorporating the working context into the search process – an area which so far has not yet been well studied

Improving materials management on construction projects

An essential factor adversely affecting the performance of construction projects is the improper handling of materials during site activities. Materials management is made problematic by materials shortages, delays in supply, price fluctuations, damage and wastage, and lack of storages pace. In addition, paper-based reports are mostly used to record and exchange information related to the materials component within a supply chain which is problematic, error-prone, and inefficient. Generally, modem technologies are not being adequately used to overcome human error and are not well integrated with project management systems to make the tracking and management of materials easier and faster. Thus, this research focuses on the development of a mechanism to improve materials management on construction projects through the integration of materials tracking and resource modelling systems. A multi-facetted research approach was adopted. Initially, a literature review on materials management process in the construction project was conducted. This was followed by case studies involving six construction projects in order to investigate current practice in materials management to establish key problem areas and elements of good practice. The case studies also explored the requirements for integrating materials management and resource modelling in project management systems. The case study findings underpinned by literature results were used to develop a real-time framework for integrating RFID-based materials tracking and resource modelling. The framework was encapsulated in a computer-based prototype system based on Microsoft Visual Basic. NET. The prototype system was developed by amalgamation of all the software and hardware chosen such as MS Access (database system), MS Project (resource modelling) and RFID (automated materials tracking) to provide the mechanisms for integrating materials management and resource modelling in the construction industry. Evaluation of the prototype system was carried out by a series of interviews with industry practitioners to assess its appropriateness and functionality. It also established the skills and other requirements for the effective use of the real-time materials tracking system. The evaluation established that the prototype system demonstrated many benefits and is suitable for use in materials tracking and inventory management processes. It is concluded that the prototype system developed can improve materials management on construction projects, particularly with regard to materials tracking and integrating materials utilisation with the resource modelling subsystem in project management applications. Adoption of the approaches suggested in the thesis will enable the construction industry to improve the real-time management of materials on sites, and hence improve project performance.EThOS - Electronic Theses Online ServiceGBUnited Kingdo