Evaluating Bad Query Abandonment in an Iterative SMS-Based FAQ Retrieval System

In this paper, we investigate how many iterations users are willing to tolerate in an iterative Frequently Asked Ques- tion (FAQ) system that provides information on HIV/AIDS. This is part of work in progress that aims to develop an automated Frequently Asked Question system that can be used to provide answers on HIV/AIDS related queries to users in Botswana. Our system engages the user in the question answering process by following an iterative interaction approach in order to avoid giving inappropriate answers to the user. Our findings provide us with an indication of how long users are willing to engage with the system. We sub- sequently use this to develop a novel evaluation metric to use in future developments of the system. As an additional finding, we show that the previous search experience of the users has a significant effect on their future behaviour

Detecting missing content queries in an SMS-Based HIV/AIDS FAQ retrieval system

Automated Frequently Asked Question (FAQ) answering systems use pre-stored sets of question-answer pairs as an information source to answer natural language questions posed by the users. The main problem with this kind of information source is that there is no guarantee that there will be a relevant question-answer pair for all user queries. In this paper, we propose to deploy a binary classifier in an existing SMS-Based HIV/AIDS FAQ retrieval system to detect user queries that do not have the relevant question-answer pair in the FAQ document collection. Before deploying such a classifier, we first evaluate different feature sets for training in order to determine the sets of features that can build a model that yields the best classification accuracy. We carry out our evaluation using seven different feature sets generated from a query log before and after retrieval by the FAQ retrieval system. Our results suggest that, combining different feature sets markedly improves the classification accuracy

A semi-automated FAQ retrieval system for HIV/AIDS

This thesis describes a semi-automated FAQ retrieval system that can be queried by users through short text messages on low-end mobile phones to provide answers on HIV/AIDS related queries. First we address the issue of result presentation on low-end mobile phones by proposing an iterative interaction retrieval strategy where the user engages with the FAQ retrieval system in the question answering process. At each iteration, the system returns only one question-answer pair to the user and the iterative process terminates after the user's information need has been satisfied. Since the proposed system is iterative, this thesis attempts to reduce the number of iterations (search length) between the users and the system so that users do not abandon the search process before their information need has been satisfied. Moreover, we conducted a user study to determine the number of iterations that users are willing to tolerate before abandoning the iterative search process. We subsequently used the bad abandonment statistics from this study to develop an evaluation measure for estimating the probability that any random user will be satisfied when using our FAQ retrieval system. In addition, we used a query log and its click-through data to address three main FAQ document collection deficiency problems in order to improve the retrieval performance and the probability that any random user will be satisfied when using our FAQ retrieval system. Conclusions are derived concerning whether we can reduce the rate at which users abandon their search before their information need has been satisfied by using information from previous searches to: Address the term mismatch problem between the users' SMS queries and the relevant FAQ documents in the collection; to selectively rank the FAQ document according to how often they have been previously identified as relevant by users for a particular query term; and to identify those queries that do not have a relevant FAQ document in the collection. In particular, we proposed a novel template-based approach that uses queries from a query log for which the true relevant FAQ documents are known to enrich the FAQ documents with additional terms in order to alleviate the term mismatch problem. These terms are added as a separate field in a field-based model using two different proposed enrichment strategies, namely the Term Frequency and the Term Occurrence strategies. This thesis thoroughly investigates the effectiveness of the aforementioned FAQ document enrichment strategies using three different field-based models. Our findings suggest that we can improve the overall recall and the probability that any random user will be satisfied by enriching the FAQ documents with additional terms from queries in our query log. Moreover, our investigation suggests that it is important to use an FAQ document enrichment strategy that takes into consideration the number of times a term occurs in the query when enriching the FAQ documents. We subsequently show that our proposed enrichment approach for alleviating the term mismatch problem generalise well on other datasets. Through the evaluation of our proposed approach for selectively ranking the FAQ documents, we show that we can improve the retrieval performance and the probability that any random user will be satisfied when using our FAQ retrieval system by incorporating the click popularity score of a query term t on an FAQ document d into the scoring and ranking process. Our results generalised well on a new dataset. However, when we deploy the click popularity score of a query term t on an FAQ document d on an enriched FAQ document collection, we saw a decrease in the retrieval performance and the probability that any random user will be satisfied when using our FAQ retrieval system. Furthermore, we used our query log to build a binary classifier for detecting those queries that do not have a relevant FAQ document in the collection (Missing Content Queries (MCQs))). Before building such a classifier, we empirically evaluated several feature sets in order to determine the best combination of features for building a model that yields the best classification accuracy in identifying the MCQs and the non-MCQs. Using a different dataset, we show that we can improve the overall retrieval performance and the probability that any random user will be satisfied when using our FAQ retrieval system by deploying a MCQs detection subsystem in our FAQ retrieval system to filter out the MCQs. Finally, this thesis demonstrates that correcting spelling errors can help improve the retrieval performance and the probability that any random user will be satisfied when using our FAQ retrieval system. We tested our FAQ retrieval system with two different testing sets, one containing the original SMS queries and the other containing the SMS queries which were manually corrected for spelling errors. Our results show a significant improvement in the retrieval performance and the probability that any random user will be satisfied when using our FAQ retrieval system

Text messaging and retrieval techniques for a mobile health information system

Mobile phones have been identified as one of the technologies that can be used to overcome the challenges of information dissemination regarding serious diseases. Short message services, a much used function of cell phones, for example, can be turned into a major tool for accessing databases. This paper focuses on the design and development of a short message services-based information access algorithm to carefully screen information on human immunodeficiency virus/acquired immune deficiency syndrome within the context of a frequently asked questions system. However, automating the short message services-based information search and retrieval poses significant challenges because of the inherent noise in its communications. The developed algorithm was used to retrieve the best-ranked question–answer pair. Results were evaluated using three metrics: average precision, recall and computational time. The retrieval efficacy was measured and it was confirmed that there was a significant improvement in the results of the proposed algorithm when compared with similar retrieval algorithms

Usability of disaster apps : understanding the perspectives of the public as end-users : a dissertation presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Emergency Management at Massey University, Wellington, New Zealand

Listed in 2020 Dean's List of Exceptional ThesesMultiple smartphone applications (apps) exist that can enhance the public’s resilience to disasters. Despite the capabilities of these apps, they can only be effective if users find them usable. Availability does not automatically translate to usability nor does it guarantee continued usage by the target users. A disaster app will be of little or no value if a user abandons it after the initial download. It is, therefore, essential to understand the users’ perspectives on the usability of disaster apps. In the context of disaster apps, usability entails providing the elements that effectively facilitate users in retrieving critical information, and thus enabling them to make decisions during crises. Establishing good usability for effective systems relies upon focussing on the user whereby technological solutions match the user’s needs and expectations. However, most studies on the usability of disaster context technologies have been conducted with emergency responders, and only a few have investigated the publics’ perspectives as end-users. This doctoral project, written within a ‘PhD-thesis-with-publication’ format, addresses this gap by investigating the usability of disaster apps through the perspectives of the public end-users. The investigation takes an explicitly perceived usability standpoint where the experiences of the end-users are prioritised. Data analysis involved user-centric information to understand the public’s context and the mechanisms of disaster app usability. A mixed methods approach incorporates the qualitative analysis of app store data of 1,405 user reviews from 58 existing disaster apps, the quantitative analysis of 271 survey responses from actual disaster app users, and the qualitative analysis of usability inquiries with 18 members of the public. Insights gathered from this doctoral project highlight that end-users do not anticipate using disaster apps frequently, which poses particular challenges. Furthermore, despite the anticipated low frequency of use, because of the life-safety association of disasters apps, end-users have an expectation that the apps can operate with adequate usability when needed. This doctoral project provides focussed outcomes that consider such user perspectives. First, an app store analysis investigating user reviews identified new usability concerns particular to disaster apps. It highlighted users’ opinion on phone resource usage and relevance of content, among others. More importantly, it defined a new usability factor, app dependability, relating to the life-safety context of disaster apps. App dependability is the degree to which users’ perceive that an app can operate dependably during critical scenarios. Second, the quantitative results from this research have contributed towards producing a usability-continuance model, highlighting the usability factors that affect end-users’ intention to keep or uninstall a disaster app. The key influences for users’ intention to keep disaster apps are: (1) users’ perceptions as to whether the app delivers its function (app utility), (2) whether it does so dependably (app dependability), and (3) whether it presents information that can be easily understood (user-interface output). Subsequently, too much focus on (4) user-interface graphics and (5) user-interface input can encourage users to uninstall apps. Third, the results from the qualitative analysis of the inquiry data provide a basis for developing guidelines for disaster app usability. In the expectation of low level of engagement with disaster app users, the guidelines list recommendations addressing information salience, cognitive load, and trust. This doctoral project provides several contributions to the body of knowledge for usability and disaster apps. It reiterates the importance of investigating the usability of technological products for disasters and showcases the value of user-centric data in understanding usability. It has investigated usability with particular attention to the end-users’ perspectives on the context of disaster apps and, thus, produces a theoretical usability-continuance model to advance disaster app usability research and usability guidelines to encourage responsible design in practice

Usability analysis of contending electronic health record systems

In this paper, we report measured usability of two leading EHR systems during procurement. A total of 18 users participated in paired-usability testing of three scenarios: ordering and managing medications by an outpatient physician, medicine administration by an inpatient nurse and scheduling of appointments by nursing staff. Data for audio, screen capture, satisfaction rating, task success and errors made was collected during testing. We found a clear difference between the systems for percentage of successfully completed tasks, two different satisfaction measures and perceived learnability when looking at the results over all scenarios. We conclude that usability should be evaluated during procurement and the difference in usability between systems could be revealed even with fewer measures than were used in our study. © 2019 American Psychological Association Inc. All rights reserved.Peer reviewe