Benchmarking: A methodology for ensuring the relative quality of recommendation systems in software engineering

This chapter describes the concepts involved in the process of benchmarking of recommendation systems. Benchmarking of recommendation systems is used to ensure the quality of a research system or production system in comparison to other systems, whether algorithmically, infrastructurally, or according to any sought-after quality. Specifically, the chapter presents evaluation of recommendation systems according to recommendation accuracy, technical constraints, and business values in the context of a multi-dimensional benchmarking and evaluation model encompassing any number of qualities into a final comparable metric. The focus is put on quality measures related to recommendation accuracy, technical factors, and business values. The chapter first introduces concepts related to evaluation and benchmarking of recommendation systems, continues with an overview of the current state of the art, then presents the multi-dimensional approach in detail. The chapter concludes with a brief discussion of the introduced concepts and a summary

Evaluating websites using a practical quality model.

Many of the existing website evaluation methods and criteria for evaluating website quality are not able to sufficiently assess the performance and quality of a website, and most of them focus on usability and accessibility. This thesis aims at proposing the website quality metrics and methods to measure the website interface and reputation quality factors. The evaluation metrics has a framework which can be viewed as a hierarchical tree with three levels. The first level is composed of five quality characteristics: Aesthetics, Ease of Use, Multimedia, Rich Content and Reputation. The second level breaks down the first level quality characteristics into sub-characteristics and the third level further breaks down the second level sub-characteristics into measurable criteria. This thesis is particularly concerned with two major quality characteristics: Aesthetics and Reputation, and also the several website measurable criteria (indicators) that now apply to almost all live websites. A website evaluation tool is provided by this study to measure website quality automatically. It includes a traversal unit, parsing unit, data metrics unit and user interface unit. Also some effective algorithms are used in each unit: data crawler, recursive, parser and data transmission. According to relative issues in previous research about website evaluation metrics, there are only a few of them that use same methods as this study to completely measure the website metrics, and highlight the entire website quality scores that meet the users’ requirements

A Cross-Cultural User Evaluation of Product Recommender Interfaces

We present a cross-cultural user evaluation of an organization-based product recommender interface, by comparing it with the traditional list view. The results show that it performed significantly better, for all study participants, in improving on their competence perceptions, including perceived recommendation quality, perceived ease of use and perceived usefulness, and positively impacting users' behavioral intentions such as intention to save effort in the next visit. Additionally, oriental users were observed reacting more significantly strongly to the organization interface regarding some subjective aspects, compared to western subjects. Through this user study, we also identified the dominating role of the recommender system's decision-aiding competence in stimulating both oriental and western users' return intention to an e-commerce website where the system is applied

Using an intelligent self-service solution to support telecommunication service provisioning

The increasing competition between telecommunication service providers (TSPs) and the large amounts of human resource costs incurred to provision telecommunication services to small, medium and micro enterprises (SMMEs), highlight the need for effective self-service provisioning methods to reduce costs. Existing service provisioning methods do not effectively support SMMEs in the service provisioning process. The current methods used are manual and Web-based. The manual service provisioning method is labour and time-intensive. TSPs are being forced to cut human resource costs as the profit margins for provisioning services are narrow due to competition. The current web-based systems do not allow SMMEs to effectively purchase services as they provide long textual lists to consult and limited information about services. The objective of this research was to develop an intelligent web-based self-service system to support SMMEs in telecommunication service provisioning, using a Design Science Research methodology. A constraint-based, knowledge-based recommender system was selected as the most appropriate intelligent system to support telecommunication service provisioning. A prototype called SelPro was developed as a solution to address the shortcomings of the current ineffective service provisioning methods available for SMMEs. SelPro incorporates an interactive wizard-style user interface with dynamic recommendations to support novice users as well as an expert interface which provides less user support. An expert review and a user study were conducted to determine if SelPro was usable, useful and effective. SelPro was perceived as being usable for SMMEs wishing to purchase telecommunication services. Expert participants were satisfied with the usability of SelPro and perceived it as being useful, easy to use and easy to learn. SelPro was considered effective because of the high task success ratings provided by the participants using the two interfaces. The research determined that intelligent web-based self-service provisioning can be used to effectively support SMMEs in telecommunication service provisioning. General design recommendations for similar systems were proposed and future work will involve using intelligent systems to support large enterprises in telecommunication service provisioning

Using an intelligent self-service solution to support telecommunication service provisioning

The increasing competition between telecommunication service providers (TSPs) and the large amounts of human resource costs incurred to provision telecommunication services to small, medium and micro enterprises (SMMEs), highlight the need for effective self-service provisioning methods to reduce costs. Existing service provisioning methods do not effectively support SMMEs in the service provisioning process. The current methods used are manual and Web-based. The manual service provisioning method is labour and time-intensive. TSPs are being forced to cut human resource costs as the profit margins for provisioning services are narrow due to competition. The current web-based systems do not allow SMMEs to effectively purchase services as they provide long textual lists to consult and limited information about services. The objective of this research was to develop an intelligent web-based self-service system to support SMMEs in telecommunication service provisioning, using a Design Science Research methodology. A constraint-based, knowledge-based recommender system was selected as the most appropriate intelligent system to support telecommunication service provisioning. A prototype called SelPro was developed as a solution to address the shortcomings of the current ineffective service provisioning methods available for SMMEs. SelPro incorporates an interactive wizard-style user interface with dynamic recommendations to support novice users as well as an expert interface which provides less user support. An expert review and a user study were conducted to determine if SelPro was usable, useful and effective. SelPro was perceived as being usable for SMMEs wishing to purchase telecommunication services. Expert participants were satisfied with the usability of SelPro and perceived it as being useful, easy to use and easy to learn. SelPro was considered effective because of the high task success ratings provided by the participants using the two interfaces. The research determined that intelligent web-based self-service provisioning can be used to effectively support SMMEs in telecommunication service provisioning. General design recommendations for similar systems were proposed and future work will involve using intelligent systems to support large enterprises in telecommunication service provisioning

User decision improvement and trust building in product recommender systems

As online stores are offering an almost unlimited shelf space, users must increasingly rely on product search and recommender systems to find their most preferred products and decide which item is the truly best one to buy. However, much research work has emphasized on developing and improving the underlying algorithms whereas many of the user issues such as preference elicitation and trust formation received little attention. In this thesis, we aim at designing and evaluating various decision technologies, with emphases on how to improve users' decision accuracy with intelligent preference elicitation and revision tools, and how to build their competence-inspired subjective constructs via trustworthy recommender interfaces. Specifically, two primary technologies are proposed: one is called example critiquing agents aimed to stimulate users to conduct tradeoff navigation and freely specify feedback criteria to example products; another termed as preference-based organization interfaces designed to take two roles: explaining to users why and how the recommendations are computed and displayed, and suggesting critique suggestions to guide users to understand existing tradeoff potentials and to make concrete decision navigations from the top candidate for better choices. To evaluate the two technologies' true performance and benefits to real-users, an evaluation framework was first established, that includes important assessment standards such as the objective/subjective accuracy-effort measures and trust-related subjective aspects (e.g., competence perceptions and behavioral intentions). Based on the evaluation framework, a series of nine experiments has been conducted and most of them were participated by real-users. Three user studies focused on the example critiquing (EC) agent, which first identified the significant impact of tradeoff process with the help of EC on users' decision accuracy improvement, and then in depth explored the advantage of multi-item strategy (for critiquing coverage) against single-item display, and higher user-control level reflected by EC in supporting users to freely compose critiquing criteria for both simple and complex tradeoffs. Another three experiments studied the preference-based organization technique. Regarding its explanation role, a carefully conducted user survey and a significant-scale quantitative evaluation both demonstrated that it can be likely to increase users' competence perception and return intention, and reduce their cognitive effort in information searching, relative to the traditional "why" explanation method in ranked list views. In addition, a retrospective simulation revealed its superior algorithm accuracy in predicting critiques and product choices that real-users intended to make, in comparison with other typical critiquing generation approaches. Motivated by the empirically findings in terms of the two technologies' respective strengths, a hybrid system has been developed with the purpose of combining them into a single application. The final three experiments evaluated its two design versions and particularly validated the hybrid system's universal effectiveness among people from different types of cultural backgrounds: oriental culture and western culture. In the end, a set of design guidelines is derived from all of the experimental results. They should be helpful for the development of a preference-based recommender system, making it capable of practically benefiting its users in improving decision accuracy, expending effort they are willing to invest, and even promoting trust in the system with resulting behavioral intentions to purchase chosen products and return to the system for repeated uses

Establishing User Requirements for a Recommender System in an Online Union Catalogue: an Investigation of WorldCat.org

This project, undertaken in collaboration with OCLC, aimed to investigate the potential role of recommendations within WorldCat, the publicly accessible union catalogue of libraries participating in the OCLC global cooperative. The goal of the project was a set of conceptual design guidelines for a WorldCat.org recommender system, based on a comprehensive understanding of the systems users and their needs. Taking a mixed-methods approach, the investigation consisted of four phases. Phase one consisted of twenty-one focus groups with key user goups held in three locations; the UK, the US, and Australia and New Zealand. Phase 2 consisted of a pop-up survey implemented on WorldCat.org, and gathered 2,918 responses. Phase three represented an analysis of two months of WorldCat.org transaction log data, consisting of over 15,000,000 sessions. Phase four was a lab based user study investigating and comparing the use of WorldCat.org with Amazon. Findings from each strand were integrated, and the key themes to emerge from the research are discussed. Different methods of classifying the WorldCat.org user population are presented, along with a taxonomy of work- and search-tasks. Key perspectives on the utility of a recommender system are considered, along with a reflection on how the information search behaviour exhibited by users interacting with recommendations while undertaking typical catalogue tasks can be interpreted. Based on the enriched perspective of the system, and the role of recommendation in the catalogue, a series of conceptual design specifications are presented for the development of a WorldCat.org recommender system