Layered evaluation of interactive adaptive systems : framework and formative methods

Peer reviewedPostprin

A Bayesian Network Estimation of the Service-Profit Chain for Transport Service Satisfaction

Bayesian network methodology is used to model key linkages of the service-profit chain within the context of transportation service satisfaction. Bayesian networks offer some advantages for implementing managerially focused models over other statistical techniques designed primarily for evaluating theoretical models. These advantages are (1) providing a causal explanation using observable variables within a single multivariate model, (2) analysis of nonlinear relationships contained in ordinal measurements, (3) accommodation of branching patterns that occur in data collection, and (4) the ability to conduct probabilistic inference for prediction and diagnostics with an output metric that can be understood by managers and academics. Sample data from 1,101 recent transport service customers are utilized to select and validate a Bayesian network and conduct probabilistic inference

CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender System

While personalization increases the utility of recommender systems, it also brings the issue of filter bubbles. E.g., if the system keeps exposing and recommending the items that the user is interested in, it may also make the user feel bored and less satisfied. Existing work studies filter bubbles in static recommendation, where the effect of overexposure is hard to capture. In contrast, we believe it is more meaningful to study the issue in interactive recommendation and optimize long-term user satisfaction. Nevertheless, it is unrealistic to train the model online due to the high cost. As such, we have to leverage offline training data and disentangle the causal effect on user satisfaction. To achieve this goal, we propose a counterfactual interactive recommender system (CIRS) that augments offline reinforcement learning (offline RL) with causal inference. The basic idea is to first learn a causal user model on historical data to capture the overexposure effect of items on user satisfaction. It then uses the learned causal user model to help the planning of the RL policy. To conduct evaluation offline, we innovatively create an authentic RL environment (KuaiEnv) based on a real-world fully observed user rating dataset. The experiments show the effectiveness of CIRS in bursting filter bubbles and achieving long-term success in interactive recommendation. The implementation of CIRS is available via https://github.com/chongminggao/CIRS-codes.Comment: 11 pages, 9 figure

What Airbnb Reviews can Tell us? An Advanced Latent Aspect Rating Analysis Approach

There is no doubt that the rapid growth of Airbnb has changed the lodging industry and tourists’ behaviors dramatically since the advent of the sharing economy. Airbnb welcomes customers and engages them by creating and providing unique travel experiences to “live like a local” through the delivery of lodging services. With the special experiences that Airbnb customers pursue, more investigation is needed to systematically examine the Airbnb customer lodging experience. Online reviews offer a representative look at individual customers’ personal and unique lodging experiences. Moreover, the overall ratings given by customers are reflections of their experiences with a product or service. Since customers take overall ratings into account in their purchase decisions, a study that bridges the customer lodging experience and the overall rating is needed. In contrast to traditional research methods, mining customer reviews has become a useful method to study customers’ opinions about products and services. User-generated reviews are a form of evaluation generated by peers that users post on business or other (e.g., third-party) websites (Mudambi & Schuff, 2010). The main purpose of this study is to identify the weights of latent lodging experience aspects that customers consider in order to form their overall ratings based on the eight basic emotions. This study applied both aspect-based sentiment analysis and the latent aspect rating analysis (LARA) model to predict the aspect ratings and determine the latent aspect weights. Specifically, this study extracted the innovative lodging experience aspects that Airbnb customers care about most by mining a total of 248,693 customer reviews from 6,946 Airbnb accommodations. Then, the NRC Emotion Lexicon with eight emotions was employed to assess the sentiments associated with each lodging aspect. By applying latent rating regression, the predicted aspect ratings were generated. With the aspect ratings, , the aspect weights, and the predicted overall ratings were calculated. It was suggested that the overall rating be assessed based on the sentiment words of five lodging aspects: communication, experience, location, product/service, and value. It was found that, compared with the aspects of location, product/service, and value, customers expressed less joy and more surprise than they did over the aspects of communication and experience. The LRR results demonstrate that Airbnb customers care most about a listing location, followed by experience, value, communication, and product/service. The results also revealed that even listings with the same overall rating may have different predicted aspect ratings based on the different aspect weights. Finally, the LARA model demonstrated the different preferences between customers seeking expensive versus cheap accommodations. Understanding customer experience and its role in forming customer rating behavior is important. This study empirically confirms and expands the usefulness of LARA as the prediction model in deconstructing overall ratings into aspect ratings, and then further predicting aspect level weights. This study makes meaningful academic contributions to the evolving customer behavior and customer experience research. It also benefits the shared-lodging industry through its development of pragmatic methods to establish effective marketing strategies for improving customer perceptions and create personalized review filter systems

Analyzing heterogeneity on the value, satisfaction, word-of-mouth relationship in retailing

Purpose - The literature recognizes the need to study differences in consumer behavior in highly competitive and dynamic markets. In this paper, the authors look at how the heterogeneous evaluation of retailing influences customer satisfaction and loyalty. The purpose of this paper is to analyze unobserved heterogeneity on customer value dimensions perceptions in retail establishments, and their potential effects on positive forms of behavioral outcomes considering customer satisfaction as a mediating variable. Design/methodology/approach - On a sample of 820 retail customers, the authors apply a finite mixture structural equation modeling that analyzes unobserved heterogeneity simultaneously. In this model, the authors study the influence of heterogeneous perceptions of excellence, efficiency, entertainment and aesthetics on customer satisfaction and of satisfaction on word-of-mouth (WOM) referral and WOM activity. Findings - The results show two latent segments where the intensity of causal relations varies, which means that the effect of value dimensions and satisfaction are over or underestimated when heterogeneity is ignored. Originality/value - The main value of the paper has been to analyze the potential heterogeneity of value dimensions (intravariable approach), and their links with satisfaction and some dimensions of loyalty (intervariable approach). Customer heterogeneity must be studied to understand the satisfaction process and WOM responses in order to design more efficient and effective relationship marketing strategies

The Current State of Performance Appraisal Research and Practice: Concerns, Directions, and Implications

On the surface, it is not readily apparent how some performance appraisal research issues inform performance appraisal practice. Because performance appraisal is an applied topic, it is useful to periodically consider the current state of performance research and its relation to performance appraisal practice. This review examines the performance appraisal literature published in both academic and practitioner outlets between 1985 and 1990, briefly discusses the current state of performance appraisal practice, highlights the juxtaposition of research and practice, and suggests directions for further research

(How) Do research and administrative duties affect university professors’ teaching?

We analyze the interaction between university professors’ teaching quality and their research and administrative activities. Our sample is a high-quality individual panel data set from a medium size public Spanish university that allows us to avoid several types of biases frequently encountered in the literature. Although researchers teach roughly 20% more than non-researchers, their teaching quality is also 20% higher. Instructors with no research are 5 times more likely than the rest to be among the worst teachers. Over much of the relevant range, we find a nonlinear and positive relationship between research output and teaching quantity on teaching quality. Our conclusions may be useful for decision makers in universities and governments

Visualizing Historical Book Trade Data: An Iterative Design Study with Close Collaboration with Domain Experts

The circulation of historical books has always been an area of interest for historians. However, the data used to represent the journey of a book across different places and times can be difficult for domain experts to digest due to buried geographical and chronological features within text-based presentations. This situation provides an opportunity for collaboration between visualization researchers and historians. This paper describes a design study where a variant of the Nine-Stage Framework was employed to develop a Visual Analytics (VA) tool called DanteExploreVis. This tool was designed to aid domain experts in exploring, explaining, and presenting book trade data from multiple perspectives. We discuss the design choices made and how each panel in the interface meets the domain requirements. We also present the results of a qualitative evaluation conducted with domain experts. The main contributions of this paper include: 1) the development of a VA tool to support domain experts in exploring, explaining, and presenting book trade data; 2) a comprehensive documentation of the iterative design, development, and evaluation process following the variant Nine-Stage Framework; 3) a summary of the insights gained and lessons learned from this design study in the context of the humanities field; and 4) reflections on how our approach could be applied in a more generalizable way

Strategic I/O Psychology and the Role of Utility Analysis Models

In the 1990’s, the significance of human capital in organizations has been increasing,and measurement issues in human resource management have achieved significant prominence. Yet, I/O psychology research on utility analysis and measurement has actually declined. In this chapter we propose a decision-based framework to review developments in utility analysis research since 1991, and show that through lens of this framework there are many fertile avenues for research. We then show that both I/O psychology and strategic HRM research and practice can be enhanced by greater collaboration and integration, particularly regarding the link between human capital and organizational success. We present an integrative framework as the basis for that integration, and illustrate its implications for future research