Log-based software monitoring: a systematic mapping study

Modern software development and operations rely on monitoring to understand how systems behave in production. The data provided by application logs and runtime environment are essential to detect and diagnose undesired behavior and improve system reliability. However, despite the rich ecosystem around industry-ready log solutions, monitoring complex systems and getting insights from log data remains a challenge. Researchers and practitioners have been actively working to address several challenges related to logs, e.g., how to effectively provide better tooling support for logging decisions to developers, how to effectively process and store log data, and how to extract insights from log data. A holistic view of the research effort on logging practices and automated log analysis is key to provide directions and disseminate the state-of-the-art for technology transfer. In this paper, we study 108 papers (72 research track papers, 24 journals, and 12 industry track papers) from different communities (e.g., machine learning, software engineering, and systems) and structure the research field in light of the life-cycle of log data. Our analysis shows that (1) logging is challenging not only in open-source projects but also in industry, (2) machine learning is a promising approach to enable a contextual analysis of source code for log recommendation but further investigation is required to assess the usability of those tools in practice, (3) few studies approached efficient persistence of log data, and (4) there are open opportunities to analyze application logs and to evaluate state-of-the-art log analysis techniques in a DevOps context

Improving reliability of service oriented systems with consideration of cost and time constraints in clouds

Web service technology is more and more popular for the implementation of service oriented systems. Additionally, cloud computing platforms, as an efficient and available environment, can provide the computing, networking and storage resources in order to decrease the budget of companies to deploy and manage their systems. Therefore, more service oriented systems are migrated and deployed in clouds. However, these applications need to be improved in terms of reliability, for certain components have low reliability. Fault tolerance approaches can improve software reliability. However, more redundant units are required, which increases the cost and the execution time of the entire system. Therefore, a migration and deployment framework with fault tolerance approaches with the consideration of global constraints in terms of cost and execution time may be needed. This work proposes a migration and deployment framework to guide the designers of service oriented systems in order to improve the reliability under global constraints in clouds. A multilevel redundancy allocation model is adopted for the framework to assign redundant units to the structure of systems with fault tolerance approaches. An improved genetic algorithm is utilised for the generation of the migration plan that takes the execution time of systems and the cost constraints into consideration. Fault tolerant approaches (such as NVP, RB and Parallel) can be integrated into the framework so as to improve the reliability of the components at the bottom level. Additionally, a new encoding mechanism based on linked lists is proposed to improve the performance of the genetic algorithm in order to reduce the movement of redundant units in the model. The experiments compare the performance of encoding mechanisms and the model integrated with different fault tolerance approaches. The empirical studies show that the proposed framework, with a multilevel redundancy allocation model integrated with the fault tolerance approaches, can generate migration plans for service oriented systems in clouds with the consideration of cost and execution time

The Determinants of Customer Perceptions in a Dynamic Business Environment: An Exploratory Analysis of the ASP Business Model

Outsourcing attracted much attention in 1989 when Kodak outsourced its data center operation to IBM (International Business Machines Corp.). Nowadays, this strategy has become more popular. At the beginning of this century, the ASP (Application Service Provider) model was considered one of the typical solutions of Internet-based IT (Information Technology) outsourcing. Although this model has been transformed and renamed (e.g. SaaS - Software as a Service), the principle concept of providing IT service through the Internet or wide area network is still there. This study attempts to explore the determinants of customer perception of Internet-based IT outsourcing by obtaining a comprehensive understanding of the ASP model. The research dimensions not only include factors affecting users' perception of service quality but also ASP business position (i.e. the firm origin of ASP and its provider type) and services utilized by the customers. Through the study of firm history, two important theoretical themes of this research - path-dependence and Ansoff's product/ market growth matrix - are taken account of in exploring the influence of the determinants. Web-based questionnaire survey research is conducted together with a documentation study to collect data. Targeting the customers of the top 50 ASPs selected by ASPnews.com during the period 2001-2004, the researcher contacted 597 potential respondents, and 196 responses were returned. The valid sample consisted of 175 responses, and 124 of them not only provided full information for satisfaction evaluation but also the information for tracking their ASP vendors' business position. The GLM (General Linear Model) and the Pearson correlation coefficient were the major statistical approaches used to evaluate the survey data for developing a structural model. The research findings indicated that the factors associated with service competitiveness, such as capability and performance, reliability and trustworthiness, affordability, integration and customization, have positive effects on customer perceived satisfaction; whereas lock-in has a negative effect. More specifically, the origin of the ASP firm has a direct effect on capacity and performance, and also directly influences the use of IT adoption services. Based on this finding, a descriptive analysis and qualitative research shows that two mechanisms for path-dependence - existing expertise and perceived expertise - can affect the satisfaction level of capacity and performance of ASP services. On the other hand, provider type has a direct effect on affordability and also directly influences the use of facility supporting services. On this basis, another two mechanisms for path-dependence - transaction cost and standardization - can indirectly impact customer's perception of this business model via affordability. In addition to those major findings, some other determinants (e.g. software applications, brand of applications, and intensity of service used) were also identified in this study. The study result can be used for theoretical understanding about the determinants of ASP customer's perception. It not only indicates a new perspective to enhance the current body of research on this topic, but can also be more broadly applied to any fast-growth firm, rapid-change business, or technology intensive industry. Acknowledgements I would like to sincerely thank the following people for their contribution to this research project. Dr. Scott Koslow, my chief supervisor, for his continued encouragement, patience and guidance to ensure the completion of this project. His speciality in statistics has provided appropriate and valuable guidance in the data analysis for my research. Dr. Steven Lim, my second supervisor, for his advice, coherence, and support over the years. I also appreciate his constructive comments on my drafts and the shaping of my research. Dr. Bob McQueen and Dr. Jim Corner, for their assistance and advice in the early stages of my study. My parents, Yu-Ho and Lee-Chiung Liang, and my brother Ken, my sisters Annie, Eva, and Nancy, my brothers-in-law, J.C. and Chen, and Alice, my sister-in-law, for their emotional support throughout the length of my study. I also thank Bessie, my best friend for her assistance in data collection and her loving support, as well as Ted, Kevin, Mark, Frank, and Shirley, my study mates for their encouragement and friendship. Special thanks goes to Dr. Kuang-Ya Wang, the principal of Yu Da High School of Commerce and Home Economics, Taiwan, and also to the staff over there for their concern and assistance in data collection. Most importantly, my heartfelt appreciation goes to Warren, my husband. I am deeply grateful to him for his understanding, patience, and practical help. Without his enduring support I could not have done this study. Finally, my thanks and gratitude goes to those people who patiently answered my survey questionnaire as their kind assistance made it possible to complete this research

Prioritisation of requests, bugs and enhancements pertaining to apps for remedial actions. Towards solving the problem of which app concerns to address initially for app developers

Useful app reviews contain information related to the bugs reported by the app’s end-users along with the requests or enhancements (i.e., suggestions for improvement) pertaining to the app. App developers expend exhaustive manual efforts towards the identification of numerous useful reviews from a vast pool of reviews and converting such useful reviews into actionable knowledge by means of prioritisation. By doing so, app developers can resolve the critical bugs and simultaneously address the prominent requests or enhancements in short intervals of apps’ maintenance and evolution cycles. That said, the manual efforts towards the identification and prioritisation of useful reviews have limitations. The most common limitations are: high cognitive load required to perform manual analysis, lack of scalability associated with limited human resources to process voluminous reviews, extensive time requirements and error-proneness related to the manual efforts. While prior work from the app domain have proposed prioritisation approaches to convert reviews pertaining to an app into actionable knowledge, these studies have limitations and lack benchmarking of the prioritisation performance. Thus, the problem to prioritise numerous useful reviews still persists. In this study, initially, we conducted a systematic mapping study of the requirements prioritisation domain to explore the knowledge on prioritisation that exists and seek inspiration from the eminent empirical studies to solve the problem related to the prioritisation of numerous useful reviews. Findings of the systematic mapping study inspired us to develop automated approaches for filtering useful reviews, and then to facilitate their subsequent prioritisation. To filter useful reviews, this work developed six variants of the Multinomial Naïve Bayes method. Next, to prioritise the order in which useful reviews should be addressed, we proposed a group-based prioritisation method which initially classified the useful reviews into specific groups using an automatically generated taxonomy, and later prioritised these reviews using a multi-criteria heuristic function. Subsequently, we developed an individual prioritisation method that directly prioritised the useful reviews after filtering using the same multi-criteria heuristic function. Some of the findings of the conducted systematic mapping study not only provided the necessary inspiration towards the development of automated filtering and prioritisation approaches but also revealed crucial dimensions such as accuracy and time that could be utilised to benchmark the performance of a prioritisation method. With regards to the proposed automated filtering approach, we observed that the performance of the Multinomial Naïve Bayes variants varied based on their algorithmic structure and the nature of labelled reviews (i.e., balanced or imbalanced) that were made available for training purposes. The outcome related to the automated taxonomy generation approach for classifying useful review into specific groups showed a substantial match with the manual taxonomy generated from domain knowledge. Finally, we validated the performance of the group-based prioritisation and individual prioritisation methods, where we found that the performance of the individual prioritisation method was superior to that of the group-based prioritisation method when outcomes were assessed for the accuracy and time dimensions. In addition, we performed a full-scale evaluation of the individual prioritisation method which showed promising results. Given the outcomes, it is anticipated that our individual prioritisation method could assist app developers in filtering and prioritising numerous useful reviews to support app maintenance and evolution cycles. Beyond app reviews, the utility of our proposed prioritisation solution can be evaluated on software repositories tracking bugs and requests such as Jira, GitHub and so on