Mining, Understanding and Integrating User Preferences in Software Refactoring Using Computational Search, Machine Learning, and Dimensionality Reduction

Search-Based Software Engineering (SBSE) is a software development practice which focuses on couching software engineering problems as optimization problems using metaheuristic techniques to automate the search for near optimal solutions to those problems. While SBSE has been successfully applied to a wide variety of software engineering problems, our understanding of the extent and nature of how software engineering problems can be formulated as automated or semi-automated search is still lacking. The majority of software engineering solutions are very subjective and present difficulties to formally define fitness functions to evaluate them. Current studies focus on guiding the search of optimal solutions rather than performing it. It is unclear yet the degree of interaction required with software engineers during the optimization process and how to reduce it. In this work, we focus on search-based software maintenance and evolution problems including software refactoring and software remodularization to improve the quality of systems. We propose to address the following challenges: • A major challenge in adapting a search-based technique for a software engineering problem is the definition of the fitness function. In most cases, fitness functions are ill-defined or subjective. • Most existing refactoring studies do not include the developer in the loop to analyze suggested refactoring solutions, and give their feedback during the optimization process. In addition, some quality metrics are cost-expensive leading to cost-expensive fitness functions. Moreover, while quality metrics evaluate the structural improvements of the refactored system, it is impossible to evaluate the semantic coherence of the design without user interactions. • Finally, several metrics can be dependent and correlated, thus it may be possible to reduce the number of objectives/dimensions when addressing refactoring problems. To address the above challenges, this work provides new techniques and tools to formulate software refactoring as scalable and learning-based search problem. We proposed novel interactive learning-based techniques using machine learning to incorporate developers knowledge and preferences in the search, resulting in more efficient and cost-effective search-based refactoring recommendation systems. We designed and implemented novel objective reduction SBSE methodologies to support scalable number of objectives. The proposed solutions were empirically evaluated in academic (open-source systems) and industrial settings.Ph.D.College of Engineering and Computer ScienceUniversity of Michigan-Dearbornhttps://deepblue.lib.umich.edu/bitstream/2027.42/138970/1/Dea Final Dissertation.pdfDescription of Dea Final Dissertation.pdf : DissertationDescription of Troh Josselin Dea Signed Certification Form.pdf : Committee signature fil

Commits Analysis for Software Refactoring Documentation and Recommendation

Software projects frequently evolve to meet new requirements and/or to fix bugs. While this evolution is critical, it may have a negative impact on the quality of the system. To improve the quality of software systems, the first step is “detection" of code antipatterns to be restructured which can be considered as “refactoring opportunities". The second step is the “prioritization" of code fragments to be refactored/fixed. The third step is “recommendation" of refactorings to fix the detected quality issues. The fourth step is “testing" the recommended refactorings to evaluate their correctness. The fifth step is the “documentation" of the applied refactorings. In this thesis, we addressed the above five steps: 1. We designed a bi-level multi-objective optimization approach to enable the generation of antipattern examples that can improve the efficiency of detection rules for bad quality designs. 2. Regarding refactoring recommendations, we first identify refactoring opportunities by analyzing developer commit messages and quality of changed files, then we distill this knowledge into usable context driven refactoring recommendations to complement static and dynamic analysis of code. 3. We proposed an interactive refactoring recommendation approach that enables developers to pinpoint their preferences simultaneously in the objective (quality metrics) and decision (code location) spaces. 4. We proposed a semi-automated refactoring documentation bot that helps developers to interactively check and validate the documentation of the refactorings and/or quality improvements at the file level for each opened pull-request before being reviewed or merged to the master 5. We performed interviews with and a survey of practitioners as well as a quantitative analysis of 1,193 commit messages containing refactorings to establish a refactoring documentation model as a set of components. 6. We formulated the recommendation of code reviewers as a multi-objective search problem to balance the conflicting objectives of expertise, availability, and history of collaborations. 7. We built a dataset composed of 50,000+ composite code changes pertaining to more than 7,000 open-source projects. Then, we proposed and evaluated a new deep learning technique to generate commit messages for composite code changes based on an attentional encoder-decoder with two encoders and BERT embeddings.Ph.D.College of Engineering & Computer ScienceUniversity of Michigan-Dearbornhttp://deepblue.lib.umich.edu/bitstream/2027.42/169486/1/Soumaya Rebai final dissertation.pdfDescription of Soumaya Rebai final dissertation.pdf : Dissertatio

NPS AUV Integrated Simulation

The development and testing of Autonomous Underwater Vehicle (AUV) hardware and software is greatly complicated by vehicle inaccessibility during operation. Integrated simulation remotely links vehicle components and support equipment with graphics simulation workstations, allowing complete real-time, pre-mission, pseudo-mission and post-mission visualization and analysis in the lab environment. Integrated simulator testing of software and hardware is a broad and versatile method that supports rapid and robust diagnosis and correction of system faults. This method is demonstrated using the Naval Postgraduate School (NPS) AUV. High-resolution three-dimensional graphics workstations can provide real-time representations of vehicle dynamics, control system behavior, mission execution, sensor processing and object classification. Integrated simulation is also useful for development of the variety of sophisticated artificial intelligence applications needed by an AUV. Examples include sonar classification using an expert system and path planning using a circle world model. The flexibility and versatility provided by this approach enables visualization and analysis of all aspects of AUV development. Integrated simulator networking is recommended as a fundamental requirement for AUV research and deployment.http://archive.org/details/npsauvintegrated00brutLieutenant Commander, United States NavyApproved for public release; distribution is unlimited

Reconstruction of Software Component Architectures and Behaviour Models using Static and Dynamic Analysis

Model-based performance prediction systematically deals with the evaluation of software performance to avoid for example bottlenecks, estimate execution environment sizing, or identify scalability limitations for new usage scenarios. Such performance predictions require up-to-date software performance models. This book describes a new integrated reverse engineering approach for the reconstruction of parameterised software performance models (software component architecture and behaviour)

Anomaly Detection and Exploratory Causal Analysis for SAP HANA

Nowadays, the good functioning of the equipment, networks and systems will be the key for the business of a company to continue operating because it is never avoidable for the companies to use information technology to support their business in the era of big data. However, the technology is never infallible, faults that give rise to sometimes critical situations may appear at any time. To detect and prevent failures, it is very essential to have a good monitoring system which is responsible for controlling the technology used by a company (hardware, networks and communications, operating systems or applications, among others) in order to analyze their operation and performance, and to detect and alert about possible errors. The aim of this thesis is thus to further advance the field of anomaly detection and exploratory causal inference which are two major research areas in a monitoring system, to provide efficient algorithms with regards to the usability, maintainability and scalability. The analyzed results can be viewed as a starting point for the root cause analysis of the system performance issues and to avoid falls in the system or minimize the time of resolution of the issues in the future. The algorithms were performed on the historical data of SAP HANA database at last and the results gained in this thesis indicate that the tools have succeeded in providing some useful information for diagnosing the performance issues of the system

Knowledge Representation in Engineering 4.0

This dissertation was developed in the context of the BMBF and EU/ECSEL funded projects GENIAL! and Arrowhead Tools. In these projects the chair examines methods of specifications and cooperations in the automotive value chain from OEM-Tier1-Tier2. Goal of the projects is to improve communication and collaborative planning, especially in early development stages. Besides SysML, the use of agreed vocabularies and on- tologies for modeling requirements, overall context, variants, and many other items, is targeted. This thesis proposes a web database, where data from the collaborative requirements elicitation is combined with an ontology-based approach that uses reasoning capabilities. For this purpose, state-of-the-art ontologies have been investigated and integrated that entail domains like hardware/software, roadmapping, IoT, context, innovation and oth- ers. New ontologies have been designed like a HW / SW allocation ontology and a domain-specific "eFuse ontology" as well as some prototypes. The result is a modular ontology suite and the GENIAL! Basic Ontology that allows us to model automotive and microelectronic functions, components, properties and dependencies based on the ISO26262 standard among these elements. Furthermore, context knowledge that influences design decisions such as future trends in legislation, society, environment, etc. is included. These knowledge bases are integrated in a novel tool that allows for collabo- rative innovation planning and requirements communication along the automotive value chain. To start off the work of the project, an architecture and prototype tool was developed. Designing ontologies and knowing how to use them proved to be a non-trivial task, requiring a lot of context and background knowledge. Some of this background knowledge has been selected for presentation and was utilized either in designing models or for later immersion. Examples are basic foundations like design guidelines for ontologies, ontology categories and a continuum of expressiveness of languages and advanced content like multi-level theory, foundational ontologies and reasoning. Finally, at the end, we demonstrate the overall framework, and show the ontology with reasoning, database and APPEL/SysMD (AGILA ProPErty and Dependency Descrip- tion Language / System MarkDown) and constraints of the hardware / software knowledge base. There, by example, we explore and solve roadmap constraints that are coupled with a car model through a constraint solver.Diese Dissertation wurde im Kontext des von BMBF und EU / ECSEL gefördertem Projektes GENIAL! und Arrowhead Tools entwickelt. In diesen Projekten untersucht der Lehrstuhl Methoden zur Spezifikationen und Kooperation in der Automotive Wertschöp- fungskette, von OEM zu Tier1 und Tier2. Ziel der Arbeit ist es die Kommunikation und gemeinsame Planung, speziell in den frühen Entwicklungsphasen zu verbessern. Neben SysML ist die Benutzung von vereinbarten Vokabularen und Ontologien in der Modellierung von Requirements, des Gesamtkontextes, Varianten und vielen anderen Elementen angezielt. Ontologien sind dabei eine Möglichkeit, um das Vermeiden von Missverständnissen und Fehlplanungen zu unterstützen. Dieser Ansatz schlägt eine Web- datenbank vor, wobei Ontologien das Teilen von Wissen und das logische Schlussfolgern von implizitem Wissen und Regeln unterstützen. Diese Arbeit beschreibt Ontologien für die Domäne des Engineering 4.0, oder spezifischer, für die Domäne, die für das deutsche Projekt GENIAL! benötigt wurde. Dies betrifft Domänen, wie Hardware und Software, Roadmapping, Kontext, Innovation, IoT und andere. Neue Ontologien wurden entworfen, wie beispielsweise die Hardware-Software Allokations-Ontologie und eine domänen-spezifische "eFuse Ontologie". Das Ergebnis war eine modulare Ontologie-Bibliothek mit der GENIAL! Basic Ontology, die es erlaubt, automotive und mikroelektronische Komponenten, Funktionen, Eigenschaften und deren Abhängigkeiten basierend auf dem ISO26262 Standard zu entwerfen. Des weiteren ist Kontextwissen, welches Entwurfsentscheidungen beinflusst, inkludiert. Diese Wissensbasen sind in einem neuartigen Tool integriert, dass es ermöglicht, Roadmapwissen und Anforderungen durch die Automobil- Wertschöpfungskette hinweg auszutauschen. On tologien zu entwerfen und zu wissen, wie man diese benutzt, war dabei keine triviale Aufgabe und benötigte viel Hintergrund- und Kontextwissen. Ausgewählte Grundlagen hierfür sind Richtlinien, wie man Ontologien entwirft, Ontologiekategorien, sowie das Spektrum an Sprachen und Formen von Wissensrepresentationen. Des weiteren sind fort- geschrittene Methoden erläutert, z.B wie man mit Ontologien Schlußfolgerungen trifft. Am Schluss wird das Overall Framework demonstriert, und die Ontologie mit Reason- ing, Datenbank und APPEL/SysMD (AGILA ProPErty and Dependency Description Language / System MarkDown) und Constraints der Hardware / Software Wissensbasis gezeigt. Dabei werden exemplarisch Roadmap Constraints mit dem Automodell verbunden und durch den Constraint Solver gelöst und exploriert

A reference architecture for flexibly integrating machine vision within manufacturing

A reference architecture provides an overall framework that may embrace models, methodologies and mechanisms which can support the lifecycle of their target domain. The work described in this thesis makes a contribution to establishing such a generally applicable reference architecture for supporting the lifecycIe of a new generation of integrated machine vision systems. Contemporary machine vision systems consist of a complex combination of mechanical engineering, the hardware and software of an electronic processor, plus optical, sensory and lighting components. "This thesis is concerned with the structure of the software which characterises the system application. The machine vision systems which are currently used within manufacturing industry are difficult to integrate within the information systems required within modem manufacturing enterprises. They are inflexible in all but the execution of a range of similar operations, and their design and implementation is often such that they are difficult to update in the face of the required change inherent within modem manufacturing. The proposed reference architecture provides an overall framework within which a number of supporting models, design methodologies, and implementation mechanisms can combine to provide support for the rapid creation and maintenance of highly structured machine vision applications. These applications comprise modules which can be considered as building blocks of CIM systems. Their integrated interoperation can be enabled by the emerging infrastructural tools which will be required to underpin the next generation of flexibly integrated manufacturing systems. The work described in this thesis concludes that the issues of machine vision applications and the issues of integration of these applications within manufacturing systems are entirely separate. This separation is reflected in the structure of the thesis. PART B details vision application issues while PAIIT C deals with integration. The criteria for next generation integrated machine vision systems, derived in PART A of the thesis, are extensive. In order to address these criteria and propose a complete architecture, a "thin slice" is taken through the areas of vision application, and integration at the lifecycle stages of design, implementation, runtime and maintenance. The thesis describes the reference architecture, demonstrates its use though a proof of concept implementation and evaluates the support offered by the architecture for easing the problems of software change

Supporting IT Service Fault Recovery with an Automated Planning Method

Despite advances in software and hardware technologies, faults are still inevitable in a highly-dependent, human-engineered and administrated IT environment. Given the critical role of IT services today, it is imperative that faults, having once occurred, have to be dealt with eciently and eeffectively to avoid or reduce the actual losses. Nevertheless, the complexities of current IT services, e.g., with regard to their scales, heterogeneity and highly dynamic infrastructures, make the recovery operation a challenging task for operators. Such complexities will eventually outgrow the human capability to manage them. Such diculty is augmented by the fact that there are few well-devised methods available to support fault recovery. To tackle this issue, this thesis aims at providing a computer-aided approach to assist operators with fault recovery planning and, consequently, to increase the eciency of recovery activities.We propose a generic framework based on the automated planning theory to generate plans for recoveries of IT services. At the heart of the framework is a planning component. Assisted by the other participants in the framework, the planning component aggregates the relevant information and computes recovery steps accordingly. The main idea behind the planning component is to sustain the planning operations with automated planning techniques, which is one of the research fields of articial intelligence. Provided with a general planning model, we show theoretically that the service fault recovery problem can be indeed solved by automated planning techniques. The relationship between a planning problem and a fault recovery problem is shown by means of reduction between these problems. After an extensive investigation, we choose a planning paradigm that based on Hierarchical Task Networks (HTN) as the guideline for the design of our main planning algorithm called H2MAP. To sustain the operation of the planner, a set of components revolving around the planning component is provided. These components are responsible for tasks such as translation between dierent knowledge formats, persistent storage of planning knowledge and communication with external systems. To ensure extendibility in our design, we apply dierent design patterns for the components. We sketch and discuss the technical aspects of implementations of the core components. Finally, as proof of the concept, the framework is instantiated to two distinguishing application scenarios

Explainable, Security-Aware and Dependency-Aware Framework for Intelligent Software Refactoring

As software systems continue to grow in size and complexity, their maintenance continues to become more challenging and costly. Even for the most technologically sophisticated and competent organizations, building and maintaining high-performing software applications with high-quality-code is an extremely challenging and expensive endeavor. Software Refactoring is widely recognized as the key component for maintaining high-quality software by restructuring existing code and reducing technical debt. However, refactoring is difficult to achieve and often neglected due to several limitations in the existing refactoring techniques that reduce their effectiveness. These limitation include, but not limited to, detecting refactoring opportunities, recommending specific refactoring activities, and explaining the recommended changes. Existing techniques are mainly focused on the use of quality metrics such as coupling, cohesion, and the Quality Metrics for Object Oriented Design (QMOOD). However, there are many other factors identified in this work to assist and facilitate different maintenance activities for developers: 1. To structure the refactoring field and existing research results, this dissertation provides the most scalable and comprehensive systematic literature review analyzing the results of 3183 research papers on refactoring covering the last three decades. Based on this survey, we created a taxonomy to classify the existing research, identified research trends and highlighted gaps in the literature for further research. 2. To draw attention to what should be the current refactoring research focus from the developers’ perspective, we carried out the first large scale refactoring study on the most popular online Q&A forum for developers, Stack Overflow. We collected and analyzed posts to identify what developers ask about refactoring, the challenges that practitioners face when refactoring software systems, and what should be the current refactoring research focus from the developers’ perspective. 3. To improve the detection of refactoring opportunities in terms of quality and security in the context of mobile apps, we designed a framework that recommends the files to be refactored based on user reviews. We also considered the detection of refactoring opportunities in the context of web services. We proposed a machine learning-based approach that helps service providers and subscribers predict the quality of service with the least costs. Furthermore, to help developers make an accurate assessment of the quality of their software systems and decide if the code should be refactored, we propose a clustering-based approach to automatically identify the preferred benchmark to use for the quality assessment of a project. 4. Regarding the refactoring generation process, we proposed different techniques to enhance the change operators and seeding mechanism by using the history of applied refactorings and incorporating refactoring dependencies in order to improve the quality of the refactoring solutions. We also introduced the security aspect when generating refactoring recommendations, by investigating the possible impact of improving different quality attributes on a set of security metrics and finding the best trade-off between them. In another approach, we recommend refactorings to prioritize fixing quality issues in security-critical files, improve quality attributes and remove code smells. All the above contributions were validated at the large scale on thousands of open source and industry projects in collaboration with industry partners and the open source community. The contributions of this dissertation are integrated in a cloud-based refactoring framework which is currently used by practitioners.Ph.D.College of Engineering & Computer ScienceUniversity of Michigan-Dearbornhttp://deepblue.lib.umich.edu/bitstream/2027.42/171082/1/Chaima Abid Final Dissertation.pdfDescription of Chaima Abid Final Dissertation.pdf : Dissertatio