An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

Deep Neural Networks (DNNs) are being adopted as components in software systems. Creating and specializing DNNs from scratch has grown increasingly difficult as state-of-the-art architectures grow more complex. Following the path of traditional software engineering, machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) and fine-tune these models for downstream tasks. Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency management. We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems. In this work, we present the first empirical investigation of PTM reuse. We interviewed 12 practitioners from the most popular PTM ecosystem, Hugging Face, to learn the practices and challenges of PTM reuse. From this data, we model the decision-making process for PTM reuse. Based on the identified practices, we describe useful attributes for model reuse, including provenance, reproducibility, and portability. Three challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks. We substantiate these identified challenges with systematic measurements in the Hugging Face ecosystem. Our work informs future directions on optimizing deep learning ecosystems by automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries

Developing knowledge factory system as a best practice platform in software process environment

Knowledge’ is one of the main outcomes systematic approaches of software engineering. It is considered as a crucial resource of each organization and, therefore, needs to be managed carefully especially for the software development process and the roles involved in each phase. The effectiveness application of software engineering relies on experiences, and organizations need to capitalize on their experiences by capturing and reusing them. In all approaches of experience reuse, repositories play a central role in enhancing the knowledge in experience transfer of process knowledge between projects. Like in a factory beside the production there are the people who produce. Experiences should not be stored on a dusty shelf, but engineered into best practices and processes that could be guidance to all roles in the software project especially for the developer. This paper presents the approach to the implementation of the project knowledge based on software development life cycle.It also describe on how the knowledge factory solution can be used as a decision support system for software project managers in day-to-day development activities. The paper concludes with an evaluation of our approach to date and a description of future research directions

QuESo: A quality model for open source software ecosystems

© 2014 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Open source software has witnessed an exponential growth in the last two decades and it is playing an increasingly important role in many companies and organizations leading to the formation of open source software ecosystems. In this paper we present a quality model that will allow the evaluation of those ecosystems in terms of their relevant quality characteristics such as health or activeness. To design this quality model we started by analysing the quality measures found during the execution of a systematic literature review on open source software ecosystems and, then, we classified and reorganized the set of measures in order to build a solid quality model.Peer ReviewedPostprint (author's final draft

Software Reuse in Agile Development Organizations - A Conceptual Management Tool

The reuse of knowledge is considered a major factor for increasing productivity and quality. In the software industry knowledge is embodied in software assets such as code components, functional designs and test cases. This kind of knowledge reuse is also referred to as software reuse. Although the benefits can be substantial, software reuse has never reached its full potential. Organizations are not aware of the different levels of reuse or do not know how to address reuse issues. This paper proposes a conceptual management tool for supporting software reuse. Furthermore the paper presents the findings of the application of the management tool in an agile development organization

Exploring regression testing and software product line testing - research and state of practice

In large software organizations with a product line development approach a selective testing of product variants is necessary in order to keep pace with the decreased development time for new products, enabled by the systematic reuse. The close relationship between products in product line indicates an option to reduce the testing effort due to redundancy. In many cases test selection is performed manually, based on test leaders’ expertise. This makes the cost and quality of the testing highly dependent on the skills and experience of the test leaders. There is a need in industry for systematic approaches to test selection. The goal of our research is to improve the control of the testing and reduce the amount of redundant testing in the product line context by applying regression test selection strategies. In this thesis, the state of art of regression testing and software product line testing are explored. Two extensive systematic reviews are conducted as well as an industrial survey of regression testing state of practice and an industrial evaluation of a pragmatic regression test selection strategy. Regression testing is not an isolated one-off activity, but rather an activity of varying scope and preconditions, strongly dependent on the context in which it is applied. Several techniques for regression test selection are proposed and evaluated empirically but in many cases the context is too specific for a technique to be easily applied directly by software developers. In order to improve the possibility for generalizing empirical results on regression test selection, guidelines for reporting the testing context are discussed in this thesis. Software product line testing is a relatively new research area. The understanding about challenges is well established but when looking for solutions to these challenges, we mostly find proposals, and empirical evaluations are sparse. Regression test selection strategies proposed in literature are not easily applicable in the product line context. Instead, control may be increased by increased visibility of the effects of testing and proper measurements of software quality. Focus of our future work will be on how to guide the planning and assessment of regression testing activities in large, complex reuse based systems, by visualizing the quality achieved in different parts of the system and evaluating the effects of different selection strategies when applied in various regression testing situations

Software similarity measurements using UML diagrams: A systematic literature review

Every piece of software uses a model to derive its operational, auxiliary, and functional procedures. Unified Modeling Language (UML) is a standard displaying language for determining, recording, and building a software product. Several algorithms have been used by researchers to measure similarities between UML artifacts. However, there no literature studies have considered measurements of UML diagram similarities. This paper presents the results of a systematic literature review concerning similarity measurements between the UML diagrams of different software products. The study reviews and identifies similarity measurements of UML artifacts, with class diagram, sequence diagram, statechart diagram, and use case diagram being UML diagrams that are widely used as research objects for measuring similarity. Measuring similarity enables resolution of the problem domains of software reuse, similarity measurement, and clone detection. The instruments used to measure similarity are semantic and structural similarity. The findings indicate opportunities for future research regarding calculating other UML diagrams, compiling calculation information for each diagram, adapting semantic and structural similarity calculation methods, determining the best weight for each item in the diagram, testing novel proposed methods, and building or finding good datasets for use as testing material

A pattern language for evolution reuse in component-based software architectures

Context: Modern software systems are prone to a continuous evolution under frequently varying requirements and changes in operational environments. Architecture-Centric Software Evolution (ACSE) enables changes in a system’s structure and behaviour while maintaining a global view of the software to address evolution-centric trade-offs. Lehman’s law of continuing change demands for long-living and continuously evolving architectures to prolong the productive life and economic value of software. Also some industrial research shows that evolution reuse can save approximately 40% effort of change implementation in ACSE process. However, a systematic review of existing research suggests a lack of solution(s) to support a continuous integration of reuse knowledge in ACSE process to promote evolution-off-the-shelf in software architectures. Objectives: We aim to unify the concepts of software repository mining and software evolution to discover evolution-reuse knowledge that can be shared and reused to guide ACSE. Method: We exploit repository mining techniques (also architecture change mining) that investigates architecture change logs to discover change operationalisation and patterns. We apply software evolution concepts (also architecture change execution) to support pattern-driven reuse in ACSE. Architecture change patterns support composition and application of a pattern language that exploits patterns and their relations to express evolution-reuse knowledge. Pattern language composition is enabled with a continuous discovery of patterns from architecture change logs and formalising relations among discovered patterns. Pattern language application is supported with an incremental selection and application of patterns to achieve reuse in ACSE. The novelty of the research lies with a framework PatEvol that supports a round-trip approach for a continuous acquisition (mining) and application (execution) of reuse knowledge to enable ACSE. Prototype support enables customisation and (semi-) automation for the evolution process. Results: We evaluated the results based on the ISO/IEC 9126 - 1 quality model and a case study based validation of the architecture change mining and change execution processes. We observe consistency and reusability of change support with pattern-driven architecture evolution. Change patterns support efficiency for architecture evolution process but lack a fine-granular change implementation. A critical challenge lies with the selection of appropriate patterns to form a pattern language during evolution. Conclusions: The pattern language itself continuously evolves with an incremental discovery of new patterns from change logs over time. A systematic identification and resolution of change anti-patterns define the scope for future research

Requirements for software engineering languages

This paper analyzes the concepts of software construction embodied in the Draco system. The analysis relates specific mechanisms in Draco to particular software engineering (SE) principles and suggests future research needed to extend the approach. The purpose of the analysis is to help researchers understand Draco better and thus be able to direct in productive directions future research on this type of software engineering tool

Software development: A paradigm for the future

A new paradigm for software development that treats software development as an experimental activity is presented. It provides built-in mechanisms for learning how to develop software better and reusing previous experience in the forms of knowledge, processes, and products. It uses models and measures to aid in the tasks of characterization, evaluation and motivation. An organization scheme is proposed for separating the project-specific focus from the organization's learning and reuse focuses of software development. The implications of this approach for corporations, research and education are discussed and some research activities currently underway at the University of Maryland that support this approach are presented