Software engineering for deep learning applications: usage of SWEng and MLops tools in GitHub repositories

The rising popularity of deep learning (DL) methods and techniques has invigorated interest in the topic of SE4DL, the application of software engineering (SE) practices on deep learning software. Despite the novel engineering challenges brought on by the data-driven and non-deterministic paradigm of DL software, little work has been invested into developing AI-targeted SE tools. On the other hand, tools tackling more general engineering issues in DL are actively used and referred to under the umbrella term of ``MLOps tools''. Furthermore, the available literature supports the utility of conventional SE tooling in DL software development. Building upon previous MSR research on tool usage in open-source software works, we identify conventional and MLOps tools adopted in popular applied DL projects that use Python as the main programming language. About 70% of the GitHub repositories mined contained at least one conventional SE tool. Software configuration management tools are the most adopted, while the opposite applies to maintenance tools. Substantially fewer MLOps tools were in use, with only 9 tools out of a sample of 80 used in at least one repository. The majority of them were open-source rather than proprietary. One of these tools, TensorBoard, was found to be adopted in about half of the repositories in our study. Consequently, the use of conventional SE tooling demonstrates its relevance to DL software. Further research is recommended on the adoption of MLOps tooling by open-source projects, focusing on the relevance of particular tool types, the development of required tools, as well as ways to promote the use of already available tools

The transition from static to dynamic web design

University of Technology, Sydney. Faculty of Design, Architecture and Building.Recent developments in open source software are enabling web designers to configure dynamic web sites quickly, easily and cheaply. The promise of open source CMS software, ultimately is a web environment in which users are programmers. On the surface it would appear that web site creation is moving in the direction of no longer needing expensive custom solutions by professional web developers. These new software systems use a variety of approaches from mere configuration of the system to script implementation. They may require limited programming experience beyond the web designer’s CSS and XHTML capabilities. Based on the Open Source platform, PHP and MySQL offer an easily learned and supported framework to web designers. Web designers now have a tremendous opportunity to harness open source content management systems and extend what they are offering to clients and their employers. Increasingly, it will become the duty of web instructors and of tertiary institutions to equip students with these capabilities, so that they can function effectively in the new environment. This research uses surveys, interviews, and a research workshop with longitudinal results to explore the questions posed. The collation of materials is hosted on a dedicated web site. The findings of this research indicate that students would benefit from a practical approach to web development through the introduction of a CMS in their studies. These are skills employers consider valuable. The main conclusions and recommendations are firstly that many design students see themselves as performing an exclusively aesthetic role in the development of web projects, and secondly, that these students, with proper instruction in the usage of open source CMS’s, are more than capable of making and implementing decisions concerning the functionality or programming of web projects. As a result of this study it is my belief that design students with a deeper understanding of open source CMS’s will be able to occupy a role currently falling between the occupations of graphic design, web development and object-oriented programming. It is my hope that this project will inform and educate so that the wider design community, and in particular design educators, can make more informed decisions about the transition from static to dynamic web design

Can open-source projects (re-) shape the SDN/NFV-driven telecommunication market?

Telecom network operators face rapidly changing business needs. Due to their dependence on long product cycles they lack the ability to quickly respond to changing user demands. To spur innovation and stay competitive, network operators are investigating technological solutions with a proven track record in other application domains such as open source software projects. Open source software enables parties to learn, use, or contribute to technology from which they were previously excluded. OSS has reshaped many application areas including the landscape of operating systems and consumer software. The paradigmshift in telecommunication systems towards Software-Defined Networking introduces possibilities to benefit from open source projects. Implementing the control part of networks in software enables speedier adaption and innovation, and less dependencies on legacy protocols or algorithms hard-coded in the control part of network devices. The recently proposed concept of Network Function Virtualization pushes the softwarization of telecommunication functionalities even further down to the data plane. Within the NFV paradigm, functionality which was previously reserved for dedicated hardware implementations can now be implemented in software and deployed on generic Commercial Off-The Shelf (COTS) hardware. This paper provides an overview of existing open source initiatives for SDN/NFV-based network architectures, involving infrastructure to orchestration-related functionality. It situates them in a business process context and identifies the pros and cons for the market in general, as well as for individual actors

Support for collaborative component-based software engineering

Collaborative system composition during design has been poorly supported by traditional CASE tools (which have usually concentrated on supporting individual projects) and almost exclusively focused on static composition. Little support for maintaining large distributed collections of heterogeneous software components across a number of projects has been developed. The CoDEEDS project addresses the collaborative determination, elaboration, and evolution of design spaces that describe both static and dynamic compositions of software components from sources such as component libraries, software service directories, and reuse repositories. The GENESIS project has focussed, in the development of OSCAR, on the creation and maintenance of large software artefact repositories. The most recent extensions are explicitly addressing the provision of cross-project global views of large software collections and historical views of individual artefacts within a collection. The long-term benefits of such support can only be realised if OSCAR and CoDEEDS are widely adopted and steps to facilitate this are described. This book continues to provide a forum, which a recent book, Software Evolution with UML and XML, started, where expert insights are presented on the subject. In that book, initial efforts were made to link together three current phenomena: software evolution, UML, and XML. In this book, focus will be on the practical side of linking them, that is, how UML and XML and their related methods/tools can assist software evolution in practice. Considering that nowadays software starts evolving before it is delivered, an apparent feature for software evolution is that it happens over all stages and over all aspects. Therefore, all possible techniques should be explored. This book explores techniques based on UML/XML and a combination of them with other techniques (i.e., over all techniques from theory to tools). Software evolution happens at all stages. Chapters in this book describe that software evolution issues present at stages of software architecturing, modeling/specifying, assessing, coding, validating, design recovering, program understanding, and reusing. Software evolution happens in all aspects. Chapters in this book illustrate that software evolution issues are involved in Web application, embedded system, software repository, component-based development, object model, development environment, software metrics, UML use case diagram, system model, Legacy system, safety critical system, user interface, software reuse, evolution management, and variability modeling. Software evolution needs to be facilitated with all possible techniques. Chapters in this book demonstrate techniques, such as formal methods, program transformation, empirical study, tool development, standardisation, visualisation, to control system changes to meet organisational and business objectives in a cost-effective way. On the journey of the grand challenge posed by software evolution, the journey that we have to make, the contributory authors of this book have already made further advances

Extracting Build Changes with BUILDDIFF

Build systems are an essential part of modern software engineering projects. As software projects change continuously, it is crucial to understand how the build system changes because neglecting its maintenance can lead to expensive build breakage. Recent studies have investigated the (co-)evolution of build configurations and reasons for build breakage, but they did this only on a coarse grained level. In this paper, we present BUILDDIFF, an approach to extract detailed build changes from MAVEN build files and classify them into 95 change types. In a manual evaluation of 400 build changing commits, we show that BUILDDIFF can extract and classify build changes with an average precision and recall of 0.96 and 0.98, respectively. We then present two studies using the build changes extracted from 30 open source Java projects to study the frequency and time of build changes. The results show that the top 10 most frequent change types account for 73% of the build changes. Among them, changes to version numbers and changes to dependencies of the projects occur most frequently. Furthermore, our results show that build changes occur frequently around releases. With these results, we provide the basis for further research, such as for analyzing the (co-)evolution of build files with other artifacts or improving effort estimation approaches. Furthermore, our detailed change information enables improvements of refactoring approaches for build configurations and improvements of models to identify error-prone build files.Comment: Accepted at the International Conference of Mining Software Repositories (MSR), 201

Environments to support collaborative software engineering

With increasing globalisation of software production, widespread use of software components, and the need to maintain software systems over long periods of time, there has been a recognition that better support for collaborative working is needed by software engineers. In this paper, two approaches to developing improved system support for collaborative software engineering are described: GENESIS and OPHELIA. As both projects are moving towards industrial trials and eventual publicreleases of their systems, this exercise of comparing and contrasting our approaches has provided the basis for future collaboration between our projects particularly in carrying out comparative studies of our approaches in practical use

Software Reuse in Agile Development Organizations - A Conceptual Management Tool

The reuse of knowledge is considered a major factor for increasing productivity and quality. In the software industry knowledge is embodied in software assets such as code components, functional designs and test cases. This kind of knowledge reuse is also referred to as software reuse. Although the benefits can be substantial, software reuse has never reached its full potential. Organizations are not aware of the different levels of reuse or do not know how to address reuse issues. This paper proposes a conceptual management tool for supporting software reuse. Furthermore the paper presents the findings of the application of the management tool in an agile development organization

A DevOps approach to integration of software components in an EU research project

We present a description of the development and deployment infrastructure being created to support the integration effort of HARNESS, an EU FP7 project. HARNESS is a multi-partner research project intended to bring the power of heterogeneous resources to the cloud. It consists of a number of different services and technologies that interact with the OpenStack cloud computing platform at various levels. Many of these components are being developed independently by different teams at different locations across Europe, and keeping the work fully integrated is a challenge. We use a combination of Vagrant based virtual machines, Docker containers, and Ansible playbooks to provide a consistent and up-to-date environment to each developer. The same playbooks used to configure local virtual machines are also used to manage a static testbed with heterogeneous compute and storage devices, and to automate ephemeral larger-scale deployments to Grid5000. Access to internal projects is managed by GitLab, and automated testing of services within Docker-based environments and integrated deployments within virtual-machines is provided by Buildbot

An open source collaboration infrastructure for Calibre

The study of Free and Open Source (Libre) software and the benefits provided by its processes and products to collaborative software development has been somewhat ad hoc. Each project wishing to use tools and techniques drawn from Libre software conducts its own research, thus duplicating effort, consequently there is a lack of established community practice on which new projects can draw. Long-standing intuitive theories of Libre development lack empirical validation. The long-term goal is to provide a resource to guide the evolution of Libre-software projects, from inception to maturity. The CALIBRE project is a co-ordination action aiming to address these issues through its research, its wider educational goals, and with an open invitation to the community to contribute. To succeed, the CALIBRE project needs an effective technological infrastructure which must support internal and external collaboration, communication and contribution to the project. The requirements of CALIBRE are similar to those of a Libre software project; this suggests that adopting a SourceForge-style environment which will be incrementally enhanced with further specialised tools as the requirements become better understood will be a sensible strategy