Requirements Engineering in the Context of Big Data Software Applications

Big Data applications, like traditional applications, serve end-user needs except that underlying the software system is Big Data which the system operates upon to improve or provide different end-user experience with the application. In comparison to traditional software development where the development processes are usually well-established, the development of Big Data applications is - to our knowledge - not explored to any significant degree. With Big Data, characterised by the well-known V attributes, questions arise as to how to elicit, specify, analyse, and document system requirements. While requirements engineering (RE) has long been recognised as critical for downstream development of computer systems, the field is currently passive about how to deal with characteristics of data in the RE process in the development of Big Data software applications. This problem is compounded by the fact that the RE field had no domain model (until recently) for Big Data systems depicting the various artefacts, activities, and relationships amongst them that, in turn, can be used to support RE specifications, product design, project decisions, and maintenance. In this thesis research, we investigated empirically a number of issues in RE involving Big Data applications, leading to the following research contributions: (i) knowledge concerning (a) the state of RE research involving Big Data applications, and (b) RE practices on real-world Big Data applications projects; (ii) a set of RE challenges in creating Big Data applications; (iii) a meta-model depicting the various RE artefacts and their inter-relationships in the context of Big Data software development projects; (iv) a goal-oriented approach (composed of a systematic process, requirements logging templates, checklists, and a requirements language) for modelling quality requirements for Big Data applications; and (v) a prototype tool that implements the proposed Big Data goal-oriented requirements language. These results lay a foundation in RE research involving Big Data applications development with anticipated impact in real-world projects and in RE research

Software Engineering for Big Data Systems

Software engineering is the application of a systematic approach to designing, operating and maintaining software systems and the study of all the activities involved in achieving the same. The software engineering discipline and research into software systems flourished with the advent of computers and the technological revolution ushered in by the World Wide Web and the Internet. Software systems have grown dramatically to the point of becoming ubiquitous. They have a significant impact on the global economy and on how we interact and communicate with each other and with computers using software in our daily lives. However, there have been major changes in the type of software systems developed over the years. In the past decade owing to breakthrough advancements in cloud and mobile computing technologies, unprecedented volumes of hitherto inaccessible data, referred to as big data, has become available to technology companies and business organizations farsighted and discerning enough to use it to create new products, and services generating astounding profits. The advent of big data and software systems utilizing big data has presented a new sphere of growth for the software engineering discipline. Researchers, entrepreneurs and major corporations are all looking into big data systems to extract the maximum value from data available to them. Software engineering for big data systems is an emergent field that is starting to witness a lot of important research activity. This thesis investigates the application of software engineering knowledge areas and standard practices, established over the years by the software engineering research community, into developing big data systems by: - surveying the existing software engineering literature on applying software engineering principles into developing and supporting big data systems; - identifying the fields of application for big data systems; - investigating the software engineering knowledge areas that have seen research related to big data systems; - revealing the gaps in the knowledge areas that require more focus for big data systems development; and - determining the open research challenges in each software engineering knowledge area that need to be met. The analysis and results obtained from this thesis reveal that recent advances made in distributed computing, non-relational databases, and machine learning applications have lured the software engineering research and business communities primarily into focusing on system design and architecture of big data systems. Despite the instrumental role played by big data systems in the success of several businesses organizations and technology companies by transforming them into market leaders, developing and maintaining stable, robust, and scalable big data systems is still a distant milestone. This can be attributed to the paucity of much deserved research attention into more fundamental and equally important software engineering activities like requirements engineering, testing, and creating good quality assurance practices for big data systems

Embrace the Challenges: Software Engineering in a Big Data World

Abstract-The design and development of data-intensive software systems-systems that generate, collect, store, process, analyze, query, and visualize large sets of data-is fraught with significant challenges both technical and social. Project EPIC has been designing and developing data-intensive systems in support of crisis informatics research since Fall 2009. Our experience working on Project EPIC has provided insight into these challenges. In this paper, we share our experience working in this design space and describe the choices we made in tackling these challenges and their attendant trade-offs. We highlight the lack of developer support tools for data-intensive systems, the importance of multidisciplinary teams, the use of highly-iterative life cycles, the need for deep understanding of the frameworks and technologies used in data intensive systems, how simple operations transform into significant challenges at scale, and the paramount significance of data modeling in producing systems that are scalable, robust, and efficient

Determinación de la calidad de producto, persona y proceso en entornos de desarrollo con GitHub: un estudio sistemático de la literatura

La comparación entre costo, tiempo y alcance de un proyecto de desarrollo de software y el resultado final, ha brindado una manera simple, aunque sesgada de medir la eficiencia y eficacia de un equipo de proyecto. Consideramos necesario comprender que algunas características de cada desarrollador más los resultados de un proceso creativo del equipo de trabajo dentro del ciclo de vida del Producto, incide directamente en la calidad de éste. Proponemos entonces considerar la calidad de las 3Ps: Producto, Persona y Proceso como una manera holística de determinar la calidad total en la Ingeniería de Software. Pretendemos encontrar métricas objetivas para las 3Ps cuya fuente de datos sea GitHub. De un estudio sistemático de la literatura realizado, los resultados muestran que los estudios previos se centraron en la definición de métricas calculables desde metadata de GitHub para proyectos de código abierto y sin foco en las 3Ps.Sociedad Argentina de Informática e Investigación Operativ