Data locality in Hadoop

Current market tendencies show the need of storing and processing rapidly growing amounts of data. Therefore, it implies the demand for distributed storage and data processing systems. The Apache Hadoop is an open-source framework for managing such computing clusters in an effective, fault-tolerant way. Dealing with large volumes of data, Hadoop, and its storage system HDFS (Hadoop Distributed File System), face challenges to keep the high efficiency with computing in a reasonable time. The typical Hadoop implementation transfers computation to the data, rather than shipping data across the cluster. Otherwise, moving the big quantities of data through the network could significantly delay data processing tasks. However, while a task is already running, Hadoop favours local data access and chooses blocks from the nearest nodes. Next, the necessary blocks are moved just when they are needed in the given ask. For supporting the Hadoop’s data locality preferences, in this thesis, we propose adding an innovative functionality to its distributed file system (HDFS), that enables moving data blocks on request. In-advance shipping of data makes it possible to forcedly redistribute data between nodes in order to easily adapt it to the given processing tasks. New functionality enables the instructed movement of data blocks within the cluster. Data can be shifted either by user running the proper HDFS shell command or programmatically by other module like an appropriate scheduler. In order to develop such functionality, the detailed analysis of Apache Hadoop source code and its components (specifically HDFS) was conducted. Research resulted in a deep understanding of internal architecture, what made it possible to compare the possible approaches to achieve the desired solution, and develop the chosen one

A practical example of a software factory: building a custom application for analysing EU Cyber Physical System (CPS) projects using Open Source software components

This paper is a retrospective analysis describing the development of a custom tool to organise data snippets derived from a substantial body of information, and a summary of the insights that this means of analysis provided in a very short time scale. The creation of data driven visualisations are of particular interest as they uncovered more cross-domain aspects of Cyber-Physical Systems projects than expert opinion had anticipated. These findings will be discussed fully in a second paper. The focus here is the development of the "Vulture" data scavenging tool using Open Source software as system components to create a custom application to serve the data collection and analysis requirements of a REA (Rapid Evidence Assessment) work-package within an EU funded project, Road2CPS

Lessons Learned from Applying Social Network Analysis on an Industrial Free/Libre/Open Source Software Ecosystem

Many software projects are no longer done in-house by a single organization. Instead, we are in a new age where software is developed by a networked community of individuals and organizations, which base their relations to each other on mutual interest. Paradoxically, recent research suggests that software development can actually be jointly-developed by rival firms. For instance, it is known that the mobile-device makers Apple and Samsung kept collaborating in open source projects while running expensive patent wars in the court. Taking a case study approach, we explore how rival firms collaborate in the open source arena by employing a multi-method approach that combines qualitative analysis of archival data (QA) with mining software repositories (MSR) and Social Network Analysis (SNA). While exploring collaborative processes within the OpenStack ecosystem, our research contributes to Software Engineering research by exploring the role of groups, sub-communities and business models within a high-networked open source ecosystem. Surprising results point out that competition for the same revenue model (i.e., operating conflicting business models) does not necessary affect collaboration within the ecosystem. Moreover, while detecting the different sub-communities of the OpenStack community, we found out that the expected social tendency of developers to work with developers from same firm (i.e., homophily) did not hold within the OpenStack ecosystem. Furthermore, while addressing a novel, complex and unexplored open source case, this research also contributes to the management literature in coopetition strategy and high-tech entrepreneurship with a rich description on how heterogeneous actors within a high-networked ecosystem (involving individuals, startups, established firms and public organizations) joint-develop a complex infrastructure for big-data in the open source arena.Comment: As accepted by the Journal of Internet Services and Applications (JISA

Fotovoltaic excess management and visualization system

L'objectiu principal del projecte és el desenvolupament d'un prototip funcional per a un sistema de gestió i visualització d'excedents fotovoltaics basat en la integració de tecnologies ja existents. En termes més concrets, implica la implementació d'una solució software capaç de gestionar els excedents d'energia d'una casa intel·ligent o instal·lació similar (seguiment de l'ús de l'energia importada/exportada, decidir quan i com utilitzar l'excés d'energia, etc.), així com visualitzar-lo (consum de diferents dispositius, càlcul de potènciacostos/beneficis, recursos hardware, etc.)The project's main objective is the development of a working prototype for a photovoltaic excess management and visualization system based on the integration of already existing technologies. In more concrete terms, this means the implementation of a software-based solution capable of managing excess power from a smart home or similar installation (tracking use of imported/exported power, deciding when and how to use excess power, etc.), as well as visualizing it (consumption of different devices, computation of power costs/benefits, hardware resources, etc.

The Relationship Between Interaction Patterns on an Online Idea Generation Community and the Implementation of Ideas

The purpose of the study was to investigate a socially-networked online idea-generation community. Specifically, the study examined the specific interaction patterns on an online social network and the emergence of ideas. Using social-network analysis, the interaction among the network participants was studied. This analysis included examining the relationships among the network participants in the generation and implementation of ideas. Comparisons were made between networks with ideas that were implemented and those that were not. The findings revealed that activities on the community can be expressed from a network perspective and that insights were found about participant relationships and positions in the networks. Differences among participants with respect to idea implementability were found between the networks. Also, the idea networks were consistent with the behavior of a complex adaptive entity. Based on the findings, practical implications for action for professional and academic disciplines were presented

Delivering Computer-Based Training Within State Prisons

Internet-connected devices are ubiquitous, and our built environment allows us to tap into formerly impossible solutions. As our world increasingly depends on technology to operate, one demographic cannot gain exposure to internet-connected devices or web-based educational programs: incarcerated individuals. The Department of Justice reports that 5 out of 6 State Prisoners are arrested and returned to prison within nine years of their initial release. Research shows that education is a pathway to reducing the U.S. prison population. Individuals who participated in any educational program are 43 percent less likely to return to prison.1 Prisons in the United States often partner with local organizations to provide job training and certification to incarcerated individuals, but few offer technical-vocational skills. Many programs rely exclusively on a partnership with a local college or lack the knowledge base to teach computer networking. By utilizing Docker containers to provide secure and isolated computerbased training, currently incarcerated individuals can increase their education beyond teacher-led classroom instruction. Providing computer access in prison presents significant security and accessibility challenges. The NIST Risk Management Framework implements proven cyber security access frameworks that help prevent unauthorized use. This study shows that combining robust cyber security frameworks with isolated Docker containers running the ToyNet learning suite helps break the cycle of recidivism among individuals incarcerated across the United States

Open source software contributors\u27 motivations in a community of practice

The success of open source software is gaining more attention from software users as well as educators. A variety of open source Software exists for different operating systems (Windows, Macintosh, and Linux) for users in many languages contributed and maintained primarily by volunteers. To learn more about what drives them to devote their time and expertise to creating, debugging, and supporting these widely-used applications, an online survey with Likert-scaled items measuring different types of motivations was distributed to contributors to Mozilla, Moodle, OpenOffice, Koha, and Limesurvey. The survey included comments that were used to check the validity of the Likert-scaled items and open-ended questions that allowed respondents to express their reasons for participating in these open source communities. The Likert-scaled items showed that the open source contributors (n=110, 38 paid and 72 volunteers) are motivated primarily by intrinsic desire: altruism, creation, and learning. Receiving payment for their work did not significantly impact reasons for contributing to OSS projects. The comments and open-ended questions validated the findings and indicated that building a Utopian community--the desire to help for the greater good worldwide--is one of the most important motivators. Also, the freedom to create free software and share a pool of knowledge with those from inside and outside the community is a main reason why contributors join and remain members of open source communities. The conclusion suggests using the community of open source software as an example of collaboration not only in the online learning but also for participation in classrooms

Community-based co-design of a crowdsourcing task management application for safeguarding indigenous knowledge

Teaching indigenous knowledge (IK) to African youth has become more complicated due to a variety of reasons such as urban migration, loss of interest in it, the dominance of scientific knowledge and the technological revolution. Therefore, there is a considerable movement towards using technologies to safeguard IK before it becomes obsolete. It is noteworthy that research conducted and software development perspectives being used are mainly based on Western worldviews that are inappropriate for African socio-cultural contexts. IK holders are often not in charge of the digitisation process and merely treated as subjects. In this study, we explored a suitable development approach of a crowdsourcing task management application (TMA) as an auxiliary tool for safeguarding IK. Moreover, the study sought to provide an opportunity for the indigenous communities to make requests of three-dimensional (3D) models of their traditional objects independently. The delivered traditional 3D models are imported into the communities' IK visualisation tools used by the IK holders to teach the youth about their cultural heritage. The main objective of this study was to ascertain how the indigenous rural communities could appropriate a foreign technological concept such as crowdsourcing. This brought about our first research theme: investigating the necessary conditions to establish and maintain beneficial embedded community engagement. The second theme was to determine the suitable methods for technology co-design. Thirdly, to discover what does the communities' appropriated crowdsourcing concept entail. We applied a consolidated research method based on Community-based CoDesign (CBCD) extended with Afrocentric research insights and operationalised with Action Research cycle principles of planning, action and reflection. CBCD was conducted in three cycles with Otjiherero speaking indigenous rural communities from Namibia. Reflections from the first cycle revealed that the rural communities would require unique features in their crowdsourcing application. During the second cycle of co-designing with the ovaHimba community, we learnt that CBCD is matured through mutual trust, reciprocity and skills transfer and deconstructing mainstream technologies to spark co-design ideas. Lastly, in our third cycle of CBCD, we showcased that communities of similar cultures and knowledge construction had common ideas of co-designing the TMA. We also simulated that the construction of traditional 3D models requires indigenous communities to provide insight details of the traditional object to minimise unsatisfactory deliverables. The findings of this study are contributing in two areas (1) research approach and (2) appropriation of technology. We provide a synthesis of Oundu moral values and Afrocentricity as a foundation for conducting Afrocentric research to establish and maintain humanness before CBCD can take place. With those taken as inherent moral values, Afrocentricity should then solely be focused on knowledge construction within an African epistemology. For the appropriation of technology, we share codesign techniques on how the indigenous rural communities appropriated the mainstream crowdsourcing concept through local meaning-making. CBCD researchers should incorporate Afrocentricity for mutual learning, knowledge construction, and sharing for the benefit of all

Universal Plug-n-Play Sensor Integration for Advanced Navigation

This research investigates the potential for Plug-n-Play sensor integration for navigation and other applications. Specifically, the requirements of such a system are outlined and attempts are made to achieve them using two separate systems: one using Robot Operating System (ROS) as middleware and the other using more traditional software design patterns. The end result is not so much a deliverable in terms of software, but more of a feasibility analysis comparing the two approaches