Performance Evaluation of Distributed Computing Environments with Hadoop and Spark Frameworks

Recently, due to rapid development of information and communication technologies, the data are created and consumed in the avalanche way. Distributed computing create preconditions for analyzing and processing such Big Data by distributing the computations among a number of compute nodes. In this work, performance of distributed computing environments on the basis of Hadoop and Spark frameworks is estimated for real and virtual versions of clusters. As a test task, we chose the classic use case of word counting in texts of various sizes. It was found that the running times grow very fast with the dataset size and faster than a power function even. As to the real and virtual versions of cluster implementations, this tendency is the similar for both Hadoop and Spark frameworks. Moreover, speedup values decrease significantly with the growth of dataset size, especially for virtual version of cluster configuration. The problem of growing data generated by IoT and multimodal (visual, sound, tactile, neuro and brain-computing, muscle and eye tracking, etc.) interaction channels is presented. In the context of this problem, the current observations as to the running times and speedup on Hadoop and Spark frameworks in real and virtual cluster configurations can be very useful for the proper scaling-up and efficient job management, especially for machine learning and Deep Learning applications, where Big Data are widely present.Comment: 5 pages, 1 table, 2017 IEEE International Young Scientists Forum on Applied Physics and Engineering (YSF-2017) (Lviv, Ukraine

Development of a Virtual Business Community in an Agricultural Cluster

Virtual business communities (VBC) are social aggregations, based on Information and Communication Technology (ICT) platforms, created to sustain activities of agents of a specific industry or sector that are economically and socially linked. To study the main challenges in creating a VBC to support a not-well-structured agricultural cluster, we developed and tested an ICT platform and analyzed its implementation process. The design research method grounded this process. The main result of this work in-progress is the analysis of challenges faced on the community level and on the economic agents’ level regarding the development of an ICT platform in order to build a VBC

Cluster Juggler - PC cluster virtual reality

Interactive computer graphics are being used as a routine tool in many disciplines, and there is a growing demand to move these interactive tools into immersive environments as technology advances. Immersive environments (or virtual reality) require highly specialized equipment and skilled technical people to develop the applications and operate the systems. These requirements prevent the widespread acceptance of visual reality in research and industrial communities. Our work aims to bring virtual reality to a level that allows these groups with basic technical computer skills and limited resources to use this technology. To reach this goal, this work focuses on taking advantage of recent advances in commodity hardware and low-end graphics systems to create a development framework for virtual reality applications. To achieve this, we have designed a software system that enables a cluster of PCs or low-end workstations to replace a large shared-memory computer as the driving system for complex virtual reality environments. This software system, Cluster Juggler, is implemented as an extension to the virtual reality software VR Juggler. We have tested our software on a cluster VR system to ensure the performance is adequate for running VR applications. Using actual VR applications, we compared the performance of our cluster system with the performance of a VR system driven by a specialized shared-memory computer and found comparable results between the two. With Cluster Juggler we provide the ability for developers with basic technical experience to develop and run virtual reality applications on commodity hardware. In doing so, we aim to make virtual reality more accessible and affordable to all areas of research.

Communities of practice in academia

Up to now, the relationships among the fundamental notions of communities of practice (CoPs), i.e. knowledge, participation, identity, and artefact development have been based mainly on results from qualitative studies; they are not yet sufficiently based on quantitative evidence. Starting from a literature review, we formulate a quantitative, causal model of CoPs that describes these variables in the context of academic communities, and aim to validate this model in two academic CoPs with a total of N = 208 participants. A cluster analysis classifies the participants into clusters that are in line with the core-periphery structure known from previous qualitative studies. A regression analysis provides evidence for the hypothesized model on the basis of quantitative data. Suggested directions for future research are to focus on factors that determine CoP participants’ contributions to artefact development and on approaches to automated monitoring of virtual CoPs

Understanding Next-Generation VR: Classifying Commodity Clusters for Immersive Virtual Reality

Commodity clusters offer the ability to deliver higher performance computer graphics at lower prices than traditional graphics supercomputers. Immersive virtual reality systems demand notoriously high computational requirements to deliver adequate real-time graphics, leading to the emergence of commodity clusters for immersive virtual reality. Such clusters deliver the graphics power needed by leveraging the combined power of several computers to meet the demands of real-time interactive immersive computer graphics.However, the field of commodity cluster-based virtual reality is still in early stages of development and the field is currently adhoc in nature and lacks order. There is no accepted means for comparing approaches and implementers are left with instinctual or trial-and-error means for selecting an approach.This paper provides a classification system that facilitates understanding not only of the nature of different clustering systems but also the interrelations between them. The system is built from a new model for generalized computer graphics applications, which is based on the flow of data through a sequence of operations over the entire context of the application. Prior models and classification systems have been too focused in context and application whereas the system described here provides a unified means for comparison of works within the field

Automating first-principles phase diagram calculations

Devising a computational tool that assesses the thermodynamic stability of materials is among the most important steps required to build a “virtual laboratory,” where materials could be designed from first principles without relying on experimental input. Although the formalism that allows the calculation of solid-state phase diagrams from first principles is well established, its practical implementation remains a tedious process. The development of a fully automated algorithm to perform such calculations serves two purposes. First, it will make this powerful tool available to a large number of researchers. Second, it frees the calculation process from arbitrary parameters, guaranteeing that the results obtained are truly derived from the underlying first-principles calculations. The proposed algorithm formalizes the most difficult step of phase diagram calculations, namely the determination of the “cluster expanison,” which is a compact representation of the configurational dependence of the alloy’s energy. This is traditionally achieved by a fit of the unknown interaction parameters of the cluster expansion to a set of structural energies calculated from first principles. We present a formal statistical basis for the selection of both the interaction parameters to include in the cluster expansion and the structures to use to determine them. The proposed method relies on the concepts of cross-validation and variance minimization. An application to the calculation of the phase diagram of the Si-Ge, CaO-MgO, Ti-Al, and Cu-Au systems is presented

Development Methods and a Scenegraph Animation API for Cluster Driven Immersive Applications

This paper presents a scenegraph animation application programming interface (API), known as the Animation Engine, which was constructed for software developers to easily perform smooth transitions and manipulations to scenegraph nodes. A developer can use one line of code to enter the property, end state and number of frames to describe the animation, then the Animation Engine handles the rest in the background. The goal of the Animation Engine is to provide a simple API that integrates into existing applications with minimal effort. Additionally, techniques to improve virtual reality (VR) application performance on a large computer cluster are presented. These techniques include maintaining high frame rates with 4096 × 4096 pixel textures, eliminating extraneous network traffic and reducing long model loading time. To demonstrate the Animation Engine and the development techniques, an application known as the Virtual Universe was created. The Virtual Universe, designed to run in a six walled CAVE, allows users to freely explore a set of space themed environments. The architecture and development techniques for writing a stable immersive VR application on a large computer cluster, in addition to the creation of the Animation Engine, is presented in this paper

Can knowledge management save regional development?

Australia needs to create innovative regions to sustain economic prosperity and regional development. In order to do this, regions will need to systematically address their knowledge needs and identify tools that are appropriate in maximising their effectiveness. Many initiatives have focused on information and communication technology (ICT) to enable knowledge exchange and stimulate knowledge generation, but active knowledge management (KM) strategies are required if ICTs are to be used effectively. These strategies must respond to the regional economic and social environments which incorporate small and medium enterprises (SMEs). This paper outlines the importance of KM for supporting regional cluster development and the key ways in which communities of practice (CoPs), a KM technique, have been used to add value in similar contexts. How CoPs and their online counterpart, virtual communities of practice (VCoPs), can be used and developed in regional areas of Australia is considered along with a program for further research.<br /