Hadoop Performance Analysis Model with Deep Data Locality

Background: Hadoop has become the base framework on the big data system via the simple concept that moving computation is cheaper than moving data. Hadoop increases a data locality in the Hadoop Distributed File System (HDFS) to improve the performance of the system. The network traffic among nodes in the big data system is reduced by increasing a data-local on the machine. Traditional research increased the data-local on one of the MapReduce stages to increase the Hadoop performance. However, there is currently no mathematical performance model for the data locality on the Hadoop. Methods: This study made the Hadoop performance analysis model with data locality for analyzing the entire process of MapReduce. In this paper, the data locality concept on the map stage and shuffle stage was explained. Also, this research showed how to apply the Hadoop performance analysis model to increase the performance of the Hadoop system by making the deep data locality. Results: This research proved the deep data locality for increasing performance of Hadoop via three tests, such as, a simulation base test, a cloud test and a physical test. According to the test, the authors improved the Hadoop system by over 34% by using the deep data locality. Conclusions: The deep data locality improved the Hadoop performance by reducing the data movement in HDFS

Enabling portable I/O analysis of commercially sensitive HPC applications through workload replication

Benchmarking and analyzing I/O performance across high performance computing (HPC) platforms is necessary to identify performance bottlenecks and guide effective use of new and existing storage systems. Doing this with large production applications, which can often be commercially sensitive and lack portability, is not a straightforward task and the availability of a representative proxy for I/O workloads can help to provide a solution. We use Darshan I/O characterization and the MACSio proxy application to replicate five production workloads, showing how these can be used effectively to investigate I/O performance when migrating between HPC systems ranging from small local clusters to leadership scale machines. Preliminary results indicate that it is possible to generate datasets that match the target application with a good degree of accuracy. This enables a predictive performance analysis study of a representative workload to be conducted on five different systems. The results of this analysis are used to identify how workloads exhibit different I/O footprints on a file system and what effect file system configuration can have on performance

Quality-of-Service differentiation in an integrated services GSM/GPRS network

We develop and analyse a generic model for performance evaluation, parameter optimisation and dimensioning in a \textsc{gsm}/\textsc{gprs} network. The model enables analytical evaluation for a scenario of integrated speech, video and data services, potentially offered in distinct priority classes. While a speech call is assigned a single traffic channel for its entire duration, both video and data calls can handle varying channel assignments. The principal distinction between these elastic call types, is that in case of video calls, a more generous channel assignment implies a better throughput and thus call quality, while for data calls the increased throughput implies a reduced sojourn time. Although a broader variety of models can be designed and analysed within the generic framework, the analytical and numerical results are presented for the \textsc{svd} model integrating speech, video and data calls, and for the \textsc{shl} model, integrating speech and two priority classes of data calls. In both models, an access queue is maintained for data calls which cannot be served immediately upon arrival. Markov chain analysis is applied to derive basic performance measures such as the expected channel utilization, service-specific blocking probabilities (\textsc{gos}), expected video \textsc{qos} (throughput) and expected (priority class-specific) data \textsc{qos} (sojourn times). Furthermore, closed-form expressions are derived for the expected video and data \textsc{qos}, conditional on the call duration or file size, respectively, and on the system state at arrival. As a potential application, these measures can be fed back to the caller as an indication of the expected \textsc{qos}. The included numerical study demonstrates the merit of the presented generic model and performance analysis, and provides \textsc{gsm}/\textsc{gprs} network operators with valuable insight in the \textsc{gos} and \textsc{qos} tradeoffs involved in balancing the various controllable system parameters

The new services in nagios network bandwidth utility email notification and sms alert in improving the network performance

A new feature of services in Nagios has been added to the existing system which has no such services. The bandwidth monitoring and notification system are configured for alerting the network administrators when the bandwidth of the network in an organization hits a certain threshold settings. The system sent an email alert and sms notification to the network administrator for taking further action in order to maintain the Quality of Service (QoS) in the network. All the logs file of the Nagios actions is saved in the Nagios File Logs. The analysis was conducted from the case study and problem statements. Network Development Life Cycle (NDLC) was chosen as a methodology for implementing this system in the network. Nagios is installed inside Ubuntu 10 Operating System along with Multi-Router Traffic Grapher (MRTG) and Mail Postfix. MRTG and Mail Postfix were configured to be integrated with the Nagios System. On the client side, NSClient++ has been installed, for monitoring the bandwidth and performance of windows based on operating system. The Nagios services have been improved with the implementation of sms and emails notifications since the existing services have no such utilities. With the implementation of these services to Nagios, the performance could be even better for the futur

The new services in nagios: network bandwidth utility, email notification and sms alert in improving the network performance

a new feature of services in Nagios has been added to the existing system which has no such services. The bandwidth monitoring and notification system are configured for alerting the network administrators when the bandwidth of the network in an organization hits a certain threshold settings. The system sent an email alert and sms notification to the network administrator for taking further action in order to maintain the Quality of Service (QoS) in the network. All the logs file of the Nagios actions is saved in the Nagios File Logs. The analysis was conducted from the case study and problem statements. Network Development Life Cycle (NDLC) was chosen as a methodology for implementing this system in the network. Nagios is installed inside Ubuntu 10 Operating System along with Multi-Router Traffic Grapher (MRTG) and Mail Postfix. MRTG and Mail Postfix were configured to be integrated with the Nagios System. On the client side, NSClient++ has been installed, for monitoring the bandwidth and performance of windows based on operating system. The Nagios services have been improved with the implementation of sms and emails notifications since the existing services have no such utilities. With the implementation of these services to Nagios, the performance could be even better for the future

Towars web site user's profile: log file analysis.

The Internet is a remote, innovative, extremely dynamic and widely accessible communication medium. As in all other human communication formats, we observe the development and adoption of its own language, inherent to its multimedia aspects. The Embrapa Satellite Monitoring is using the Internet as a dissemination medium of its research results and interaction with clients, partners and web site users for more than one decade. In order to evaluate the web site usage and performance of the e-communication system the Webalizer software has been used to track and to calculate statistics based on web server log file analysis. The objective of the study is to analyze the data and evaluate the indicators related to requests origin (search string, country, time), actions performed by users (entry pages, agents) and system performance (error messages). It will help to remodel the web site design to improve the interaction dynamics and also develop a customized log file analyser. This tool would retrieve coherent and real information

Raptorqp2P: Maximize The Performance Of P2P File Distribution With Raptorq Coding

BitTorrent is the most popular Peer-to-Peer (P2P) file sharing system widely used for distributing large files over the Internet. It has attracted extensive attentions from both network operators and researchers for investigating its deployment and performance. For example, recent studies have shown that under steady state, its rarest first scheme with the tit-for-tat mechanism can work very effectively and make BitTorrent near optimal for the generic file downloading process. However, in practice, the highly dynamic network environment, especially the notorious user churns prevalently existing in most peer-to-peer systems, can severely degrade the downloading performance. In this thesis, we first study on the limitations of BitTorrent under dynamic network environments, focusing on two scenarios where with our preliminary modeling and analysis, we clearly identify how network dynamics and peer churns can significantly degrade the performance. With these findings, we further propose a novel protocol named RaptorQP2P, which is based on RaptorQ coding, to overcome the limitations of current BitTorrent design and maximize the performance of P2P file distribution. The new protocol features two levels of RaptorQ encoding. At the top layer, the entire file is RaptorQ encoded to yield a collection of source blocks and repair blocks, and then each source and repair block is RaptorQ encoded independently to yield a collection of source symbols and repair symbols for the block. The symbols are independently transferred among the peers and when a sufficient number of distinct symbols for a particular block have been received, whether source or repair, the block can be reconstructed. The file can be reconstructed using a sufficient arbitrary number of distinct blocks. Our results show that RaptorQP2P can well handle the network dynamics as well as peer churns and significantly shorten the downloading completion time by up to 41.4% with excellent scalability on both file size and user population

ENHANCEMENT OF INFORMATION MANAGEMENT CAPABILITIES IN MDO FRAMEWORK

Multidisciplinary Design Optimization (MDO) frameworks have been developed to facilitate the integration of disciplinary analysis codes and optimization techniques. Recent advances in MDO frameworks have addressed issues related to data exchange, distributed computing, process integration and trade study. However, managing, storing and sharing MDO problem information have not yet been fully addressed. In this research a software configuration is proposed. The configuration is built upon a structured repository, common file system and software applications. The configuration is integrated into a commercially available MDO framework to manage, store and share MDO problem information. A common file system proposed in this research provides a structure to store MDO components and enable sharing of components over the network. The ModelCenter framework is selected for the integration of the repository based on the evaluation of the MDO frameworks against a set of extended information management requirements. The repository is a relational database which provides an information model to store information related to MDO problems. A Java interface is utilized to provide access to the structured repository and the common file system in the ModelCenter framework. Java applications are developed to demonstrate the benefits offered by the proposed repository and the common file system. The proposed features and the Java applications are tested for the functionality and performance utilizing IEEE software testing standards