Redundant Arrays of IDE Drives

The next generation of high-energy physics experiments is expected to gather prodigious amounts of data. New methods must be developed to handle this data and make analysis at universities possible. We examine some techniques that use recent developments in commodity hardware. We test redundant arrays of integrated drive electronics (IDE) disk drives for use in offline high-energy physics data analysis. IDE redundant array of inexpensive disks (RAID) prices now equal the cost per terabyte of million-dollar tape robots! The arrays can be scaled to sizes affordable to institutions without robots and used when fast random access at low cost is important. We also explore three methods of moving data between sites; internet transfers, hot pluggable IDE disks in FireWire cases, and writable digital video disks (DVD-R).Comment: Submitted to IEEE Transactions On Nuclear Science, for the 2001 IEEE Nuclear Science Symposium and Medical Imaging Conference, 8 pages, 1 figure, uses IEEEtran.cls. Revised March 19, 2002 and published August 200

Evaluating atomicity, and integrity of correct memory acquisition methods

AbstractWith increased use of forensic memory analysis, the soundness of memory acquisition becomes more important. We therefore present a black box analysis technique in which memory contents are constantly changed via our payload application with a traceable access pattern. This way, given the correctness of a memory acquisition procedure, we can evaluate its atomicity and one aspect of integrity as defined by Vömel and Freiling (2012). We evaluated our approach on several memory acquisition techniques represented by 12 memory acquisition tools using a Windows 7 64-bit operating system running on a i5-2400 with 2 GiB RAM. We found user-mode memory acquisition software (ProcDump, Windows Task Manager), which suspend the process during memory acquisition, to provide perfect atomicity and integrity for snapshots of process memory. Cold-boot attacks (memimage, msramdump), virtualization (VirtualBox) and emulation (QEMU) all deliver perfect atomicity and integrity of full physical system memory snapshots. Kernel level software acquisition tools (FTK Imager, DumpIt, win64dd, WinPmem) exhibit memory smear from concurrent system activity reducing their atomicity. There integrity is reduced by running within the imaged memory space, hence overwriting part of the memory contents to be acquired. The least amount of atomicity is exhibited by a DMA attack (inception using IEEE 1394). Further, even if DMA is performed completely in hardware, integrity violations with respect to the point in time of the acquisition let this method appear inferior to all other methods. Our evaluation methodology is generalizable to examine further memory acquisition procedures on other operating systems and platforms

A SYSTEM TO IMPLEMENT A LINEAGE AUTO CAPTURE PLUGIN FOR HIVE

The present disclosure provides a system and a method for lineage auto capture plugin for hive. The system receives a query using a user interface and hive clients, process the query using a driver, execute the query via an execution engine. Further, the system implements the functionality of hive lineage connector to perform lineage auto capture for hive. Thereby, managing data efficiently and addressing issues related to data efficiently

Virtualisation and Thin Client : A Survey of Virtual Desktop environments

This survey examines some of the leading commercial Virtualisation and Thin Client technologies. Reference is made to a number of academic research sources and to prominent industry specialists and commentators. A basic virtualisation Laboratory model is assembled to demonstrate fundamental Thin Client operations and to clarify potential problem areas

Does Big Data Require Complex Systems? A Performance Comparison Between Spark and Unicage Shell Scripts

The paradigm of big data is characterized by the need to collect and process data sets of great volume, arriving at the systems with great velocity, in a variety of formats. Spark is a widely used big data processing system that can be integrated with Hadoop to provide powerful abstractions to developers, such as distributed storage through HDFS and resource management through YARN. When all the required configurations are made, Spark can also provide quality attributes, such as scalability, fault tolerance, and security. However, all of these benefits come at the cost of complexity, with high memory requirements, and additional latency in processing. An alternative approach is to use a lean software stack, like Unicage, that delegates most control back to the developer. In this work we evaluated the performance of big data processing with Spark versus Unicage, in a cluster environment hosted in the IBM Cloud. Two sets of experiments were performed: batch processing of unstructured data sets, and query processing of structured data sets. The input data sets were of significant size, ranging from 64 GB to 8192 GB in volume. The results show that the performance of Unicage scripts is superior to Spark for search workloads like grep and select, but that the abstractions of distributed storage and resource management from the Hadoop stack enable Spark to execute workloads with inter-record dependencies, such as sort and join, with correct outputs.Comment: 10 pages, 14 figure

In-network machine learning using programmable network devices: a survey

Machine learning is widely used to solve networking challenges, ranging from traffic classification and anomaly detection to network configuration. However, machine learning also requires significant processing and often increases the load on both networks and servers. The introduction of in-network computing, enabled by programmable network devices, has allowed to run applications within the network, providing higher throughput and lower latency. Soon after, in-network machine learning solutions started to emerge, enabling machine learning functionality within the network itself. This survey introduces the concept of in-network machine learning and provides a comprehensive taxonomy. The survey provides an introduction to the technology and explains the different types of machine learning solutions built upon programmable network devices. It explores the different types of machine learning models implemented within the network, and discusses related challenges and solutions. In-network machine learning can significantly benefit cloud computing and next-generation networks, and this survey concludes with a discussion of future trends

A cross-stack, network-centric architectural design for next-generation datacenters

This thesis proposes a full-stack, cross-layer datacenter architecture based on in-network computing and near-memory processing paradigms. The proposed datacenter architecture is built atop two principles: (1) utilizing commodity, off-the-shelf hardware (i.e., processor, DRAM, and network devices) with minimal changes to their architecture, and (2) providing a standard interface to the programmers for using the novel hardware. More specifically, the proposed datacenter architecture enables a smart network adapter to collectively compress/decompress data exchange between distributed DNN training nodes and assist the operating system in performing aggressive processor power management. It also deploys specialized memory modules in the servers, capable of performing general-purpose computation and network connectivity. This thesis unlocks the potentials of hardware and operating system co-design in architecting application-transparent, near-data processing hardware for improving datacenter's performance, energy efficiency, and scalability. We evaluate the proposed datacenter architecture using a combination of full-system simulation, FPGA prototyping, and real-system experiments

A scalable silicon photonic chip-scale optical switch for high performance computing systems

This paper discusses the architecture and provides performance studies of a silicon photonic chip-scale optical switch for scalable interconnect network in high performance computing systems. The proposed switch exploits optical wavelength parallelism and wavelength routing characteristics of an Arrayed Waveguide Grating Router (AWGR) to allow contention resolution in the wavelength domain. Simulation results from a cycle-accurate network simulator indicate that, even with only two transmitter/receiver pairs per node, the switch exhibits lower end-to-end latency and higher throughput at high (> 90%) input loads compared with electronic switches. On the device integration level, we propose to integrate all the components (ring modulators, photodetectors and AWGR) on a CMOS-compatible silicon photonic platform to ensure a compact, energy efficient and cost-effective device. We successfully demonstrate proof-of-concept routing functions on an 8 x 8 prototype fabricated using foundry services provided by OpSIS-IME. (C) 2013 Optical Society of Americ