LERC: Coordinated Cache Management for Data-Parallel Systems

Memory caches are being aggressively used in today's data-parallel frameworks such as Spark, Tez and Storm. By caching input and intermediate data in memory, compute tasks can witness speedup by orders of magnitude. To maximize the chance of in-memory data access, existing cache algorithms, be it recency- or frequency-based, settle on cache hit ratio as the optimization objective. However, unlike the conventional belief, we show in this paper that simply pursuing a higher cache hit ratio of individual data blocks does not necessarily translate into faster task completion in data-parallel environments. A data-parallel task typically depends on multiple input data blocks. Unless all of these blocks are cached in memory, no speedup will result. To capture this all-or-nothing property, we propose a more relevant metric, called effective cache hit ratio. Specifically, a cache hit of a data block is said to be effective if it can speed up a compute task. In order to optimize the effective cache hit ratio, we propose the Least Effective Reference Count (LERC) policy that persists the dependent blocks of a compute task as a whole in memory. We have implemented the LERC policy as a memory manager in Spark and evaluated its performance through Amazon EC2 deployment. Evaluation results demonstrate that LERC helps speed up data-parallel jobs by up to 37% compared with the widely employed least-recently-used (LRU) policy

A Framework to Build a Big Data Ecosystem Oriented to the Collaborative Networked Organization

A Collaborative Networked Organization (CNO) is a set of entities that operate in heterogeneous contexts and aim to collaborate to take advantage of a business opportunity or solve a problem. Big data allows CNOs to be more competitive by improving their strategy, management and business processes. To support the development of big data ecosystems in CNOs, several frameworks have been reported in the literature. However, these frameworks limit their application to a specific CNO manifestation and cannot conduct intelligent processing of big data to support decision making at the CNO. This paper makes two main contributions: (1) the proposal of a metaframework to analyze existing and future frameworks for the development of big data ecosystems in CNOs and (2) to show the Collaborative Networked Organizations–big data (CNO-BD) framework, which includes guidelines, tools, techniques, conceptual solutions and good practices for the building of a big data ecosystem in different kinds of Collaborative Networked Organizations, overcoming the weaknesses of previous issues. The CNO-BD framework consists of seven dimensions: levels, approaches, data fusion, interoperability, data sources, big data assurance and programmable modules. The framework was validated through expert assessment and a case study

Scalability of mobile cloud storage

Today, there are high demands on Mobile Cloud Storage (MCS) services that need to manage the increasing number of works with stable performance. This situation brings a challenge for data management systems because when the number of works increased MCS needs to manage the data wisely to avoid latency occur. If latency occurs it will slow down the data performance and it should avoid that problem when using MCS. Moreover, MCS should provide users access to data faster and correctly. Hence, the research focuses on the scalability of mobile cloud data storage management, which is study the scalable on how deep the data folder itself that increase the number of works

Internet of Things: A Review of Surveys Based on Context Aware Intelligent Services

The Internet of Things (IoT) has made it possible for devices around the world to acquire information and store it, in order to be able to use it at a later stage. However, this potential opportunity is often not exploited because of the excessively big interval between the data collection and the capability to process and analyse it. In this paper, we review the current IoT technologies, approaches and models in order to discover what challenges need to be met to make more sense of data. The main goal of this paper is to review the surveys related to IoT in order to provide well integrated and context aware intelligent services for IoT. Moreover, we present a state-of-the-art of IoT from the context aware perspective that allows the integration of IoT and social networks in the emerging Social Internet of Things (SIoT) term.This work has been partially funded by the Spanish Ministry of Economy and Competitiveness (MINECO/FEDER) under the granted Project SEQUOIA-UA (Management requirements and methodology for Big Data analytics) TIN2015-63502-C3-3-R, by the University of Alicante, within the program of support for research, under project GRE14-10, and by the Conselleria de Educación, Investigación, Cultura y Deporte, Comunidad Valenciana, Spain, within the program of support for research, under project GV/2016/087. This work has also been partially funded by projects from the Spanish Ministry of Education and Competitivity TIN2015-65100-R and DIIM2.0 (PROMETEOII/2014/001)

Selling the Fourth Revolution: The Correlation between C-Suite Architecture and a Big Data Mindset as Portrayed in the Letter to the Shareholders

Big data’s diverse applications for the modern data deluge span problems and industries. While offering titular possibilities, is big data an area of serious corporate inquiry or is it a source of hype? This research seeks to add to the growing body of management literature on big data. C-suite architecture additions of a CIO and/or a CTO demonstrate an environment for and increased BDA mindset. A review of the annual letter to the shareholder is a proxy for the external narrative of a big data strategy. To measure correlation between C-suite structure and outward narrative, a regression for eight industries, from 2011 to 2014, measure correlation between external and organizational positioning. Additional regressions examine industry-sensitivity and leader-laggard dynamics. Five of the eight industries demonstrate correlation between architectural repositioning and perceived investor support for big data, indicating that big data intensive C-suite architecture correlates with letter to the shareholder big data emphasis

Monitorització d'incendis amb imatges des de satèl·lit i amb UAV

Desenvolupament d'algorisme de detecció d'incendis amb imatge de satèl·lit i validació amb imatges des d'avió.Wildfires are of major concern now days because, even though they have also a natural role, they have negative impacts in environmental, economic and social terms. The main purposes of this project are the study, combination, comparison of different ways to sense fires. It focus on the images and data -that remotely sensed by instruments- provided and transmitted from satellites and drones. Its uttermost purpose is to design ways to detect, trace, measure the burned areas from UAV and give a comprehensive view of the wildfire while it is taking place. It deepens into the nature of forest fires and relates how factors like wind or terrain affect their propagation and growth. In the case of UAV, based on the information received from the sensors at an altitude between 10km and 20km, the project discusses how to measure the burned areas. Despite there is little data or imagery recorded with specific instruments from this altitude while a forest fire is taking place at this stage, it is enough to draw some lines of action. Meanwhile, satellite images and data of forest fires already occurred are easily available, we had no forest fires images recorded by UAVs or other airplanes at 10 or 20km height. Consequently, many algorithm of detection and monitoring using only images from satellite (both when there is no fire and when there is burning a fire) and images from sensors flying on UAVs.. With all that information regarding the sensors and the patterns of the fire it is possible to discern the movements of the fire. Once the movements and growth are recorded and known, a drone-based easy-to-understand chart of the wildfire is delivered.Los incendios constituyen una gran preocupación hoy en dia, a pesar que a veces juegan un papel natural, también tienen un impacto negativo en términos medioambientales, económicos y sociales. Este proyecto consiste en el estudio , la combinación , la comparación de diferentes maneras para detectar, medir y entender los incendios. Se centra en el estudio de las imágenes y datos captados remotamente por sensores -y camaras- desde satélite y desde aviones no tripulados. Su máximo objetivo es diseñar maneras de detectar , rastrear , medir las áreas quemadas y dar una visión comprehensiva de un incendio forestal mientras este tiene lugar. Se profundiza en la naturaleza del fuego durante los incendios forestales y como factores como el viento o el terreno afecta su propagación y crecimiento. Basándose en información recibida desde sensores a una altitud de entre 10km y 20km, el proyecto discute como medir la áreas quemadas. A pesar que no hay muchos datos o imágenes de incendios grabados desde drones a esta altitud con ciertos instrumentos mientras este ocurre, eso es suficiente para plantear unas lineas de estudio y trabajo. Con las características de ciertos sensores y los patrones descritos por el fuego es posible discernir los movimientos del incendio. Una vez grabados y conocidos tales movimientos debería ser posible mostrar un mapa o diagrama del incendio fácil de entender basado en drones para la gente que esta sobre el terreno.Els focs forestals constitueixen una gran preocupació avui en dia, malgrat que a vegades juguen un paper natural, també tenen un impacte negatiu en termes ambientals, econòmics y socials. Aquest projecte consisteix en l'estudi, la combinació , la comparació de diferents maneres per detectar incendis. Es centra en el estudi de images i dades produides i transmeses per sensors desde satel·lits i per avions no tripulats. El seu màxim objectiu és dissenyar maneres de detectar , rastrejar , mesurar les àrees cremades y donar una visión comprensiva d'in incendi forestal mentre aquest es produeix. S'aprofundeix en la naturalesa del foc durant els incendis forestals i com factors com el vent o el terreny afecten la seva propagació i creixement. En el cas dels UAV o drones, basant-se en la informació rebuda des dels sensors a una altitud d'entre 10km i 20km, el projecte discuteix com mesurar les arees cremades. Malgrat que no existeix moltes dades o imatges sobre incendis a aquestes altituts grabades per drones, es suficient per plantejar unes lines d'estudi, treball i acció. Amb les caracteristiques del sensors i els patrons descrits pel foc seria possible discernir els moviments del incendi. Llavors un cop grabats i coneguts aquest creixement hauria de ser possible mostrar un mapa o diagrames utils i fàcils d'entendre basat en drones per a la gent que esta sobre el terreny

Technological Innovation, Data Analytics, and Environmental Enforcement

Technical innovation is ubiquitous in contemporary society and contributes to its extraordinarily dynamic character. Sometimes these innovations have significant effects on the state of the environment or on human health and they have stimulated efforts to develop second order technologies to ameliorate those effects. The development of the automobile and its impact on life in the United States and throughout the world is an example. The story of modern environmental regulation more generally includes chapters filled with examples of similar efforts to respond to an enormous array of technological advances. This Article uses a different lens to consider the role of technological innovation. In particular, it considers how technological advances have the potential to shape governance efforts in the compliance realm. The Article demonstrates that such technological advances – especially new and improved monitoring capacity, advances in information dissemination through e-reporting and other techniques, and improved capacity to analyze information – have significant potential to transform governance efforts to promote compliance. Such transformation is likely to affect not only the “how” of compliance promotion, but also the “who.” Technological innovation is likely to contribute to new thinking about the roles key actors can and should play in promoting compliance with legal norms. The Article discusses some of the potential benefits of these types of technological innovation in the context of the Environmental Protection Agency (EPA)’s ongoing efforts to improve its compliance efforts by taking advantage of emerging technologies. We also identify some of the pitfalls or challenges that agencies such as EPA need to be aware of in opening this emerging bundle of new tools and making use of them to address real-world environmental needs

Technological Innovation, Data Analytics, and Environmental Enforcement

Technical innovation is ubiquitous in contemporary society and contributes to its extraordinarily dynamic character. Sometimes these innovations have significant effects on the state of the environment or on human health and they have stimulated efforts to develop second order technologies to ameliorate those effects. The development of the automobile and its impact on life in the United States and throughout the world is an example. The story of modern environmental regulation more generally includes chapters filled with examples of similar efforts to respond to an enormous array of technological advances. This Article uses a different lens to consider the role of technological innovation. In particular, it considers how technological advances have the potential to shape governance efforts in the compliance realm. The Article demonstrates that such technological advances – especially new and improved monitoring capacity, advances in information dissemination through e-reporting and other techniques, and improved capacity to analyze information – have significant potential to transform governance efforts to promote compliance. Such transformation is likely to affect not only the “how” of compliance promotion, but also the “who.” Technological innovation is likely to contribute to new thinking about the roles key actors can and should play in promoting compliance with legal norms. The Article discusses some of the potential benefits of these types of technological innovation in the context of the Environmental Protection Agency (EPA)’s ongoing efforts to improve its compliance efforts by taking advantage of emerging technologies. We also identify some of the pitfalls or challenges that agencies such as EPA need to be aware of in opening this emerging bundle of new tools and making use of them to address real-world environmental needs