Big Data and the Internet of Things

Advances in sensing and computing capabilities are making it possible to embed increasing computing power in small devices. This has enabled the sensing devices not just to passively capture data at very high resolution but also to take sophisticated actions in response. Combined with advances in communication, this is resulting in an ecosystem of highly interconnected devices referred to as the Internet of Things - IoT. In conjunction, the advances in machine learning have allowed building models on this ever increasing amounts of data. Consequently, devices all the way from heavy assets such as aircraft engines to wearables such as health monitors can all now not only generate massive amounts of data but can draw back on aggregate analytics to "improve" their performance over time. Big data analytics has been identified as a key enabler for the IoT. In this chapter, we discuss various avenues of the IoT where big data analytics either is already making a significant impact or is on the cusp of doing so. We also discuss social implications and areas of concern.Comment: 33 pages. draft of upcoming book chapter in Japkowicz and Stefanowski (eds.) Big Data Analysis: New algorithms for a new society, Springer Series on Studies in Big Data, to appea

A System-of-Systems Approach to Enterprise Analytics Design: Acquisition Support in the Age of Machine Learning and Artificial Intelligence

Excerpt from the Proceedings of the Nineteenth Annual Acquisition Research SymposiumSystem-of-Systems (SoS) capability emerges from the collaboration of multiple systems, which are acquired from independent organizations. Even though the systems contribute to and benefit from the larger SoS, the data analytics and decision-making about the independent system is rarely shared across the SoS stakeholders. The objective of the research presented in this paper is to identify how the sharing of datasets and the corresponding analytics among SoS stakeholders can lead to an improved SoS capability. Our objective is to characterize how appropriate use of data sets may lead to deployment of different predictive (predicting an outcome from data) and prescriptive (determining a preferred strategy) analytics and lead to better decision outcomes at the SoS level. We build and demonstrate a framework for this objective, based on extensive literature review, which utilizes appropriate predictive and prescriptive methodologies for SoS analysis. Additionally, we propose to utilize machine learning techniques to predict the achievable SoS capability and identify sources of uncertainty derived by sharing partial datasets. A case study demonstrates the use of the framework and prospects for future improvements.Approved for public release; distribution is unlimited

A System-of-Systems Approach to Enterprise Analytics Design: Acquisition Support in the Age of Machine Learning and Artificial Intelligence

Excerpt from the Proceedings of the Nineteenth Annual Acquisition Research SymposiumSystem-of-Systems (SoS) capability emerges from the collaboration of multiple systems, which are acquired from independent organizations. Even though the systems contribute to and benefit from the larger SoS, the data analytics and decision-making about the independent system is rarely shared across the SoS stakeholders. The objective of the research presented in this paper is to identify how the sharing of datasets and the corresponding analytics among SoS stakeholders can lead to an improved SoS capability. Our objective is to characterize how appropriate use of data sets may lead to deployment of different predictive (predicting an outcome from data) and prescriptive (determining a preferred strategy) analytics and lead to better decision outcomes at the SoS level. We build and demonstrate a framework for this objective, based on extensive literature review, which utilizes appropriate predictive and prescriptive methodologies for SoS analysis. Additionally, we propose to utilize machine learning techniques to predict the achievable SoS capability and identify sources of uncertainty derived by sharing partial datasets. A case study demonstrates the use of the framework and prospects for future improvements.Approved for public release; distribution is unlimited

Machine Learning and Integrative Analysis of Biomedical Big Data.

Recent developments in high-throughput technologies have accelerated the accumulation of massive amounts of omics data from multiple sources: genome, epigenome, transcriptome, proteome, metabolome, etc. Traditionally, data from each source (e.g., genome) is analyzed in isolation using statistical and machine learning (ML) methods. Integrative analysis of multi-omics and clinical data is key to new biomedical discoveries and advancements in precision medicine. However, data integration poses new computational challenges as well as exacerbates the ones associated with single-omics studies. Specialized computational approaches are required to effectively and efficiently perform integrative analysis of biomedical data acquired from diverse modalities. In this review, we discuss state-of-the-art ML-based approaches for tackling five specific computational challenges associated with integrative analysis: curse of dimensionality, data heterogeneity, missing data, class imbalance and scalability issues

Semantics-Empowered Big Data Processing with Applications

We discuss the nature of Big Data and address the role of semantics in analyzing and processing Big Data that arises in the context of Physical-Cyber-Social Systems. We organize our research around the Five Vs of Big Data, where four of the Vs are harnessed to produce the fifth V - value. To handle the challenge of Volume, we advocate semantic perception that can convert low-level observational data to higher-level abstractions more suitable for decision-making. To handle the challenge of Variety, we resort to the use of semantic models and annotations of data so that much of the intelligent processing can be done at a level independent of heterogeneity of data formats and media. To handle the challenge of Velocity, we seek to use continuous semantics capability to dynamically create event or situation specific models and recognize relevant new concepts, entities and facts. To handle Veracity, we explore the formalization of trust models and approaches to glean trustworthiness. The above four Vs of Big Data are harnessed by the semantics-empowered analytics to derive value for supporting practical applications transcending physical-cyber-social continuum

Big Data Analytics for Earth Sciences: the EarthServer approach

Big Data Analytics is an emerging field since massive storage and computing capabilities have been made available by advanced e-infrastructures. Earth and Environmental sciences are likely to benefit from Big Data Analytics techniques supporting the processing of the large number of Earth Observation datasets currently acquired and generated through observations and simulations. However, Earth Science data and applications present specificities in terms of relevance of the geospatial information, wide heterogeneity of data models and formats, and complexity of processing. Therefore, Big Earth Data Analytics requires specifically tailored techniques and tools. The EarthServer Big Earth Data Analytics engine offers a solution for coverage-type datasets, built around a high performance array database technology, and the adoption and enhancement of standards for service interaction (OGC WCS and WCPS). The EarthServer solution, led by the collection of requirements from scientific communities and international initiatives, provides a holistic approach that ranges from query languages and scalability up to mobile access and visualization. The result is demonstrated and validated through the development of lighthouse applications in the Marine, Geology, Atmospheric, Planetary and Cryospheric science domains

Secure big data ecosystem architecture : challenges and solutions

Big data ecosystems are complex data-intensive, digital–physical systems. Data-intensive ecosystems offer a number of benefits; however, they present challenges as well. One major challenge is related to the privacy and security. A number of privacy and security models, techniques and algorithms have been proposed over a period of time. The limitation is that these solutions are primarily focused on an individual or on an isolated organizational context. There is a need to study and provide complete end-to-end solutions that ensure security and privacy throughout the data lifecycle across the ecosystem beyond the boundary of an individual system or organizational context. The results of current study provide a review of the existing privacy and security challenges and solutions using the systematic literature review (SLR) approach. Based on the SLR approach, 79 applicable articles were selected and analyzed. The information from these articles was extracted to compile a catalogue of security and privacy challenges in big data ecosystems and to highlight their interdependencies. The results were categorized from theoretical viewpoint using adaptive enterprise architecture and practical viewpoint using DAMA framework as guiding lens. The findings of this research will help to identify the research gaps and draw novel research directions in the context of privacy and security in big data-intensive ecosystems. © 2021, The Author(s)

Technologies and Applications for Big Data Value

This open access book explores cutting-edge solutions and best practices for big data and data-driven AI applications for the data-driven economy. It provides the reader with a basis for understanding how technical issues can be overcome to offer real-world solutions to major industrial areas. The book starts with an introductory chapter that provides an overview of the book by positioning the following chapters in terms of their contributions to technology frameworks which are key elements of the Big Data Value Public-Private Partnership and the upcoming Partnership on AI, Data and Robotics. The remainder of the book is then arranged in two parts. The first part “Technologies and Methods” contains horizontal contributions of technologies and methods that enable data value chains to be applied in any sector. The second part “Processes and Applications” details experience reports and lessons from using big data and data-driven approaches in processes and applications. Its chapters are co-authored with industry experts and cover domains including health, law, finance, retail, manufacturing, mobility, and smart cities. Contributions emanate from the Big Data Value Public-Private Partnership and the Big Data Value Association, which have acted as the European data community's nucleus to bring together businesses with leading researchers to harness the value of data to benefit society, business, science, and industry. The book is of interest to two primary audiences, first, undergraduate and postgraduate students and researchers in various fields, including big data, data science, data engineering, and machine learning and AI. Second, practitioners and industry experts engaged in data-driven systems, software design and deployment projects who are interested in employing these advanced methods to address real-world problems

A System-of-Systems Approach to Enterprise Analytics Design: Acquisition Support in the Age of Machine Learning and Artificial Intelligence

Acquisition Research Program Sponsored Report SeriesSponsored Acquisition Research & Technical ReportsSystem-of-Systems (SoS) capability emerges from the collaboration of multiple systems, which are acquired from independent organizations. Even though the systems contribute to and benefit from the larger SoS, the data analytics and decision-making about the independent system is rarely shared across the SoS stakeholders. The objective of this work is to identify how the sharing of datasets and the corresponding analytics among SoS stakeholders can lead to an improved SoS capability. Our objective is to characterize how the sharing of connected data sets may lead to deployment of different predictive (predicting an outcome from data) and prescriptive (determining a preferred strategy) analytics and lead to better decision outcomes at the SoS level. We build and demonstrate a framework for this objective based on extensive literature review and generating appropriate predictive and prescriptive methodologies that can be used for SoS analysis: Additionally, we propose to utilize machine learning techniques to predict the SoS capability achievable by sharing pertinent datasets and to prescribe the information links between systems to enable this sharing. Two case studies demonstrate the use of the framework and prospects for meeting the objective. Highlights of our study are summarized next.Approved for public release; distribution is unlimited.Approved for public release; distribution is unlimited