Governance in the age of social machines: the web observatory

The World Wide Web has provided unprecedented access to information; as humans and machines increasingly interact with it they provide more and more data. The challenge is how to analyse and interpret this data within the context that it was created, and to present it in a way that both researchers and practitioners can more easily make sense of. The first step is to have access to open and interoperable data sets, which Governments around the world are increasingly subscribing to. But having ‘open’ data is just the beginning and does not necessarily lead to better decision making or policy development. This is because data do not provide the answers – they need to be analysed, interpreted and understood within the context of their creation, and the business imperative of the organisation using them. The major corporate entities, such as Google, Amazon, Microsoft, Apple and Facebook, have the capabilities to do this, but are driven by their own commercial imperatives, and their data are largely siloed and held within ‘walled gardens’ of information. All too often governments and non-profit groups lack these capabilities, and are driven by very different mandates. In addition they have far more complex community relationships, and must abide by regulatory constraints which dictate how they can use the data they hold. As such they struggle to maximise the value of this emerging ‘digital currency’ and are therefore largely beholden to commercial vendors. What has emerged is a public-private data ecosystem that has huge policy implications (including the twin challenges of privacy and security). Many within the public sector lack the skills to address these challenges because they lack the literacy required within the digital context. This project seeks to address some of these problems by bringing together a safe and secure Australian-based data platform (facilitating the sharing of data, analytics and visualisation) with policy analysis and governance expertise in order to create a collaborative working model of a ‘Government Web Observatory’. This neutral space, hosted by an Australian university, can serve as a powerful complement to existing Open Data initiatives in Australia, and enable research and education to combine to support the development of a more digitally literate public service. The project aims to explore where, and in which contexts, people, things, data and the Internet meet and result in evolving observable phenomena which can inform better government policy development and service delivery.&nbsp

From Social Data Mining to Forecasting Socio-Economic Crisis

Socio-economic data mining has a great potential in terms of gaining a better understanding of problems that our economy and society are facing, such as financial instability, shortages of resources, or conflicts. Without large-scale data mining, progress in these areas seems hard or impossible. Therefore, a suitable, distributed data mining infrastructure and research centers should be built in Europe. It also appears appropriate to build a network of Crisis Observatories. They can be imagined as laboratories devoted to the gathering and processing of enormous volumes of data on both natural systems such as the Earth and its ecosystem, as well as on human techno-socio-economic systems, so as to gain early warnings of impending events. Reality mining provides the chance to adapt more quickly and more accurately to changing situations. Further opportunities arise by individually customized services, which however should be provided in a privacy-respecting way. This requires the development of novel ICT (such as a self- organizing Web), but most likely new legal regulations and suitable institutions as well. As long as such regulations are lacking on a world-wide scale, it is in the public interest that scientists explore what can be done with the huge data available. Big data do have the potential to change or even threaten democratic societies. The same applies to sudden and large-scale failures of ICT systems. Therefore, dealing with data must be done with a large degree of responsibility and care. Self-interests of individuals, companies or institutions have limits, where the public interest is affected, and public interest is not a sufficient justification to violate human rights of individuals. Privacy is a high good, as confidentiality is, and damaging it would have serious side effects for society.Comment: 65 pages, 1 figure, Visioneer White Paper, see http://www.visioneer.ethz.c

Mapping Big Data into Knowledge Space with Cognitive Cyber-Infrastructure

Big data research has attracted great attention in science, technology, industry and society. It is developing with the evolving scientific paradigm, the fourth industrial revolution, and the transformational innovation of technologies. However, its nature and fundamental challenge have not been recognized, and its own methodology has not been formed. This paper explores and answers the following questions: What is big data? What are the basic methods for representing, managing and analyzing big data? What is the relationship between big data and knowledge? Can we find a mapping from big data into knowledge space? What kind of infrastructure is required to support not only big data management and analysis but also knowledge discovery, sharing and management? What is the relationship between big data and science paradigm? What is the nature and fundamental challenge of big data computing? A multi-dimensional perspective is presented toward a methodology of big data computing.Comment: 59 page

Generalised network architectures for environmental sensing: case studies for a digitally enabled environment

A digitally enabled environment is a setting which incorporates sensors coupled with reporting and analytics tools for understanding, observing or managing that environment. Large scale data collection and analysis are a part of the emerging digitally enabled approach for the characterisation and understanding of our environment. It is recognised as offering an effective methodology for addressing a range of complex and interrelated social, economic and environmental concerns. The development and construction of the approach requires advances in analytics control linked with a clear definition of the issues pertaining to the interaction between elements of these systems. This paper presents an analysis of selected issues in the field of analytics control. It also discusses areas of progress, and areas in need of further investigation as sensing networks evolve. Three case studies are described to illustrate these points. The first is a physical analytics test kit developed as a part of the “Reinvent the Toilet Challenge” (RTTC) for process control in a range of environments. The second case study is the Cranfield Urban Observatory that builds on elements of the RTTC and is designed to allow users to develop user interfaces to monitor, characterise and compare a variety of environmental and infrastructure systems plus behaviours (e.g., water distribution, power grids). The third is the Data and Analytics Facility for National Infrastructure, a cloud-based high-performance computing cluster, developed to receive, store and present such data to advanced analytical and visualisation tools.Engineering and Physical Sciences Research Council (EPSRC): EP/P016782/1, EP/R013411/1, EP/R012202/1 and EP/R017727/1. Bill & Melinda Gates Foundatio

Technologies and Applications for Big Data Value

This open access book explores cutting-edge solutions and best practices for big data and data-driven AI applications for the data-driven economy. It provides the reader with a basis for understanding how technical issues can be overcome to offer real-world solutions to major industrial areas. The book starts with an introductory chapter that provides an overview of the book by positioning the following chapters in terms of their contributions to technology frameworks which are key elements of the Big Data Value Public-Private Partnership and the upcoming Partnership on AI, Data and Robotics. The remainder of the book is then arranged in two parts. The first part “Technologies and Methods” contains horizontal contributions of technologies and methods that enable data value chains to be applied in any sector. The second part “Processes and Applications” details experience reports and lessons from using big data and data-driven approaches in processes and applications. Its chapters are co-authored with industry experts and cover domains including health, law, finance, retail, manufacturing, mobility, and smart cities. Contributions emanate from the Big Data Value Public-Private Partnership and the Big Data Value Association, which have acted as the European data community's nucleus to bring together businesses with leading researchers to harness the value of data to benefit society, business, science, and industry. The book is of interest to two primary audiences, first, undergraduate and postgraduate students and researchers in various fields, including big data, data science, data engineering, and machine learning and AI. Second, practitioners and industry experts engaged in data-driven systems, software design and deployment projects who are interested in employing these advanced methods to address real-world problems

COSMOS: Towards an integrated and scalable service for analysing social media on demand

The growing number of people using social media to publish their opinions, share expertise, make social connections and promote their ideas to an international audience is creating data on an epic scale. This enables social scientists to conduct research into ethnography, discourse analysis and analysis of social interactions, providing insight into today's society, which is largely augmented by social computing. The tools available for such analysis are often proprietary and expensive, and often non-interoperable, meaning the rapid marshalling of large data-sets through a range of analyses is arduous and difficult to scale. The collaborative online social media observatory (COSMOS), an integrated social media analysis tool is presented, developed for open access within academia. COSMOS is underpinned by a scalable Hadoop infrastructure and can support the rapid analysis of large data-sets and the orchestration of workflows between tools with limited human effort. We describe an architecture and scalability results for the computational analysis of social media data, and comment on the storage, search and retrieval issues associated with massive social media data-sets. We also provide an insight into the impact of such an integrated on-demand service in the social science academic community

DEVELOPING A REAL-TIME DATA ANALYTICS FRAMEWORK FOR TWITTER STREAMING DATA

Twitter is an online social networking service with more than 300 million users, generating a huge amount of information every day. Twitter's most important characteristic is its ability for users to tweet about events, situations, feelings, opinions, or even something totally new, in real time. Currently there are different workflows offering real-time data analysis for Twitter, presenting general processing over streaming data. This study will attempt to develop an analytical framework with the ability of in-memory processing to extract and analyze structured and unstructured Twitter data. The proposed framework includes data ingestion and stream processing and data visualization components with the Apache Kafka messaging system that is used to perform data ingestion task. Furthermore, Spark makes it possible to perform sophisticated data processing and machine learning algorithms in real time. We have conducted a case study on tweets about the earthquake in Japan and the reactions of people around the world with analysis on the time and origin of the tweets