Designing Traceability into Big Data Systems

Providing an appropriate level of accessibility and traceability to data or process elements (so-called Items) in large volumes of data, often Cloud-resident, is an essential requirement in the Big Data era. Enterprise-wide data systems need to be designed from the outset to support usage of such Items across the spectrum of business use rather than from any specific application view. The design philosophy advocated in this paper is to drive the design process using a so-called description-driven approach which enriches models with meta-data and description and focuses the design process on Item re-use, thereby promoting traceability. Details are given of the description-driven design of big data systems at CERN, in health informatics and in business process management. Evidence is presented that the approach leads to design simplicity and consequent ease of management thanks to loose typing and the adoption of a unified approach to Item management and usage.Comment: 10 pages; 6 figures in Proceedings of the 5th Annual International Conference on ICT: Big Data, Cloud and Security (ICT-BDCS 2015), Singapore July 2015. arXiv admin note: text overlap with arXiv:1402.5764, arXiv:1402.575

JSB Composability and Web Services Interoperability Via Extensible Modeling & Simulation Framework (XMSF), Model Driven Architecture (MDA), Component Repositories, and Web-based Visualization

Study Report prepared for the U. S. Air Force, Joint Synthetic Battlespace Analysis of Technical Approaches (ATA) Studies & Prototyping Overview: This paper summarizes research work conducted by organizations concerned with interoperable distributed information technology (IT) applications, in particular the Naval Postgraduate School (NPS) and Old Dominion University (ODU). Although the application focus is distributed modeling & simulation (M&S) the results and findings are in general easily applicable to other distributed concepts as well, in particular the support of operations by M&S applications, such as distributed mission operations. The core idea of this work is to show the necessity of applying open standards for component description, implementation, and integration accompanied by aligned management processes and procedures to enable continuous interoperability for legacy and new M&S components of the live, virtual, and constructive domain within the USAF Joint Synthetic Battlespace (JSB). JSB will be a common integration framework capable of supporting the future emerging simulation needs ranging from training and battlefield rehearsal to research, system development and acquisition in alignment with other operational requirements, such as integration of command and control, support of operations, integration of training ranges comprising real systems, etc. To this end, the study describes multiple complementary Integrated Architecture Framework approaches and shows, how the various parts must be orchestrated in order to support the vision of JSB effectively and efficiently. Topics of direct relevance include Web Services via Extensible Modeling & Simulation Framework (XMSF), the Object Management Group (OMG)’s Model Driven Architecture (MDA), XML-based resource repositories, and Web-based X3D visualization. To this end, the report shows how JSB can − Utilize Web Services throughout all components via XMSF methodologies, − Compose diverse system visualizations using Web-based X3D graphics, − Benefit from distributed modeling methods using MDA, and − Best employ resource repositories for broad and consistent composability. Furthermore, the report recommends the establishment of necessary management organizations responsible for the necessary alignment of management processes and procedures within the JSB as well as with neighbored domains. Continuous interoperability cannot be accomplished by technical standards alone. The application of technical standards targets the implementation level of the system of systems, which results in an interoperable solution valid only for the actual 2 implementation. To insure continuity, the influence of updates, upgrades and introduction of components on the system of systems must be captured in the project management procedures of the participating systems. Finally, the report proposes an exemplifying set of proof-of-capability demonstration prototypes and a five-year technical/institutional transformation plan. All key references are online available at http://www.movesinstitute.org/xmsf/xmsf.html (if not explicitly stated otherwise)

Impliance: A Next Generation Information Management Appliance

ably successful in building a large market and adapting to the changes of the last three decades, its impact on the broader market of information management is surprisingly limited. If we were to design an information management system from scratch, based upon today's requirements and hardware capabilities, would it look anything like today's database systems?" In this paper, we introduce Impliance, a next-generation information management system consisting of hardware and software components integrated to form an easy-to-administer appliance that can store, retrieve, and analyze all types of structured, semi-structured, and unstructured information. We first summarize the trends that will shape information management for the foreseeable future. Those trends imply three major requirements for Impliance: (1) to be able to store, manage, and uniformly query all data, not just structured records; (2) to be able to scale out as the volume of this data grows; and (3) to be simple and robust in operation. We then describe four key ideas that are uniquely combined in Impliance to address these requirements, namely the ideas of: (a) integrating software and off-the-shelf hardware into a generic information appliance; (b) automatically discovering, organizing, and managing all data - unstructured as well as structured - in a uniform way; (c) achieving scale-out by exploiting simple, massive parallel processing, and (d) virtualizing compute and storage resources to unify, simplify, and streamline the management of Impliance. Impliance is an ambitious, long-term effort to define simpler, more robust, and more scalable information systems for tomorrow's enterprises.Comment: This article is published under a Creative Commons License Agreement (http://creativecommons.org/licenses/by/2.5/.) You may copy, distribute, display, and perform the work, make derivative works and make commercial use of the work, but, you must attribute the work to the author and CIDR 2007. 3rd Biennial Conference on Innovative Data Systems Research (CIDR) January 710, 2007, Asilomar, California, US

Engineering Agile Big-Data Systems

To be effective, data-intensive systems require extensive ongoing customisation to reflect changing user requirements, organisational policies, and the structure and interpretation of the data they hold. Manual customisation is expensive, time-consuming, and error-prone. In large complex systems, the value of the data can be such that exhaustive testing is necessary before any new feature can be added to the existing design. In most cases, the precise details of requirements, policies and data will change during the lifetime of the system, forcing a choice between expensive modification and continued operation with an inefficient design.Engineering Agile Big-Data Systems outlines an approach to dealing with these problems in software and data engineering, describing a methodology for aligning these processes throughout product lifecycles. It discusses tools which can be used to achieve these goals, and, in a number of case studies, shows how the tools and methodology have been used to improve a variety of academic and business systems