This report contains information gathered during the execution of E-ARK Pilot 7: SIP Access to Databases

Alföldi, István

Anderson, David

Anderson, Janet

Lux, Zoltán

Mezei, József

English

Mezei,  József

ZENODO

Access to Databases (Full Scale Pilot 7)

NEUROSURGERY ENTHUSIASTIC WOMEN SOCIETY

University of Brighton Research Portal

   D.2.4.P7  Full-scale Pilot 7 - Access to Databases       DOI: 10.5281/zenodo.1171715     Grant Agreement Number: 620998    Project Title:  European Archival Records and    Knowledge Preservation   Release Date:   12th February 2018                                  Contributors  Name Affiliation   Zoltán Lux    National Archives of Slovenia   József Mezei    National Archives of Slovenia   István Alföldi    National Archives of Hungary   David Anderson    University of Brighton   Janet Anderson    University of Brighton  D2.4 Pilot Documentation Table of Contents 1. EXECUTIVE SUMMARY .............................................................................................................................1 2. PILOT DOCUMENTATION .........................................................................................................................2 2.1 SCENARIOSŰ ............................................................................................................................................. 2 2.2 DATASETS ................................................................................................................................................ 2 2.3 DETAILED SCENARIO DESCRIPTION .................................................................................................................. 4 2.3.1 Scenario 1 ....................................................................................................................................... 4 2.3.2 Scenario 2 ....................................................................................................................................... 6 2.3.3 Scenario 3 ....................................................................................................................................... 7 2.3.4 Scenario 4 ....................................................................................................................................... 8 2.3.5 Scenario 5 ..................................................................................................................................... 10 2.4 ORGANISATIONS ...................................................................................................................................... 10 2.5 TOOLS ................................................................................................................................................... 10 2.5.1 E-ARK WEB ................................................................................................................................... 102.5.2 RODA-in ........................................................................................................................................ 11 2.5.3 DBPTK ........................................................................................................................................... 11 2.5.4 BÜRke ........................................................................................................................................... 11 2.5.5 Data Explorer (Oracle APEX) .......................................................................................................... 12 2.5.6 Oracle Warehouse Builder ............................................................................................................. 12 2.5.7 Oracle Data Integrator .................................................................................................................. 12 2.5.8 Oracle BI ....................................................................................................................................... 12 3. INSTALLATION MANUALS ...................................................................................................................... 12 3.1 INFRASTRUCTURE ..................................................................................................................................... 12 3.1.1 PCs ............................................................................................................................................... 13 3.1.2 Server ........................................................................................................................................... 13 D2.4 Pilot Documentation Page 1 of 13 1. Executive SummaryThis document is part of the deliverable: D2.4) Pilot documentation Pilot documentation: This package of documentation will provide technical and end-user guidance to support not only the pilot sites but also possible future deployments thereafter. [month 33] (from DoW) Structure of this deliverable The deliverable is a package if linked documents. This Summary contains the common information and short overview of the pilots, along with links to the final version of the Pilot Definition excel files and Pilot Documentation Packages. The Pilot Definition excel provides detailed information about scenarios, data sets and step-by-step preparation and process step instructions. The Pilot Documentation Package is created by the pilot staff responsible for the pilot execution. This package contains additional information along with screenshots (and videos in some cases) of the tools during the execution of the pilot.  Summary (this document) – Created by WP2 Pilot Package – Pilot 1 - Pilot Definition (Final version) – Created by WP2 and Pilot 1 responsible- Pilot Documentation files – Created by Pilot 1Pilot Package – Pilot 2- Pilot Definition (Final version) – Created by WP2 and Pilot 2 responsible- Pilot Documentation files – Created by Pilot 2Pilot Package – Pilot 3- Pilot Definition (Final version) – Created by WP2 and Pilot 3 responsible- Pilot Documentation files – Created by Pilot 3Pilot Package – Pilot 4- Pilot Definition (Final version) – Created by WP2 and Pilot 4 responsible- Pilot Documentation files – Created by Pilot 4Pilot Package – Pilot 5- Pilot Definition (Final version) – Created by WP2 and Pilot 5 responsible- Pilot Documentation files – Created by Pilot 5Pilot Package – Pilot 6- Pilot Definition (Final version) – Created by WP2 and Pilot 1 responsible- Pilot Documentation files – Created by Pilot 6Pilot Package – Pilot 7- Pilot Definition (Final version) – Created by WP2 and Pilot 7 responsible- Pilot Documentation files – Created by Pilot 7D2.4 Pilot Documentation Page 2 of 13 2. Pilot documentation2.1 Scenariosű Ingest Scenario 1: SIP Creation and Ingest of old (not normalized) database in SIARD 2.0 format Scenario 2: SIP Creation and Ingest of unstructured files AccessScenario 3: Extract SIARD Package from Preservica/E-ARK AIP (APEX/Oracle BI access) Scenario 4: Search and present SIARD based information with E-ARK access tools Scenario 5: Access information from unstructured files 2.2 Datasets Dataset 1 Hungarian Prosecution Office database Description Old (not normalized) database in CSV exports of DBASE files. D2.4 Pilot Documentation Page 3 of 13 Data type CSV files Metadata format none Quantity more then 300.000 cases and 500.000 name. (1,6 GB) Dataset 2 Scanned meeting minutes of the Central Committee of the Hungarian Socialist Party Description Scanned documents in file systems in PDF file and corresponding metadata (EAD) Data type PDF/JPG files (representations) Metadata format EAD Quantity 123.225 files. (101 GB) D2.4 Pilot Documentation Page 4 of 13 2.3 Detailed scenario description 2.3.1 Scenario 1 The Hungarian Prosecution Office delivered a data collection in CSV format from their DBASE database to the Hungarian National Archives.  The database contains the criminal records in Hungary between 1994 and 1999. 1 - BÜRke D2.4 Pilot Documentation Page 5 of 13 2 - One Prosecution office's database 3 - A file in csv format There were no documentation, neither to the original user program (by witch the data were maintained), nor to their DBASE database itself. In order to create an appropriate SIARD 2.0 package, we had loaded the original content into our Oracle database with BÜRke, and we added detailed documentation as comments to the database objects (tables, fields) within the database. SIARD 2.0 files had been created of the oracle database with DBPTK created by KEEPS SOLUTIONS: D2.4 Pilot Documentation Page 6 of 13 5 - Making and export with DBPTK E-ARK SIPs had been created, after ingested into our local E-ARK WEB and Preservica.2.3.2 Scenario 2 3703 E-ARK SIPs had been created of the scanned meeting minutes of the Central Committee of the Hungarian Worker Party (MSZMP). The original creation time interval is 1957 to 1989. SIPs had been loaded into our local E-ARK WEB:  D2.4 Pilot Documentation Page 7 of 13 6 - E-ARK WEB AIPs also into Preservica: 7 - SDB Preservita AIPs 2.3.3 Scenario 3 Access the database information of the Hungarian Prosecution Office in SIARD 2.0, format using Data Explorer (Oracle APEX). For a more logical presentation of the data if needed, Oracle Warehouse D2.4 Pilot Documentation Page 8 of 13 Builder can be used, for transforming the original data model into a data warehouse model, in that case APEX Data Explorer model will be applied to the Data Warehouse schema. 8 - Data Explorer (Oracle APEX) 2.3.4 Scenario 4 Access database information of the Hungarian Prosecution Office in SIARD 2.0 format using HADOOP based search and access with Lily Presentation in local environment. 9 - E-ARK WEB We load the SIARD 2.0 package into a live Oracle database by DBPTK. To presentation the data to the user, Oracle Data Integrator and Oracle Warehouse Builder is used.  D2.4 Pilot Documentation Page 9 of 13 By OWB the original model can be converted into a Data Warehouse model. Oracle Data Integrator enables us to design data marts in order to present the data according to specific business requirements. Using ODI, in the presentation layer, we can create a user friendly model which hides the complexity of the data (BI) model from the user and present the data in an understandable form, and also provides a broad opportunity to visualize the data.   10 - ORACLE BI D2.4 Pilot Documentation Page 10 of 13 2.3.5 Scenario 5 Create DIP from scanned documents of the Meeting minutes of the Central Committee of the Hungarian Socialist Party.  11 - EARK WEB Search The image files are in PDF format with EAD metadata in E-ARK Web HDFS storage and Preservica. 2.4 Organisations The organizations involved in this pilot are: KEEP SOLUTIONS (creator and developer of RODA-in, DBPTK, DBVTK), AIT (creator and developer of an open source archiving and digital preservation system, E-ARK WEB), XperTeam (developer of Data Explorer, Data Warehouse, and the local Oracle BI infrastructure), Hungarian Prosecution Office 2.5 Tools The tools involved in this pilot are: E-ARK WEB, RODA-in, DBPTK, Oracle (OLAP Viewer), 2.5.1 E-ARK WEB E-ARK Web is an open source archiving and digital preservation system. It is OAIS-oriented whichmeans that data ingest, archiving and dissemination functions operate on information packagesbundling content and metadata in contiguous containers. The information package format uses METSto represent the structure and PREMIS to record digital provenance information.D2.4 Pilot Documentation Page 11 of 13 E-ARK Web offers functionality for the three types of information packages defined in the OAISreference model: the Submission Information Package (SIP) which is the information sent from theproducer to the archive, the Archival Information Package (AIP) which is the information stored by thearchive, and the Dissemination Information Package (DIP) which is the information sent to a user whenrequested. The system allows executing different types of actions, such as information extraction,validation, or transformation operations, on information packages to support ingesting a SIP, archivingan AIP, and creating a DIP from a set of AIPs.E-ARK Web consists of a frontend web application together with a task execution system based onCelery which allows synchronous and asynchronous processing of information packages by means ofprocessing units which are called “tasks”.More information: https://github.com/eark-project/earkweb Installation manual: 2.5.2 RODA-in RODA-in is a tool specially designed for producers and archivists to create Submission Information Packages (SIP) ready to be submitted to an Open Archival Information System (OAIS). The tool creates SIPs from files and folders available on the local file system. In version 2 we revolutionized the way SIPs are created to satisfy the need for mass processing of data. In this version you can create thousands of valid SIPs with just a few clicks, complete with data and metadata. More information: http://rodain.roda-community.org/ 2.5.3 DBPTK The Database Preservation Toolkit allows conversion between Database formats, including connection to live systems, for purposes of digitally preserving databases. The toolkit allows conversion of live or backed-up databases into preservation formats such as SIARD, a XML-based format created for the purpose of database preservation. The toolkit also allows conversion of the preservation formats back into live systems to allow the full functionality of databases. This toolkit was part of the RODA project and now has been released as a project by its own due to the increasing interest on this particular feature. It is now being further developed in the EARK project together with a new version of the SIARD preservation format. The toolkit is created as a platform that uses input and output modules. Each module supports read and/or write to a particular database format or live system. New modules can easily be added by implementation of a new interface and adding of new drivers. More information: http://www.database-preservation.com/ 2.5.4 BÜRke Special local tool, developed to load CSV files into a pre-created Schema of a live Oracle Database (Oracle RDBMS), where the objects of the database are provided with detailed comments. BÜRke is standalone tool. A configuration file have to be set, which contains the path of the CSV files, and the connection informations to the Oracle database. D2.4 Pilot Documentation Page 12 of 13 2.5.5 Data Explorer (Oracle APEX) Data Explorer (Oracle APEX) is a local tool developed in Oracle APEX framework. The new version of APEX enables user to create interactive reports by which user can create queries on multiple tables, using simple graphical user interface. More information: https://apex.oracle.com/en/ Installation manual: 2.5.6 Oracle Warehouse Builder Oracle Warehouse Builder is a single, comprehensive tool for all aspects of data integration. Warehouse Builder leverages Oracle Database to transform data into high-quality information. It provides data quality, data auditing, fully integrated relational and dimensional modeling, and full lifecycle management of data and metadata. Warehouse Builder enables you to create data warehouses, migrate data from legacy systems, consolidate data from disparate data sources, clean and transform data to provide quality information, and manage corporate metadata. More information: https://docs.oracle.com/cd/B28359_01/owb.111/b31278/concept_overview.htm 2.5.7 Oracle Data Integrator A widely used data integration software product, Oracle Data Integrator provides a new declarative design approach to defining data transformation and integration processes, resulting in faster and simpler development and maintenance. More information: https://docs.oracle.com/cd/E23943_01/integrate.1111/e12641/overview.htm#ODIGS111 2.5.8 Oracle BI Oracle BI Enterprise Edition (sometimes simply referred to as Oracle Business Intelligence) provides a full range of business intelligence capabilities that allow you to:  Collect up-to-date data from your organization Present the data in easy-to-understand formats (such as tables and graphs) Deliver data in a timely fashion to the employees in your organizationMore information: http://docs.oracle.com/cd/E28280_01/bi.1111/e10544/getstart.htm#BIEUG11055 3. Installation manualsOracle services by XperTeam: https://www.dropbox.com/sh/h2wa22gm1abzvn1/AACcq6yyR0JdJgNJylc_31kba?dl=0 E-ARK WEB by AIT: https://www.dropbox.com/s/me211o0duxt6y8p/NAH-Pilot-Installation-AIT.pdf?dl=03.1 Infrastructure The infrastructure we used in this pilot to create SIPs AIPs, DIPs, and visualize data. PCs for the ingest (preparation) part, programs like RODA-in, Server for storage, management, and services, like E-ARK WEB, Oracle, DBPTK. D2.4 Pilot Documentation Page 13 of 13 3.1.1 PCs  DELL WST3500 – Windows 10 – Intel Xeon W3503 -2 core – 2.4ghz Fujitsu MI4W-D3171 – Windows 7 – Intel Core i3 2130 – 4 core – 3.4 ghz3.1.2 Server  HS22 in RAC: RedHat Linux, Oracle 12c -  Xeon MP 2,64 ghz – 12 core/blade, 96GB memory

https://cris.brighton.ac.uk/ws/files/4696378/E_ARK_D.2.4.P.7.pdf

Access to Databases (Full Scale Pilot 7)

Abstract

Similar works

Full text

Available Versions

ZENODO

NEUROSURGERY ENTHUSIASTIC WOMEN SOCIETY

University of Brighton Research Portal