The design and engineering of a unified data access layer : bridging data consumers and diverse data sources

Abstract

In the research and development process at Océ, considerable amounts of data are being generated by either machines or services. This data is used by different data consumers at Océ for various purposes such as analysis and reporting. However, since the data is generated by different machines and for different purposes, the data sources are incompatible in the sense of the format and the access technology. When a data consumer requires data from different data sources, it has to write its own integration code. This leads to effort and time for each data consumer. Therefore, there is a need to have a unified Data Access Layer between data sources and data consumers to reduce the effort and time for each data consumer. In addition, for solving the incompatibility of data formats, the goal is to provide a schema and type system, in which data structures are mapped with each other. Design and implementation of the data access layer is an important goal to show the validation of the chosen technology. Design challenges are: 1) mapping the schema of the data sources to the foreign tables. 2) automatic process of creating foreign tables when the schema evolves, and 3) designing reusable foreign data wrappers. These challenges are addressed in the design. The system quality is validated based on the user feedback. The report also lists several recommendations for future improvements

    Similar works