A Typical Case of Recommending the Use of a Hierarchical Data Format

Abstract

Storing large amounts of data in an efficient way from the point of view of very fast retrieval is animportant requirement for many industries like KEMA, a company offering consulting, testing andcertification in the energy business. Because their relational database could not cope fast enough with thelarge amounts of data involved, an alternative way for data storage was proposed, called HierarchicalData Format 5 (HDF5). HDF5 is a data model, a library, and file format for storing and managing data.Four hierarchical designs for storing and retrieving the large amounts of data involved were investigated.A benchmark was carried out in order to know which hierarchical structure would perform best.Eventually, a benchmark between HDF5 and MS SQL Server was carried out. It could be shown thatHDF5 performs four (4) times better for inserting and even 200 times better for retrieving data than theMS. SQL Server