Düpmeier, Clemens

Schmitt, Christian

English

KITopen

KIT – University of Baden-Württemberg and National Research Center of the Helmholtz Alliance Institute of Applied Computer Science (IAI) www.kit.edu Storing and Analyzing Bibliographic Metadata with ElasticSearch Clemens Düpmeier, Christian Schmitt (KIT / IAI) Institute for Applied Computer Science (IAI) 2 03.07.2014 Overview Search based web applications and the openTA project A short overview on ElasticSearch and its features Storing and analyzing bibliographic metadata Summary and outlook Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 3 03.07.2014 Modern web user interfaces The design principles of web user interfaces have changed  Instead of complex forms and static hierarchical navigation =>  responsive and natural language based interfaces  Search engine technology can support these types of user interfaces well  Research group “ Web based  Information Systems“ Push Search Institute for Applied Computer Science (IAI) 4 03.07.2014 Search based web applications Use search engine technology  as key element for data access  Available data is a mixture  of unstructured semi-structured structured       information coming from      different sources  Use semantic technologies for aggregation,  normalization and  classification of data  And a natural language approach for data access  Research group “ Web based  Information Systems“ Search based environmental  information portal Institute for Applied Computer Science (IAI) 5 03.07.2014 The openTA portal is search based too Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 6 03.07.2014 The openTA portal DfG project with partners  (IAI, ITAS, KIT library)  from KIT   Web portal for the Network Technology Assessment  (NTA) Aggregates and provides  information about  members, organizations news and events scientific publications Provides services and web technology (widgets) to use  the aggregated information remotely Research group “ Web based  Information Systems“ Homepage of openTA with  aggregated news feed and calendar Institute for Applied Computer Science (IAI) 7 03.07.2014 Search driven architecture of openTA Research group “ Web based  Information Systems“ Search Engine (Aggregation, Analysis) Structured indices OpenTA Calendar News Service Data Services Portal Web  UI Portal Service APIs Standard Functionalities (User Management, CMS, Social) Data and Service Access RSS / Atom ICalendar Pub.-Form. NTA Organization NTA Organization NTA Organization NTA Organization User User Most information will be ingested  using service interfaces and crawling.  Data Import / Crawler Publication Service Institute for Applied Computer Science (IAI) 8 03.07.2014 What kind of search technology? Well known open source products are: Solr, ElasticSearch Both are based on Apache Lucene They both provide Document oriented, structured search indices Scalable architecture by distributing and processing index entries on multiple servers („Sharding“) Advanced features for data aggregation, analysis and retrieval The openTA project uses ElasticSearch Newer and better architecture (compared to Solr) Completely REST and JSON format based Supports hierarchical document structures and parent  / child relationships Supports queries against multiple indices at once And other interesting features explained on the next slides  Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 9 03.07.2014 Some elements of the search language  (Lucine based) Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 10 03.07.2014 Indexing data with ElasticSearch Send JSON documents to server, e.g. use REST API  No schema necessary => ElasticSearch determines type of attributes But’s possible to explicitly specify types for attributes Like string, byte, short, integer, long, float, double, boolean, date Analysis of  (text) attributes Word extraction, reduction of words to their base form (stemming) Stop words Support for multiple languages Can be extended via plugins for quite complex analysis Entity recognition De-duplication Generate identifier for data sets; duplicate recognition  Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 11 03.07.2014 Query Possibilities Combined search on different attributes and different indices Many possibilities for full-text search on attribute values Exact, non-exact, proximity (phrases), partial match  Support well-known logical operators (And / or, …) Range queries (i.e. date ranges) … Control relevance and ranking of search results, sort them Boost relevance while indexing Boost or ignore relevance while querying Different possibilities to sort search results otherwise Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 12 03.07.2014 More advanced features Multi-tenant  Spatial data queries Search suggestions Real time aggregation of search data Statistical calculations (sums, mean value, max, min, …) Faceting By using terms Statistical calculations Classification ( Grouping by using ranges Filter rules By geographical distance Percolators Warm up of indices  Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 13 03.07.2014 Importing bibliographic data in ElasticSearch Readers transform source format to record-oriented  Metamorph format (in Java) Metafacture / Metamorph is a rule based transformation engine for bibliographic data (open source as part of the CultureGraph platform, DNB) Transformation rules applied by Metamorph transform data into internal objects, which are then serialized to JSON EL-Search Writer stores the JSON objects in ElasticSearch Research group “ Web based  Information Systems“ BibTex BibTex Reader Metamorph JSON- Object EL-Search Writer Index Transform. rules Institute for Applied Computer Science (IAI) 14 03.07.2014 Some features of Metamorph Change grouping of attributes, change name of attribute Split / join attributes Substring and pattern extraction Transformation between data types Map values to other values using lookup tables Build records from attributes, convert data records in attributes Supports recursion and conditions in transformation rules Allows to plugin your own code at different places of the transformation  Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 15 03.07.2014 Example of Metamorph transformation rules Research group “ Web based  Information Systems“ Transformation rules Split authors field at character | Convert into lower case and map type by using lookup table Institute for Applied Computer Science (IAI) 16 03.07.2014 Role of ElasticSearch  While storing Generate unique identifier und keys to find duplicates Group documents which seem to describe same entities Further analysis of attributes (especially attributes containing text) Entity recognition Keyword extraction / Classification Querying in realtime Search across multiple fields, calculate ranking Hits in different fields can be given different ranking factors (i.e. finding an author name in the author field) Or matching the publication date Aggregation of data; calculations for faceted search Publication year Top 10 authors (with most publications) Type of publication   Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 17 03.07.2014 Prototype: Search with „Energie  –Rohracher“ Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 18 03.07.2014 Summary and outlook The search based architecture of openTA works really well First prototype of publication service looks promising too Implemented import pipeline and usage of Metamorph allows for a modular extension to support more formats ElasticSearch already showed its great potential but only the basic functionality is used at the moment Publication service will be continually enhanced  New sources and formats will be added More facets and filter functionality will be implemented in the UI Explore more advanced features and extensions of ElasticSearch to solve some challenges Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 19 03.07.2014 openTA website Research group “ Web based  Information Systems“ http://www.openta.net Institute for Applied Computer Science (IAI) 20 03.07.2014 Metamorph: Map Publication Types Metamorph allows to transform values by map lookup Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 21 03.07.2014 BibTex -> JSON Mapping Text based format  -> Object based format Split and rename fields Normalization of data type notions (e.g. date) Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 22 03.07.2014 BibTex Format Text based format consisting of several text lines wrapped in @article{} construct Research group “ Web based  Information Systems“ Institute for Applied Computer Science (IAI) 23 03.07.2014 Example of metadata as JSON Text version of JSON format used by openTA Research group “ Web based  Information Systems“ 

Storing and Analyzing Bibliographic Metadata with ElasticSearch

https://publikationen.bibliothek.kit.edu/1000041752/3131119

Storing and Analyzing Bibliographic Metadata with ElasticSearch

Abstract

Similar works

Full text

Available Versions

KITopen