Building scalable digital library ingestion pipelines using microservices

CORE, a harvesting service offering access to millions of open access research papers from around the world, has shifted its harvesting process from following a monolithic approach to the adoption of a microservices infrastructure. In this paper, we explain how we rearranged and re-scheduled our old ingestion pipeline, present CORE's move to managing microservices and outline the tools we use in a new and optimised ingestion system. In addition, we discuss the ineffciencies of our old harvesting process, the advantages, and challenges of our new ingestion system and our future plans. We conclude that via the adoption of microservices architecture we managed to achieve a scalable and distributed system that would assist with CORE's future performance and evolution

CORE

Le présent document présente le CORE, un service de recherche largement utilisé, qui donne accès aux publications de recherche en libre accès, provenant d’un réseau mondial de dépôts. CORE a été créé dans le but de permettre l’exploration de textes et de données de la littérature scientifique et donc de soutenir la découverte scientifique, mais il est maintenant utilisé dans un large éventail de cas