1 research outputs found
Managing the Complexity of Processing Financial Data at Scale -- an Experience Report
Financial markets are extremely data-driven and regulated. Participants rely
on notifications about significant events and background information that meet
their requirements regarding timeliness, accuracy, and completeness. As one of
Europe's leading providers of financial data and regulatory solutions vwd
processes a daily average of 18 billion notifications from 500+ data sources
for 30 million symbols. Our large-scale geo-distributed systems handle daily
peak rates of 1+ million notifications/sec. In this paper we give practical
insights about the different types of complexity we face regarding the data we
process, the systems we operate, and the regulatory constraints we must comply
with. We describe the volume, variety, velocity, and veracity of the data we
process, the infrastructure we operate, and the architecture we apply. We
illustrate the load patterns created by trading and how the markets' attention
to the Brexit vote and similar events stressed our systems.Comment: 12 pages, 2 figures, to be published in the proceedings of the 10th
Complex Systems Design & Management conference (CSD&M'19) by Springe