Managing the Complexity of Processing Financial Data at Scale -- an Experience Report

Financial markets are extremely data-driven and regulated. Participants rely on notifications about significant events and background information that meet their requirements regarding timeliness, accuracy, and completeness. As one of Europe's leading providers of financial data and regulatory solutions vwd processes a daily average of 18 billion notifications from 500+ data sources for 30 million symbols. Our large-scale geo-distributed systems handle daily peak rates of 1+ million notifications/sec. In this paper we give practical insights about the different types of complexity we face regarding the data we process, the systems we operate, and the regulatory constraints we must comply with. We describe the volume, variety, velocity, and veracity of the data we process, the infrastructure we operate, and the architecture we apply. We illustrate the load patterns created by trading and how the markets' attention to the Brexit vote and similar events stressed our systems.Comment: 12 pages, 2 figures, to be published in the proceedings of the 10th Complex Systems Design & Management conference (CSD&M'19) by Springe

Poster: A Real-World Distributed Infrastructure for Processing Financial Data at Scale

Financial markets are event- and data-driven to an extremely high degree. For making decisions and triggering actions stakeholders require notifications about significant events and reliable background information that meet their individual requirements in terms of timeliness, accuracy, and completeness. As one of Europe's leading providers of financial data and regulatory solutions vwd processes an average of 18 billion event notifications from 500+ data sources for 30 million symbols per day. Our large-scale distributed event-based systems handle daily peak rates of 1+ million event notifications per second and additional load generated by singular pivotal events with global impact. In this poster we give practical insights into our IT systems. We outline the infrastructure we operate and the event-driven architecture we apply at vwd. In particular we showcase the (geo)distributed publish/subscribe broker network we operate across locations and countries to provide market data to our customers with varying quality of information (QoI) properties.Comment: Authors' version of the accepted submission; final version published by ACM as part of the proceedings of DEBS '19: The 13th ACM International Conference on Distributed and Event-based Systems (DEBS '19); 2 pages, 1 figure; vwd Vereinigte Wirtschaftsdienste GmbH is by now known as Infront Financial Technology GmbH (part of the Infront group

Poster: Benchmarking Financial Data Feed Systems

Data-driven solutions for the investment industry require event-based backend systems to process high-volume financial data feeds with low latency, high throughput, and guaranteed delivery modes. At vwd we process an average of 18 billion incoming event notifications from 500+ data sources for 30 million symbols per day and peak rates of 1+ million notifications per second using custom-built platforms that keep audit logs of every event. We currently assess modern open source event-processing platforms such as Kafka, NATS, Redis, Flink or Storm for the use in our ticker plant to reduce the maintenance effort for cross-cutting concerns and leverage hybrid deployment models. For comparability and repeatability we benchmark candidates with a standardized workload we derived from our real data feeds. We have enhanced an existing light-weight open source benchmarking tool in its processing, logging, and reporting capabilities to cope with our workloads. The resulting tool wrench can simulate workloads or replay snapshots in volume and dynamics like those we process in our ticker plant. We provide the tool as open source. As part of ongoing work we contribute details on (a) our workload and requirements for benchmarking candidate platforms for financial feed processing; (b) the current state of the tool wrench.Comment: Authors' version of the accepted submission; final version published by ACM as part of the proceedings of DEBS '19: The 13th ACM International Conference on Distributed and Event-based Systems (DEBS '19); 2 pages, 2 figure

Runtime Support for Quality of Information Requirements in Event-based Systems

Modern reactive software systems turn fine-granular real-time notifications about processes in the physical world into information and knowledge to react in time. Push-based Event-based Systems (EBSs) complement pull-based architectures, such as Service-oriented Architectures (SOAs), and enable enterprises to react to meaningful events in a timely manner. Applications for algorithmic trading, energy-aware reactive data center management, or smart supply chain management are just three examples of reactive systems where information provided by heterogeneous data sources has to be interpreted and where false alarms, missed events or otherwise data of inadequate Quality of Information (QoI) carries a cost. Whether the QoI of notifications is adequate depends on the purpose, the information is intended to be used for by each receiver. This purpose is application-specific and changes at runtime. Thus, the notion of QoI combines objectively measurable properties of a notification and their application-specific assessment that determines the Value of Information (VoI) for a receiver. Receiving only data that conforms to their QoI requirements is crucial for reactive applications. Current support for QoI, however, is limited in terms of expressiveness and effectiveness. In this dissertation, we introduce the concept of expectations, capabilities and feedback as a holistic concept to express, negotiate and enforce QoI requirements at runtime in push-based systems. Participants express requirements and define individual trade-offs between them as expectations; the ability of the system to support properties by adapting itself is captured by capabilities that include the individual costs of participants. Feedback to participants is a central component of our approach and is used to coordinate the adaptation of participants at runtime. We show that our approach is more expressive and supports a wider range of properties than current approaches; our approach actively enforces complex requirements about QoI in an effective way without deteriorating the system’s performance. The work presented in this dissertation contributes to the challenge of runtime QoI support in push-based architectures on a conceptual and practical level. On the conceptual level, we contribute a generic and extensible model to express and manage requirements about arbitrary QoI properties, algorithms for negotiation and enforcing these requirements at runtime as well as a concept for effective runtime monitoring in a distributed and decentralized EBS. The conceptual part of this dissertation synthesizes and expands approaches devised in pull- and push-based systems as well as in economics into a novel concept to support QoI at runtime in reactive software systems. On the practical level, we contribute a reference architecture for runtime support of QoI requirements, two prototypes built on a centralized and a decentralized Message-oriented Middleware (MOM), examples for existing applications enhanced with our approach as well as an extensive evaluation of our prototypes built on the industry-strength Apache ActiveMQ platform and the academic REconfigurable Dispatching System (REDS). The evaluation uses industry-strength benchmarks and systems to quantify the benefits and execution costs for participants. The practical part of this dissertation shows the practicability of our approach and quantifies the benefits of actively enforcing QoI requirements using feedback

Runtime Support for Quality of Information Requirements in Event-based Systems

Modern reactive software systems turn fine-granular real-time notifications about processes in the physical world into information and knowledge to react in time. Push-based Event-based Systems (EBSs) complement pull-based architectures, such as Service-oriented Architectures (SOAs), and enable enterprises to react to meaningful events in a timely manner. Applications for algorithmic trading, energy-aware reactive data center management, or smart supply chain management are just three examples of reactive systems where information provided by heterogeneous data sources has to be interpreted and where false alarms, missed events or otherwise data of inadequate Quality of Information (QoI) carries a cost. Whether the QoI of notifications is adequate depends on the purpose, the information is intended to be used for by each receiver. This purpose is application-specific and changes at runtime. Thus, the notion of QoI combines objectively measurable properties of a notification and their application-specific assessment that determines the Value of Information (VoI) for a receiver. Receiving only data that conforms to their QoI requirements is crucial for reactive applications. Current support for QoI, however, is limited in terms of expressiveness and effectiveness. In this dissertation, we introduce the concept of expectations, capabilities and feedback as a holistic concept to express, negotiate and enforce QoI requirements at runtime in push-based systems. Participants express requirements and define individual trade-offs between them as expectations; the ability of the system to support properties by adapting itself is captured by capabilities that include the individual costs of participants. Feedback to participants is a central component of our approach and is used to coordinate the adaptation of participants at runtime. We show that our approach is more expressive and supports a wider range of properties than current approaches; our approach actively enforces complex requirements about QoI in an effective way without deteriorating the system’s performance. The work presented in this dissertation contributes to the challenge of runtime QoI support in push-based architectures on a conceptual and practical level. On the conceptual level, we contribute a generic and extensible model to express and manage requirements about arbitrary QoI properties, algorithms for negotiation and enforcing these requirements at runtime as well as a concept for effective runtime monitoring in a distributed and decentralized EBS. The conceptual part of this dissertation synthesizes and expands approaches devised in pull- and push-based systems as well as in economics into a novel concept to support QoI at runtime in reactive software systems. On the practical level, we contribute a reference architecture for runtime support of QoI requirements, two prototypes built on a centralized and a decentralized Message-oriented Middleware (MOM), examples for existing applications enhanced with our approach as well as an extensive evaluation of our prototypes built on the industry-strength Apache ActiveMQ platform and the academic REconfigurable Dispatching System (REDS). The evaluation uses industry-strength benchmarks and systems to quantify the benefits and execution costs for participants. The practical part of this dissertation shows the practicability of our approach and quantifies the benefits of actively enforcing QoI requirements using feedback

FIT for SOA? Introducing the F.I.T.-Metric to Optimize the Availability of Service Oriented Architectures

Abstract The paradigm of service-oriented architectures (SOA) is by now accepted for application integration and in widespread use. As an underlying key-technology of cloud computing and because of unresolved issues during operation and maintenance it remains a hot topic. SOA encapsulates business functionality in services, combining aspects from both the business and infrastructure level. The reuse of services results in hidden chains of dependencies that affect governance and optimization of service-based systems. To guarantee the cost-effective availability of the whole service-based application landscape, the real criticality of each dependency has to be determined for IT Service Management (ITSM) to act accordingly. We propose the FIT-metric as a tool to characterize the stability of existing service configurations based on three components: functionality, integration and traffic. In this paper we describe the design of FIT and apply it to configurations taken from a production-strength SOA-landscape. A prototype of FIT is currently being implemented at Deutsche Post MAIL.