216 research outputs found
The End of Slow Networks: It's Time for a Redesign
Next generation high-performance RDMA-capable networks will require a
fundamental rethinking of the design and architecture of modern distributed
DBMSs. These systems are commonly designed and optimized under the assumption
that the network is the bottleneck: the network is slow and "thin", and thus
needs to be avoided as much as possible. Yet this assumption no longer holds
true. With InfiniBand FDR 4x, the bandwidth available to transfer data across
network is in the same ballpark as the bandwidth of one memory channel, and it
increases even further with the most recent EDR standard. Moreover, with the
increasing advances of RDMA, the latency improves similarly fast. In this
paper, we first argue that the "old" distributed database design is not capable
of taking full advantage of the network. Second, we propose architectural
redesigns for OLTP, OLAP and advanced analytical frameworks to take better
advantage of the improved bandwidth, latency and RDMA capabilities. Finally,
for each of the workload categories, we show that remarkable performance
improvements can be achieved
NaNet: a Low-Latency, Real-Time, Multi-Standard Network Interface Card with GPUDirect Features
While the GPGPU paradigm is widely recognized as an effective approach to
high performance computing, its adoption in low-latency, real-time systems is
still in its early stages.
Although GPUs typically show deterministic behaviour in terms of latency in
executing computational kernels as soon as data is available in their internal
memories, assessment of real-time features of a standard GPGPU system needs
careful characterization of all subsystems along data stream path.
The networking subsystem results in being the most critical one in terms of
absolute value and fluctuations of its response latency.
Our envisioned solution to this issue is NaNet, a FPGA-based PCIe Network
Interface Card (NIC) design featuring a configurable and extensible set of
network channels with direct access through GPUDirect to NVIDIA Fermi/Kepler
GPU memories.
NaNet design currently supports both standard - GbE (1000BASE-T) and 10GbE
(10Base-R) - and custom - 34~Gbps APElink and 2.5~Gbps deterministic latency
KM3link - channels, but its modularity allows for a straightforward inclusion
of other link technologies.
To avoid host OS intervention on data stream and remove a possible source of
jitter, the design includes a network/transport layer offload module with
cycle-accurate, upper-bound latency, supporting UDP, KM3link Time Division
Multiplexing and APElink protocols.
After NaNet architecture description and its latency/bandwidth
characterization for all supported links, two real world use cases will be
presented: the GPU-based low level trigger for the RICH detector in the NA62
experiment at CERN and the on-/off-shore data link for KM3 underwater neutrino
telescope
NaNet:a low-latency NIC enabling GPU-based, real-time low level trigger systems
We implemented the NaNet FPGA-based PCI2 Gen2 GbE/APElink NIC, featuring
GPUDirect RDMA capabilities and UDP protocol management offloading. NaNet is
able to receive a UDP input data stream from its GbE interface and redirect it,
without any intermediate buffering or CPU intervention, to the memory of a
Fermi/Kepler GPU hosted on the same PCIe bus, provided that the two devices
share the same upstream root complex. Synthetic benchmarks for latency and
bandwidth are presented. We describe how NaNet can be employed in the prototype
of the GPU-based RICH low-level trigger processor of the NA62 CERN experiment,
to implement the data link between the TEL62 readout boards and the low level
trigger processor. Results for the throughput and latency of the integrated
system are presented and discussed.Comment: Proceedings for the 20th International Conference on Computing in
High Energy and Nuclear Physics (CHEP
Peak Performance – Remote Memory Revisited
Many database systems share a need for large amounts of
fast storage. However, economies of scale limit the utility
of extending a single machine with an arbitrary amount
of memory. The recent broad availability of the zero-copy
data transfer protocol RDMA over low-latency and high
throughput network connections such as InfiniBand prompts
us to revisit the long-proposed usage of memory provided
by remote machines. In this paper, we present a solution
to make use of remote memory without manipulation of the
operating system, and investigate the impact on database
performance
Building Near-Real-Time Processing Pipelines with the Spark-MPI Platform
Advances in detectors and computational technologies provide new
opportunities for applied research and the fundamental sciences. Concurrently,
dramatic increases in the three Vs (Volume, Velocity, and Variety) of
experimental data and the scale of computational tasks produced the demand for
new real-time processing systems at experimental facilities. Recently, this
demand was addressed by the Spark-MPI approach connecting the Spark
data-intensive platform with the MPI high-performance framework. In contrast
with existing data management and analytics systems, Spark introduced a new
middleware based on resilient distributed datasets (RDDs), which decoupled
various data sources from high-level processing algorithms. The RDD middleware
significantly advanced the scope of data-intensive applications, spreading from
SQL queries to machine learning to graph processing. Spark-MPI further extended
the Spark ecosystem with the MPI applications using the Process Management
Interface. The paper explores this integrated platform within the context of
online ptychographic and tomographic reconstruction pipelines.Comment: New York Scientific Data Summit, August 6-9, 201
Performance characterization of containerization for HPC workloads on InfiniBand clusters: an empirical study
Containerization technology offers an appealing alternative for encapsulating and operating applications (and all their dependencies) without being constrained by the performance penalties of using Virtual Machines and, as a result, has got the interest of the High-Performance Computing (HPC) community to obtain fast, customized, portable, flexible, and reproducible deployments of their workloads. Previous work on this area has demonstrated that containerized HPC applications can exploit InfiniBand networks, but has ignored the potential of multi-container deployments which partition the processes that belong to each application into multiple containers in each host. Partitioning HPC applications has demonstrated to be useful when using virtual machines by constraining them to a single NUMA (Non-Uniform Memory Access) domain. This paper conducts a systematical study on the performance of multi-container deployments with different network fabrics and protocols, focusing especially on Infiniband networks. We analyze the impact of container granularity and its potential to exploit processor and memory affinity to improve applications’ performance. Our results show that default Singularity can achieve near bare-metal performance but does not support fine-grain multi-container deployments. Docker and Singularity-instance have similar behavior in terms of the performance of deployment schemes with different container granularity and affinity. This behavior differs for the several network fabrics and protocols, and depends as well on the application communication patterns and the message size. Moreover, deployments on Infiniband are also more impacted by the computation and memory allocation, and because of that, they can exploit the affinity better.We thank Lenovo for providing the testbed to run the experiments in this paper. This work was partially supported by Lenovo as part of Lenovo-BSC collaboration agreement, by the Spanish Government under contract PID2019-107255GB-C22, and by the Generalitat de Catalunya under contract 2017-SGR-1414 and under Grant No. 2020 FI-B 00257.Peer ReviewedPostprint (published version
RDMA mechanisms for columnar data in analytical environments
Dissertação de mestrado integrado em Engenharia InformáticaThe amount of data in information systems is growing constantly and, as a consequence, the
complexity of analytical processing is greater. There are several storage solutions to persist
this information, with different architectures targeting different use cases. For analytical
processing, storage solutions with a column-oriented format are particularly relevant due
to the convenient placement of the data in persistent storage and the closer mapping to
in-memory processing.
The access to the database is typically remote and has overhead associated, mainly when
it is necessary to obtain the same data multiple times. Thus, it is desirable to have a cache
on the processing side and there are solutions for this. The problem with the existing so lutions is the overhead introduced by network latency and memory-copy between logical
layers. Remote Direct Memory Access (RDMA) mechanisms have the potential to help min imize this overhead. Furthermore, this type of mechanism is indicated for large amounts of
data because zero-copy has more impact as the data volume increases. One of the problems
associated with RDMA mechanisms is the complexity of development. This complexity is
induced by its different development paradigm when compared to other network commu nication protocols, for example, TCP.
Aiming to improve the efficiency of analytical processing, this dissertation presents a dis tributed cache that takes advantage of RDMA mechanisms to improve analytical processing
performance. The cache abstracts the intricacies of RDMA mechanisms and is developed
as a middleware making it transparent to take advantage of this technology. Moreover, this
technique could be used in other contexts where a distributed cache makes sense, such as
a set of replicated web servers that access the same database.A quantidade de informação nos sistemas informáticos tem vindo a aumentar e consequentemente, a complexidade do processamento analítico torna-se maior. Existem diversas soluções para o armazenamento de dados com diferentes arquiteturas e indicadas para determinados casos de uso. Num contexto de processamento analítico, uma solução com o modelo de dados colunar e especialmente relevante devido à disposição conveniente dos dados em disco e a sua proximidade com o mapeamento em memória desses mesmos dados. Muitas vezes, o acesso aos dados é feito remotamente e isso traz algum overhead, principalmente quando é necessário aceder aos mesmos dados mais do que uma vez. Posto isto, é vantajoso fazer caching dos dados e já existem soluções para esse efeito. O overhead introduzido pela latência da rede e cópia de buffers entre camadas lógicas é o principal problema das soluções existentes. Os mecanismos de acesso direto à memória remota (RDMA - Remote Direct Memory Access) tem o potencial de melhorar o desempenho neste cenário. Para além disso, este tipo de tecnologia faz sentido em sistemas com grandes quantidades de dados, nos quais o acesso direto pode ter um impacto ainda maior por ser zero-copy. Um dos problemas associados com mecanismos RDMA é a complexidade de desenvolvimento. Esta complexidade é causada pelo paradigma de desenvolvimento completamente diferente de outros protocolos de comunicação, como por exemplo, TCP. Tendo em vista melhorar a eficiência do processamento analítico, esta dissertação propõe uma solução de cache distribuída que tira partido de mecanismos de acesso direto a memoria remota (RDMA). A cache abstrai as particularidades dos mecanismos RDMA e é disponibilizada como middleware, tornando a utilização desta tecnologia completamente transparente. Esta solução visa os sistemas de processamento analítico, mas poderá ser utilizada noutros contextos em que uma cache distribuída faça sentido, como por exemplo num conjunto de servidores web replicados que acedem a mesma base de dados
- …