Hybrid MPI+threads programming is gaining prominence as an alternative to the
traditional "MPI everywhere'" model to better handle the disproportionate
increase in the number of cores compared with other on-node resources. Current
implementations of these two models represent the two extreme cases of
communication resource sharing in modern MPI implementations. In the
MPI-everywhere model, each MPI process has a dedicated set of communication
resources (also known as endpoints), which is ideal for performance but is
resource wasteful. With MPI+threads, current MPI implementations share a single
communication endpoint for all threads, which is ideal for resource usage but
is hurtful for performance.
In this paper, we explore the tradeoff space between performance and
communication resource usage in MPI+threads environments. We first demonstrate
the two extreme cases---one where all threads share a single communication
endpoint and another where each thread gets its own dedicated communication
endpoint (similar to the MPI-everywhere model) and showcase the inefficiencies
in both these cases. Next, we perform a thorough analysis of the different
levels of resource sharing in the context of Mellanox InfiniBand. Using the
lessons learned from this analysis, we design an improved resource-sharing
model to produce \emph{scalable communication endpoints} that can achieve the
same performance as with dedicated communication resources per thread but using
just a third of the resources
