We discuss the possibility of implementing asynchronous replica-exchange (or
parallel tempering) molecular dynamics. In our scheme, the exchange attempts
are driven by asynchronous messages sent by one of the computing nodes, so that
different replicas are allowed to perform a different number of time-steps
between subsequent attempts. The implementation is simple and based on the
message-passing interface (MPI). We illustrate the advantages of our scheme
with respect to the standard synchronous algorithm and we benchmark it for a
model Lennard-Jones liquid on an IBM-LS21 blade center cluster.Comment: Preprint of Proceeding for CSFI 200

Bussi, Giovanni

English

arXiv

We discuss the possibility of implementing asynchronous replica-exchange (or parallel tempering) molecular dynamics. In our scheme, the exchange attempts are driven by asynchronous messages sent by one of the computing nodes, so that different replicas are allowed to perform a different number, of time, steps between subsequent attempts. The implementation is simple and based on the message-passing interface (MPI). illustrate the advantages of our scheme with respect to the standard synchronous algorithm and we benchmark it for a model Lennard-Jones liquid on an IBM-LS21 blade center cluster

Bussi, G.

Sissa Digital Library

A simple asynchronous replica-exchange implementation

Bussi G

SISSA Digital Library

We discuss the possibility of implementing asynchronous replicaexchange (or parallel tempering) molecular dynamics. In our scheme, the Exchange attempts are driven by asynchronous messages sent by one of the computing nodes,

so that different replicas are allowed to perform a different number of time steps between subsequent attempts. The implementation is simple and based on the

message-passing interface (MPI). We illustrate the advantages of our scheme with respect to the standard synchronous algorithm and we benchmark it for a model

Lennard-Jones liquid on an IBM-LS21 blade center cluster

Scientific Open-access Literature Archive and Repository

DOI 10.1393/ncc/i2009-10369-8Colloquia: CSFI 2008IL NUOVO CIMENTO Vol. 32 C, N. 2 Marzo-Aprile 2009A simple asynchronous replica-exchange implementationGiovanni Bussi(∗)Dipartimento di Fisica, Universita` di Modena e Reggio EmiliaCNR-INFM Center on nanoStructures and bioSystems at Surfaces (S3)via Campi 213, 41100 Modena, Italy(ricevuto il 15 Maggio 2009; pubblicato online il 3 Agosto 2009)Summary. — We discuss the possibility of implementing asynchronous replica-exchange (or parallel tempering) molecular dynamics. In our scheme, the exchangeattempts are driven by asynchronous messages sent by one of the computing nodes,so that different replicas are allowed to perform a different number of time stepsbetween subsequent attempts. The implementation is simple and based on themessage-passing interface (MPI). We illustrate the advantages of our scheme withrespect to the standard synchronous algorithm and we benchmark it for a modelLennard-Jones liquid on an IBM-LS21 blade center cluster.PACS 89.20.Ff – Computer science and technology.PACS 05.10.-a – Computational methods in statistical physics and nonlinear dy-namics.PACS 02.70.-c – Computational techniques; simulations.Parallel tempering is a popular method used to enhance sampling in Monte Carlo ormolecular dynamics simulations [1-3], with applications ranging from molecular biologyto statistical physics. In parallel tempering simulations, many replicas of the system aresimulated at different temperatures and their temperatures are exchanged in a MonteCarlo fashion. The algorithm is very flexible, and can be extended to simulation pa-rameters other than the temperature (see, e.g., refs. [4-6]). In this generalized form themethod is usually referred to as replica exchange. Furthermore, parallel tempering (orreplica exchange) can be easily combined with other enhanced-sampling methods [7, 8].A very important feature of replica-exchange simulations is that they allow exploitinglarge parallel machines: the standard implementation is done assigning a different com-puting node to each replica, so that inter-process communications are only needed whenattempting an exchange move, which is typically done with a prefixed stride. Moreover,parallel tempering can be coded in such a way that temperatures are exchanged insteadof coordinates. In this case, the amount of data that are transmitted between computingnodes is negligible, resulting in a theoretically perfect scalability.(∗) E-mail: gbussi@unimore.itc© Societa` Italiana di Fisica 6162 GIOVANNI BUSSIHowever, practical problems arise when the number of replicas grows. In fact, forhundreds of computing nodes, even the overhead of synchronization can be large, espe-cially when using short strides between subsequent attempts as suggested, e.g., in ref. [9].Furthermore, there is no warranty that the different nodes will take the same time to per-form the same number of steps. As an example, the frequency of updates of the neighborlist depends on the diffusion coefficient [10], which in turns depends on the temperature,so that high-temperature simulations are expected to be slower. The situation is evenworse when heterogeneous machines are used.A possible solution for these problems is the adoption of serial methods [11,12]. In thispaper we will discuss a different approach, namely the implementation of replica exchangein an asynchronous manner. The present implementation is simple and based on the MPIlibrary [13]. It is designed as a master-slave scheme, where a node is asynchronouslydriving the exchanges for the other replicas. To the best of our knowledge, the onlyother attempt to implement asynchronous parallel tempering is the SALSA frameworkintroduced in refs. [14,15].1. – ImplementationIn our implementation temperatures are exchanged rather than coordinates. Thisallows to minimize the amount of data that has to be communicated, at the price of aslightly more complex analysis of the resulting trajectories. Our asynchronous algorithmconsists of the iteration of the following steps:– Each node sends (asynchronously) its temperature to the master node, then startsto integrate molecular dynamics.– When Nx steps are elapsed since the last exchange attempt, the master node collectsthe temperatures of all the other nodes, sorts them, and establishes the exchangepattern (i.e. which pairs of nodes should attempt the exchange). Then, it sends(asynchronously) a message to each node (including itself) containing the rank ofthe partner node.– At every step, each node checks if a message arrived from the master. If theanswer is negative, computation is continued. Otherwise, a message is sent (asyn-chronously) to the partner node, containing the actual temperature, potential en-ergy and a uniformly distributed random number. Then, the node waits for thecorresponding message from the parter node. Once both temperatures and energiesare available, the random number generated by one of the two nodes (arbitrarilyfixed to be that with the lowest rank) is used to establish if the exchange is successfulusing the standard Metropolis rule [3]. In case of positive answer, the temperatureand velocities are properly scaled.As a further improvement, the rank of the master node is changed at each exchange,so that the master overhead (sorting replicas and sending/receiving many messages) isdistributed over the full set of nodes.Only two MPI calls need to be done in a blocking way: a) the collection of temper-atures done by the master node, since all the temperatures should be known to preparethe exchange pattern; b) the reception of the triplet of temperature, energy and randomnumber, which is needed to determine whether the temperature has to be changed ornot. All the other calls to the MPI library are asynchronous and allows for overlap ofA SIMPLE ASYNCHRONOUS REPLICA-EXCHANGE IMPLEMENTATION 63communication and computation. The scheme can be trivially extended to support non-nearest neighbor exchanges. When compared with the SALSA framework [14, 15], ourscheme is simpler and has the advantage that it is based on a standard MPI library anddo not require multi-thread capabilities.As a reference, we also implement a synchronous version, where the collection of thetemperature is done at every node with the prefixed stride using a MPI ALLGATHERcall. Here all the nodes are able to decide the exchange pattern and perform the exchangeafter the same prefixed number of steps.2. – ExampleWe test our implementation using a Lennard-Jones liquid. This system does not ex-hibit significant barriers, and parallel tempering is in principles not needed to properlysample the phase space. However, the diffusion of the replicas in temperature space issimilar to what is usually observed for simulation of solvated molecules, so that it can beused as a meaningful test case. We simulate 4000 particles at density ρ = 0.844, with cut-off in the interparticle interaction at 2.5. Throughout this section we use Lennard-Jonesunits for distances, energies and time. A neighbor-list containing pairs up to a distance3.0 is used, and it is updated when the displacement of at least one particle is largerthan half the skin width [10]. The integration time step is 0.005, and temperature is con-trolled using stochastic velocity rescaling [16]. Nr replicas are simulated at exponentiallydistributed temperatures in the range from Tmin = 0.7 to Tmax = Tmin × 1.0275Nr−1,which gives an exchange acceptance of approximately 30%, irrespectively of Nr. In theasynchronous implementation, exchanges between replicas are attempted when the mas-ter node has done Nx steps since the last attempt. Since the number of steps performedby the other nodes is not prefixed, the actual average number of steps between attempts〈Nx〉 is not equal to Nx. On the other hand, when the synchronous implementation isused the same number of steps is performed by all the nodes between subsequent at-tempts, so that 〈Nx〉 = Nx. All the calculations are performed on the BCX cluster atCINECA.2.1. Diffusion in temperature space. – A very important parameter in replica-exchangesimulations is the speed at which each replica diffuses in the temperature space. Thisspeed is inversely related to the average time required for a replica to cross the fulltemperature range from the lowest to the highest temperature. We here calculate thediffusion coefficient of the logarithm of the temperature in the case Nr = 48 for differentchoices of Nx in both the synchronous and asynchronous implementation. The resultsare plotted in fig. 1(a) as a function of the observed exchange stride 〈Nx〉. As is seen,diffusion in both cases is equivalent and only depends on the effective exchange stride.Moreover, as has been observed in ref. [9], the optimal performance is obtained by usingan exchange stride which is as small as possible.2.2. Scalability . – We then consider the overhead due to the exchanges as the numberof replicas Nr grows. To this aim we fix Nx = 5 for the synchronous implementationand Nx = 1 for the asynchronous implementation. For the latter, this choice gives apractical waiting time of 〈Nx〉 ≈ 5, so that the two algorithms are comparable. We thendefine the parallel efficiency λ as the ratio between the total number of steps performedby all the replicas in a prefixed wall-clock time and the same number as obtained duringa simulation without exchanges. In practice, this number should be equal to 1 if there64 GIOVANNI BUSSI0 10 20 30 40 50<Nx>1×10-32×10-33×10-34×10-35×10-3Dsyncasync0 25 50 75 100 125Nr0.20.40.60.81.0λ(b)(a)Fig. 1. – (a) Diffusion coefficient D in the log-temperature space, as a function of the averagenumber of waiting steps 〈Nx〉. (b) Parallel efficiency λ, as a function of the number of replicasNr. Both quantities are plotted for the synchronous (diamonds) and asynchronous (circles)implementations. See text for details.was no computational overhead, and decreases towards zero as the overhead increases.In fig. 1(b) we plot the efficiency λ as a function of Nr. It is clearly seen that thesynchronous implementation is much less efficient than the asynchronous one, and thatit gets slower and slower as Nr increases. On the other hand, the efficiency of theasynchronous implementation is very close to 1 even when more than a hundred replicasare used.3. – ConclusionsWe have shown an asynchronous implementation of replica-exchange molecular dy-namics, where the exchange attempts are driven by external commands sent by one of thereplicas. This implementation is based on the MPI library. Its performance is excellentand allows for a significant saving of computational effort in simulations with hundredsof replicas. The code is available on request.∗ ∗ ∗The author is grateful to C. Cavazzoni for carefully reading the manuscript. TheCINECA supercomputing center is acknowledged for the computational time.REFERENCES[1] Swendsen R. and Wang J., Phys. Rev. Lett., 57 (1986) 2607.[2] Hansmann U. H. E., Chem. Phys. Lett., 281 (1997) 140.[3] Sugita Y. and Okamoto Y., Chem. Phys. Lett., 314 (1999) 141.[4] Fukunishi H., Watanabe O. and Takada S., J. Chem. Phys., 116 (2002) 9058.[5] Jang S., Shin S. and Pak Y., Phys. Rev. Lett., 91 (2003) 58305.[6] Liu P., Kim B., Friesner R. A. and Berne B. J., Proc. Natl. Acad. Sci. U.S.A., 102(2005) 13749.[7] Sugita Y., Kitao A. and Okamoto Y., J. Chem. Phys., 113 (2000) 6042.A SIMPLE ASYNCHRONOUS REPLICA-EXCHANGE IMPLEMENTATION 65[8] Bussi G., Gervasio F. L., Laio A. and Parrinello M., J. Am. Chem. Soc., 128 (2006)13435.[9] Sindhikara D., Meng Y. and Roitberg A., J. Chem. Phys., 128 (2008) 024103.[10] Allen M. P. and Tildesley D. J., Computer Simulation of Liquids (Oxford UniversityPress, Oxford) 1987.[11] Marinari E. and Parisi G., Europhys. Lett., 19 (1992) 451.[12] Hagen M., Kim B., Liu P., Friesner R. and Berne B., J. Phys. Chem. B., 111 (2007)1416.[13] http://www.mpi-forum.org/.[14] Zhang L., Parashar M., Gallicchio E. and Levy R., Salsa: Scalable AsynchronousReplica Exchange for Parallel Molecular Dynamics Applications, in Proceedings of the 2006International Conference on Parallel Processing (IEEE Computer Society Washington,DC, USA) 2006, pp. 127-134.[15] Gallicchio E., Levy R. and Parashar M., J. Comput. Chem., 29 (2008) 788.[16] Bussi G., Donadio D. and Parrinello M., J. Chem. Phys., 126 (2007) 014101.

http://eprints.bice.rm.cnr.it/16497/1/ncc9383.pdf

A simple asynchronous replica-exchange implementation

Abstract

Similar works

Full text

Available Versions

Sissa Digital Library

SISSA Digital Library

Scientific Open-access Literature Archive and Repository