Parallel tempering (PT), also known as replica exchange, is the go-to
workhorse for simulations of multi-modal distributions. The key to the success
of PT is to adopt efficient swap schemes. The popular deterministic even-odd
(DEO) scheme exploits the non-reversibility property and has successfully
reduced the communication cost from O(P2) to O(P) given sufficiently many
P chains. However, such an innovation largely disappears in big data due to
the limited chains and few bias-corrected swaps. To handle this issue, we
generalize the DEO scheme to promote non-reversibility and propose a few
solutions to tackle the underlying bias caused by the geometric stopping time.
Notably, in big data scenarios, we obtain an appealing communication cost
O(PlogP) based on the optimal window size. In addition, we also adopt
stochastic gradient descent (SGD) with large and constant learning rates as
exploration kernels. Such a user-friendly nature enables us to conduct
approximation tasks for complex posteriors without much tuning costs.Comment: Accepted by AAAI 202