Generative flow networks (GFlowNets) are a family of algorithms for training
a sequential sampler of discrete objects under an unnormalized target density
and have been successfully used for various probabilistic modeling tasks.
Existing training objectives for GFlowNets are either local to states or
transitions, or propagate a reward signal over an entire sampling trajectory.
We argue that these alternatives represent opposite ends of a gradient
bias-variance tradeoff and propose a way to exploit this tradeoff to mitigate
its harmful effects. Inspired by the TD(位) algorithm in reinforcement
learning, we introduce subtrajectory balance or SubTB(位), a GFlowNet
training objective that can learn from partial action subsequences of varying
lengths. We show that SubTB(位) accelerates sampler convergence in
previously studied and new environments and enables training GFlowNets in
environments with longer action sequences and sparser reward landscapes than
what was possible before. We also perform a comparative analysis of stochastic
gradient dynamics, shedding light on the bias-variance tradeoff in GFlowNet
training and the advantages of subtrajectory balance.Comment: ICML 202