The second-order training methods can converge much faster than first-order
optimizers in DNN training. This is because the second-order training utilizes
the inversion of the second-order information (SOI) matrix to find a more
accurate descent direction and step size. However, the huge SOI matrices bring
significant computational and memory overheads in the traditional architectures
like GPU and CPU. On the other side, the ReRAM-based process-in-memory (PIM)
technology is suitable for the second-order training because of the following
three reasons: First, PIM's computation happens in memory, which reduces data
movement overheads; Second, ReRAM crossbars can compute SOI's inversion in
O(1) time; Third, if architected properly, ReRAM crossbars can
perform matrix inversion and vector-matrix multiplications which are important
to the second-order training algorithms.
Nevertheless, current ReRAM-based PIM techniques still face a key challenge
for accelerating the second-order training. The existing ReRAM-based matrix
inversion circuitry can only support 8-bit accuracy matrix inversion and the
computational precision is not sufficient for the second-order training that
needs at least 16-bit accurate matrix inversion. In this work, we propose a
method to achieve high-precision matrix inversion based on a proven 8-bit
matrix inversion (INV) circuitry and vector-matrix multiplication (VMM)
circuitry. We design \archname{}, a ReRAM-based PIM accelerator architecture
for the second-order training. Moreover, we propose a software mapping scheme
for \archname{} to further optimize the performance by fusing VMM and INV
crossbar. Experiment shows that \archname{} can achieve an average of
115.8×/11.4× speedup and 41.9×/12.8×energy saving
compared to a GPU counterpart and PipeLayer on large-scale DNNs.Comment: 13pages, 13 figure