311 research outputs found
TinyKG: Memory-Efficient Training Framework for Knowledge Graph Neural Recommender Systems
There has been an explosion of interest in designing various Knowledge Graph
Neural Networks (KGNNs), which achieve state-of-the-art performance and provide
great explainability for recommendation. The promising performance is mainly
resulting from their capability of capturing high-order proximity messages over
the knowledge graphs. However, training KGNNs at scale is challenging due to
the high memory usage. In the forward pass, the automatic differentiation
engines (\textsl{e.g.}, TensorFlow/PyTorch) generally need to cache all
intermediate activation maps in order to compute gradients in the backward
pass, which leads to a large GPU memory footprint. Existing work solves this
problem by utilizing multi-GPU distributed frameworks. Nonetheless, this poses
a practical challenge when seeking to deploy KGNNs in memory-constrained
environments, especially for industry-scale graphs.
Here we present TinyKG, a memory-efficient GPU-based training framework for
KGNNs for the tasks of recommendation. Specifically, TinyKG uses exact
activations in the forward pass while storing a quantized version of
activations in the GPU buffers. During the backward pass, these low-precision
activations are dequantized back to full-precision tensors, in order to compute
gradients. To reduce the quantization errors, TinyKG applies a simple yet
effective quantization algorithm to compress the activations, which ensures
unbiasedness with low variance. As such, the training memory footprint of KGNNs
is largely reduced with negligible accuracy loss. To evaluate the performance
of our TinyKG, we conduct comprehensive experiments on real-world datasets. We
found that our TinyKG with INT2 quantization aggressively reduces the memory
footprint of activation maps with , only with loss in accuracy,
allowing us to deploy KGNNs on memory-constrained devices
Multi-Target Prediction: A Unifying View on Problems and Methods
Multi-target prediction (MTP) is concerned with the simultaneous prediction
of multiple target variables of diverse type. Due to its enormous application
potential, it has developed into an active and rapidly expanding research field
that combines several subfields of machine learning, including multivariate
regression, multi-label classification, multi-task learning, dyadic prediction,
zero-shot learning, network inference, and matrix completion. In this paper, we
present a unifying view on MTP problems and methods. First, we formally discuss
commonalities and differences between existing MTP problems. To this end, we
introduce a general framework that covers the above subfields as special cases.
As a second contribution, we provide a structured overview of MTP methods. This
is accomplished by identifying a number of key properties, which distinguish
such methods and determine their suitability for different types of problems.
Finally, we also discuss a few challenges for future research
- …