2 research outputs found
TabDDPM: Modelling Tabular Data with Diffusion Models
Denoising diffusion probabilistic models are currently becoming the leading
paradigm of generative modeling for many important data modalities. Being the
most prevalent in the computer vision community, diffusion models have also
recently gained some attention in other domains, including speech, NLP, and
graph-like data. In this work, we investigate if the framework of diffusion
models can be advantageous for general tabular problems, where datapoints are
typically represented by vectors of heterogeneous features. The inherent
heterogeneity of tabular data makes it quite challenging for accurate modeling,
since the individual features can be of completely different nature, i.e., some
of them can be continuous and some of them can be discrete. To address such
data types, we introduce TabDDPM -- a diffusion model that can be universally
applied to any tabular dataset and handles any type of feature. We extensively
evaluate TabDDPM on a wide set of benchmarks and demonstrate its superiority
over existing GAN/VAE alternatives, which is consistent with the advantage of
diffusion models in other fields. Additionally, we show that TabDDPM is
eligible for privacy-oriented setups, where the original datapoints cannot be
publicly shared.Comment: code https://github.com/rotot0/tab-ddp
TabR: Unlocking the Power of Retrieval-Augmented Tabular Deep Learning
Deep learning (DL) models for tabular data problems are receiving
increasingly more attention, while the algorithms based on gradient-boosted
decision trees (GBDT) remain a strong go-to solution. Following the recent
trends in other domains, such as natural language processing and computer
vision, several retrieval-augmented tabular DL models have been recently
proposed. For a given target object, a retrieval-based model retrieves other
relevant objects, such as the nearest neighbors, from the available (training)
data and uses their features or even labels to make a better prediction.
However, we show that the existing retrieval-based tabular DL solutions provide
only minor, if any, benefits over the properly tuned simple retrieval-free
baselines. Thus, it remains unclear whether the retrieval-based approach is a
worthy direction for tabular DL.
In this work, we give a strong positive answer to this question. We start by
incrementally augmenting a simple feed-forward architecture with an
attention-like retrieval component similar to those of many (tabular)
retrieval-based models. Then, we highlight several details of the attention
mechanism that turn out to have a massive impact on the performance on tabular
data problems, but that were not explored in prior work. As a result, we design
TabR -- a simple retrieval-based tabular DL model which, on a set of public
benchmarks, demonstrates the best average performance among tabular DL models,
becomes the new state-of-the-art on several datasets, and even outperforms GBDT
models on the recently proposed ``GBDT-friendly'' benchmark (see the first
figure).Comment: Code: https://github.com/yandex-research/tabular-dl-tab