Large pre-trained neural networks are ubiquitous and critical to the success
of many downstream tasks in natural language processing and computer vision.
However, within the field of web information retrieval, there is a stark
contrast in the lack of similarly flexible and powerful pre-trained models that
can properly parse webpages. Consequently, we believe that common machine
learning tasks like content extraction and information mining from webpages
have low-hanging gains that yet remain untapped.
We aim to close the gap by introducing an agnostic deep graph neural network
feature extractor that can ingest webpage structures, pre-train self-supervised
on massive unlabeled data, and fine-tune to arbitrary tasks on webpages
effectually.
Finally, we show that our pre-trained model achieves state-of-the-art results
using multiple datasets on two very different benchmarks: webpage boilerplate
removal and genre classification, thus lending support to its potential
application in diverse downstream tasks.Comment: Submitted to CIKM '2