Complex feature extractors are widely employed for text representation
building. However, these complex feature extractors make the NLP systems prone
to overfitting especially when the downstream training datasets are relatively
small, which is the case for several discourse parsing tasks. Thus, we propose
an alternative lightweight neural architecture that removes multiple complex
feature extractors and only utilizes learnable self-attention modules to
indirectly exploit pretrained neural language models, in order to maximally
preserve the generalizability of pre-trained language models. Experiments on
three common discourse parsing tasks show that powered by recent pretrained
language models, the lightweight architecture consisting of only two
self-attention layers obtains much better generalizability and robustness.
Meanwhile, it achieves comparable or even better system performance with fewer
learnable parameters and less processing time