Among the many tasks that Large Language Models (LLMs) have revolutionized is
text classification. However, existing approaches for applying pretrained LLMs
to text classification predominantly rely on using single token outputs from
only the last layer of hidden states. As a result, they suffer from limitations
in efficiency, task-specificity, and interpretability. In our work, we
contribute an approach that uses all internal representations by employing
multiple pooling strategies on all activation and hidden states. Our novel
lightweight strategy, Sparsify-then-Classify (STC) first sparsifies
task-specific features layer-by-layer, then aggregates across layers for text
classification. STC can be applied as a seamless plug-and-play module on top of
existing LLMs. Our experiments on a comprehensive set of models and datasets
demonstrate that STC not only consistently improves the classification
performance of pretrained and fine-tuned models, but is also more efficient for
both training and inference, and is more intrinsically interpretable.Comment: 23 pages, 5 figures, 8 tables Code available at
https://github.com/difanj0713/Sparsify-then-Classif