Large Language Models (LLMs) have demonstrated remarkable performance across
a wide range of natural language processing tasks. However, their remarkable
parameter size and their impressive high requirement of computing resources
pose challenges for their practical deployment. Recent research has revealed
that specific capabilities of LLMs, such as numerical reasoning, can be
transferred to smaller models through distillation. Some studies explore the
potential of leveraging LLMs to perform table-based reasoning. Nevertheless,
prior to our work, there has been no investigation into the prospect of
specialising table reasoning skills in smaller models specifically tailored for
table-to-text generation tasks. In this paper, we propose a novel table-based
reasoning distillation, with the aim of distilling distilling LLMs into
tailored, smaller models specifically designed for table-based reasoning task.
Experimental results have shown that a 0.22 billion parameter model
(Flan-T5-base) fine-tuned using distilled data, not only achieves a significant
improvement compared to traditionally fine-tuned baselines but also surpasses
specific LLMs like gpt-3.5-turbo on the scientific table-to-text generation
dataset (SciGen). The code and data are released in
https://github.com/Bernard-Yang/TableDistill