Large Language Models (LLMs) can justify or criticize their predictions
through discussion with other models or humans, thereby enhancing their
intrinsic understanding of instances. While proactive discussions enhance
performance, this approach is currently limited to the inference phase. In this
context, we posit a hypothesis: learning interactive discussions during
training can improve understanding for the instances in the training step and
proficiency in logical/critical thinking ability and verbalized expression of
the model in the inference step. Our proposed SAIE training method involves
both supportive and adversarial discussions between the learner and partner
models. The learner model receives a remark from the partner through the
discussion, and the parameters of the learner model are then updated based on
this remark. That is, the teacher signal dynamically adjusts in response to the
evolving model output throughout the training step. By bolstering the capacity
for discussion and comprehension of instances, our experiments across datasets,
including GSM8K, CommonsenseQA, and MMLU, reveal that models fine-tuned with
our method consistently surpass those trained with standard fine-tuning
techniques. Moreover, our approach demonstrates superior performance in
multi-agent inference scenarios, boosting the models' reasoning abilities at
the inference step.Comment: Work in progres