The emergent few-shot reasoning capabilities of Large Language Models (LLMs)
have excited the natural language and machine learning community over recent
years. Despite of numerous successful applications, the underlying mechanism of
such in-context capabilities still remains unclear. In this work, we
hypothesize that the learned \textit{semantics} of language tokens do the most
heavy lifting during the reasoning process. Different from human's symbolic
reasoning process, the semantic representations of LLMs could create strong
connections among tokens, thus composing a superficial logical chain. To test
our hypothesis, we decouple semantics from the language reasoning process and
evaluate three kinds of reasoning abilities, i.e., deduction, induction and
abduction. Our findings reveal that semantics play a vital role in LLMs'
in-context reasoning -- LLMs perform significantly better when semantics are
consistent with commonsense but struggle to solve symbolic or
counter-commonsense reasoning tasks by leveraging in-context new knowledge. The
surprising observations question whether modern LLMs have mastered the
inductive, deductive and abductive reasoning abilities as in human
intelligence, and motivate research on unveiling the magic existing within the
black-box LLMs. On the whole, our analysis provides a novel perspective on the
role of semantics in developing and evaluating language models' reasoning
abilities. Code is available at {\url{https://github.com/XiaojuanTang/ICSR}}