We introduce an analysis technique for understanding compositional structure present in the vector representations used by neural networks. The inner workings of neural networks are notoriously difficult to understand, and in particular it is far from clear how they manage to perform remarkably well on tasks that depend on compositional structure even though they use continuous vector representations with no obvious compositional structure. Using our analysis technique, we show that the representations of these models can be closely approximated by Tensor Product Representations, a type of interpretable structure that lends significant insight into the workings of these hard-to-interpret models
