In this essay we are searching for neural correlates of ‘doing mathematical physics'. We introduce a toy model of a mathematical physicist, a brain connected with the outside world only by vision and saccadic eye movements and interacting with a computer screen. First, we describe the neuroanatomy of the visuo-saccadic system and Listing's law, which binds saccades and the optics of the eye. Then we explain space-time transformations in the superior colliculus, the performance of a canonical cortical circuit in the frontal eye field and finally the recurrent interaction of both areas, which leads to a coherent percept of space in spite of saccades. This sets the stage in the brain for doing mathematical physics, which is analyzed in simple example
