We are constantly processing sequential information in our day-to-day life, from listening to a piece of music (processing a stream of notes), watching a movie (processing a series of scenes), to having conversations with people around us (processing a stream of syllables, words, and sentences). What are the neural and computational principles underlying this ubiquitous cognitive process? In this thesis, I first review the background and prior studies regarding the neural and computational mechanisms of real-life sequence processing and present our research questions. I then present four research projects to answer those questions: By combining neuroimaging data analysis and computational modeling, I discovered the neural phenomena of integrating and forgetting temporal information during naturalistic sequence processing in the human cerebral cortex. Furthermore, I identified computational principles (e.g., hierarchical architecture) and processes (e.g., dynamical context gating) which can help to explain the neural state changes observed during naturalistic processing. These neural and computational findings not only validate the existing components of hierarchical temporal integration theory, but also rule out alternative models, and propose important new elements of the theory, including context gating at event boundaries. I next explored the computations for natural language processing in brains and machines, by (1) applying our neuroscience-inspired methods to examine the timescale and functional organization of neural network language models, thereby revealing their own architecture for processing information over multiple timescales; and by (2) investigating the context and entity representations in two neural networks with brain-inspired architectures, thereby revealing a gap between brain-inspired and performance-optimized architectures. Finally, I discuss the positions and contributions of our findings in the field and some future directions
