This thesis aimed to identify the underlying mechanism which drives fish to evolve into the schooling behaviour under predation. Previous works used to explain this formation by group benefits, individual differences or evolutionary trade-offs. However, conflicts from these explanations to the natural fish are considerable. Based on agent-based simulations and game-theoretic analyses, this thesis demonstrated that the intraspecies competition in a relatively homogeneous population is sufficient to cause the evolution of fish schools. It has been shown that when predators exhibit the ‘marginal predation’, that is, only catching prey on the margin of a group, prey fish must evolve to form a selfish herd, which will be increasingly crowded until no outer fish can enter it. In this case, fish should evolve to leave the group margin together, as the originally reported ‘collective departure strategy’, to expose the inner fish and share the risk. This adaptation then leads to the emergence of fish schools. The thesis further showed that the ‘marginal predation’ of predators and the ‘collective departure’ of prey is able to form an evolutionarily stable state in a coevolutionary system, which completed the explanation of the evolution of fish schools in nature
