The presence of widespread terrigenous material of an ice-rafted origin in Quaternary sediments of the Southern Ocean has been recognized for almost 150 years. Normally this material has been ascribed to deposits from icebergs of continental Antarctic origin. However, during Quaternary glaciations there have been periods of extensive land ice across the sub-Antarctic, on both islands scattered around most of the circumpolar extent of the Southern Ocean, as well as in Patagonia, so providing alternative sources for debris-carrying icebergs. Here a relatively high resolution ocean and iceberg model is used to study the potential distribution of ice-rafted debris (IRD) from the range of past ice sources around the Southern Ocean. It is shown that IRD found in marine cores of the Southern Ocean is most likely to have derived from the Antarctic continent in some regions, particularly of the South Atlantic, but that for extensive regions of the Southern Ocean sub-Antarctic sources of IRD, rather than the continent itself, are more likely. This is particularly true equatorward of 55oS, away from the core continental iceberg outflow from the Weddell gyre. It is argued that the glaciated sub-Antarctic cannot be neglected in explaining past IRD records in the Southern Ocean. This has implications not just for reconstructing the history of glaciation in the sub-Antarctic, but also for understanding past variation in the upper ocean circulation within the Quaternary Southern Ocean