Flow experiments are conducted in a two-stage compound open-channel, with varying intensity of the velocity difference between the main channel (deep part) and the floodplain (shallower part), using a large-scale free surface PIV technique (LS-PIV). For all investigated flows, a shear layer develops at the interface between main channel and floodplain, characterised by a peak of turbulent shear stress. Yet, two different kinds of shear layer could be identified. The first kind is characterised by the presence of large-scale quasi-periodic structures of Kelvin-Helmholtz type which are growing in downstream direction, whereas the second kind is characterised by smaller-scale vortical structures without quasi-periodicity and which do not grow in downstream direction. The shear parameter λ=(U2−U1)/(U2+U1), where U1 and U2 are defined as the velocities outside the shear layer, is identified as a key parameter to distinguish between these two types of shear layers, supporting a result from Proust et al. (Water Resour Res 53: 3387–3406, 2017). A physical interpretation of the λ-criterion is proposed, based on the inhibiting effect of ambient turbulence (the turbulence level outside the shear layer) on the emergence of Kelvin-Helmholtz structures. Accordingly, the threshold value of λ, above which large-scale structures can develop, is dependent on the level of the ambient turbulence. Despite their very different behaviours, the two types of shear layer have the same efficiency to generate turbulent shear stress for a given velocity difference across the shear layer, except for λ-values close to the threshold value