Lipschitz continuity is a simple yet crucial functional property of any
predictive model for it lies at the core of the model's robustness,
generalisation, as well as adversarial vulnerability. Our aim is to thoroughly
investigate and characterise the Lipschitz behaviour of the functions realised
by neural networks. Thus, we carry out an empirical investigation in a range of
different settings (namely, architectures, losses, optimisers, label noise, and
more) by exhausting the limits of the simplest and the most general lower and
upper bounds. Although motivated primarily by computational hardness results,
this choice nevertheless turns out to be rather resourceful and sheds light on
several fundamental and intriguing traits of the Lipschitz continuity of neural
network functions, which we also supplement with suitable theoretical
arguments. As a highlight of this investigation, we identify a striking double
descent trend in both upper and lower bounds to the Lipschitz constant with
increasing network width -- which tightly aligns with the typical double
descent trend in the test loss. Lastly, we touch upon the seeming
(counter-intuitive) decline of the Lipschitz constant in the presence of label
noise