The formation of exotic orders in the vicinity of magnetic quantum critical points is a promising research field. Particularly motivating is the discovery of high temperature superconductors with novel ordering symmetries. In this thesis we extend the fermionic quantum order by disorder approach to analyse phase formation in the vicinity of antiferromagnetic and spin-triplet nematic quantum critical points. We first investigate a fundamental model for itinerant antiferromagnetism. Already at mean field level this model exhibits exciting behaviour. Nesting and van-Hove physics drive incommensurate order and a discontinuous transition to the antiferromagnetic phase. These phenomena can be related to the Fulde-Ferrell-Larkin-Ovchinnikov state of superconductivity. Additional phenomena are driven by fluctuations. Applying the fermionic order by disorder approach we discover a multitude of phases. In addition to the antiferromagnetic order we consider d-wave superconductivity and d-wave bond density wave order. Finally, we analyse mode-mode coupling between all pairs of these three orders. The resulting phase diagram is in qualitative agreement with experimental findings in cuprate superconductors and offers an alternative perspective to the standard theoretical approaches. In the absence of an antiferromagnetic quantum critical point we instead consider superconducting order driven by ferromagnetic spin fluctuations. We analyse a model for spin-triplet nematic order and investigate its susceptibility to the formation of spatially modulated nematic order, fluctuation driven p-wave superconductivity and composite pair density wave order. Here, the presence of spin-triplet nematic order enhances the superconducting transition temperature dramatically
