We propose a unified electric-magnetic symmetry group in Yang-Mills theory, which we call the skeleton group. We work in the context of non-abelian unbroken gauge symmetry, and provide evidence for our proposal by relating the representation theory of the skeleton group to the labelling and fusion rules of charge sectors, and by showing how the skeleton group arises naturally in a gauge-fixed description of the theory. In particular we show that the labels of electric, magnetic and dyonic sectors in non-abelian Yang-Mills theory can be interpreted in terms of irreducible representations of the skeleton group. Decomposing tensor products of these representations thus gives candidate fusion rules for these charge sectors. We demonstrate consistency of these fusion rules with the known fusion rules of the purely electric or magnetic sectors, and extract new predictions for the fusion rules of dyonic sectors in particular cases. We also implement S-duality and show that the fusion rules obtained from the skeleton group commute with S-duality. As further evidence for the relevance of the skeleton group we consider a generalisation of ’t Hooft’s abelian gauge fixing procedure. We show that the skeleton group plays the role of an effective symmetry in this gauge, and argue that this gauge is particularly useful for exploring phases of the theory which generalise Alice electrodynamics