Two-level quantum systems, qubits, are not the only basis for quantum
computation. Advantages exist in using qudits, d-level quantum systems, as the
basic carrier of quantum information. We show that color codes, a class of
topological quantum codes with remarkable transversality properties, can be
generalized to the qudit paradigm. In recent developments it was found that in
three spatial dimensions a qubit color code can support a transversal
non-Clifford gate, and that in higher spatial dimensions additional
non-Clifford gates can be found, saturating Bravyi and K\"onig's bound [Phys.
Rev. Lett. 110, 170503 (2013)]. Furthermore, by using gauge fixing techniques,
an effective set of Clifford gates can be achieved, removing the need for state
distillation. We show that the qudit color code can support the qudit analogues
of these gates, and show that in higher spatial dimensions a color code can
support a phase gate from higher levels of the Clifford hierarchy which can be
proven to saturate Bravyi and K\"onig's bound in all but a finite number of
special cases. The methodology used is a generalisation of Bravyi and Haah's
method of triorthogonal matrices [Phys. Rev. A 86 052329 (2012)], which may be
of independent interest. For completeness, we show explicitly that the qudit
color codes generalize to gauge color codes, and share the many of the
favorable properties of their qubit counterparts.Comment: Authors' final cop
