We investigate refined algebraic quantisation with group averaging in a
constrained Hamiltonian system whose gauge group is the connected component of
the lower triangular subgroup of SL(2,R). The unreduced phase space is
T^*R^{p+q} with p>0 and q>0, and the system has a distinguished classical
o(p,q) observable algebra. Group averaging with the geometric average of the
right and left invariant measures, invariant under the group inverse, yields a
Hilbert space that carries a maximally degenerate principal unitary series
representation of O(p,q). The representation is nontrivial iff (p,q) is not
(1,1), which is also the condition for the classical reduced phase space to be
a symplectic manifold up to a singular subset of measure zero. We present a
detailed comparison to an algebraic quantisation that imposes the constraints
in the sense H_a Psi = 0 and postulates self-adjointness of the o(p,q)
observables. Under certain technical assumptions that parallel those of the
group averaging theory, this algebraic quantisation gives no quantum theory
when (p,q) = (1,2) or (2,1), or when p>1, q>1 and p+q is odd.Comment: 30 pages. LaTeX with amsfonts, amsmath, amssymb. (v4: Typos
corrected. Published version.