We apply periodic orbit theory to a two-dimensional non-integrable billiard
system whose boundary is varied smoothly from a circular to an equilateral
triangular shape. Although the classical dynamics becomes chaotic with
increasing triangular deformation, it exhibits an astonishingly pronounced
shell effect on its way through the shape transition. A semiclassical analysis
reveals that this shell effect emerges from a codimension-two bifurcation of
the triangular periodic orbit. Gutzwiller's semiclassical trace formula, using
a global uniform approximation for the bifurcation of the triangular orbit and
including the contributions of the other isolated orbits, describes very well
the coarse-grained quantum-mechanical level density of this system. We also
discuss the role of discrete symmetry for the large shell effect obtained here.Comment: 14 pages REVTeX4, 16 figures, version to appear in Phys. Rev. E.
Qualities of some figures are lowered to reduce their sizes. Original figures
are available at http://www.phys.nitech.ac.jp/~arita/papers/tricirc
