We report the first observation of ionization of Rydberg atoms by subpicosecond, circularly polarized THz radiation. The field amplitude in these pulses is non-negligible for only one-quarter of an optical cycle. The experiment is performed in the short-pulse regime, where the duration of the ionizing pulse is shorter than the classical Kepler period of the Rydberg electron. We find that the ionization probability for these atoms is remarkably insensitive to the time-varying polarization of the THz field