We perform a comprehensive analysis of the superorbital modulation in the ultracompact X-ray source 4U 1820-303, consisting of a white dwarf accreting onto a neutron star. Based on RXTE data, we measure the fractional amplitude of the source superorbital variability (with a 170-d quasi-period) in the folded and averaged light curves, and find it to be by a factor of about 2. As proposed before, the superorbital variability can be explained by oscillations of the binary eccentricity. We now present detailed calculations of the eccentricity-dependent flow through the inner Lagrangian point, and find a maximum of the eccentricity of about 0.004 is sufficient to explain the observed fractional amplitude. We then study hierarchical triple models yielding the required quasi-periodic eccentricity oscillations through the Kozai process. We find the resulting theoretical light curves to match well the observed ones. We constrain the ratio of the semimajor axes of the outer and inner systems, the component masses, and the inclination angle between the inner and outer orbits. Last but not least, we discover a remarkable and puzzling synchronization between the observed period of the superorbital variability (equal to the period of the eccentricity oscillations in our model) and the period of the general-relativistic periastron precession of the binary
