We investigate the response of self-interacting dark matter (SIDM) halos to the growth of galaxy potentials using idealized simulations, with each run in tandem with collisionless cold dark matter (CDM). We find that if the stellar potential strongly dominates in the central parts of a galaxy, then SIDM halos can be as dense as CDM halos on observable scales. For extreme cases, core collapse can occur, leading to SIDM halos that are denser and cuspier than their CDM counterparts. If the stellar potential is not dominant, then SIDM halos retain isothermal cores with densities far below CDM predictions. When a disk is present, the inner SIDM halo becomes more flattened in the disk plane than the CDM halo. These results are in excellent quantitative agreement with the predictions of Kaplinghat et al. We also simulated a cluster halo with a central stellar distribution similar to the brightest central galaxy of the cluster A2667. An SIDM halo simulated with the cross-section over mass provides a good match to the measured dark matter (DM) density profile, while an adiabatically contracted CDM halo is denser and cuspier. The profile of the same halo simulated with is not dense enough. Our findings are in agreement with previous results that is disfavored for DM collision velocities above about 1500 km s-1. More generally, the interaction between baryonic potentials and SIDM densities offers new directions for constraining SIDM cross-sections in galaxies where baryons are dynamically important