We study the instanton effective action of four-dimensional deformed N=4
supersymmetric Yang-Mills theory in the presence of constant, self-dual
Ramond-Ramond (R-R) 3-form background in type IIB superstring theory. We
compare the instanton effective action with the low-energy effective action on
D(-1)-branes in the D3-D(-1) system in the same background. We find that
discrepancy appears at the second order of the R-R background, which was also
observed in deformed N=2 theory. This discrepancy is resolved if the action of
the deformed gauge theory is improved by introducing a term with
coordinate-dependent gauge coupling. We obtain the same deformed instanton
effective action from super Yang-Mills theory in ten-dimensional
Omega-background by dimensional reduction. We also discuss another type of R-R
3-form background which corresponds to massive deformations of the instanton
effective action.Comment: 33 pages, no figures, typos corrected, version to appear in JHE