We develop a method for calculating the correlation structure of the Cosmic
Microwave Background (CMB) using Feynman diagrams, when the CMB has been
modified by gravitational lensing, Faraday rotation, patchy reionization, or
other distorting effects. This method is used to calculate the bias of the
Hu-Okamoto quadratic estimator in reconstructing the lensing power spectrum up
to O(\phi^4) in the lensing potential ϕ. We consider both the diagonal
noise TTTT, EBEB, etc. and, for the first time, the off-diagonal noise TTTE,
TBEB, etc. The previously noted large O(\phi^4) term in the second order noise
is identified to come from a particular class of diagrams. It can be
significantly reduced by a reorganization of the ϕ expansion. These
improved estimators have almost no bias for the off-diagonal case involving
only one B component of the CMB, such as EEEB.Comment: 17 pages, 17 figure