Difference target propagation

Backpropagation has been the workhorse of recent successes of deep learning but it relies on infinitesimal effects (partial derivatives) in order to perform credit assignment. This could become a serious issue as one considers deeper and more non-linear functions, e.g., consider the extreme case of non-linearity where the relation between parameters and cost is actually discrete. Inspired by the biological implausibility of Backpropagation, this thesis proposes a novel approach, Target Propagation. The main idea is to compute targets rather than gradients, at each layer in which feedforward and feedback networks form Auto-Encoders. We show that a linear correction for the imperfectness of the Auto-Encoders, called Difference Target Propagation is very effective to make Target Propagation actually work, leading to results comparable to Backpropagation for deep networks with discrete and continuous units, Denoising Auto-Encoders and achieving state of the art for stochastic networks. In Chapters 1, we introduce several classical learning rules in Deep Neural Networks, including Backpropagation and more biological plausible learning rules. In Chapters 2 and 3, we introduce a novel approach, Target Propagation, more biological plausible learning rule than Backpropagation. In addition, we show that Target Propagation is comparable to Backpropagation in Deep Neural Networks.L'algorithme de r etropropagation a et e le cheval de bataille du succ es r ecent de l'apprentissage profond, mais elle s'appuie sur des e ets in nit esimaux (d eriv ees partielles) a n d'e ectuer l'attribution de cr edit. Cela pourrait devenir un probl eme s erieux si l'on consid ere des fonctions plus profondes et plus non lin eaires, avec a l'extr^eme la non-lin earit e o u la relation entre les param etres et le co^ut est r eellement discr ete. Inspir ee par la pr esum ee invraisemblance biologique de la r etropropagation, cette th ese propose une nouvelle approche, Target Propagation. L'id ee principale est de calculer des cibles plut^ot que des gradients a chaque couche, en faisant en sorte que chaque paire de couches successive forme un auto-encodeur. Nous montrons qu'une correction lin eaire, appel ee Di erence Target Propaga- tion, est tr es e cace, conduisant a des r esultats comparables a la r etropropagation pour les r eseaux profonds avec des unit es discr etes et continues et des auto- encodeurs et atteignant l' etat de l'art pour les r eseaux stochastiques

Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation

We introduce Equilibrium Propagation, a learning framework for energy-based models. It involves only one kind of neural computation, performed in both the first phase (when the prediction is made) and the second phase of training (after the target or prediction error is revealed). Although this algorithm computes the gradient of an objective function just like Backpropagation, it does not need a special computation or circuit for the second phase, where errors are implicitly propagated. Equilibrium Propagation shares similarities with Contrastive Hebbian Learning and Contrastive Divergence while solving the theoretical issues of both algorithms: our algorithm computes the gradient of a well defined objective function. Because the objective function is defined in terms of local perturbations, the second phase of Equilibrium Propagation corresponds to only nudging the prediction (fixed point, or stationary distribution) towards a configuration that reduces prediction error. In the case of a recurrent multi-layer supervised network, the output units are slightly nudged towards their target in the second phase, and the perturbation introduced at the output layer propagates backward in the hidden layers. We show that the signal 'back-propagated' during this second phase corresponds to the propagation of error derivatives and encodes the gradient of the objective function, when the synaptic update corresponds to a standard form of spike-timing dependent plasticity. This work makes it more plausible that a mechanism similar to Backpropagation could be implemented by brains, since leaky integrator neural computation performs both inference and error back-propagation in our model. The only local difference between the two phases is whether synaptic changes are allowed or not

Atmospheric Noise in Interferometer Radars -Observaf/ons on GE Mod III Tracking System

Recent low elevation angle space missions have provided a means of detecting random atmospheric propagation noise in range rate difference radar data while tracking a moving target. The range rate difference data noise increases as the elevation angle approaches the horizon. The noise increases as an approximate function of E esc 2 E down to about 4 degrees elevation and increases less rapidly below 4 degrees

Optoelectronic System Measures Distances to Multiple Targets

An optoelectronic metrology apparatus now at the laboratory-prototype stage of development is intended to repeatedly determine distances of as much as several hundred meters, at submillimeter accuracy, to multiple targets in rapid succession. The underlying concept of optoelectronic apparatuses that can measure distances to targets is not new; such apparatuses are commonly used in general surveying and machining. However, until now such apparatuses have been, variously, constrained to (1) a single target or (2) multiple targets with a low update rate and a requirement for some a priori knowledge of target geometry. When fully developed, the present apparatus would enable measurement of distances to more than 50 targets at an update rate greater than 10 Hz, without a requirement for a priori knowledge of target geometry. The apparatus (see figure) includes a laser ranging unit (LRU) that includes an electronic camera (photo receiver), the field of view of which contains all relevant targets. Each target, mounted at a fiducial position on an object of interest, consists of a small lens at the output end of an optical fiber that extends from the object of interest back to the LRU. For each target and its optical fiber, there is a dedicated laser that is used to illuminate the target via the optical fiber. The targets are illuminated, one at a time, with laser light that is modulated at a frequency of 10.01 MHz. The modulated laser light is emitted by the target, from where it returns to the camera (photodetector), where it is detected. Both the outgoing and incoming 10.01-MHz laser signals are mixed with a 10-MHz local-oscillator to obtain beat notes at 10 kHz, and the difference between the phases of the beat notes is measured by a phase meter. This phase difference serves as a measure of the total length of the path traveled by light going out through the optical fiber and returning to the camera (photodetector) through free space. Because the portion of the path length inside the optical fiber is not ordinarily known and can change with temperature, it is also necessary to measure the phase difference associated with this portion and subtract it from the aforementioned overall phase difference to obtain the phase difference proportional to only the free-space path length, which is the distance that one seeks to measure. Therefore, the apparatus includes a photodiode and a circulator that enable measurement of the phase difference associated with propagation from the LRU inside the fiber to the target, reflection from the fiber end, and propagation back inside the fiber to the LRU. Because this phase difference represents twice the optical path length of the fiber, this phase difference is divided in two before subtraction from the aforementioned total-path-length phase difference. Radiation-induced changes in the photodetectors in this apparatus can affect the measurements. To enable calibration for the purpose of compensation for these changes, the apparatus includes an additional target at a known short distance, located inside the camera. If the measured distance to this target changes, then the change is applied to the other targets

Electron impact double ionization of helium from classical trajectory calculations

With a recently proposed quasiclassical ansatz [Geyer and Rost, J. Phys. B 35 (2002) 1479] it is possible to perform classical trajectory ionization calculations on many electron targets. The autoionization of the target is prevented by a M\o{}ller type backward--forward propagation scheme and allows to consider all interactions between all particles without additional stabilization. The application of the quasiclassical ansatz for helium targets is explained and total and partially differential cross sections for electron impact double ionization are calculated. In the high energy regime the classical description fails to describe the dominant TS1 process, which leads to big deviations, whereas for low energies the total cross section is reproduced well. Differential cross sections calculated at 250 eV await their experimental confirmation.Comment: LaTeX, 22 pages, 10 figures, submitted to J. Phys.