Do-arnt oom, ROMT O 36-412Bca~e¢5o 'o.-- ; of ElectronicsI -I .T inoogyTHEORY OF NONLINEAR TRANSDUCERSH. E. SINGLETONL O ~ A )TECHNICAL REPORT NO. 160AUGUST 12, 1950RESEARCH LABORATORY OF ELECTRONICSMASSACHUSETTS INSTITUTE OF TECHNOLOGY.W*. Ca~~V The research reported in this document was made possiblethrough support extended the Massachusetts Institute of Tech-nology, Research Laboratory of Electronics, jointly by the ArmySignal Corps, the Navy Department (Office of Naval Research)and the Air Force (Air Materiel Command), under Signal CorpsContract No. W36-039-sc-32037, Project No. 102B; Departmentof the Army Project No. 3-99-10-022.MASSACHUSETTS INSTITUTE OF TECHNOLOGYRESEARCH LABORATORY OF ELECTRONICSTechnical Report No. 160 August 12, 1950THEORY OF NONLINEAR TRANSDUCERSH. E. SingletonAbstractTransducers are classified according to the manner in which they make use ofstorage to control their output. The state of a transducer is specified by the positionof a point in multi-dimensional space. The system function of the transducer is thendetermined by attaching to each state point a set of numbers or probabilities charac-teristic of the output for the corresponding state. The technique of synthesizing non-linear transducers is considered, and standard forms for synthesizing the various kindsof transducers are obtained. It is shown that a necessary and sufficient condition foran invariant finite state transducer to be capable of synthesis in terms of a finite numberof linear elements and rectifiers is that the state-defining regions be bounded by hyper-plane surfaces. It is found that any nonlinear system function is determined by certainhigher-order autocorrelation functions of the input and crosscorrelations between theinput and output.The problem of optimum design of nonlinear filters is discussed. It is shown thatif the probability distributions of noise and signal are gaussian, the optimumrhmean-square filter is linear. For finite-state filters, the criterion of minimizing the proba-bility of error is employed. The problem is to specify optimum boundaries for thestate-defining regions, and specific design equations are obtained. Several examplesof filter design are given, including an application to the radar search problem._.·-_LI-----^-^ -- I-IITHEORY OF NONLINEAR TRANSDUCERSIntroductionWith the publication of Wiener's work, "The Extrapolation, Interpolation, andSmoothing of Stationary Time Series", network theory gained additional purpose anddirection, and achieved a new maturity (1). In this classical treatise Wiener appliedstatistical methods to the problem of selecting optimum linear filter response charac-teristics. But the book, originally published in 1942 as an N.D.R.C. report, had agreater significance than is explicit in its treatment of linear filters. In its statementof the concept that communication signals be regarded as stationary time series andstudied with the tools of mathematical statistics, it laid the groundwork for the develop-ment of the science of information theory, and was itself a first example of the workingof this powerful new discipline.Information theory has given precise meaning to the notion of quantity of information(2, 3, 4), and has made it possible to devise ways of measuring the amount of informationcarried by a signal. The statistical theory has also provided the foundation for studiesof the theoretical limitations on the rate of transmission of information in the presenceof noise (5, 6, 7). In the further development of communication systems, informationtheory may be expected to assume the responsibility of judging how well the systemsdo their jobs. The networks in the systems will be designed in accordance with thedemands of information theory to insure the most effective utilization of noisy orlimited media for the transmission of information (8).The same decade that has seen the development of information theory has alsowitnessed a more general introduction of communication systems which make use ofpulse modulation. The use of pulse techniques tends to draw network design emphasisaway from linearity and equalization, since by means of nonlinear devices the signalcan be reshaped or regenerated when required. Moreover, multi-channel pulse com-munication systems make use of time division multiplex, mixing and separating channelsin the time domain, and avoiding the use of highly selective filters. It is no accidentthat the development of information theory and the exploitation of pulse modulationshould occur together; for it has been shown that the performance of certain quantizedpulse communication systems can be made to approach the limiting ideal performancespecified by information theory (5, 9, 10).The use of pulse regeneration in the repeaters and decoders of pulse communicationsystems implies that the corresponding network theory should include an additionalelement, namely the electronic switch. Although electronic switches and other nonlinearcircuit elements are in common use in transducers employed in pulse communicationsystems, there has been very little study of such transducers from a general point ofview. There has been no published work at all on the optimum design of such trans-ducers.A principal purpose of the present paper is to develop methods of selecting optimum-1-_______II·___ _ · ·I __response characteristics for a certain class of nonlinear transducers. The function ofthe transducers may be described as "information processing", and may include noisediscrimination, filtering, prediction, decoding, etc. Interest is centered chiefly ontransducers which are especially useful in the processing of pulse communicationsignals. An additional purpose of the paper is to develop some of the properties ofsuch transducers, in particular those properties which are related to the statisticalbehavior of the transducers. In this connection it should be mentioned that Wiener hasrecently suggested a method for making use of the statistical response to random noiseof an unknown nonlinear transducer to obtain a complete mathematical characterizationof the device. This method will be described in a later section.Wiener's filter theory provided the solution to the problem of selecting optimumresponse characteristics for linear transducers operating on prescribed ensembles ofnoise and signal. Beyond the requirement of invariance in time, the statistics of theensembles were not restricted, but the transducers were required to be linear. In thepresent paper, on the other hand, the transducers will be allowed to become nonlinear,while the statistics characterizing the signals will often be restricted. In most of theillustrative examples it will be required that the signal statistics conform to those ofone of the quantized pulse communication systems. It is natural that such restrictionsbe employed, since it is likely that pulse modulation systems will be the first to makeuse of optimum nonlinear information processing devices. Furthermore, as has beenindicated, information theory has shown that certain quantized communication systemsare able to approach the limiting rate of ideal systems.For the development of the optimum linear filter theory there was already availablea highly developed theory of linear networks themselves, including necessary andsufficient conditions on transfer functions for physical realizability ( 1, 12), and anumber of synthesis procedures (11, 12, 13, 14). On the other hand, no nonlinear trans-ducer theory corresponding to the theory of linear transducers is available to providea basis for the study of optimum nonlinear systems. A part of the present paper isaccordingly devoted to developing some of the properties of nonlinear transducers. Ingeneral, the properties investigated are those which are pertinent to the selection ofoptimum nonlinear systems, or which depend on or characterize the statistical behaviorof nonlinear systems. Principal emphasis of this part of the paper is also placed ontransducers which are especially suitable for processing quantized signals, but a fewproperties of more general transducers are also discussed.I. Classification of Transducers1.1 General RemarksBy a transducer is meant a device which operates on one time series (the input) toproduce another time series (the output). Since we are interested in the way transducersfunction as information processing devices, we shall restrict the inputs to be considered-2-to those produced by information sources. The information produced by a source canbe measured in terms of a set of probability distribution functions (2), and so from thepoint of view of information processing, a source is completely described by the set ofprobability distribution functions which govern its operation. Thus any transducer isalways discussed with reference to the information source with which it is associated.There is a certain contrast here with the linear theory, where h(t) affords a completedescription of the linear transducer. In the nonlinear theory, the system function whichspecifies the behavior of the transducer may in general be different for different sources.For example, if the input is quantized, it is convenient to make use of a system functionwhich is defined only for the possible inputs, since a more general description mightbe unnecessarily complex. This is not a real limitation, because any given transducerwill usually be designed to work with a particular source, and its response to othersources is not of interest.The transducers to be studied are restricted to those which have a finite memory,and which operate on a finite portion T of the immediate past of the input. If the inputis a continuous function of time, it might at first appear that complete description of along portion T of the past of the input could not be stored in a memory system havinga finite capacity. If our sources were completely general, this would in fact be thecase, but the sources considered are always restricted in some way. In particular,if the source is band limited to the bandwidth W, the past of its input over the timeinterval T may be specified by 2TW numbers (5). These numbers could be, for example,the values of the function at 2TW epochs equally spaced on the time interval T. Ifinstead of a bandwidth limitation, the source is restricted to produce voltage pulses atquantized time intervals, the past of the input is specified by the values of the pulsesproduced during the interval T. Whatever the method chosen for specifying the pastof the input, it will always be done in terms of a finite set of numbers.Since the transducer operates on a portion of the past of the input covering a timeinterval T, the only information available to the transducer on which to base its outputis the finite set of numbers which characterize the past of its input. Let us assumethat there are s of the numbers. We may consider the state of the transducer as definedby the values of these s numbers. If we think of the values over which the s numberscan range as being measured along coordinate axes in an s-dimensional space, calledthe transducer space, the coordinates of a point in the space can be used to specifythe state of the transducer. Since the s numbers which characterize the past of theinput to the transducer are functions of time, the state point will move about as timeprogresses.1. 2 Classification of TransducersWe are now in a position to give a more precise definition of various kinds of trans-ducers:Invariant transducers. If a definite number can be assigned to each point in-3-transducer space, such that the output of the transducer equals this number when thetransducer is in the state defined by the point, the transducer is called time-invariant,or simply invariant.Stationary transducers. If a fixed probability distribution can be assigned to eachpoint in transducer space, such that the output of the transducer has this probabilitydistribution when the transducer is in the state defined by the point, the transducerwill be said to be stationary.Finite state transducers. If the transducer can produce only a finite set of outputvalues, it is called finite state.If a transducer is invariant, it is also stationary, but the converse is not generallytrue. An invariant transducer will be finite state if the input to the transducer isquantized, so that the state point can occupy only a finite number of different positions.On the other hand, even if the input is not quantized and the state point varies over aninfinite set of points, an invariant transducer can still be finite state. This will occurif the transducer space is broken up into a finite number of volume elements, and theoutput of the transducer remains constant as long as the state point is in any one volumeelement. Stationary transducers will be finite state under the same conditions asinvariant transducers, provided in addition that all the defining probability distributionsof the output are discrete.1.3 Other Kinds of TransducersIt is of course possible to define many other classes of transducers. One importantclass might be transducers whose output depends on certain characteristics of the pastof the input which are not restricted to have been determined in any particular timeinterval. A number of dimensions in transducer space would thus be devoted to storingcertain types of information which might depend on portions of the past of the inputindefinitely remote, rather than on just the immediately preceding time interval T. Inthe case of a discrete input, a typical datum of this type might be a record of the totalnumber of pulses received up to the present, or a record of the value of the largestpulse so far received.As has been pointed out, we are primarily interested in the application of nonlineartransducers to the processing of pulse communication signals. The natural transducerto use for this purpose is the invariant finite state transducer. Accordingly, we nextdescribe a standard form in which any invariant finite state transducer can be repre-sented. We then go on to discuss some of the properties of such transducers.II. Synthesis of Invariant Transducers2. 1 Synthesis in Terms of Linear Elements and RectifiersVarious standard forms for representing linear transducers have been known forsome time. These are embodied in the Wiener-Lee canonical form, the Darlington-4-Iinsertion loss theory, the Cauer canonical form, and the Gewertz theory. The Wiener-Lee form (13, 14) is especially suitable for modification to allow the introduction ofnonlinearity. Ways in which this modification may be accomplished will next be dis-cussed.We shall prove the following theorem: any invariant transducer driven by a finitequantized source can be synthesized by means of a finite number of resistors, capacitors,inductors, rectifiers*, and direct-current sources.We prove this first for the case where the input symbols to the transducer consistof quantized voltage pulses of constant duration, uniformly spaced in time and having afinite number of different amplitudes. An example is shown in Fig. 1. It is unimportantwhether the duration of the pulseOUTPUTTIMEFig. 1 Typical output of a finitequantized source.The output of the transducer canSince the function F describes thecan occur, it may be convenientlyThe function F, and therefore theschematically as shown in Fig. 2.transducer will be finite state.s equal the interval between pulses or not.Let the quantized input pulses have N possibledifferent amplitudes, and let the transducer be cap-able of storing at most s input symbols. Then thetransducer can have at most N s possible states.Let the variable Si designate the various states,where i ranges from 1 through NS. Correspondingto each state there will be a specific output symbol.be represented by a function of the state:Output = F(Si)behavior of the transducer under all conditions whichregarded as the system function of the transducer.behavior of the transducer, can be representedIt is clear, since the input is quantized, that theF(Si)Fig. 2 Graphical representation of the systemfunction of an invariant finite statetransducer.I 2 3 4 5 6 7 8 ---- NsSi STATE OF TRANSDUCERThe past input symbols which determine the output at any instant of time can bestored in a delay line consisting of a ladder network of inductors and capacitors, ter-minated in a resistor. The delay line is tapped at intervals equal to the time intervalbetween input symbols. Although in the present quantized case this implicitly limitsthe transducer to one output symbol per input symbol, this limitation will be removedwhen we consider continuous inputs.*By a rectifier is meant an element which is open-circuited for one polarity of current,and short-circuited for the other polarity.-5-iI I ---- I-j3 AAAmII'Fig. 3 An invariant finite state transducer driven by a finite quantized source.-6-... 1.C>9 V" T- I- AVV\l- T-- - -- ^VA.-- - . I-The arrangement for producing an arbitrary output as in Fig. 2 from an input suchas that of Fig. 1 is as shown in Fig. 3. For simplicity the transducer is shown ashaving storage capacity for only one past symbol, so that its output at any time is afunction of the input symbol which is being produced and of the immediately precedingsymbol. Also for simplicity the source is assumed to produce only three differentsymbols, say pulses having amplitudes of a, b, and c volts, where 0 < a < b < c andA> 0 is chosen such that a > A, b- a >A, and c - b >A. Each stored symbol is appliedto a set of rectifiers which are biassed to voltages indicated on the diagram. Currentwill flow in the load resistors when the input symbol is greater than the biassing voltage.By means of a resistor-rectifier clipper which is biassed A volts higher than the corre-sponding load resistors, the voltage taken from each load resistor is limited to exactlyA volts. The voltages obtained from adjacent load resistors are applied to opposite endsof the center-tapped primary of an ideal transformer. It is evident that only one of thesecondaries of the three output transformers will be energized for each input pulse. Theresistor-rectifier clippers at the right-hand terminals of the center-tapped primariesare to prevent coupling from one transformer to the next. The various possible com-binations of secondary windings are connected to conventional coincidence circuits, onlyone of which will be energized at a time. The required amount of voltage is taken fromeach coincidence circuit by means of a resistive voltage divider, and the outputs of allthe voltage dividers are mixed by means of a set of diodes connected to a common loadresistor.It is not necessary that the period and duration of the quantized pulses remain con-stant, as has been assumed up to now. If the leading and trailing edges of the pulsesare quantized to discrete epochs kto, k = 1, 2, ... , the argument remains unchanged,except to note that the taps on the delay line should be placed at intervals equal to to .Each possible state of the transducer will therefore still produce a unique result.Although the discussion has been carried through for a particular number of possiblesymbols and for a specific amount of storage, these restrictions are in no way necessary.This completes the proof of the theorem.For the case of an unquantized source, we obtain the following result: a necessaryand sufficient condition that any finite state transducer be capable of synthesis by meansof a finite number of resistors, inductors, capacitors, rectifiers, and direct-currentsources, is that the regions defining the states of the transducer be bounded by plane(or hyperplane) surfaces.It makes little difference whether the source is discrete or continuous. If it isdiscrete, the taps on the delay line are placed at intervals equal to the separationbetween symbols. If the source is continuous, we assume that the bandwidth occupiedby its output is finite. Let W be the bandwidth of the source. Then the taps on thedelay line are placed at intervals equal to 1/2W. As is well known, the past of the inputis then specified in the range t = 0 to t = s/2W, where s is the number of taps on thedelay line.-7-We first prove the sufficiency of the condition. To that end, we describe a trans-ducer which will indicate in which of a set of sub-ranges a voltage varying over a con-tinuous range lies. In Fig. 4 the ranges are defined by means of the voltages to whichFig. 4 Transducer to indicate range in which a varying voltage lies.the upper rectifiers are biassed. The ideal transformer across each of the rectifierload resistors steps up the voltage by a very large factor. The resulting voltage isclipped by a resistor-rectifier clipper biassed to a small positive voltage. By makingthe voltage step-up of the ideal transformer sufficiently large, a standard voltage isobtained at the output of the clipper, independent of how much greater than the biasvoltage the input voltage is. These resulting voltages are then applied in pairs toopposite ends of the center-tapped primaries of ideal transformers, as in the earlierfigure. Only one of the transformer secondaries will be energized, in accordance withthe range in which the input voltage lies. An exception occurs when the input voltagelies just above and very close to one of the biassing voltages. In this case, the voltageacross the secondary of the step-up transformer will not be large enough to be clipped,so that a nonstandard output will be obtained from two range indicators, rather than astandard output from one. The proportion of time during which such failures occurcan be made arbitrarily small by increasing the step-up ratio of the transformer anddecreasing the clipping bias voltage. It should be noted that this peculiarity is notlimited to the representation selected here, but is characteristic, in one form oranother, of any finite-state transducer driven by an unquantized continuous source.That is, the peculiarity should be present in the representation.We next note that any plane surface in the s-dimensional space of the transducer canbe represented by an equation of the forma1 x1 + a 2 2 + ... + a s X =C (1)-8-For all points[x]= (xl x2 , ... , xs) on one side of the plane,Za i x i > c, while for allpoints [x] on the other side of the plane, Xai xi < c. This is true since Eq. 1 is con-tinuous in the x i , and it is therefore impossible to pass continuously from a point whereai x i > c to a point where Za i x i < c without passing through a point in the plane.Now assume that the region Rk which defines the state Sk is bounded by a set of nplane surfaces:Z aji x i = cj, j = 1, ... , n . (2)i=lIn general some of the planes represented by Eq. 2 will not only bound, but will alsopass through the region R k, breaking it up into a number of sub-regions. Each of thesesub-regions will be bounded by a subset of the planes represented by Eq. 2. The con-dition that a point [x] lie within a particular one of sub-regions bounded by the planesaji Xi = Cj, j = jl, 2 (3)i=lis that [x] satisfy simultaneously a set of inequalities of the form:ajix i > c,=j, j lJ(4) i=lsince as long as the point [x] stays within a sub-region it cannot cross any of the boundingplanes. The condition that a point lie within the region Rk is then that it satisfy one ofthe sets of inequalities of the form of Eq. 4, which correspond to the various sub-regions.For each of the sub-regions of Rk, each of the linear combinationss+ Z aji Xi, = l' j2' ...i=lcan be formed as shown in Fig. 5, by adding together the weighted output of the varioustaps. By means of a transducer such as the one already described which indicates inwhich set of ranges a given voltage lies, the comparison5+ aji xi > cj (5)i=lSOURCEFig. 5 Formation of weighted sumof past samples.Oli-9-can be made. By doing this for each of the inequalities for which j = j l , j 2 ' .'.. a setof outputs is obtained. A set of coincidence circuits is used to indicate whether all theconditions are met, and if they all are, the coincidence circuit produces the outputwhich corresponds to the region R k. A similar process is carried out for the othersub-regions of Rk, and the outputs of all the sub-regions are mixed. The other statedefining regions of the transducer are treated the same way as Rk.This completes the proof of the sufficiency of the condition that an invariant finitestate transducer be capable of synthesis with linear elements, rectifiers, and direct-current sources. The proof has been obtained by producing a transducer which meetsthe condition and is composed of the stated elements. A typical two-dimensionalexample is shown in Fig. 6.Fig. 6 Schematic representation of a possible two-dimensional finite state transducer.To show that the condition is also necessary, we note that the systems we have beendiscussing are essentially switched systems which are governed by linear laws betweentransitions. If we choose any point [x] 0 continuous in [x] on a bounding surface, an expan-sion of the surface about the point is linear if the bounding surface is planar. Hence ifwe vary one of the x's, say xi , in such a way that x passes through the bounding surfacein the neighborhood of [x] 0 , the value of x i at which a transition of state occurs will be alinear function of the other x's. On the other hand, if we assume that a certain portionof a bounding surface in the neighborhood of [x] 0 is not planar, an expansion of the surfaceabout [x]0 must include terms higher than the first. If now we vary x i so that x] passesthrough the bounding surface in the neighborhood of [x]0 , the value of x i at which a transi-tion occurs will be a nonlinear function of the other x's. Even for infinitesimal changesin the x's a continuous nonlinearity is obtained. Such behavior cannot be exhibited by anideal rectifier whose output is either linear or discontinuous. It follows that if idealrectifiers and linear elements are to be used, the surfaces must be planar.Any invariant finite state transducer can of course be approximated by a transducercomposed of linear elements, rectifiers, and direct-current sources, since any surfacecan be approximated by a set of planes. In fact, the standard form which has beenobtained is also applicable to time-invariant nonfinite state transducers, in the sensethat it will approximate the output of such a transducer arbitrarily closely. Considerthe s-dimensional space xl, ... , xs of the transducer. From the point of view of thetransducer, the relevant part of the past of the input to the transducer is described bythe position of a point in this space, as we have seen, and the point will move about astime progresses. The transducer is completely specified by attaching a number (equal-10-to the output of the transducer for the corresponding state) to each point in the space.If the input signal is bounded the space of the transducer is finite. If the numbersrepresenting the output of the transducer are continuous in the space coordinates, it ispossible to break the space up into a finite number of cells, and to assign a cell-numberto each cell such that the output for any point in the cell will be arbitrarily close to thiscell-number. A finite state transducer whose states are defined by the cells and whoseoutput for each state is equal to the corresponding cell-number, will thus approximatearbitrarily closely to the output of the original transducer. The above argument maybe considered as a proof of the following theorem: any invariant transducer can beapproximated arbitrarily closely by means of a transducer composed of resistors,capacitors, inductors, rectifiers, and direct-current sources.2.2 Synthesis in Terms of Function GeneratorsWe have completed the discussion of the synthesis of transducers whose only non-linear elements are rectifiers, and now go on to consider the use of other nonlinearelements. As long as we are dealing with finite state transducers, each state Si maybe associated with a region R i in[x] -space, where R i is bounded by a set of surfaces:fj([x])= O, j= 1, ... , n . (6)In a later section we shall see that an important practical case arises when none of thesurfaces f. = 0 passes through the region Ri. Under this restriction, the condition thata point xlbe interior to Ri is that+fj([x])> 0, j = 1, ... , n . (7)If a function generator is available for producing each of the functions fj([x]), a set ofcomparison circuits and a coincidence circuit is all that is required to find out whether[x] belongs to R i or not. The transducer can be synthesized in the form shown in Fig. 7.There will be as many coincidence circuits as there are states Si .Fig. 7 Finite state transducer with regions having non-planar boundaries.-11-As an example we write out the equations for the two-dimensional transducer ofFig. 8.The condition that the point belong to R i isthat(x l -a) 2 + (x 2 -a) 2 < r .For R 2 the conditions are:xiFig. 8 Schematic representation of apossible two-dimensional finitestate transducer.O < x1 < aX1 -X2 < 0For R3:(xl - a) 2 + (xZ - a) 2 > r< 2 < aX1- X2 > (x- a)2 + (x2 -a)2> rx1 >ax2 >a2. 3 Expansion in SeriesFor future reference, we mention one other synthesis technique, a method especiallysuitable for invariant continuous output transducers. Let the set of numbers [x] =(x 1, . . ., Xs) specify the state S of the transducer. Then the output of the transducerat time t is given byy(t) = F(S) = F ([x(t)])= F[xl (t), . . ., xs(t)] , (8)and if y(t) is continuous in the x i and has continuous derivatives, y may be expanded ina power series:y(t) =xxi J h'j' ' 'h 1 "'...i h= Z A, Xa (t)a.(9)(10)where Xa (t)ai'j' ''h'containing Mi hcorrespondingrepresents one of the products x x2 ... xS and A is the correspondingIf we wish to approximate the infinite expansion by a finite expansionterms, we may limit the range of values which the exponents i, j, ... , h12-X2And for R4:can assume to 0 to N - 1, and a may be identified as follows:= i + j N +...+ h N s , (11)so that a ranges from 0 to N s - 1 = M. The transducer in this case includes adding andmultiplying circuits; the form of the transducer would be as shown in Fig. 9.Fig. 9 Series form of nonfinite statetransducer.We shall also find it convenient in a later application to use, instead of a powerseries representation, a series expansion in orthogonal functions, as employed byWiener (15).III. Synthesis of Stationary TransducersBy making use of the finite state transducer synthesis technique which has beendeveloped in the preceding section, we are in a position to describe a standard formfor stationary transducers. Instead of a definite output voltage for each state, a voltagewhich has a fixed probability distribution dependent on the state must be produced.Although the optimum synthesis we later discuss is based on invariant rather thanstationary transducers, it is perhaps worth noting that stationary transducers havepossible value in the synthesis of secrecy systems, in studying the effects of noise intransmission systems, and in the study of information sources. We shall show, forexample, that any information source whose output depends statistically on a fixedfinite portion of its past can be represented in terms of an appropriate stationary trans-ducer whose output is its own input. In providing a building block for the synthesis ofinformation sources, it is possible that stationary transducers may also serve as usefullaboratory tools.To provide a basis for the stationary transducer synthesis, we assume that we havea certain invariant finite state transducer called a state evaluator. The state evaluatorprovides an indication of the state the transducer is in. A schematic representation isshown in Fig. 10.INPUTFig. 10 Block diagram of state evaluator.SI S2 S3 SM-I SM-13-Fig. 1 1 Distribution control circuit.p(x)!X -- C -4 -3 -2 -I04.30.2DI 1(6) 5 J(I)0 i 2 3 4Fig. 12 Separation of gaussian distribution in ranges having assigned probabilities.Fig. 13 Output probability distribution for separation according to Fig. 12.Fig. 14 Stationary transducer used as a source.-14-A unit voltage appears on the output line corresponding to the state of the transducer.As the state point moves in transducer space from one state-defining region to another,the output line which is energized changes accordingly. The state evaluator is evidentlycomposed of sets of comparison circuits and coincidence circuits, as described in Sect.II, and its design need not be discussed any further here.In order to convert the state evaluator to a stationary transducer, it is sufficientto connect each of the output lines S 1 , ... , S M of Fig. 10 to separate circuits which areeach adjusted to produce an output voltage which has an arbitrary prearranged proba-bility distribution.A method of producing a voltage having an arbitrary discrete probability distributionwill now be described. The elements are shown in block diagram form in Fig. 11. Tocomplete the stationary transducer, one of these arrangements is connected to eachstate indicator line. A wide band noise source is sampled at the time the state indicatorline is energized, and the resulting sample is stored on a capacitor in the samplingcircuit. The stored voltage is applied to an amplitude distributor which produces astandard voltage on one of a number of output lines, according to the voltage range inwhich the stored sample lies. By means of the potentiometers shown at the output ofthe amplitude distributor, any required proportion of the standard voltage is applied tothe output circuit. When the transducer again changes state, the capacitor is discharged.The probability distribution of the noise source is gaussian. The biases in theamplitude distributor are adjusted in such a way as to break the total area under thegaussian curve into sub-areas having predetermined values. Figure 12 shows anexample of a possible distribution. The samples of noise are broken up into six ampli-tude ranges, such that the probabilities of falling in ranges one through six are respec-tively 1/6, 1/3, 1/10, 2/10, 1/10, and 1/10. Arbitrary output voltages of 4, - 2, 0,- 1, 2, and 1 are then produced for sample voltages falling in these ranges, as shownin Fig. 13.In Fig. 14 is shown a diagram of a stationary transducer whose input is providedby its own output. The blocks labelled "distribution control" are circuits such as thoseof Fig. 11. Since the distribution control circuits may be set to have any desired dis-crete distribution, and since the state evaluator may be connected in such a way as toprovide an arbitrary correspondence between distributions and states, it follows thatthe arrangement is a model for any discrete statistical source whose output dependson a finite portion of its past.IV. Statistical Analysis of Transducers4. 1 Correlation Analysis of TransducersIt has been shown (16, 17) that for a linear transducero00io(T) h(t) ii (t-T) dt (12)--o015-For the discrete case this may be writtenx(n) y (n + T) = a x(n) x(n+T) + a2 x(n) x(n +T - 1)+... + a s x(n) x(n + T - S + 1) (13)where the output of the transducer is given byy(n) = al x(n) + a 2 x(n- 1) +.. + as x(n- s + 1) . (14)If we know the value of ~ii(T) and io(T) for T = 0 to T = - 1, the averages in Eq. 13are known. Letting T range over the values T = 0 to T = s - 1 in Eq. 13 provides a setof s simultaneous equations in the s unknowns al, ... , a s . By solving the equationsfor the ai, the system function is determined. Thus the crosscorrelation function ofthe input with the output, and the autocorrelation of the input are together sufficientto define a linear system. This rather striking result comes about because the formof the equation defining the transducer is known. The problem of completely describinga transducer mathematically is greatly simplified when the form of the equation relatinginput and output is specified. The remainder of the problem of specifying the transduceris that of finding the coefficients which must be fitted into the known form.In the nonlinear case it may also happen that the form of the equation describingthe transducer is known. This will happen, for example, if the kind of nonlinearelements available is restricted. In this case the coefficients which are required tocomplete the characterization of the transducer can be found by making use of statisticalparameters similar to those employed in the linear analysis.Let the transducer operate on a finite portion of the past of the input (t, t - T) whereT = s - 1/2W, and let the input be band limited to the band W. Then the equation of thetransducer can be writteny(t) = F [x(t), x(t-), . , x(t- ZW )=Z' ai, j, . . , h [x(t)] i [x(t -2-)] j * [x(t--2 )]hi h=ZA X (t) (15)awhere Xa (t) represents one of the products[x(t)] i [x(t- ZI)] j... [x(t_- -)]hand A is the corresponding value of ai j. h If to N - 1 is the range of valueswhich the exponents i, j, ... , h can assume, we may identify a as follows:a = i + j N +. . .+ h Ns - 1 (16)-16-so that a ranges from 0 to N s - 1 = M.On the basis of the assumed form for the nonlinear transducer, a complete descrip-tion is provided by specifying the coefficients A a . In order to determine the values ofAa in terms of statistical parameters, we multiply Eq. 15 by Xp(t), and average overall time:XP(t) y(t) =-A a X3(t) Xa() (17)axy() - A, xx(a, P) (18)awhere the higher order correlation functions xy and xx are written in place of thecorresponding time averages. Eq. 18 must hold for all , and so provides a set ofM linear simultaneous equations in M unknowns which leads to the evaluation of the A a .4. 2 Experimental Study of Unknown TransducersAn interesting technique for experimentally evaluating the defining coefficients ofan unknown transducer can be based on the above result. Let it be required to evaluatethe system function of a transducer having a prescribed form of nonlinearity for asignal which is band limited to the band W. The experimental arrangement shown inblock diagram form in Fig. 15 could be used. We choose a noise source having abandwidth large compared to W. Successive samples of the noise are obtained by meansFig. 15 Experimental study of unknownnonlinear transducer.of the sampler tube and a pulse generator having a period 1/2W. The amplitudes of thesamples are all independent of each other, since the autocorrelation function of the noisebecomes negligible in a time small compared to the sampling period. The values of thecorrelation functions xx (a, ) are therefore all easily computed, and are in fact givenby00o ooxx(a'1) = * I Xa(Xl', ' x )xs (x,) X( xl ) p(x. ) dx. .. d .9)- -It remains to evaluate xy (P). This is accomplished experimentally, as shown in Fig. 15,by first computing XP(t) and then crosscorrelating the result with the output y(t) of theunknown nonlinear transducer. This amounts to multiplying the two functions and-17-averaging the product. By repeating this measurement for each value of P, the completeset of simultaneous equations represented by Eq. 18 can be set up, and the coefficientsA evaluated.The experimental procedure which has been described is based on a method origi-nally suggested by Wiener (15). We now describe Wiener's method, which has thevirtue of not requiring the solution of any simultaneous equations. The idea is to usenormal and orthogonal functions in place of the products Xa (t) in the expansion of thesystem function in Eq. 15. We first rewrite Eq. 15, replacing x(t), . .. , x(t - s - 1/2W)by a set of variables ul(t), ... , u s (t) which characterize the past of the input. Thusy(t) = F [u l (t), u 2 (t) .. .. , us (t)] . (20)The system function F is next expanded in a series of Hermite functions (19), givingy(t) = F(ul, .. , Us) =a.i, .. . .. r Hi (ul)... Hh(Us) exp[ +i hU2 u +...+u2= Aa H(a) exp2 1s (21)where H(a) represents the polynomial Hi(u 1).. .Hh(us), and a is defined as in Eq. 16.If we now multiply by H(P) and average over all time we findUl2 +''+y(t) H(A) =A a H(a) H(P) exp _ - (22)aIn order to calculate the time averages in the summation, we make use of the ergodictheorem to replace the time averages by phase averages. The phase averages can beconveniently calculated if we choose an input whose parameters ul, ... , us haveindependent gaussian distributions, and which have been normalized to have a standarddeviation of unity. Then the joint probability distribution of u .. , u s is(21r)-s/2 exp [- 2+ uand the sum of the phase averages becomes:(21T) -s/ A l-*5 H(a) H(P) exp (u + ... + uS )du 1 ... du (23)a -ooSince the Hermite functions are orthogonal over(-oo, o ), expression 23 reduces to(21r) - s / 2 K2 A[ and we obtain:A = (2T) y(t) H(P) . (24)P-18-Here the constant KP is the product of the normalizing factors associated with theHermite polynomials whose product equals H(P). Thus (19)K [s/2 2 i+j+. .+h (i! )(j! )... (h!l)] 1/2 (25)Wiener's method lends itself readily to the experimental evaluation of unknown systemfunctions, but even if the representation requires the use of only a few Hermite functions,the complexities involved in automatically computing H(P) become very great. Never-theless, as electronic methods for multiplication continue to be improved and becomemore available, Wiener's method of studying unknown nonlinear transducers mayreceive considerable application.In the equations which have been derived, we related the characteristic parametersof a nonlinear transducer to the higher order correlation functionsxy(U 1, ... , Us) and xx(U1, ... , Us)of the input and output signals. These correlation functions represent certain averages,which can of course be computed from a knowledge of the joint probability distributionsP(ul, ..., us) and P (y: u 1 , ... , us)It is perhaps worth noting that since we are dealing with invariant transducers, thedistribution P(y: ul, ... , us) is alone sufficient to specify the transducer; because foreach set of values ul, ... , u s there is only one possible output, so thatP(y: U 1 , ... , s) = 1 for y = F(ul, ... , us)= 0 otherwise . (26)Also, if the variables ul, ... , u s are quantized so that the transducer can have onlya finite number of states, and if a unique value of y corresponds to each state, then thetransducer is completely characterized by the set of distributionsP[yj: uki], k = 1, . . ., s .where P [yj: uk ] is the probability that y has its jth value when uk has its ith value.For we may wrieP(ui: j) P(ukj) P(Yj : Uki)PP~uki ukY-) =_ )- (27)P(uki) P(y ki) P( ki)and since there is only one state for each yj, it follows that P(uki : yj) will be unity forone value of i and zero for all others. Thus by holding j constant and ranging over allk we can find the state S. which corresponds to yj. By carrying out the same procedurefor each value of j we completely specify the transducer. We need not actually knowP(uki) in order to carry out this process, since P(yj: uki) will differ from P(uki: yj)only by the factor P(uki)/P(yj), which is a positive number for all i, j. Hence P(yj : ki )will differ from zero for each combination of k and yj only for one uki.-19-4. 3 Influence of Transducers on Probability DistributionsThis section on the statistical analysis of transducers will be concluded with a fewremarks regarding the influence of a transducer on the statistics of the signal on whichit operates. The transducer and the statistics of the input are assumed to be known,and it is required to calculate the statistics of output. In the corresponding linearcase it is usual to consider only the spectra, or correlation functions. Thus if we havea linear transducer with a certain frequency response characteristic and the input is awhite noise, we say that the power spectrum of the output has the same form as thefrequency response. Such statistical assertions are of great value in judging the per-formance of transducers, and have in fact provided the basis for much linear filterdesign.We shall consider a fairly general case, in which it is desired to calculate the con-ditional probability distribution functions of the output when the transducer is specifiedand the set of input probability distribution functions is known. Let the input timeseries consist of a sequence of uniformly spaced quantized pulses x(n), where n is adiscrete time parameter, and x(n) is the nth pulse applied to the input of the transducer.The sequence x(n) is assumed to be a Markoff process (20) of order r, so that it isgoverned by the joint probability distribution functionp [x(n), x(n- 1), ... , x(n -r)]. (28)Let each pulse have one of N different amplitudes. Since the transducer operates on sinput pulses, it can have a maximum of N s = M possible states Sm' where m = 1, ... , M.If we designate by Xu(k) the value of x stored in the kth position in the transducer, whereu(k) is an index which ranges from 1 to N, the possible states Sm can be orderedaccording to the equationm = u() + [u(2)- 1] N +... + [u(s) - 1 N (29)The probability of occurrence of the state S is thenmP[Sm = P [Xu(l)) X( 2)' . Xu(s)] (30)If s < r, p[Sm] is found by summing over the extra variables in expression 28. If s > r,so that s - r = a, p [Sm] is found fromP [Sm] = P[Xu(a)' Xu(a+l) ... u(s)P [xu(l) xu(). xxu(0 -L) xu(a)' ... Xu(s)] (31)ButP u()' Xu(2) -. Xu(a-l) Xu(a)- p'. xu(s=a- P[Xu(a-k)' Xu(a-k-l) ... , Xu(r+a-k)] (32)k=l P[xu(a-k-l)' u(a-k-2)' ... ' u(r+a-k)]-20-Similarly the conditional probability that state j follow state i, wherei = v(l) + [v(2)- 1] N . + [v(s)- 1] N 1and j = w(l)+[w(2)-1 ] N+...+ [w(s)- 1] N - 1is given by p [Sj(n+l): Si(n = P[Xw(l) Xv((), X( 2) Xv(s)],p Sj(n+2) : Si(n] = p [Xw(2 ) Xw() v(1)' Xv( 2 ) Xv(s)]etc. (33)On the basis of the probability distributions determined for the states, we may nowfind the probability distributions of the output. Let the equation of the transducer beA m= F m, (34)where A m is the output when the transducer is in state Sm . If the output time seriesis represented by y(n), each value of y will equal one of the numbers Am. It followsthat the output probability distribution isP(yt) = P [Sj] (35)where the summation is over all j for which Aj = Yt.SimilarlyP[ynq(n ) : Yt(n)l = P[Sk(n + 1): Sj(n (36)k jwhere the summation is over all j and k such that Ak = yq and Aj = Yt.If a transducer has a unique output corresponding to each state and s > r, theprobability distribution of the next output symbol is completely determined by thepresent output symbol. It follows that the output in this case is a first order Markoffprocess, i. e., r = 1, where r is the order of the output Markoff process.On the other hand, if s < r, the probability distribution of the next input symbol,and hence of the next output symbol, is dependent not only on the s stored symbols butalso on the r - s additional symbols which are no longer stored. These r - s inputsymbols may be determined, however, if the preceding r - s output symbols are known.The probability distribution of the next input symbol, and hence of the next output symbol,is determined by these r - s output symbols plus the present output symbol. That is,the order of the output Markoff process is r = r- s + 1.-21 -V. Synthesis of Optimum Nonlinear Filters5.1 General RemarksThe problem to be considered in this section is that of separating two interferingsignals, one of which may be a noise. If the distribution of the energy in the signalas a function of frequency is different from that of the noise, a partial separation canbe affected by means of a linear filter. It has been shown by Wiener (1) that if thenoise and signal are independent, the power density spectra of the signal and noise aresufficient to prescribe the best possible linear filter in the mean square error sense.In this incoherent case, the treatment given the filter problem by Lee (18) makesit clear that essentially none of the noise energy which lies in a band where the signalenergy is relatively large will be effectively removed by the optimum linear filter. Inparticular, if the noise and signal have the same spectrum, no separation can be accom-plished, in spite of the fact that other statistical parameters of the noise and signal maydiffer considerably. The ability of a linear device to separate noise and signal is thuslimited because direct use is not made of parameters which provide a more completestatistical characterization.The foregoing considerations indicate that in the general case, a nonlinear devicecan be expected to do a better job of separating noise and signal than a linear filter(Cf. Sect. 5, 6), and that the operation of the device should be governed by statisticalparameters of the noise and signal. It is possible to consider a noise separating deviceas having for its function to make the information in a corrupted signal more readilyunderstandable to a user (receiver, decoder). Thus the noise discriminator is aninformation processing device. The information carried by the signal is defined by setsof amplitude probability distribution functions (2). It follows that the behavior of thedevice should take into account the same sets of probability distributions. This is done,of course, in the synthesis of optimum linear filters, since the power spectra may beobtained from the probability functions. It may be noted that as many different timeseries governed by different sets of probability distributions as desired can be constructedwhich lead to the same power spectrum. All these time series would be processed bythe same linear filter: on the other hand, the optimum nonlinear filter would in generalbe different for each of the series.5.2 An ExampleBefore going on to formulate the filter problem on a general basis, it may be ofinterest to illustrate some of the ideas involved by working out a simple example. Theexample chosen is that of a signal and noise having identical spectra.The signal consists of a sequence of uniformly spaced independent pulses which mayhave amplitudes of +1, 0, and - 1, with the following probabilities:*If the noise and signal are not independent, the cross-power spectrum must also beknown.-22-(37)The noise consists of a similar sequence of uniformly spaced independent pulses whichmay have amplitudes of + 2, 0, and -2, with probabilities:1 5p(+2)= p(-2) = 12 ' (0) = . (38)This situation corresponds to interference between two signals on the same channel,with the signals having the same general form but different statistical characteristics.The signal and noise have identical power spectra and equal power, so that no separationcan be carried out by means of a linear filter. A nonlinear filter is easily designed,however, by taking into account the probability distributions.The noise and signal are combined additively, so that the input to the transducerconsists of a sequence of pulses with amplitudes ranging from + 3 to -3. The possiblecombinations and the corresponding probabilities are shown in Table I.Table IPossible CombinationsSignal Noise+10+1-10--1+10-1+2+20+200--2-2--2Probability13- 136101613-10361036136131 I1366From the table the values which the signal and noise must have are evident for allreceived pulses other than + 1 or - 1. One or the other of these two ambiguous pulsesis received 22/36 of the time. An ambiguous pulse can be caused by either of twopossible combinations, one of which is of high probability (10/36), and the other of lowprobability (1/36). Since the magnitude of the error caused by a wrong selection isindependent of which of the two possible combinations is chosen, it follows that the com-bination of high probability should always be selected to represent an ambiguous pulse.The optimum transducer can thus be specified by the input-output relation given in-23-ReceivedSignal+3+2+1+10-1--2-3Table II. Any number of graphs can be constructed which will pass through the pointsgiven in Table II, and a suitable graph which can be conveniently synthesized is shownin Fig. 16.Table IIInput+3+2+10-1-2-3Output+10+10-10-1A block diagram which will produce the input-output relation of Fig. 16 is shownin Fig. 17. The upper flipflop is arranged to trigger when the input exceeds + 1.5, andwhen it triggers it adds -2 to the original input. The lower flipflop triggers when theinput is less than- 1. 5, and when it triggers it adds + 2 to the original input.OUTPUTFig. 16 Response for minimum probabilityof error.The result is the input-output curve ofFig. 16. A circuit realization of theblock diagram would require two doubletriodes and a number of resistors, capa-citors, and crystal diodes.The effectiveness of the transducerin this case is measured by its reduc-tion of the mean square error from 6/9to 2/9, and by its reduction of the pro-bability of error from 3/18 to 1/18.5. 3 Formulation of the Filter ProblemWe consider a message fm(t) whosepast over the time interval (t, t - T)can be described by the set of para-meters u l (t), u2 (t), . . ., us (t). SuchFig. 17 Nonlinear transducer with response a description will be possible, as weof Fig. 16.have seen, if for example the messageis band limited to the band W, and the values ul, ... , u s represent samples of themessage at the points t, t- 1/2W, ... , t- s- 1/2W = t-T, where s = 2TW + 1. Thedescription in terms of ul, ... , us is of course especially applicable to pulse communi-cation signals.-24-We assume that the message as received at some point is additively perturbed by anoise fn(t) whose past over the same time interval (t, t - T) can be described by the setof parameters v1 (t), v2 (t), ... , vs (t). The received combination of signal and noisewill then be represented byul(t) + vl(t), . . . us(t) + Vs(t) = Wl(t), ... Ws(t) ·We denote the set of functions ul(t), ... , us(t) by [u(t)], the set of functions vl(t), ... ,vS(t) by [v(t)], and the set of functions wl(t), ... , ws(t) by [w(t)]. We assume that[u]and [v] are stationary random functions with given characteristic probability distributions,so that the probability distributions p([u]) = p(ul, ... , us), p([v]) = p(v l , ... , vs), andp([w]) = p(w1 , . .. , Ws) can be calculated.The general filter problem may now be stated as follows: what operator F([w(t)] )best approximates, in a prescribed statistical sense, a desired function f [fm(t)]The problem as formulated includes prediction, detection, decoding, and ordinaryfiltering as special cases. If, for example, the function f fm(t)] = f([u(t)]), we mighthave a filter or a decoder. The transducer does not know the values of ul, ..., u s ,but is nevertheless required to produce the best approximation to some function of them.As another example, if f [fm(t)] = fm(t + a), a > 0, we would have a problem in prediction.The answer to the problem as formulated will evidently have to be expressed in termsof the probability distributions of [u], v], and [w] . As we have noted, Wiener obtainedthe solution to the problem for the case in which f and F are linear operators on themessage and on the past of the message plus noise respectively. The result is expressedin the Wiener-Hopf equation: oo4id(+ a) =I h(t) ii(t- T) dt, T > 0 (39)where h(t) is the linear operator on the past of the message plus noise which is requiredto be found, and the correlation functions are determined by the probability distributionsand the desired operation on the message. The statistical sense in which h(t) is the bestlinear operator is that it gives minimum average error power.We now proceed to investigate some of the techniques which can be used to solve theproblem when the functions f and F are not restricted to be linear. We shall usuallyrequire that F assumes a prescribed mathematical form. The form may of course bevery inclusive, as in the case of a power series or series of Hermite functions. Orthe form of F may be such that the space of [w] is broken up into a large number ofsmall cells, and F( [w] ) required to be constant in each cell. By making the numberof cells large enough, F can be made to approximate arbitrarily closely to the mostgeneral invariant transducer (Cf. Sect. II).-25-5.4 Determination of Series Representation forMinimum Mean Square ErrorAs we saw in Sect. II and Sect. IV, we may represent F ([w]) by a series:My(t) = F( [w]) = Aa Wa(t) (40)a=lwhere the W(t) are known functions, such as products of the functions w, ... , ws, orproducts of Hermite functions of the wi. The filter problem becomes that of findingthe best values of Aa for approximating to f[fm(t)] = fd(t). The values of the A a forminimum mean square error can be obtained as follows:Let the error at time t be given byE(t) = y(t)- fd(t) (41)The mean square error isTE = [e(t)]2 = i 5 I [y(t)-fd(t)]2 dt-T-T aT Tlim [fd]2 dt- T W(t) fd(t) dt-T a -TT+ Ala Ap T-m 2T1 W ((t ) fWt) dt (43)a, -TThe limits in this equation are higher order correlation functions, and they may beevaluated in terms of the known probability distributions. Thus we haveT(O ) = lim 1 5 [f (t)] 2 dtqdd 0=-T o 2T dt-Too- d[fd] P [fdfd (44)_00Tlim 1iwd(a) = T- 2T Wa(t) fd(t) dt-T-26-= Wa fP [Wa, fd] d[w] dfd (45)- 00oo - 00Tlir 1 www( 1) = Tmoo Z-- W cLa(t) Wp(t) dt-T03WWp P[Wv. W] d [w] (46)- 00Substituting,E = dd(0)-2ZAa dwd() + E AAp ww(, (47)a a, ,In order to make E a minimum, we must haveaE 0 for all . (48)yThusc E - Zw() + 2 A Pw,(,¥ 0 (49)orqwd( Y)= AcL ww,, ( . 1, ., M .(507aCThis is a set of M simultaneous equations in M unknowns, and hence leads to an evaluationof the A and of the system function F.We now show that this necessary condition specified by Eq. 50 that E be a minimumis also sufficient. That the values of A actually specify a minimum and not a maximum,may be verified by observing that aE/aA > 0 for A Y greater than its optimum value,and aE/ a A < 0 for A less than its optimum value, since b ww ( y, y) is positive. HenceE is increased by any eparture of A from optimum. Furthermore, the error isrepresented by a polynomial of the second degree in the A's so that there cannot be morethan one minimum. Hence condition 50 is sufficient.If now the A's are obtained by solving Eq. 50, the mean square error becomes, bysubstituting Eq. 50 in Eq. 47,E O= dd(O) - 2 Aa wd(La) + Aa wd(a)a aa-27-qdd (o)- AaAp qww ((, ) (51)Ca, 5.5 Example of Power Series SynthesisAs an illustration of the use of the results which have been developed, we apply themto the design of a minimum mean square error transducer for the noisy transmissionsystem used in the previous example. This will enable us to compare the system func-tions for minimum mean square error and for minimum probability of error.Since all the pulses are independent of each other, the transducer will not makeuse of storage. The output can accordingly be represented byMy(t) = Y, Ai [w(t) * (52)i=OThe input can take on only seven different values: + 3, + 2, ... , -3, and the output musttake on a unique value for each input. Because the input-output relation must be skew-symmetrical, and go through three prescribed points on each side of the origin, it isclear that Eq. 52 must contain only odd powers of w, and need have no terms of degreehigher than the fifth; i. e., M = 5. The design equations areMwd ( Y) T Aa ww ( a Y), Y = 1, 2, ... , M (53)a=lMaking use of Eqs. 45 and 46 we computebwd(Y) w¥ Yd; wd(o) = wd(2) = wd(4) =... =0,2owd(1) 3' ,wd ( 3 ) = 2, wd( 5 ) = 14a1 ;-y 3Also Ww(ay) w w = w ... 0,4 4 - 134 - 1138 10w -, w 6, w = 33383 w 3 Substituting in Eq. 53, we obtain:Ao = A 2 = A4 = 04 134 2A1 3 A 6 +A5 5 = 3134 1138A1 6 +A 3- + A 3 2134 1138A1 3 +A 3 +A 5 3338 = 14 (54)-28-The solution is:A1 = 1.20A 3 = 0. 485A5 = 0. 0424 (55)so that the equation of the optimum mean square transducer isy = 1.26 w - 0.495 w 3 + 0.0424 w5 (56)A plot of Eq. 56 is given in Fig. 18. It is clear that the probability of error is increasedas compared with the earlier design, since for the ambiguous inputs + 1 and -1, thecorrect output is never obtained. The probability of error is in fact raised from 1/18for the earlier design to 11/18 for the present design. On the other hand, the meansquare error is reduced from 2/9 = 22/99 for the earlier design to 20/99 for thepresent design.Fig. 18 Response for minimum meansquare error.A transducer having the system function of Eq. 56 is easily constructed if circuitsfor raising voltages to the third and fifth powers are available. The result would be asshown in Fig. 19. But inasmuch as the transducer is required to function only forinteger inputs ranging over (+ 3, -3), the form shown in Fig. 20 can be used. The pulseamplitude distributor can be of the crystal diode type shown in Fig. 4, or a modificationof that circuit. Because amplitude distributors are usually easier to construct thancircuits for raising voltages to powers, the arrangement of Fig. 20 may be preferable.Fig. 20 Nonlinear transducer withresponse of circled points of Fig. 18.-29-Fig. 19 Nonlinear transducer withresponse of Fig. 18.5.6 Optimum Mean Square Filter for Gaussian SignalsIt seems appropriate at this point to examine the problem of optimum mean squarefilter design from a slightly different point of view. We shall obtain equations which,while not always convenient to apply physically, nevertheless throw additional lighton the relation of the filter to the probability distributions of the input time series.In particular we shall show that if the given signal and noise have gaussian probabilitydistributions, the best mean square filter is linear, as has been pointed out by Wiener(3).The general multi-dimensional gaussian distribution of the set of variables wl, ...,Ws may be written (21)exp(~])kE-·-1·j (57)P5(2)s/2 -exp E Ajk wj wk (5)j,kwhere A is the determinant of the moment matrix[ :A] . . . . . . . (58)Ajk is the cofactor of the element Xjk of A, andii i ' Xik = iCri k (59)In Eq. 59 o'i is the standard deviation of the random variable wi, and Pik is the corre-lation coefficient between w i and wk. If all the variables wl, ... , ws belong to the sametime series f(t), all the i are equal; and if wi(t) is separated from wj(t) by the timeinterval (i-j)T, where T is a constant, then [A] becomes the correlation matrix, i.e.,(0) ( T) * . S )[A] = (T) (0) . . (s - 2) T] (60)'S - 1)T] (S - 2)T] o(0)Eq. 57 shows the general gaussian distribution to be defined completely in terms of amatrix [A] , which, in case Eq. 57 is the distribution of a single time series f(t), isin turn completely specified by the autocorrelation function (k T), k = 0, 1, ... , s - 1.It is thus clear that a knowledge of the autocorrelation function will permit the calculationof any other statistical parameter of f(t).In case two functions are involved, the only significant change in the character of[A is that some of the elements become crosscorrelations. Since the only pertinentstatistical parameters in such cases are the correlation functions, and since the corre-lation functions are sufficient to specify the optimum mean square linear filter, it might.be expected that for gaussian signal and noise the optimum mean square filter would-30-not contain any nonlinearity. That this is in fact true is verified below.We first note that if the probability distribution p(x) of a random variable x is given,the best mean square estimate of x is x; for the mean square error due to an estimatey isE = e = i (y-x) 2 p(x) dx_- coand minimizing with respect to y we find2 (y-x) p(x) dx = 0 Thusy i p(x) dx = x p(x) dx(61)(62)(63)y =x (64)We now consider the design of an optimum mean square filter for which the desiredoutput fd(t) is equal to the value of one of the members of[u], say ud. Let the jointdistribution of [w] and ud be gaussian, and let a new variable [z] be defined byZi = i i = 1,..., sZs+ = Ud (65)Let [z] have a moment matrix [A] . Thenp( [z]) = s+s+l -(2,) 2s+l1eexp -A j,k=lAjk j Zk'According to Eq. 64, the best mean square estimate of ud = Zs+l at any time when [w]= z 1, ... , z s takes on a particular value is given by the conditional mean of ud = Zs+lunder this condition. But the conditional mean of s+1 for a particular [w] is determinedfrom the conditional probability distribution p(Zs+l: Zl .. , Zs). Thuss+lP(Zs+ 1 : z' ... , z ) =o0exp(67)- 2 , Ak z zi dzs+[ jjk j k slj, k=l -31-(66)_____ ____ O0-02exp [A(As+l,s+l Zs l + 2 zs 1 Aj, S+ l 2j=1~ . ~(68)2exp - A s+5expF A (s+l, s+l s+l + 2 1 z j+ s1 z)dzEq. 68 also represents a gaussian distribution. It may be put in standard form bymultiplying both numerator and denominator by a constant which will complete thesquare of the expression in the exponential. The numerator becomesexp- Z A ( I + -. 2 (69)The conditional mean of Zs+l is therefore given bys A:l_+ Z j, s+l Zj (70)jkl As+l, s+lOr, writing fo(t) for the output of the filterfo(t) = T sA ws+1 j(t) (71)s+, s+l j jj=1This shows that in this case the optimum mean square filter is linear.Although Eq. 71 was derived for gaussian distributions of signal and noise, it alsospecifies the optimum linear filter for any distributions. This is because the optimumlinear filter depends directly on the correlation functions, so that combinations ofsignal and noise which have different sets of probability distributions, but the same setof correlation functions, result in the same optimum linear filter. Eq. 71 is in factthe same result which is obtained by solving the Wiener-Hopf equation for the case weare considering.In the general case we have from Eq. 65 for the equation of the optimum meansquare filter00i Ud P(Ud , W1 -- I W) dudfo (ud . (72dudI P(Ud, W1' . e W) dud (00-32-5.7 Finite State FiltersIf one assumes at the outset that a finite state transducer is to be used, as usuallyseems best for pulse communication systems, two convenient methods for handlingthe problem suggest themselves. One method is especially convenient for unquantizedsignals, and the other appears to be more natural for quantized signals. In the firstmethod the space of the transducer is assumed to be broken up into regions in a pre-determined way, for example into cells whose boundaries are perpendicular to coordinateaxes. The problem is then to determine a number to assign to each cell, such that ifthe output of the transducer equals this number when the input lies in the correspondingcell, the performance of the transducer is optimum in some prescribed statistical sense.The advantage of this method is that the transducer is easily synthesized in the standardform of Sect. II. The disadvantage is, of course, that while the transducer is the bestof all transducers having the prescribed call arrangement, some other transducer witha different cell arrangement might be still better. Improvement of this kind will not beimportant if the number of cells are sufficiently large.In the second method the arrangement and location of the regions are not specifiedin advance, but the numbers which are available for assignment to the various regionsare assumed to be known, since the signal is quantized. The problem is then to specifythe boundaries for each region in such a way as to give optimum performance in a pre-scribed statistical sense. This method does not suffer from the disadvantage mentionedfor the first method.We now outline briefly two procedures suitable for designing according to the firstmethod. Because our principal interest is in quantized signals, we will then go on todiscuss in detail a procedure for handling the second method.Let the condition that a point [w] = (w 1 , . .. , ws) lie within the ith cell in w space bethat wll< w pj p(L[w [u]j) for all j / i . (84)To show the necessity of the condition, we assume that the probabilities are con-tinuous in [w], that p(E) is a minimum, and that the neighborhood of a certain point[w] belongs to R i. We now ask whether there is a j such thatPi p([w] : [ui) < p p([w] : [ulj) . (85)That there is no such j follows from the fact that Pi p( [w] : ]i) is the integrand of theintegral in Eq. 83; so that if such a j existed the value of the summation would beincreased by reassigning some neighborhood of [w] to Rj. This would contradict theinitial assumption, and shows the necessity of the condition.To show the sufficiency of the condition we assume that each received point [w]belongs to R i, i = 1, ... , M, if pi p([w] : [u]i) > pj p([w] : [uj) for all j / i. We nowdivide the inequality by p([w]), obtainingPi p([w] []i) Pj P([w] [) for all j i (86)p([w]) p([w]) for all j orP([u]i : [w]) > p([u]j : [w]) for all j / i . (87)That is, the probability that [u]i was transmitted if [w] was received is greater than orequal to the probability that any other point [u]j was transmitted. Therefore, by assigning[w] to Ri, we minimize the probability of error as far as this particular [w], is con-cerned. The same procedure is followed for each received point [w], so that the totalerror p(E) is also minimized.The set of equationsPi p([w] [u]i)- p p([w] : u]j) > O for all j / i (88)and i = 1, ... , M are the design equations of the transducer. The process of designingthe optimum filter consists primarily of designing equipment which evaluates theexpressionsPi p([w] : Lu]i) - p ([w :[j) (89)-36-Ifor all i and j(j / i) each time a point [w is received. This equipment can usually beput in the form of amplitude discriminators, as will be shown. The results of theevaluations are led to M coincidence circuits, one of which corresponds to each of theregions R i . Any received point [w] will cause one of the coincidence circuits to producean output, according to the region in which [w] lies. The output of the ith coincidencecircuit, corresponding to the region Ri, is adjusted to equal f([uJi), i = 1, . . ., M.Under suitably restrictive conditions on the noise, it may be shown that the proba-bility of obtaining more than one output for an arbitrary value of [w] is zero. Forexample, if the noise has a probability distribution(v]) = (vl, ... , s) (90)which is continuous in the v i , i = 1, ... , s, we have that the boundary of any region R iconsists of sections from a set of surfaces of the formPi P([w] : [u]i)- p([w] : [u]j) = 0 (91)orPi C([W]-[u]i)-Pj ([w]-[]j) = 0 (92)For each pair of values of i, j, Eq. 92 represents a surface in w space, and the ratioof the measure of the set of points on a section of the surface to the measure of the setof points in any finite volume of the same space is zero (21). The ratio of the proba-bility that an arbitrary point [w] falls in the volume to the probability that it falls on thesurface is just the ratio of the integrals of the probability distribution over the twocorresponding sets of points. Because of the assumed continuity of the distributions,the probability function cannot be concentrated on the surface, so that the ratio mustbe zero. But more than one output can be obtained only if [w] falls on a surface. Hencethe probability of more than one output is zero.5.9 Example: Optimum P.C.M. DecoderAs a first example we choose the case of a two-dimensional transducer, since theresult may conveniently be shown on a two-dimensional graph. A two-dimensionaltransducer will generally be necessary for detecting or filtering code groups whichconsist of two pulses per group. We assume therefore that our signal consists of asequence of pulses uniformly spaced by a time interval 1/2W which can take the values+E or - E at the receiver, and that the signal is transmitted in code groups consistingof pairs of pulses over a transmission channel having a bandwidth W. This wouldcorrespond to a simple P.C.M. system. Let the code groups be governed by thefollowing set of probabilities:p(+E, +E) = a p(+E, -E) = bp(-E, -E) = a p(-E, +E) = b (93)-37-^..1---.-_-_1111_.1ItY - 11 ------- 11 ---_I - I^.·- *-PIILIIIIY·IC-. -LI--LIII·IYII·I^11·_-^111------- ---1__ 1~ IIII X llll~lll- ------ -1where 2(a + b) = 1, and let each code group be independent of all the others.The transmitted pulses are assumed to be additively perturbed by a white noisewhich is also band-limited to the bandwidth W, so that samples of the noise which areseparated by the interval 1/2W are independent and have a gaussian distribution. Letthe power in the received noise equal N. Then the joint probability distribution of thetwo noise voltages v 1 and v2 which add to the two signal pulses in a code group is givenby Z11 (V1 v2According to Eq. 84 , the opt mum N exprobability of error in(94)According to Eq. 84, the optimum design for minimizing the probability of error inthe ith code group is found from the condition thatPi p([w] : [x]i) > pj p([w] : [x]j) for all j / i[x]l = (+E, +E)[x] = (+ E, -E)[x]3 = (-E, +E)[x] 4 = (-E, -E)p([w]: [X]1) = 2wN exp[ (w- ) 2 + (w 2 - E)]p([w]: x]2) = - exp - ) (2 + E)(w1 ± E)2 + (w - E)2p([w] : x]3) exp- 2N) p([w] : [x] 4 ) = 2N exp[ (W1 + E) 2 +(w E) 2From Eq. 95 the set of conditions under which a received point [w] =to R1 is as follows:(95)(96)(97)(w 1 , w2 ) belongsa - (wl - E) 2 + (w -E)2,rN exp N 2 NbZirNexp -exp[-> exp [-2 rN'\2\w 1 T r D T kW 2 - r_ )2N(w 1 -E) 2 + ( + E)22N(w 1 + E) 2 + (w + E)22N(98)-38-Let us setThenIIi _ _ , 2 , ...Simplifying, the conditions become:w wZ w1 W2log a +E E N -EN - + + logbw w> E- - E + logbw1 w>-E - E + loga (99)orW >- 1 (N) log aW2 > - 1 () log aW1 + W2 > 0 . (100)In a similar way it is found that the conditions that [w] belong to R 2 are:> E) logWZ < 2- (E )log W1 W2 > 0 . (101)For R3 the conditions are:2<-(E) log aWz> 2() log aW1 - 2 < 0 . (102)And for R 4 the conditions arew1 < N) log aw 1 (N) log a2 <2(E) log b1 + W < 0 . (103)All of the conditions 100 -103 can be conveniently presented in graphical form, asshown in Fig. 22. The dashed lines divide w space into four regions, corresponding tothe case a/b > 1. In this case the conditions w-w 2 > 0 for R 2 and w - w2 < 0 for R 3are included in the other two conditions on R Z and R 3. With reference to the transducer-39-diagram of Fig. 22, we may say that the function of the transducer is to recognize inwhich of the four regions a received signal plus noise falls, and to produce the correctoutput corresponding to that region.The transducer diagram corresponds to whatW2 one would intuitively expect to be best. If a/b > 1is held fixed and N/E becomes very large, best(-E,+E) !i (+E+E) results will be obtained by simply taking (+E, +E)- _ _ ___- ____4,log when w + w > O and (-E, -E) when w + w2 > 0.wG; On the other hand if N/E becomes very small, thediagram reduces to the four quadrants. This result(-E,-E) g (+E,-E) would be expected in the case of a small noise, sincethe signal is then easily decoded without referenceto the probabilities.Fig. 22 Two-dimensional A transducer for carrying out the operationtransducer diagram. indicated graphically in Fig. 22 would comprise anumber of comparison circuits for comparing the input signals with the prescribedreference voltages, and a set of coincidence circuits for indicating the region in whichthe voltage lies. There would be four such coincidence circuits, one for each region.A block diagram of the arrangement is shown in Fig. 23. Successive pairs of samplesFig. 23 Block diagram of two-dimensionaltransducer.OUTPUTof signal plus noise are obtained by keying the gate tubes with synchronized timingpulses which may be obtained from the input signal. The voltages wl, w 2, and w1 + w2are then sent to separate comparison circuits. The comparison circuits produce outputvoltages on appropriate lines, as shown, in accordance with the magnitude of the samplevoltages. Each coincidence circuit functions in such a way as to produce an output onlywhen all the lines leading to it are keyed, and this will happen in accordance with the-40-region in which a received signal plus noise lies. If the transducer is to function as adecoder, each coincidence circuit can be adjusted to have an output representing theparticular code group to which it corresponds. If on the other hand it is desired toreproduce the original signal, as at a repeater station, this can be accomplished bykeying suitable circuits with a trigger pulse derived from the coincidence circuits.5. 10 Example: Radar Search ProblemAs a final example we discuss the application of the methods which have beendeveloped to an important practical problem, namely that of detecting a target by radarin the presence of interference. The interference may be due to ground clutter or seareturn, to jamming, or in the case of small or very distant targets, to random thermalnoise originating in the first stages of the receiver. In practice targets are usuallydetected by an operator observing the pattern of received echos on a cathode ray tubescreen. Experiments have shown that many targets are missed by the operator, eitherbecause he is observing some other part of the cathode ray tube screen when the targetappears, or because of fatigue (targets are missed during the latter part of a long watchmore often than during the early part).In radar systems of the type we are considering, the behavior of the electrical partof the system is essentially linear: small or distant targets give very weak indicationson the screen; larger targets give stronger indications. A nonlinearity is usually intro-duced by the human operator: if the target echo is extremely weak he does not observethe target at all, and takes no action. On the other hand, at least in the ideal case, ifthe target response passes a threshold value, the operator does observe it; and takes adecisive action. He may, for example, if the target is a submarine schnorkel, approachthe target and drop a depth bomb. From this point of view the human operator functionsas a nonlinear filter, with an all or nothing response, according as a target is or is notpresent. The response of the eye is apparently fairly linear, so the nonlinear responseof the human observer is due to operations on the input signal which are carried on in thebrain. Furthermore, the response of the human operator takes into account the statisticsof the target return and of the interference. For even though the signal level is veryweak, the statistical character of the target echo may be very different from that of thesurrounding interference, and so permit the target to be identified by an experiencedobserver.In view of experiments showing a large number of targets missed due to humanfailure, it is natural to inquire whether further electrical operation on the radar returnmight not make the human observer's job easier. In particular, it has been suggestedthat the necessary nonlinearity (which must exist in certain complete systems) be intro-duced ahead of the indicating device by means of an optimum nonlinear filter, in sucha way that either an unmistakable signal is presented to the human operator, or nosignal whatsoever appears on the indicator screen. If the highly complex human linkis to be replaced in this way, it is to be expected that the nonlinear filter which takesits place should also be complex, at least relative to more conventional filters. The-41-advantage to be gained would be freedom from oversight and fatigue.We consider the particular case of an airborne radar searching for a submarineschnorkel. In order to carry through a design with the aid of relations 84, we shallassume a certain probability distribution for the return in the absence of a target (theso-called sea return), and another distribution for the return when a target is present.The distribution assumed for a single echo in the absence of a target is of the formp(w: exp + c exp - - . (104)[~0n n I 2a i (104)If- an 2a n ° ara oThe first of the two normal distributions is due to thermal noise and has a relativelysmall r n and m n , and takes account of the fact that waves sometimes mask sizeableareas behind them. The second of the two normal distributions is due to reflectionsfrom waves. If the sea were smooth, the sea return would be absent (c = o), and thelimitation on range would be due to thermal noise in the receiver. The parametersc, ar 0, and mo are functions of the coordinates with respect to the aircraft, of theelementary area under consideration, and will moreover vary with flight conditionsand weather. These influences on the statistics of the sea return are taken into con-sideration by the human observer, and must be taken into consideration by the nonlinearfilter as well. It is probable that the parameters can be continuously evaluated, in anapproximate fashion, by an auxiliary computer associated with the filter, and theresults can be fed continuously to the filter, varying as flight altitude, weather conditions,etc., change. This is, of course, the way a good human observer works. The auxili-ary computer need not be located in the searching aircraft, but can be placed on landor on a ship, according to requirements.In the multi-dimensional case, assuming the echos on successive scans to be inde-pendent of one another, which they clearly are for sea return, the joint distribution fors echos becomes:+ exp wmo2 (105)1zv n- 2CrThe distribution of a single echo when a target is present will be assumed to begaussian, with mean ml, and standard deviation C 1:1 [ (w - i)p(w )= exp _ 2 (106)p(w: 1) I 2o. 1The parameters m nd 1 will also d pend on the coordinates of the schnorkel withThe parameters mi1 and o 1 will also depend on the coordinates of the schnorkel with-42-__respect to the aircraft, and on the flight conditions and weather. They cannot be con-tinuously evaluated during flight, and would therefore have to be evaluated experi-mentally for a wide range of conditions as part of the program of developing the non-linear filter.The joint distribution for s echos when a target is present becomes:P(wl, ... , w s : 1)-07)1)W1· = e I,:l) (107)(21r) exp 2d 11 1In order to determine the optimum filter it is now necessary to adopt a criterion ofperformance. It might at first appear that a suitable criterion would be that of mini-mizing the probability of error. Equation 84 would be then be directly applicable, andwould give, on the basis of s scans of an elementary area,P1 p(w , W,s: 1) > pO P(w 1' ***., w s : 0) (108)as the design equation of the filter, where P1 is the relative frequency with which targetsappear in the elementary area under consideration, and po = 1 - p1 . That is, when theinequality 108 is satisfied, a target is judged to be present. Since Po >> P1, it is clearthat 108 would almost never be satisfied, and the filter would indicate no target at almostall times. This result is in keeping with the criterion mentioned, and shows thatanother criterion should be chosen. The human observer has as his criterion to missas few submarines as possible while searching as much area as possible. The reductionin area searched can be considered as coming about due to false alarms, which taketime to investigate. If the cost of missing a submarine which is present is taken as a,and the cost of investigating a false alarm is taken as b, the criterion adopted by thehuman observer would be to minimize e a + eob, where e is the relative frequencyat which targets are missed and e is the relative frequency at which false alarms arecalled. From Eq. 83 we findela + eb = apl 1- R! pw: 1) dHR1+ bpo 1- p(w: 0) d[W . (109)RSince a and b are independent of R o and R 1 , the condition that a point belong to R 1 willbe found in the same way as before, from Eq. 84, givingap 1p(wl, ... , ws : 1) > bpop(wl, .. .s : ) . (110)-43-___Condition 110 can be met, as contrasted to condition 108, whenever the cost of missinga submarine is sufficiently greater than the cost of investigating a false alarm.In the same way that p([w] : 1) and p([w] : 0) depend on the coordinates of the areaunder consideration, the values of a and b can vary with these coordinates. For if atarget is out on the fringe of a search path, and we incorrectly say that no target isthere, we have very little chance of rectifying the error. On the other hand, if thetarget is dead ahead and we miss it, it may be picked up later.Substituting Eqs. 105 and 107 in Eq. 110 givesapi exp (Wi-ml)2 - (Wi- mss 2 >b 1 - exp 2(rleXPL-E 2cr2 2cr2 /1 1 n /+ c exp ( - . (111)0The optimum nonlinear filter in this case is a computer. Its function is to evaluate thetwo sides of the inequality 111, and compare the results to determine which side islarger. If the left hand side is larger, a warning is given.Condition 11 1 is based on statistics which define the coherence from pulse to pulse.In the same way it would be possible to take into consideration the statistics relatingthe returns produced by reflections of the same pulse when incident on different areas.It seems most convenient to do this in the design of the linear filters associated withthe i-f and video amplifiers.It may be expected that aC and C n will both be small compared to mn - m o . In thiscase, one of the two terms on the right side of 111 will usually be much larger than theother and the inequality may be written:(wi- m)2 bPo 1 l-- c-z w, > slog + klog 2 ap n2a2 1+ (s - k) log c (112)(k) n (s-k) owhere k is the number of returns for which1-- exp [ (wi-mn)2 c (wi - mo)21r C exp [ n21 -c exp 2 (113)n oand Z indicates summation over those terms for which 113 holds. Simplifying 112(k)-44-gives(Wi -) l bpo -1-c o1 1(wi -mn) 2 ( w i - mO) 22- ~~ zo-~2 ~(114)(k) 2 n (s-k) 20-0oThe computer which carries out the computation 114 may be of any convenientdesign, such as a modern high speed digital computer. It could be carried on a ship,for example, and operate on signals [w] received from an aircraft.It is worth noting that in case c = 1, which might correspond to the case of a fairlysmooth sea, 114 would become:(wi- ml)2 s (Wi- 2 bpo >) > Z m--, + slog a p 0 (115)1 2cr 1 1 2o' 1 a 0If now in addition ' 1 = cr, we obtains w2 s 2S Wi sm wim sm o bp oiE 1 > _O - + slog . (116)1 1 1 1 o 1 aP1Or rewriting:_ 1S i M + .2 bp.1 i > m I log (117)1 1 oAs would be expected in this simple case, relation 117 simply requires (whenm 1 > mo) that the average value of the return exceed the average of the two means plusan additional term which takes into account the probabilities p and P1 and the costs aand b.ConclusionPrincipal emphasis in this study of nonlinear transducers has been placed oninformation processing devices intended for use in receivers, or in equipment similarto receivers. Thus the field of study parallels roughly the general area studied byWiener in his book on optimum linear filters, with the difference that the devices whichare optimized are allowed to be nonlinear. Within the limits of this field it has beenfound possible to handle the problem of determining optimum invariant devices for pro-cessing various kinds of signals of importance. Specific design equations are givenin these cases. It has not been found possible to summarize the results in a closedform, as was done by Wiener for the linear case, but the methods which have beendeveloped should prove applicable, with minor modifications, to new problems-45-____I Ilr _ C1 ·/_*_YYUIII___II_I_ .lillXIX-_l·l·ll-- 11_·1__^_ 11-1--· ·- ·L-C-------CI II--·---which may arise in the general area selected for study.The results which have been obtained lead to equipment which is physically realiz-able in one or another of the standard forms which are proposed. It is possible that inmany cases simpler equipment which produces the same results can be designed, andthis possibility needs to be investigated. The problem involved is related to that of thelogical design of computers.The examples which have been discussed indicate that optimum nonlinear equipmenttends to be relatively complex. Only in the case of gaussian signal and noise is theoptimum filter found to reduce to a linear device. The examples also emphasize thata considerable body of statistical data must be obtained before optimum design can becarried out. Because of the tendency toward complexity, and also because of therequirement for large amounts of statistical data, it is not to be expected that optimumnonlinear devices will generally replace linear networks in jobs which are normallyperformed by linear networks. We may instead conclude that optimum nonlinear deviceswill probably first receive extensive use in processing information which is normallyprocessed by a nonlinear device anyway. By making use of the nonlinear theory, anoptimum system replaces what might otherwise have to be designed on an intuitive ora cut-and-try basis. Again, as has been discussed by Wiener in "Cybernetics",optimum transducer theory may make possible the design of transducers which willbe good enough to release human beings from certain routine tasks. As illustrated byour example of the radar search problem, some tasks are now performed by humanbeings, not because the human being is especially suited to the job, but because otherequipment which would do it has not been available.Closely related to the problem which has been studied is that of the design ofoptimum coding equipment. The problem of optimum coding may in a sense be con-sidered to include the problem of optimum reception; and in fact to constitute theproblem of optimum communication system design. It is hoped that some of thetechniques which have been employed in this thesis may prove useful in the study ofthis more general problem.AcknowledgementThe author wishes to express his deep appreciation to Professor J. B. Wiesner,who directed this research and led it to its present form; to Professor R. M. Fanofor much constructive criticism; to Professor Y. W. Lee, who introduced the authorto this field, and whose work provides a basis for this research; and to Messrs. A. J.Lephakis and C. A. Stutt for many helpful suggestions and comments. The author isalso greatly indebted to Professor Norbert Wiener, who by generously discussing hisown research on nonlinear networks with the author, has been a source of inspirationand encouragement.-46--- IReferences1. N. Wiener: The Extrapolation, Interpolation, and Smoothing of Stationary TimeSeries with Engineering Applications, (Wiley and Sons, N.Y. 1949).2. C. E. Shannon: B.S.T.J. 27, 3, 379-423 (1948); 27, 4, 623-656 (1948).3. N. Wiener: Cybernetics (Wiley and Sons, N.Y. 1948).4. R. M. Fano: R.L.E. Technical Report, No. 65, M.I.T. (1949).5. C. E. Shannon: Proc. I.R.E. 37, 1, 10-21 (1949).6. W. G. Tuller: Proc. I.R.E. 37, 5, 468-478 (1949).7. R. M. Fano: R.L.E. Technical Report, No. 149, M.I.T. (1950).8. R. L. Dietzhold: Elec. Eng., 67, 9, 895-899 (1948).9. B. M. Oliver, J. R. Pierce, C. E. Shannon: Proc. I.R.E. 36, 11, 1324-1331(1948).10. M. J. E. Golay: Proc. I.R.E. 37, 9, 1031 (1949).11. S. Darlington: J. Math. Phys. 18, 257-353 (1939).12. R. M. Fano,A. W. Lawson, Jr. : The Theory of Microwave Filters, 9, 9,M. I.T. Radiation Laboratory Series, (McGraw Hill, N.Y. 1946).13. Y. W. Lee: J. Math. Phys. 11, 83-113 (1932).14. N. Wiener,Y. W. Lee: U.S. Patents No. 2, 128, 257, 2, 024, 900, and 2 124, 599.15. N. Wiener: Seminar in Nonlinear Networks, Research Laboratory of Electronics,February, 1949, Unpublished.16. Y. W. Lee: Quarterly Progress Report, Research Laboratory of Electronics, M.I.T.p. 60, (January 15, 1950).17. Y. W. Lee,J. B. Wiesner: Electronics 23, 6, 86-92 (1950).18. Y. W. Lee: Course 6. 563 Class Notes, Unpublished.19. E. C. Titchmarsh: Introduction to the Theory of Fourier Integrals (Oxford UniversityPress, London, 1937).20. M. Frechet: Theorie des evenements en chaine, (Gauthier-Villars, Paris, 1938).21. H. Cramer: Mathematical Methods of Statistics, (Princeton University Press,Princeton, N.J. 1946).-47-__________11_1___111_11_111_1__-1_--1_ ··^ ·1*·-·11 -- -·141 _ 1-^1-.--L1l.1. ---_ .·.· II -·-.----C -- I ·- ^ -- I -_I_ __

Theory of nonlinear transducers

Abstract

Similar works

Full text

Available Versions

DSpace@MIT