Yngve, V. H.

Contains research objectives and reports on one research project.National Science Foundatio

DSpace@MIT

XV. MECHANICAL TRANSLATION*Dr. V. H. YngveRESEARCH OBJECTIVESThe basic objective of our program is to devise techniques by which languages canbe translated by machine. There are two aspects of the program: the linguistic and themechanical. The two must progress side by side because there is considerable influenceon the methods of each by the results of the other. In addition, a balance must bemaintained between short-range programs which aim at the immediately practicable,and longer range research which may add to the basic understanding needed for futureprogress.A. SYNTACTIC CATEGORIESOne of the major problems in linguistics is to devise objective methods for analyzingand describing language. Language can be considered as an ordered sequence ofsymbols. Not all possible sequences of symbols are allowed in a language; most areforbidden by the constraints imposed by rules of usage. A description of a languageis in essence a list of the symbols used in the language and a statement of the rules forcombining or ordering the symbols. If the symbols are taken as words, the list is adictionary and the statement of the rules is a grammar. The list of words in a languagecan be determined by fairly objective techniques, but it is difficult to find objectivemethods for discovering the rules of grammar and syntax.Our most recent effort at devising such objective methods consists of comparing asample of the language with certain features of a simple statistical model of language insuch a way that the constraints of grammar and syntax show up as deviations fromrandomness.The statistical model adopted assumes that the probability of finding a given word ata given position in the text depends only on the normalized frequency of this word, noton its position in the text. Experimentally, of course, the probability of a given wordappearing at a given point in the text is strongly influenced by the occurrence of otherwords in its environment. Thus, deviations from the model are expected.In this experiment, the influence of only the six most frequent English words uponeach other was measured, since the sample text was short (9490 words). Figure XV-1shows 8 of the 21 histograms obtained experimentally. Each histogram represents thenumber of times a given word appeared at various positions before and after anothergiven word located at the center position. If the word at the center position has noinfluence on the probability of occurrence of the given word, one would expect theaverage number of occurrences to be equal to the center horizontal line, which isthe product of the frequencies of the two words divided by the total number of words inthe text. On a statistical basis, one would expect about one-sixth of the histogram bars*This work was supported in part by the National Science Foundation.-93-(b)A-THE THE- A(C)IN -THE(f)OF-A A-OF(9)THE -IN12 (d) (h)IN-A A-IN AND-AND AND-AND4 _ _ _ _ _ _15 12 8 4 0 0 4 8 12 15 15 12 8 4 0 0 4 8 12 15GAP LENGTHSFig. XV-1The frequency of occurrence of certain words in the neighborhood of another word. Forexample, in (b), the number of occurrences of the word "a" is plotted as a function of thenumber of intervening words to the left or right of "the," which is at the center position.-94-(XV. MECHANICAL TRANSLATION)to be above the upper horizontal line and one-sixth to be below the lower horizontal line.Every statistically significant observed deviation from randomness has been corre-lated with known facts about English grammar and syntax. For example, (a) and (h)illustrate the fact that the same word tends not to be repeated at close intervals. Thesetwo histograms are, of course, symmetrical because the two words considered are thesame; (b) also shows a symmetry which is the result of the grammatical similarity of"a" and "the"; (c), (e), and (g) show the high peak immediately to the left of the centerbecause these words can all be used as prepositions. The wider peak on (g) is causedby the competition of infinitive constructions with the prepositional construction. Thelarge peak at the right of the center of (e) represents the genitive construction.The width of the observed peaks and valleys gives an idea of the length of thesequences of words in the various constructions. For example, in (a) it is clear thatconstructions with articles involve two or three words and that such constructions mighttend to repeat every four or five words, while constructions with "of" are rather morecompact. In this connection it is interesting to note in (h) that the observed probabilityof "and" in the vicinity of another "and" is lower than would be expected, even out to aseparation of 15 words. This is because one of the uses of "and" is to coordinate longclauses.The method appears to have advantages in that it gives a different viewpoint towardthe grammar of well-known languages and thus may help our understanding of language.It may also be useful as another technique at the disposal of the linguist who is trying todescribe a little-known language.-95-

Mechanical Translation

Contains reports on two research projects

XV. MECHANICAL TRANSLATIONDr. V. H. YngveA. A NEW PUBLICATIONIn cooperation with Prof. W. N. Locke of the Department of Modern Languages,M.I.T., a new publication called MECHANICAL TRANSLATION is being published ona trial basis for one year. During the year, issues will appear at irregular intervalsas material becomes available. The purpose of the publication is to provide this newand rapidly growing field with a much needed means of communication and exchange ofideas. The first issue, which appeared in March, contains an annotated bibliographyof 41 entries.B. LANGUAGE AS AN ERROR CORRECTING CODEWorkers in the field of mechanical translation have expressed surprise that a word-for-word translation is as good as it is. Their intuitive feeling apparently was that itwould be next to worthless. Experiments in which the words of the foreign language havebeen replaced by lists of all possible English equivalents for each word would give mil-lions or billions of possible sentences based on the various choices of meanings; yet aperson can find his way through such a maze with fair facil:ty and pick out the particularmeanings that are best. Such experiments have focused attention on the error correctingnature of language.The error correcting codes that have been investigated in information theory litera-ture operate by recoding the original message in a special way involving constraintsbetween successive symbols so that an error in one or more of the received symbolscan be corrected and the original message restored. One way in which this can beunderstood is to consider all possible sequences of symbols as possible receivedsequences. Now if only a fraction of these possible sequences is permitted to be trans-mitted as messages, we have introduced a form of constraint. If the permitted sequencesare different enough, an error in one or more symbols of one of these sequences willnot transform it into another permitted sequence. In fact, it may still be differentenough from all other permitted sequences to be identifiable by an appropriate recoverytechnique. The constraints that are effective in providing the error correcting natureof the code have nothing to do with the relative transmission probabilities of the variouspermitted message sequences. The important thing is that some sequences of symbolsare allowed and some are not.It is interesting, in the light of the demonstrated error correcting properties of lan-guage, to investigate the amount of the redundancy of language that is due to the fact thatcertain sequences are virtually impossible, and the amount that is due to the fact thatthe allowed sequences have widely different probabilities. Consideration of the numberof letter sequences actually assigned to words in any given language reveals that most-73-(XV. MECHANICAL TRANSLATION)of the redundancy at the word level is due to the fact that only a very small fraction ofthe possible letter sequences are actual words. Only a small amount of the redundancyis contributed by the widely varying probabilities of the allowed words. For example,Shannon calculates that the entropy is 11.82 bits per word on the basis of a vocabularyof 8727 words and a law of word probability given by pn= 0.1/n, where n is the rank ofthe word when ordered by frequency. If these 8727 words were considered equiprobable,the entropy would be log 2 8727 = 13.08 bits per word. The small difference, 1.26 bits perword, can be attributed to the effect of the hyperbolic law of probability used.There is, however, an outstanding difference between the way in which constraints.are incorporated in language and the way in which they have been incorporated in errorcorrecting codes in the past. Constraints are incorporated in language on various levels.For example, out of all possible letter sequences, very few are actually assigned towords. Out of all possible word sequences, only a few are actually assigned to phrases.Out of all possible phrase sequences, only a few are actually assigned to sentences. Outof all possible sentence sequences, only a few make coherent paragraphs. The fact thatlanguage is structured on different levels aids in reconstructing the original of a garbledmessage. At the word level, one can frequently decide whether a word has been mis-spelled, or whether there has been a misprint. At the phrase level, one can tell if a wordhas been transformed into another meaningful word, and so on. This property of incor-porating the constraints at various levels makes it possible to get a certain amount oferror correction immediately at the lowest levels, and then to have additional error cor-recting ability as the message grows longer. Occasionally we do not decide what a givenword must have been until several sentences have gone by. The efficiency of these lan-guage constraints in providing error correction has not been thoroughly investigated.On the other hand, error correcting codes in the past have worked on only one level.The material to be transmitted has been divided into blocks of a given length andrecoded into a code with special constraints which can correct a certain number oferrors in each block. However, errors are usually produced at random, and the numberof errors per block is not constant, but follows a Poisson distribution. Thus with anaverage of one error per block, a one-, three-, or five-error correcting code will leaveuncorrected over 26 percent, nearly 2 percent, or over 0.05 percent of the blocks,respectively. It has proved difficult to construct high-order error correcting codes withvery long blocks, and they would prove unwieldy in practice.It appears that error correcting codes could be designed with advantage on a layerprinciple, as is language. One could correct single errors on the lowest level. On thenext level, the multiple errors could be corrected on a word basis with fewer addi-tional symbols needed than on the lowest level. The process could be continued in thismanner up through the layers. In section IX, Prof. P. Elias has worked out such a codeand has shown it to possess certain definite advantages.-74-

XVI. MECHANICAL TRANSLATIONDr. V. H. YngveA concise description of the structure of a language is something that is being soughtmore and more by linguists. If such descriptions for various languages were availablein a fairly complete form the problem of devising mechanical translation routines wouldbe a great deal simpler.The statistics of language has been investigated by many people. In particular, muchwork has been done in counting the frequency of different words as they appear in care-fully selected samples of language.An investigation has been made of the possibility that statistical information on theoccurrence of words might be useful in finding a description of language. If a largesample of language is obtained, and certain words are selected throughout the text, onecan investigate the distribution in the lengths of the gaps that appear between selectedwords. A gap is measured between two selected words by counting the number of wordsthat occurs between the two words. The two selected words may be called the initialword and the final word, though neither is counted as part of the gap. The smallestpossible gap is zero when the two selected words are adjacent.If the initial and final words are selected at random throughout the text, the distri-bution of gap lengths follows an exponentialp(n) = f exp(-kn) with k = - z(l1-f)where p(n) is the probability of a gap of length n, and f is the frequency of the selectedwords in the text, each selected word being used once as an initial word and once as afinal word.If the initial word is selected at random, but the final word is the first occurrenceof a particular given word, the distribution of gap lengths will again be an exponential.If, on the other hand, both the initial and the final word are not selected at random,but each is a particular given word, the observed distribution of gap lengths will departfrom the exponential. Generally, if the initial word and the final word are alike, gapsof zero will be less frequent than expected because of the fact that English rarely doublesa word. In this case we may say that like words tend to repel. Certain words selectedas initials and certain other words selected as finals show an attraction. For example,the words "of" and "the" have more gaps of zero length than would be expected fromrandom behavior.The structure of language can be considered as a deviation from randomness; thusit is that the comparison of gap distributions actually observed with the distributionsexpected on a random basis can lead to information, obtained entirely by the use of sta-tistical techniques, on the structure of language. A more complete discussion of thepossibilities and difficulties of the method is being prepared for journal publication.-74-

http://dspace.mit.edu/bitstream/1721.1/51185/1/RLE_QPR_033_XV.pdf

Mechanical Translation

Abstract

Similar works

Full text

Available Versions

DSpace@MIT

DSpace@MIT

DSpace@MIT