Williams, Ronald J.

English

eScholarship - University of California

UC MercedProceedings of the Annual Meeting of the Cognitive Science SocietyTitleOn Self-Organization in Connectionist NetworksPermalinkhttps://escholarship.org/uc/item/8q306874JournalProceedings of the Annual Meeting of the Cognitive Science Society, 6(0)AuthorWilliams, Ronald J.Publication Date1984 Peer reviewedeScholarship.org Powered by the California Digital LibraryUniversity of California353 O n Self-Organizatio n i n Connectionis t  Network s Ronal d J .  William s Institut e fo r  Cognitiv e Scienc e Universit y o f  California ,  Sa n Dieg o C-01 5 La Jolla .  C A 9209 3 The ai m o f  thi s pape r  i s t o presen t  som e observation s abou t  certai n type s o f  representations ,  o r encodings ,  i n connectionist ,  o r  neural-like ,  networks .  I n particular ,  thi s pape r  wil l  cal l  attentio n t o tw o distinc t  categorie s o f  encodin g i n suc h network s an d examin e som e result s bearin g o n th e issu e o f self-organizin g network s whic h us e on e o r  th e othe r  typ e o f  encoding .  Thi s discussio n wil l  b e limite d t o th e encodin g o f  dat a whic h i s fundamentall y numerica l  (or ,  mor e precisely ,  geometric) .  I t  i s  a n interestin g questio n whethe r  semanti c dat a ca n als o b e imbedde d i n a  geometri c framework ,  bu t  suc h matter s wil l  b e ignore d here . A number of interesting attempts have been made to provide an answer to the general problem of  ho w a  networ k migh t  b e shape d t o a  particula r  environmen t  throug h self-organization .  A m o n g thes e ar e th e earl y pcrceptro n studie s o f  Rosenblat t  (1962) ,  th e investigation s int o possibl e neura l  ne t dynamic s b y Grossber g (1980) ,  th e recen t  theoretica l  approac h o f  Hinto n &  Sejnowsk i  (1983) ,  an d severa l  work s wit h th e goa l  o f  finding  way s i n whic h cell s i n visua l  corte x migh t  becom e tune d t o specifi c  feature s throug h self-organizatio n (vo n de r  Malsburg ,  1973 ;  Nas s &  Cooper ,  1975 ;  Bienenstoc k et  al. ,  1982) .  T w o recen t  work s whic h shar e a  commo n perspectiv e wit h th e approac h t o b e take n her e ar e tha t  o f  Kohone n (1982 )  an d tha t  o f  Amar i  (1983) . On the other hand, a number of non-self-organizing connectionist networks have been hand-crafte d t o perfor m particula r  sensor y o r  cognitiv e processin g task s i n way s whic h ar e generall y intende d t o accoun t  fo r  huma n performanc e dat a and/o r  b e compatibl e wit h neurobiologica l  dat a (Feldma n &  Ballard ,  1982 ;  Ballard ,  1981 ;  McClellan d &  Rumelhart ,  1981 ;  Hinton ,  1981) .  Certai n classe s o f  networ k hav e eve n bee n propose d a s havin g a  certai n universalit y i n sensor y processing ,  a t leas t  i n th e visua l  syste m (Ballard ,  1981) .  Suc h universalit y migh t  reasonabl y b e take n a s makin g suc h network s plausibl e candidate s fo r  th e actua l  implementatio n o f  thes e algorithm s i n th e brain .  I t  the n becomes reasonabl e a s wel l  t o investigat e possibl e mechanism s b y whic h suc h network s migh t  b e abl e t o self-organiz e t o som e degree ;  i f  suc h mechanism s ca n b e show n t o exist ,  i t  coul d the n b e argue d tha t  thes e type s o f  networ k represen t  a  genera l  processin g strateg y whic h coul d find  wid e applicabilit y i n th e brain . Thi s researc h wa s supporte d by « gran t  t o Davi d Zipse r  fro m th e Syste m Developmen t  Foundation . Copyright O 1984 Ronald J. Williams ;i5 4 I n wha t  follows ,  attentio n wil l  b e restricte d t o th e followin g 2-laye r  architecture : 7 ^ T f T - T ^ / \ •  •  • AI  an d A 2 ar e layer s o f  unit s an d eac h ma y optionall y hav e fixed  latera l  connections .  A s depicted , ther e ar e connection s fro m A  j  unit s t o A j  units ,  an d thes e wil l  b e assume d t o b e variabl e an d thu s sub -jec t  t o self-organization .  I n addition ,  ther e ma y b e connection s (no t  depicte d above )  fro m A 2 unit s t o AI  units ,  an d thes e ma y als o b e variable .  Th e reaso n tha t  onl y th e A i  -  A 2 connectivit y i s depicte d abov e i s tha t  i t  i s  th e A j  -  A 2 transfe r  functio n tha t  i s  o f  paramoun t  interes t  here .  Specifically ,  inpu t t o th e syste m wil l  b e assume d t o consis t  o f  a  patter n o f  activatio n i n th e A j  layer ,  an d outpu t  wil l  b e take n a s th e resultin g patter n o f  activatio n i n th e A 2 layer .  Whil e thi s networ k i s bein g considere d her e i n isolation ,  on e ma y vie w thi s mor e generall y a s simpl y a  sub-networ k consistin g o f  tw o adjacen t layer s i n a  large r  hierarchica l  network . I will consider the Aj - A2 transfer function as performing a mapping between two individual encodings ,  fro m tha t  i n th e A i  laye r  t o tha t  i n th e A 2 layer .  A  patter n i n eithe r  laye r  ca n b e con -sidere d a s a  Euclidea n vecto r  whos e coordinate s ar e simpl y th e respectiv e activatio n value s o f  al l  o f th e unit s i n tha t  layer .  Th e distinctio n I  sugges t  drawin g betwee n encoding s essentiall y  revolve s aroun d ho w usefu l  suc h a  vecto r  spac e descriptio n i s fo r  capturin g th e essentia l  dimension s alon g whic h th e lawfu l  pattern s ma y vary . A full characterization of the two types of encoding will not be given here; it will be sufficient fo r  presen t  purpose s t o simpl y giv e example s o f  eac h an d t o cit e a  closel y relate d distinctio n alread y existin g i n th e literature . Ballard (1981) calls attention to the distinction between having each unit in a network represent a particula r  poin t  i n a  paramete r  spac e (wit h it s activatio n representin g confidenc e i n th e validit y o f tha t  point )  an d havin g unit s whos e activatio n represent s th e valu e o f  a  (necessaril y  one-dimensional ) parameter .  Th e forme r  i s calle d a  valu e uni t  encodin g b y Ballar d an d i s use d extensivel y i n hi s gen -eralize d Houg h transfor m approac h t o earl y visua l  processing ;  th e latte r  i s  calle d a  variabl e uni t  encod -ing . For purposes of this paper, call any variable unit encoding a Type I encoding; the class of Type II encoding s wil l  includ e an y valu e uni t  encodin g a s wel l  a s an y representatio n typifie d b y pixel-leve l description s o f  retina l  images . As a  concret e example ,  conside r  a  1-dimensiona l  arra y o f  1 0 unit s suc h tha t  th e onl y pattern s whic h appea r  i n thi s arra y al l  consis t  o f  tw o adjacen t  I' s  wit h th e res t  O's .  Thi s i s a  Typ e I I  encodin g of  a  patter n spac e whic h ma y b e considere d essentiall y  one-dimensional ;  th e 10-dimensiona l  vector s 355 whic h represen t  th e pattern s jum p aroun d i n th e spac e i n suc h a  wa y tha t  thi s one-dimensionalit y i s not  easil y recognized .  Thi s one-dimensionalit y i s reall y a  consequenc e o f  th e manne r  i n whic h th e pattern s overlap . In contrast, this same pattern space may be given a Type I encoding in a single unit whose activatio n i s a  monotoni a functio n of ,  say ,  th e distanc e o f  th e leftmos t  1  i n th e patter n fro m th e left -hand en d o f  th e array . At this point, the central thesis of this paper can be stated: Self-organizing mappings from Type I representation s i s straightforward ;  self-crganizin g mapping s fro m Typ e I I  representations ,  i f  possibl e a t al ,  wil l  requir e th e us e o f  mechanism s ye t  t o b e discovered .  I n suppor t  o f  th e first  hal f  o f  thi s thesis ,  I presen t  th e followin g tw o example s o f  self-organizin g mappings ,  th e first  take n fro m wor k o f  Kohone n (1982 )  an d th e secon d fro m recen t  wor k o f  m y own .  Followin g thes e example s i s a  discussio n i n sup -por t  o f  th e secon d hal f  o f  thi s thesis . Example 1. Let the A2 layer have a certain pattern of lateral feedback connections so that the onl y pattern s o f  activit y whic h i t  support s ar e suc h tha t  al l  non-zer o activit y i s  confine d t o a  ver y smal l number  o f  nearb y units .  I n particular ,  assum e tha t  th e unit s ar e lai d ou t  i n 2-dimensiona I  spac e i n suc h a  wa y tha t  nearb y unit s excit e on e anothe r  bu t  mor e distan t  unit s inhibi t  on e another .  Suppos e tha t  th e A  \  laye r  consist s o f  2  units ,  wit h pattern s draw n uniforml y fro m a  conve x subse t  o f  Euclidea n 2-space .  Suppos e als o tha t  ther e ar e n o A 2 -  A x connections .  The n Kohone n (1982 )  ha s show n that , by usin g a  commo n varian t  o f  wha t  ha s com e t o b e know n a s th e Heb b learnin g rule ,  th e A j  -  i4 2 mappin g wil l  generall y self-organiz e i n suc h a  wa y tha t  nearb y unit s respon d mos t  strongl y t o nearb y patterns .  ̂  Th e resultin g mappin g re-code s th e 2-dimensiona l  patter n spac e implici t  i n th e activation s of  th e A x unit s i n suc h a  wa y tha t  it s 2-dimcnsionaIit y become s explici t  i n th e A 2 layer .  I n th e languag e o f  Ballar d (1981) ,  th e resultin g mappin g ca n b e sai d t o tur n a  variabl e uni t  encodin g int o what  i s essentiall y  a  valu e uni t  encoding ;  i n th e terminolog y o f  thi s paper ,  th e resultin g mappin g re -cast s a  Typ e I  representatio n int o a  particula r  Typ e I I  representation . Example 2. -Let the system have no laferal connections in either the Ax layer or the A2 layer, but le t  ther e b e reciproca l  A 2 -  A j  connections .  Le t  th e A x unit s appl y a  weigh t  modificatio n rul e t o thei r incomin g A 2 -  A j  connection s whic h ha s th e effec t  o f  tryin g t o mor e closel y matc h thei r  curren t  pat -tern ;  furthermore ,  le t  the m appl y thi s sam e correctio n t o thei r  outgoin g A 1 -  A 2 connections .  Then , i f  th e bottom-u p an d top-dow n connection s ar e symmetrical ,  ̂  th e syste m perform s a  principa l  com -ponen t  analysi s o f  th e trainin g stimul i  durin g self-organization .  Mor e precisely ,  le t  n 2 denot e th e number  o f  unit s i n th e A 2 layer .  Then ,  i f  thi s syste m i s traine d wit h pattern s havin g mea n 0 ,  self -organizatio n cause s th e outpu t  correspondin g t o an y give n inpu t  vecto r  t o consis t  o f  a  projectio n ont o th e subspac c spanne d b y th e eigenvector s correspondin g t o th e n 2 larges t  eigenvalue s o f  th e scatte r matri x o f  th e trainin g stimuli .  Thi s outpu t  i s expresse d i n som e orthonorma l  basi s whic h nee d no t  b e thes e eigenvector s themselves .  I n othe r  words ,  individua l  unit s i n A 2 wil l  no t  necessaril y  b e featur e detector s fo r  individua l  principa l  components ;  instea d th e outpu t  encodin g ma y b e distribute d wit h respec t  t o thes e components .  A  fulle r  accoun t  o f  th e detail s o f  thi s syste m an d a n analysi s o f  it s behavio r  wil l  appea r  elsewhere . 1. 1 hav e slightl y simplifie d th e actua l  detail s o f  Kohonen' s wor k i n orde r  t o avoi d discussio n o f  techni -cal  matter s no t  german e t o thi s presentation . 2. These weights need not be assumed symmetrical at the outset; the simple trick of allowing all weight s t o deca y slowl y wil l  accomplis h th e necessar y symmetr y eventually . 356 The ke y poin t  t o b e mad e abou t  th e syste m o f  Exampl e 2  i n th e contex t  o f  thi s pape r  i s tha t  i t readil y self-organize s a  usefu l  mappin g fro m on e Typ e I  representatio n t o another . While the work of Kohonen (1982) and Amari (1983) may leave one with the impression that certai n Typ e I I  -  Typ e I I  mapping s ma y b e self-organize d i n th e sam e wa y a s describe d i n Exampl e 1 , I  woul d clai m that ,  i n general ,  a  goo d mappin g i s no t  achieve d throug h th e applicatio n o f  suc h a learnin g rule .  Fo r  example ,  suppos e tha t  th e A j  laye r  i s a n identica l  cop y o f  th e A 2 laye r  a s describe d i n Exampl e 1  an d th e syste m i s expecte d t o self-organiz e wha t  i s essentiall y  a n identit y mapping .  Thi s i s a  simpl e versio n o f  th e proble m o f  formin g a  topographi c ma p between ,  fo r  example ,  th e retin a an d visua l  cortex .  Whil e Amar i  (1983 )  make s certai n claim s abou t  suc h self-organizatio n bein g possible ,  h e readil y concede s tha t  i t  i s  difficul t  t o obtai n a  topographi c ma p fro m suc h a  syste m i f  on e start s wit h totall y rando m initia l  connections .  I n fact ,  m y simulation s o f  suc h a  syste m woul d sugges t  tha t  unles s one start s wit h initia l  connection s ver y clos e t o wha t  on e intend s a s th e final  outcome ,  th e syste m i s ver y unlikel y t o for m a  tru e topographi c map .  Th e majo r  difficulty ,  i t  appear s t o me ,  i s tha t  th e learnin g rul e basicall y require s that ,  a t  statistica l  equilibrium ,  th e stimulu s vecto r  t o whic h eac h A 2 uni t  mos t  strongl y respond s mus t  b e equa l  t o a  weighte d averag e o f  th e stimulu s vector s t o whic h it s neighbor s (i n th e topolog y o f  th e latera l  connectivit y o f  A2 )  mos t  strongl y respond .  Thi s i s th e under -lyin g reaso n wh y suc h a  syste m work s wel l  fo r  self-organizin g conve x Typ e I  input ,  an d wh y I  clai m tha t  i t  canno t  b e expecte d t o d o th e sam e fo r  Typ e I I  input .  Indeed ,  thi s fundamenta l  differenc e i s th e mai n motivatio n behin d m y drawin g th e distinctio n betwee n thes e tw o type s o f  representation . Thi s argumen t  i n fac t  suggest s tha t  an y learnin g rul e whic h cause s a n A 2 uni t  t o lear n t o respon d t o an averag e o f  th e stimulu s vector s whic h hav e excite d i t  canno t  b e expecte d t o achiev e a  goo d map -pin g betwee n Typ e I I  encodings .  Som e othe r  learnin g rul e mus t  b e use d t o achiev e this . Thus I argue that it remains an open question whether mappings can, in general, be self-organize d fro m a  Typ e I I  representatio n t o anothe r  Typ e I I  representation .  Discover y o f  suc h a mechanis m woul d b e quit e interesting ,  sinc e i t  i s  possibl e t o specif y latera l  connectivit y pattern s i n th e A2 laye r  whic h coul d forc e an y particula r  topolog y o n th e stimulu s space .  A s a n example ,  i f  th e A 2 laye r  ha s th e connectivit y o f  a  Mobiu s ban d an d th e A i  laye r  i s a  patc h o f  2-dimensiona l  retin a upo n whic h pattern s o f  activit y ar e elongate d bar s o f  variou s orientatio n an d position ,  the n applicatio n o f thi s mechanis m shoul d lea d t o a  mappin g i n whic h eac h A 2 uni t  i s  maximall y responsiv e t o a  particula r combinatio n o f  orientatio n an d position .  On e ca n imagin e self-organizin g jus t  abou t  an y Hough-styl e transfor m i n thi s manner . Another intriguing possibility which is suggested by this work is that of self-organizing a map-pin g fro m a  Tyjs e I I  representatio n t o a  Typ e I  representation .  A s a n exampl e o f  suc h a  mapping ,  con -side r  a  descriptio n o f  a  connecte d patter n o n a  2-dimensiona I  retin a i n term s o f  Fourie r  descriptor s fo r  th e boundar y (Zah n &  Roskies ,  1972 ;  Persoo n &  Fu ,  1977 )  alon g wit h th e coordinate s o f  th e cente r  o f  mass ,  al l  encode d i n variabl e units .  Wha t  i s appealin g abou t  thi s particula r  exampl e i s tha t i t  shoul d b e muc h mor e economica l  i n bot h unit s an d connection s t o comput e a  mappin g betwee n a retina-base d descriptio n o f  a n objec t  an d a n object-base d descriptio n o f  tha t  objec t  (Hinton ,  1981 )  i f thes e description s ar e encode d i n variabl e unit s tha n i f  the y ar e encode d i n valu e units . 3b7 Reference s Amari, S. (1983). Field theory of self-organizing neural nets. IEEE Transactions on Systems, Man. and Cybernetics ,  13 ,  741-748 . Ballard, D. H. (1981). Parameter networks: Towards a theory of low-level vision. Proceedings of the Sevent h Internationa l  Join t  Conferenc e o n Artificia l  Intelligence ,  Vancouver ,  B.C. ,  Canada ,  1068-1078 . Bienenstock, E. L., Cooper, L. N., & Munro, P. W. (1982). Theory for the development of neuron selectivity :  Orientatio n specificit y an d binocula r  interactio n i n visua l  cortex .  Journa l  o f  Neurosci -ence ,  2 ,  32-48 . Feldman,  J. A., & Ballard, D. H. (1982). Conncctionist models and their properties. Cognitive Science, 6,  205-254 . Grossberg, S. (1980). How does the brain build a cognitive code? Psychological Review, 87, 1-51. Hinton, G. E. (1981). A parallel computation that assigns canonical object-based frames of reference. Proceeding s o f  th e Sevent h Internationa l  Join t  Conferenc e o n Artificia l  Intelligence ,  Vancouver ,  B.C. , Canada ,  683-685 . Hinton, G. E., & Sejnowski, T. J. (1983). Analyzing Cooperative Computation. Proceedings of the Fift h Annua l  Conferenc e o f  th e Cognitiv e Scienc e Society ,  Rochester ,  N Y ,  683-685 . Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological Cyber-netics ,  43 ,  49-69 . McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in lette r  perception ,  Par t  1 :  A n accoun t  o f  th e basi c findings.  Psychologica l  Review ,  88 ,  375-407 . Nass, M. M., & Cooper, L. N. (1975). A theory for the development of feature detecting cells in the visua l  cortex .  Biologica l  Cybernetics ,  19 ,  1-18 . Persoon, E., & Fu, K. (1977). Shape discrimination using Fourier descriptors. IEEE Transactions on Systems .  Man ,  an d Cybernetics ,  7 ,  170-179 . Rosenblatt, F. (1961). Principles of neurodynamics: Perceptrons and the theory of brain mechanisms. Washington ,  D C :  Spartan . von dcr Malsburg, C. (1973). Self-organization of orientation sensitive cells in the striate cortex. Kybernetik ,  14 ,  85-100 . Zahn, C, & Roskies, R. (1972). Fourier descriptors for plane closed curves. IEEE Transactions on Computers ,  21 ,  269-281 . 

On Self-Organization in Connectionist Networks

https://escholarship.org/content/qt8q306874/qt8q306874.pdf

On Self-Organization in Connectionist Networks

Abstract

Similar works

Full text

Available Versions

eScholarship - University of California