Case-based reasoning is deemed an important technology to alleviate the bottleneck of knowledge acquisition in Artificial Intelligence (AI). In case-based reasoning, knowledge is represented in the form of particular cases with an appropriate similarity measure rather than any form of rules. The case-based reasoning paradigm adopts the view that an Al system is dynamically changing during its life-cycle which immediately leads to learning considerations.

Within the present paper, we investigate the problem of case-based learning of indexable classes of formal languages. Prior to learning considerations, we study the problem of case-based representability and show that every indexable class is case-based representable with respect to a fixed similarity measure. Next, we investigate several models of case-based learning and systematically analyze their strengths as well as their limitations. Finally, the general approach to case-based learnability of indexable classes of formal languages is prototypically applied to so-called containmet decision lists, since they seem particularly tailored to case-based knowledge processing

Globig, Christoph

Jantke, Klaus P.

Lange, Steffen

Sakakibara, Yasubumi

English

HSSS - Hochschulschriftenserver der SLUB

On case-based learnability of languages

Qucosa

Qucosa - Publikationsserver der Universität Leipzig

An Algorithm for Generalized Principal Curveswith Adaptive Topology in Complex Data SetsG. Balzuweit1, R. Der1, M. Herrmann2, M. Welk11 Universitat Leipzig, Inst. fur Informatik,Augustusplatz 10/11, D-04109 Leipzig, Fed. Rep. of Germanyder@informatik.uni-leipzig.de2 RIKEN, Lab. for Information Representation,2-1 Hirosawa, Wako-shi, 351-01 Saitama, Japanmichael@murasaki.riken.go.jpAbstract. Generalized principal curves are capable of representing com-plex data structures as they may have branching points or may consist ofdisconnected parts. For their construction using an unsupervised learningalgorithm the templates need to be structurally adaptive. The presentalgorithm meets this goal by a combination of a competitive Hebbianlearning scheme and a self-organizing map algorithm. Whereas the Heb-bian scheme captures the main topological features of the data, in themap the neighborhood widths are automatically adjusted in order to sup-press the noisy dimensions. It is noteworthy that the procedure which isnatural in prestructured Kohonen nets could be carried over to a neuralgas algorithm which does not use an initial connectivity. The principalcurve is then given by an averaging procedure over the critical uctua-tions of the map exploiting noise-induced phase transitions in the neuralgas.1 IntroductionAnalogous to principal components, principal curves represent essential datafeatures if the underlying functional relation is non-linear. Each point of a prin-cipal curve represents the local centroid of the data in the orthogonal directionsand provides thus an estimation of a one-dimensional relation in noise-corrupteddata. To construct principal curves constitutes an elementary task in non-linearstatistics which may be approached in simple cases [2, 3, 4] non-parametricallyusing suitably modied Kohonen maps with a chain-like topology. Whereas instatistics the centroid condition is accompanied by a loosely dened smoothnessrequirement for the principal curve, in self-organizing maps both smoothnessand local averaging are combined in a most natural way.In the present contribution we will address this task in a more general man-ner. The algorithm proposed here goes beyond Kohonen maps as network struc-tures are generated in a self-adaptive manner. This allows to extract generalizedprincipal curves which may have branching points or may be disconnected, whileexhibiting the typical averaging properties in the non-exceptional regions. Thus,non- or multiply connected data structures can be represented without any priorknowledge of the number of components and their respective structure. In thisframework a representation of principal curves is given in terms of a discreteset of reference vectors which are updated by a special self-organizing map algo-rithm which combines adaptation schemes for the reference vectors, for the mapparameters and for the network structure. The smoothness condition is includedby intrinsic elasticity properties of self-organizing maps which are adaptivelymodied in dependence on the local variance of the noise. Structure adaptationis performed by a simple Hebbian learning rule, and the reference vectors form-ing the principal curve arise as averages over uctuations of the maps at theborder of stability.In the next section the main achievement of the present work, an adaptationscheme for the network topology, is derived. The following sections present thealgorithm and results for several typical examples.2 Collaboratively learning vector quantizersIn on-line learning vector quantizers, reference vectors wr are adjusted towardsa current input datum v 2 V,wr(t + 1) = wr(t) + "hr(v;w)(v  wr(t)); (1)where w = (wr jr 2 A), " is the learning rate, wr is a vector in the input spaceV, the range of the indices r is a discrete output (code) space A, and hr(v;w)involves the interaction among the units. In Kohonen's algorithm the interactionis formulated by the neighborhood functionhr(v;w)  hrs = exp kr   sk222; s = argminrkwr   vk; (2)where kr  sk is the distance between neurons r and s according to the topologyof the neuron space. For a topology preserving mapping, this topology has tobe known, which is the main disadvantage of Kohonen's algorithm. Instead, onehas to infer this topology from the data. The neural gas algorithm [6] is suitablefor this purpose while otherwise (cf. the following section) exhibiting similarproperties as Kohonen's learning rule. Namely, instead of (2) the neighborhoodfunction relies on the rank R(r) in the ordered sequence of distances kwr   vk.For the best matching unit s0 we have R(s0) = 0, the second-best has R(s1) = 1and so on. Thus,hr(v;w) = exp ( R(r)=) (3)and the weights are learned according to (1) while decreasing . Eventually, thetopology is represented in terms of a connectivity matrix arising from a simpleHebbian learning rule, cf. eq. (5) below.However, the topology learnt in this way is that of the noisy data and not ofthe generalized principal curve to be constructed. A principal curve is obtainedfrom the neural gas by keeping the interaction width  suciently high so thatthe weights are forced into chain-like structures. The algorithm given below willself-consistently adapt these widths locally to obtain the desired principal curverepresentation of the data set.3 Phase transitions in the neural gasIn self-organizing maps evolving according to Kohonen's algorithm, phase tran-sitions due to dimensional conicts between the input topology and that of theoutput lattice of neurons are well investigated. A generic example (cf. [7]) isthe mapping of a rectangular data distribution onto a one-dimensional lattice ofneurons.Viewing the rectangle as a straight line corrupted by noise, the height s ofthe rectangle measures the variance of the noise. For low noise, s < sc = 2:02where  is the neighborhood width, the chain of neurons is mapped onto theoriginal data distribution, i. e. the straight line.At s = sc the noise induces a phase transition to a new stable congurationcorresponding to a folding of the map into the input space. There is a coex-istence between two phases of similar shape where one of the two phases stillconserves topology whereas the other one is topology violating. The latter is themore stable one so that the phase transition is signaled usually by the topologyviolations.In the neural gas scenario there is no xed topology (like that of the chainin the example above). Nevertheless one observes phase transitions very similarto the one described above. The point is that for suciently large  the neuronsare mapped chain-like into the rectangular input space. This is an immediateconsequence of the cooperativity in learning introduced by the neighborhoodfunction. Now, for given neighborhood width  there is a critical noise levelsc = sc() such that for s < sc the neurons are mapped onto the median of therectangular data distribution while for s > sc we observe a wave like structurevery much similar as the one observed with the Kohonen map. Of course there isno topology violation now, since there is no neighborhood between the neuronsdened. However it can be observed that in the \folding" phase the rst andsecond winner are no longer neighbors in input space. There are further neuronslying between the two in the sense that their (Euclidean) distance from both therst and the second winner is smaller than the distance between the rst andsecond winner themselves. The number of these neurons \in between" will servein the following for xing the local value of the neighborhood function optimally.4 Topology adaptationWe start with a modication of the neural gas algorithm with real-valued con-nectivity matrix C. The connection value Crs expresses the current belief onwhether neurons r and s are connected; it is increased each time that r and sare the rst and second winner while in each step all connections are weakenedsimultaneously. In evaluation, a connection is considered active according to athreshold condition. The change of topology caused by a new connection beingcreated is used as a criterion for topology violation which controls the learning ofan individual neighborhood width. The ranks R(r) are assigned to the units ineach step in the order of their distances kwr ws0k from the rst winner (ratherthan the input vector), thus enforcing a topological aspect in weight learning.In topology adaptation, the neighborhood function (3) displays a disappoint-ing behavior insofar as single points tend to \escape" from the adapted topologyscheme. Therefore another neighborhood function was derived from (2) by re-placing kr   sk by the rank R(r).Another sort of instabilities still emerges as a result of the ranking by Eu-clidean distances. This will support clustering of neurons since for instance in acluster of two, the stimuli which make the two the rst and second winner willalways move them closer together. Note that, e.g., for a linear equidistant (chain-like) arrangement of units the rank alternates (5{3{1{0{2{4{6), thus making theimmediate neighbors of the winner learning with dierent strength, whereas un-der (2) both are co-learning with the same strength.To overcome these instabilities one wishes to assign the neighborhood ranksin analogy to the Kohonen algorithm by searching the C matrix for neighbors.However, matrix searching is of high numerical complexity. This led us to try aneighborhood function based on searching only immediate neighbors in the Cmatrix and ranking by Euclidean distance from the winner for all other units. Itturned out that such a \mixed ranking" grants essentially all the benets of afull matrix search while preserving the lower numerical complexity of Euclideanranking.As can be seen from the gures, extra connections can be observed in somecases near branching points. These represent an uncertainty in the position ofthe branching points caused by noise. For purposes like object recognition theycan be removed in a subsequent step.5 The algorithmFor each input signal v drawn from a distribution P (v) the rst and secondwinner s0, s1 are calculated. The neighborhood parameter  is initialized ate.g. N=3 (with N being the number of units) and is decreased slowly later on:r := maxfr   "r; 1g: (4)The connection values Crs develop by decreasing in each step all links simulta-neously, followed by an increase of the link between the rst two winners.Cij =  1N"cCij 8i; j and Cs0s1 = "c (5)If in this way a new link has been created (i.e. has grown across the threshold)the neighborhood parameter r is increased for all neurons with pointers wrgeometrically between ws0 and ws1, counteracting to the persistent decrease of. We consider those neurons as situated between the rst and second winnerwhich are closer to both of them than these are to each other, i.e. for whichkwr  ws0k < kws0  ws1k and kwr  ws1k < kws0   ws1k: (6)Fig. 1. This Y-shaped data distribu-tion cannot be represented appropri-ately by a linear chain. Instead, a gener-alized principal curve with one branch-ing point is constructed by adaptivelystructuring a network of 50 neurons.Fig. 2. Representation of a multiplyconnected data set by a closed general-ized principal curve with two branchingpoints; N = 70. is increased towards an estimated optimal ̂opt which is proportional to thenumber of neurons satisfying (6) with a global proportionality constant.r = (̂opt   r) if r < ̂opt (7)Further, the r must not exceed their initial value.Next, sort the list of distances kwr   ws0k in an ascending order and denea modied neighborhood function hr(v; fwg) by assigning ranks Rr = 1 to allunits which are directly connected to the winning neuron, while using the usualranks otherwise. The update rule of the algorithm nally readswr = " exp 2R2r(r + s0)2(v  wr) : (8)6 ConclusionIn the present contribution the principal curve problem was solved in a quitegeneral way. This task required an algorithm that combines learning at dierentlevels, in particular of the network structure, while retaining the capabilities ofthe learning mechanisms at the other levels. In the present algorithm the solutionof the principal curve problem was not given in terms of a convergent networkstate, but as an average over the network's dynamical behavior in the vicinity-505-505-505101520Fig. 3. Solution of a two-spirals prob-lem found in an unsupervised man-ner by the present algorithm. The con-structed principal curve consists of twodisconnected chains evolving from thesame initial state of the neural gas asin the above examples. Here, N = 120.of a topological phase transition. The algorithm is relatively ecient since themost time consuming stage is the averaging process.Forthcoming work on this topic will address a theoretical analysis of thephase transition in the neural gas algorithm analogous to the analysis in [7] aswell as a possible generalization to generalized principal manifolds of dimensiongreater than one.Acknowledgement: One of the authors (RD) gratefully acknowledges the hos-pitality of RIKEN (Tokyo) he received during a visit in February 1996.References1. Der, R., Herrmann, M.: Critical Phenomena in Self-Organized Feature Maps: AGinzburg-Landau Approach. Phys. Rev. E 49:6, 5840{5848 (1994).2. Der, R., Balzuweit, G., Herrmann, M.: Constructing Principal Manifolds in SparseData Sets by Self-Organizing Maps with Self-Regulating Neighborhood Width. Toappear in Proc. ICNN'96, Washington DC (1996).3. Herrmann, M.: Self-Organizing Feature Maps with Self-Organizing NeighborhoodWidths. Proc. 1995 IEEE Intern. Conf. on Neur. Networks, 2998{3003 (1995).4. Kiviluoto, K.: Topology Preservation in Self-Organizing Maps. Subm. to Proc. 1996IEEE Intern. Conf. on Neur. Networks (1996).5. Kohonen, T.: The Self-Organizing Map. Springer-Verlag (1995).6. Martinetz, T.: Selbstorganisierende neuronale Netzwerkmodelle zur Bewegungssteue-rung. Gesellschaft fur Informatik e.V. (1992).7. Ritter, H., Martinetz, T., Schulten, K.: Neural Computation and Self-OrganizingMaps. Addison-Wesley (1992).8. Villmann, T., Der, R., Herrmann, M., Martinetz, T.: Topology Preservation inSOFMs: Exact Denition and Measurement. To appear in IEEE Transact. on Neur.Netw. (1994).9. Yuille, A. L., Geiger, D.: Winner-take-all Mechanisms. In: Arbib, M. A.: The Hand-book of Brain Theory and Neural Networks. The MIT Press, 1056{1060 (1995).This article was processed using the LATEX macro package with LLNCS style

https://ul.qucosa.de/api/qucosa%3A32731/attachment/ATT-0/

On case-based learnability of languages

Abstract

Similar works

Full text

Available Versions

HSSS - Hochschulschriftenserver der SLUB

Qucosa

Qucosa - Publikationsserver der Universität Leipzig