G. C.  Gini

M. A.  Neri

N.  Vitucci

VITUCCI, NICOLA

ARRIGONI NERI, MARIO

GINI, GIUSEPPINA

Archivio istituzionale della ricerca - Politecnico di Milano

Using f-SHIN to represent objects:an aid to visual graspingNicola Vitucci, Mario Arrigoni Neri, and Giuseppina GiniPolitecnico di Milano - Dipartimento di Elettronica e InformazioneVia Ponzio 34/5, 20133 Milano, Italy{vitucci,arrigoni,gini}@elet.polimi.itAbstract. Description Logics (DLs) are nowadays used to face a va-riety of problems. When dealing with numerical data coming from thereal world, however, the use of traditional logics results in a loss of usefulinformation that can be otherwise exploited using more expressive log-ics. Fuzzy extensions of traditional DLs, being able to represent vagueconcepts, are well suited to reason on such objects.In this paper we present an architecture for the automatic building andquerying of a fuzzy ontology related to the representation of objects interms of their composing parts. Our approach mainly aims to face theproblem of visual grasping, which is of wide interest in the robotics field.1 IntroductionThe decomposition of an object in parts has been recognized as an importantproblem in artificial intelligence: it is considered both as a human-like way ofreasoning on objects [1] and as a good way to reduce complexity in tasks likeobject recognition [14]. Apart of the actual image decomposition phase, a majorissue is constituted by the semantic description of the extracted features andtheir mutual relationships. Due to the vagueness affecting real world data, sometolerance should be taken into account when formally representing the structureof an object; this is a reason to take advantage of novel tools as fuzzy DLs [11].Fuzzy DLs extend crisp DLs by adding imprecision and vagueness in thereasoning process, thus giving some degrees of truth in place of binary answersas yes or no. Although the available fuzzy reasoners are not yet as powerfulas their crisp counterparts, some interesting applications can be found. One ofthem lies in the robotics field, in which a symbolic representation of objectscan improve the grasping capabilities of a robot by the use of some semanticinformation, regarding both the type of grasp itself and the structure of theobject to be grasped.To the best of our knowledge, the problem of semantic part decomposition isstill an open problem and there are no tools available to automatically create afuzzy ontology from raw concepts. The use of ontologies for object recognition hasbeen investigated in some works as [4,5,6], but none of them makes explicitly useof fuzzy reasoning except for the creation of (crisp) descriptors as Very high to beused in the classical way; furthermore, they rely on a previous phase of semanticannotation by domain experts, while we focus on the automatic generation ofsimple concepts, which are sufficient for our purposes.There are some recent works in which fuzzy DLs are thoroughly used to rea-son on multimedia information (see [7,8,10,12]) but little advantage is taken fromthe expressiveness given by cardinality restrictions (when available). Generallyspeaking, this is due to the fact that, for scene understanding purposes, it issufficient to know whether a kind of object is present or not (see [15,16]). Onthe other hand, for object recognition purposes, it is often necessary to be ableto count the instances of each kind of recognized component.In this paper, we show why we use f-SHIN [9] as the underlying DL foraddressing this problem, then we describe an architecture for the automaticbuilding of a (crisp) ontology and its use for object recognition via fuzzy ABoxreasoning services; eventually, in the last section, we make some considerationsand propose some future work. The architecture we propose here is still far frombeing considered complete, yet we were able to obtain some interesting results.2 The f-SHIN logicThe f-SHIN logic is the fuzzy extension of the SHIN logic [9]. The main im-provement of this extension with respect to its crisp version is the possibility touse assertions like Concept(p)[≥ 0.7], meaning that the individual p has a min-imum degree of participation of 0.7 to the concept Concept, or role(p,q)[≤ 0.3],meaning that the individuals p and q participate in the role role with a maxi-mum degree of 0.3. The greatest lower bound (GLB) [11] is used to know “howmuch” an individual can be considered to belong to a certain class. A completedescription of the f-SHIN logic can be found in [9].For the f-SHIN logic there exists a reasoner called FiRE 1, while there existother reasoners like fuzzyDL2 which is based on the fuzzy extension of the SHIFlogic. The reason why we chose to use FiRE as reasoner is, independently fromthe supported reasoning services, the high expressivity of the underlying f-SHINlogic as it supports cardinality restrictions; on the other hand, such a choice needssome functional blocks to be added to carry out operations like the definition ofconcepts in terms of membership functions.3 ArchitectureAs anticipated in the previous section, due to the limitations of the reasoner,the whole architecture is complex and requires some functional elements to besplit among different modules (e.g. the reasoner used on the definitions ontologyis different from the one used on the objects ontology). The whole architectureof the system is depicted in Fig. 1.1 http://www.image.ece.ntua.gr/~nsimou/FiRE/2 http://gaia.isti.cnr.it/~straccia/software/fuzzyDL/fuzzyDL.htmlTypes of features to extract from the imageNames of fuzzy concepts (e.g. LongObject)Types and parameters ofmembership functionsExternal ontologyImage segmentationand part decompositionFeature extraction for each partSelection of interestingquantitative measuresImage analysisCalculation of relationships among the partsCalculation and selection of truth valuesFuzzy concepts withtruth degreesCreation of component axioms in the TBoxCreation of object axiomsin the TBoxCreation of roles related to component conceptsObjects ontology buildingCreation of component assertions in the ABoxCreation of role assertionsin the ABoxCalculation of the GLB for each componentObject recognitionCalculation of the GLBfor the whole objectFuzzy reasonerFig. 1: The general architecture of the systemThe “high level” information, which reflects the kind of knowledge that isto be extracted from the image, is encoded in the external ontology; the imageanalysis and the numerical calculations are performed with MATLABTM, whilethe intermediate steps are performed either in MATLABTMor in JavaTM. TheFiRE reasoner is standalone, thus some steps are still to be carried out by hand.As an example, we will model a fork in terms of its parts; thus, we will usethe images shown in Fig. 2.3.1 External ontologyThe external ontology, also called the “definitions ontology”, is used to specifythe kinds of membership functions to be used as well as the kinds of features tobe extracted from the objects found in the images (e.g. elongation, eccentricity,parallelism with respect to other objects and so on) and the meanings of conceptslike LongObj and SmallObj in terms of membership functions.Taking the ontology described in [2] as an example, we built a meta-ontology(based on the crisp logic SHOIN (D) with datatypes) in which the features tobe extracted from the image are subclasses of the meta-class GeometricConceptand the kinds of membership functions to use are subclasses of the meta-classMembFunc. The ontology presented in [2] makes use of some “concrete” conceptslike TrapezoidalConcreteFuzzyConcept and TriangularConcreteFuzzyConcept, eachone having several properties defined as hasParameterX (where X stands for A,B, K1 etc.) depending on the parameters needed by the considered membershipfunction; an individual tra1 represents a trapezoidal membership function withgiven parameters.In our ontology, a concept like “a long object” is modeled as an individuallongObject of meta-class Length which has, as its membership function, anotherindividual longMF of a subclass of MembFunc with the function parameters givenas datatype properties. By means of the Jena Ontology API3 and the Pelletreasoner4, information like the kind and the parameters of a membership functionrepresenting a concept related to the image is extracted to feed the image analysismodule; thus, a SPARQL query like:SELECT *WHERE {?x rdfs:subClassOf :GeometryConcept .?y rdf:type ?x .?y :hasMembershipFunction ?z .?z rdf:type ?w .?w rdf:subClassOf :MembershipFunction .FILTER (?w != :MembershipFunction) .?z :hasParameter1 ?k1 .?z :hasParameter2 ?k2 .OPTIONAL {?z :hasParameter3 ?k3} .OPTIONAL {?z :hasParameter4 ?k4}}is used to extract the individuals representing the actual fuzzy geometry concepts(e.g. LongObject) used in the objects ontology and their related membershipfunctions data (e.g. a sigmoidal function with two parameters).The ontology is built by a domain expert to reflect the physical characteristicsof the robot, so that for example an object can be considered “long” with respectto the maximum aperture of the robot hand. Although a system of measurementhas to be established, we now use only pixel measures.3 http://jena.sourceforge.net/4 http://clarkparsia.com/pellet/(a) Original image (b) Image after segmentation(c) Image after edge dilation and part decomposition (with three partsout of six put in evidence)Fig. 2: Steps of the image analysis phase3.2 Image analysisIn this phase, the original image is converted in a binary image after thresholdingand edge recognition performed by Canny method [17] (Fig. 2b); the resultingedges are dilated, then the parts having an area over a threshold are selected(Fig. 2c). This segmentation and decomposition phase is actually non-robust, sothat the use of fuzzy relationships can be better shown.After the first phase, some features like the area, the length of the majoraxis of the ellipse having the same normalized second central moments as theselected region, and so on, are extracted from each found part (see Tab. 1 forsome examples of extracted values); then, some quantitative characteristics arecomputed: for example, the measure of parallelness pi, given α and β as theangles between the major axes of the two objects and the x axis of the image,is defined as pi = | cos (α− β) |, while the distance between two parts, instead,is defined as the minimum distance between their convex hulls.Using the definitions from the external ontology, for every part we calculatethe degree of membership of each feature to its related membership functions.For example, for the feature “length” (i.e. the length of its major axis), thetruth values for the functions “LongObj”, “MediumLengthObj” and “ShortObj”are calculated; if a MediumLengthObj is associated to a generalized bell curvemembership function with parameters a = 240, b = 2.5, c = 600 and the lengthof the major axis of the considered object is 456.61 pixels, the object will belongto the class MediumLengthObj with a truth degree µ = 0.93.Table 1: Examples of features extracted from the image(a) Measures of parallelness be-tween every pair of partsp1 p2 p3 p4 p5 p6p1 1.00 0.97 0.97 0.97 0.97 0.97p2 0.97 1.00 0.89 0.90 0.88 0.89p3 0.97 0.89 1.00 0.99 0.99 1.00p4 0.97 0.90 0.99 1.00 0.99 0.99p5 0.97 0.88 0.99 0.99 1.00 0.99p6 0.97 0.89 1.00 0.99 0.99 1.00(b) Other features (area and lengths are inpixels)Area EccentricityMajor MinorAxes ratioaxis axisp1 14860 0.99 456.61 47.33 0.10p2 12351 0.95 288.23 88.41 0.30p3 2194 0.98 151.78 23.11 0.15p4 500 0.99 93.09 8.81 0.09p5 2617 0.98 181.07 25.79 0.14p6 771 0.99 141.47 9.39 0.063.3 Objects ontology buildingUsing the results from the previous phase, and taking as a working hypothesisthat all the found parts belong to the same object (i.e. there is just one object inthe scene), for each part only the membership functions which give the highesttruth value for each feature are selected; for example, if a part has a truth degreeover a threshold for the membership function “MediumLengthObj”, the conceptMediumLengthObj is added to the concept representing that part in the fuzzyontology. At the end, we obtain a concept like (for the sake of simplicity we listonly some concepts and roles):ObjClass1 ≡ MediumLengthObj u SmallObj u ≥ 5 parall u ≥ 1 near u . . .where ObjClass1 is the newly created concept related to the part which has beenconsidered. A new fuzzy concept is created only if the current analyzed part doesnot belong to any existing concept, i.e. there is no concept that fully describesthe part (it can be verified via the fuzzy reasoner). Since FiRE does not let uswrite fuzzy TBox axioms, the degrees of truth are discarded in this phase.When there are no parts left, a role for each concept is created. For example,from the class ObjClass1 the role hasObjClass1 is created, so that the class Forkcan be created using the previously found number of objects per class:Fork ≡ ≥ 1 hasObjClass1 u ≥ 4 hasObjClass2 u ≥ 1 hasObjClass3This is due to the fact that the f-SHIN logic lacks of the qualified cardinalityrestrictions, so a general hasPart role cannot be used. We use a “typographical”operation, yet the problem of role creation has been faced in [3]. For the sakeof completeness, domain and range role axioms should be added to qualify thenew roles introduced, but the used reasoner does not fully support them yet.3.4 Object recognitionOnce the objects ontology TBox has been built, it is possible to find whetheran object, after it has been decomposed in parts, belongs to a class or not (i.e.how much it can be considered to belong to the considered class with respectto a certain threshold); the image analysis steps are the same for the ontologybuilding phase.When for every part all the pertaining concepts and roles can be writtenin the ABox, the fuzzy reasoning is performed to find the GLB of that partbelonging to a certain class; then, roles like hasObjClass1 are created with thesame value of the found GLBs and, at the end, the GLB of the main object iscalculated.This procedure can be applied to determine whether a specific kind of graspcan be performed or not on the selected object. For example, given the conceptdefined as (for the sake of simplicity using no roles):GraspableByPinch ≡ MediumLengthObj u HighlyEccentricalObjrepresenting objects that are graspable by a pinch grip, we can find which partof the object (if any) can be grasped this way via a subsumption check.4 ConclusionsIn this paper we have presented a possible architecture for the generation and theuse of a fuzzy ontology for object recognition by means of objects decompositionin parts. We take advantage of the use of fuzzy cardinality restrictions which, tothe best of our knowledge, have not been fully exploited in the current fuzzy DLsapplications (e.g. multimedia retrieval). Our results are preliminar and prone toerrors, partly due to limitations in the modules in use (e.g. the fuzzy reasoneris still experimental), partly due to the approximations induced by the use of aSHIN logic, while at least qualified cardinality restrictions would be needed.As future work, we plan to take advantage of a more powerful fuzzy DLas it seems to be needed for object modeling purposes, so we will work on amore powerful reasoner and on a better integration between classical and fuzzyknowledge bases; furthermore, as we plan to use the system as an aid to thegrasping task, we will add physical information (that can obtained via differentsensors, e.g. haptic devices) and further information on the grasping types alongwith their quality measurements.References1. Biederman, I.: Recognition-by-components: A theory of human image understand-ing. Psychological Review 94 2 (1987) 115-1172. Bobillo, F., Straccia, U.: An OWL Ontology for Fuzzy OWL 2. Proceedings of the18th International Symposium on Methodologies for Intelligent Systems (2009)3. Haarslev, V., Lutz, C., Mo¨ller, R.: Foundations of spatioterminological reasoningwith description logics. Proceedings of Sixth International Conference on Principlesof Knowledge Representation and Reasoning (1998) 112–1234. Hudelot, C.: Towards a cognitive vision platform for semantic image interpretation;Application to the recognition of biological organisms. PhD thesis. University ofNice Sophia Antipolis (2005)5. Maillot, N.: Ontology based object learning and recognition. PhD thesis. Universityof Nice Sophia Antipolis (2005)6. Hudelot, C., Atif, J., Bloch, I.: Fuzzy spatial relation ontology for image interpre-tation. In: Fuzzy Sets and Systems , 159 15 (2008) 1929–19517. Stoilos, G., Stamou, G., Pan, J.Z., Simou, N., Tzouvaras, V.: Reasoning with thefuzzy description logic f-SHIN : Theory, practice and applications. In P.C.G. daCosta et al. (eds): Uncertainty Reasoning for the Semantic Web I (2008) 262–2818. Simou, N., Athanasiadis, T., Tzouvaras, V., Kollias, S.: Multimedia reasoning withf-SHIN . Second International Workshop on Semantic Media Adaptation and Per-sonalization (2007) 44–499. Stoilos, G., Stamou, G., Tzouvaras, V., Pan, J.Z., Horrocks, I.: The fuzzy descrip-tion logic f-SHIN . International Workshop on Uncertainty Reasoning For theSemantic Web (2005)10. Mylonas, P., Simou, N., Tzouvaras, V., Avrithis, Y.: Towards semantic multimediaindexing by classification and reasoning on textual metadata. Knowledge Acquisi-tion from Multimedia Content Workshop (2007)11. Lukasiewicz, T., Straccia, U.: Managing uncertainty and vagueness in DescriptionLogics for the Semantic Web. Journal of Web Semantics 6 4 (2008) 291–30812. Straccia, U.: Towards Spatial Reasoning in Fuzzy Description Logics. Proc. of the2009 IEEE International Conference on Fuzzy Systems (2009)13. Suh, I. H., Lim, G. H., Hwang, W., Suh, H., Choi, J.-H., Park, Y.-T.: Ontology-based multi-layered robot knowledge framework (OMRKF) for robot intelligence.IEEE Int. Conf. on Intelligent Robots and Systems (2007) 429–43614. Wan, L.: Parts-based 2D shape decomposition by convex hull. IEEE InternationalConference on Shape Modeling and Applications (2009) 89–9515. Dasiopoulou, S., Kompatsiaris, I., Strintzis, M.G.: Applying Fuzzy DLs in theextraction of image semantics. Journal of Data Semantics 14 (2009) 105–13216. Meghini, C., Sebastiani, F., Straccia, U.: A model of multimedia information re-trieval. Journal of ACM 48 5 (2001) 909–97017. Canny, J.: A computational approach to edge detection. IEEE Transactions onPattern Analysis and Machine Intelligence, 8 6 (1986) 679-698

Using f-SHIN to represent objects: an aid to visual grasping

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Politecnico di Milano