The paper gives a high-level overview of some ways in which logical representations and reasoning can be used in computer vision applications, such as tracking and scene interpretation. The combination of logical and statistical approaches is also considered

Bennett, B.

English

The paper gives a high-level overview of some ways in which 
logical representations and reasoning can be used in computer vision 
applications, such as tracking and scene interpretation. The combination 
of logical and statistical approaches is also considered

Bennett, Brandon

Dagstuhl Research Online Publication Server

Combining Logic and Probabilityin Tracking and Scene InterpretationBrandon BennettUniversity of Leeds, School of ComputingLeeds, LS2 9JT, UKbrandon@comp.leeds.ac.ukAbstract. The paper gives a high-level overview of some ways in whichlogical representations and reasoning can be used in computer visionapplications, such as tracking and scene interpretation. The combinationof logical and statistical approaches is also considered.Keywords. Vision, Tracking, Logic, Probability, Spatio-Temporal Con-tinuity1 IntroductionComputer Vision was among the first problems identified as a goal of ArtificialIntelligence. In these early days, logical reasoning was seen as a general purposemechanism for modelling intelligent behaviour. Hence, research in Computer Vi-sion often involved development of semantic representations and inference mech-anisms (see e.g. the papers in the collection [1]). Indeed the development of rea-soning techniques such as constraint propagation is closely associated with earlywork in computer vision (vis [2]).Since these early days there has been a marked divergence between the fieldsof Knowledge Representation and Computer Vision. Whereas, Knowledge Rep-resentation still uses logical representations and reasoning as one of its primarytools, Computer Vision research has moved away from this, embracing statis-tical techniques as the underpinning for most of its algorithms. Perhaps themain reason for this is that many problems in visual processing turned out tobe much harder than had originally been anticipated. Interpretation of visualscenes tends to be highly unreliable (except in highly constrained artificial set-ting). Hence, statistics are employed to find the most likely interpretations outof many possibilities.However, it is now becoming apparent that statistical methods alone havelimitations. In particular, it is very difficult to combine localised probabilisticinformation into a coherent overall description without taking account seman-tic constraints between separate pieces of information. Logical reasoning canprovide a powerful mechanism for determining consistent possibilities. Also, bydescribing the conceptual structure of possible situations, semantic knowledgemay be used to guide the search for plausible interpretations of incomplete orambiguous data.Dagstuhl Seminar Proceedings 08091 Logic and Probability for Scene Interpretation http://drops.dagstuhl.de/opus/volltexte/2008/16122 B. Bennett2 Enhancing Tracking by Enforcing Spatio-TemporalContinuityAs an illustration of how logic may be used in visual interpretation, I first con-sider a problem of object tracking and recognition, which was tackled at Leeds,as part of the European Cognitive Vision project.1 This work involved collabo-ration between the vision research group, whose work has resulted in a numberof tracking and object recognition systems, and the Knowledge representationand Reasoning group, whose focus has been particularly on spatial and spatio-temporal representation and reasoning [3,4,5].The goal of tracking is to extract information about the positions of movingobjects from a dynamic visual scene. A common approach is to first identifymoving objects by use of so-called ‘blob-tracker’, which uses techniques such asbackground subtraction [6,7,8]. Once areas of an image have been identified ascontaining a moving object, particular objects within these areas can be identi-fied by the same methods as for static images.With real video data, such an approach will normally give very poor results.Because the objects are moving, they will be seen from many different angles.Moreover occlusions may occur, where one object overlaps or perhaps completelyhides another. Under such conditions, it is not surprising that recognition algo-rithms developed for static images are extremely unreliable.But it is evident that by separating the tracking and recognition tasks, muchuseful information concerning the objects has been thrown away. In particular,the constraints of basic physics mean that the tracked objects must move contin-uously in time and space. Thus a possible labelling of the objects correspondingto the moving blobs detected by the tracker must be consistent with this conti-nuity constraint. Handling such constraints in terms of joint probabilities wouldrequire very sophisticated statistical modelling and intensive computation. How-ever, within a logical framework, such constraints are relatively easy to state andreason with.The tracker used in this research generates a sequence of boxes indicatingthe approximate locations of moving objects. This output can be simplified so1 COGVIS: http://www.comp.leeds.ac.uk/vision/cogvis/.Fig. 1. Two possible object labellings of tracker box output.Combining Logic and Probability in Tracking and Scene Interpretation 3that frame sequences over which there is no change in the number of boxes aretreated as a single state. The result is a series of states where between successivestates is a merge or split event. These occur when objects move together (pos-sibly occluding each other) and when clustered objects separate, as illustratedin Figure 1. This representation allows continuity to be captured by very simpleconstraints: when two boxes merge, the objects that were in those boxes mustbe present in the merged box; and, when a box splits, the objects in the originalbox must be distributed between the two resulting boxes.Figure 1 shows how an output of this kind can be given different labellingsconsistent with spatio-temporal continuity. Since the logical constraints cannotdecide between these, it is here that statistical information can be used to choosethe most likely labelling. Whereas both the statistical and logical methods usedseparately cannot decide reliably between different interpretations, our experi-mental results indicate that their combination greatly improves performance. Bytaking the logically consistent model with the best statistical support, we canachieve very high accuracy. Moreover, enforcing spatio-temporal consistency en-ables us to give correct labellings even when objects are occluded or completelyhidden.More details of how logical and statistical knowledge are combined and theresults obtained are given in the following publications: [9,10,11].3 Bi-Directionality of Visual Information ProcessingOne of the primary goals of machine vision is to extract high level descriptionsfrom low level data. Partial success has been achieved by methods that are either:– Bottom-Up — where high level entities/relationships are synthesised fromlow level data.– Top-Down — where high level models are used to guide the search for evi-dence in low level data.Improved performance is often achieved by systems that employ a combina-tion of bottom up and top down mechanisms. But it seems to be increasinglyapparent (especially in much of the work reported at this Dagstuhl workshop)that visual interpretation requires information processing in a style that is in-trinsically bi-directional.By its nature, logic is undirected in the way it manipulates information.Axioms may involve concepts and relations at different levels of abstraction.For example, consider the following formula, which characterises the high-levelconcept Pillar in terms of low-level geometrical properties:D1) ∀x[Pillar(x) ↔ (Solid(x) ∧ Cylinder(x) ∧ Vertical(princ axis(x)))]This equivalence could be exploited in either direction: we may have reasonto believe a pillar is present (perhaps because of an even higher level descriptionof a building) and use it to guide the search for this object within the geometry4 B. Bennettof a scene; or we may have an arbitrary scene which we process in terms of itslow-level geometrical structure and then use definitions such as this to identifyhigh-level objects.More generally, the pillar definition could function in both directions withina single scene interpretation algorithm. To illustrate this, I first give a furtheraxiom defining (roughly) the concept of Arch as consisting of two (or more)pillars supporting a lintel.D2) Arch(x) ↔ ∃yzw[¬(x = y) ∧ Part(y, x) ∧ Part(z, x) ∧ Part(w, x)∧ Pillar(y) ∧ Pillar(z) ∧ Lintel(w) ∧ Supports(y, w) ∧ Supports(z, w)]Given these logical definitions, an algorithm could work as follows. First ituses definitions such as D1 to look for basic objects like pillars based on theirgeometrical definitions. At this stage one would probably require a relativelyhigh degree of conformity to the geometrical specification to avoid excessivegeneration of objects that may not actually be present. Then using definitionssuch as D2 the possible existence of higher level composite objects such as anarch can be inferred, even if certain components of these composites have notbeen identified. In this case, having found one pillar, one can infer that an archmay be present if a second pillar and a lintel can be found in a suitable geometricconfiguration. The geometrical definitions of pillar and arch could then be re-applied to try to find these components to complete the arch. At this stage, aweaker level of geometrical conformity could be required, since, having previouslyidentified a pillar, one already has some evidence of the existence of an arch.General purpose logical reasoners do not constrain the directionality of infor-mation manipulation and hence may provide a natural vehicle for implementingbi-directional visual processing architectures. However, the non-directedness ofgeneral proof procedures quickly leads to intractability, so it is likely that amore task-oriented control mechanism would be required to achieve reasonableperformance. One way in which logical theories might be made more amenableto the kinds of reasoning relevant to scene interpretation is to emphasise def-initional structure within the logical theory. Definitional axioms make explicittwo of the most significant directions for inference in scene interpretation: infer-ring instances of higher level concepts from information expressed in terms oflower-level concepts; and conversely, inference from the hypothesised existence ofinstances of high-level concepts to requirements on low-level information whichcan support the hypothesis. Distinguishing the lowest level primitives from de-fined concepts also facilitates grounding and model-based reasoning techniques.4 Issues in Logical ModellingWe have seen that certain semantic constraints relevant to tracking can be cap-tured by relatively simple logical rules. But scene interpretation in general in-volves an extremely rich semantics and to cover all aspects would require acomplex and extensive theory. Moreover this domain involves many subtletieswhich present significant challenges for logical modelling.Combining Logic and Probability in Tracking and Scene Interpretation 5One aspect of visual scenes that must be taken into account is the ontologicaldistinction between actual objects and their appearances. Most formal theoriesof the physical world directly represent objects and their attributes, implicitlyassuming that the domain of objects and the conditions under which they possessthese attributes are clear. But in most visual scene analysis problems theseassumptions cannot be made. On the one hand, the domain of objects may beunknown or open-ended, and on the other we do not have completely reliablemethods for detecting attributes of objects.Although the relation between objects and appearances is a major topic inphilosophical meta-physics it is seldom covered in modern logic or AI representa-tions. The following papers may provide a staring point for developing a suitablelogical formalism: [12,13].Another important issue is the relationship between the logical theory andinformation of a statistical nature. In the tracking work described above a rathersimple approach was used, we identified a model that was consistent with thelogical theory and best supported by the statistical information. To evaluatethe statistical support we simply summed over a number of localised probabilitymeasures. Although this gave good results, it lacked any rigorous theoreticalbasis.In order to develop a more principled approach, we need to employ a muchmore sophisticated logical treatment. One possibility is to use a fully fledgedlogic of probability such as that proposed by Halpern [14]. Use of such a logicwould make explicit certain aspects of probability which are often glossed overin its application to computer vision applications. One such aspect is the distinc-tion between various ways in which probabilities may be applied to propositionalstatements. Halpern identifies the following distinct modes in which logical state-ments may be given probabilities:– Prob(φ) — probability of φ relative to a probability distribution over possibleworlds.– Probx(φ(x)) — probability of φ(x) relative to a probability distribution overthe domain of individuals.In the context of scene interpretation, these two types of probability areoften compounded together in subtle ways. For instance, a particular type ofobject, T, may have a certain probability p of possessing an attribute φ; and agiven feature detection attribute may have a certain probability q of detectingφ if present. The first figure p is is a probability distribution over individualsof the domain, whereas q is a probability relative to the distribution of possibleworlds. Hence, the overall probability that an object of type T will be detectedas possessing attribute φ involves a complex combination of statistics evaluatedover the domains of both individuals and possible worlds.In the context of reasoning about actions and changes a rich theory of how toreason about noisy sensor has been proposed by Bacchus et al. [15]. With somemodification such a theory might also be applied as a basis for integrating logicand probability. However, due to the complexity of reasoning with formal theories6 B. Bennettand the large amount of data that vision systems must deal with, it is likely thatconsiderable simplification would need to be made to achieve effectiveness in realvision applications.5 ConclusionThe aim of this short article has been to indicate some promising ways in whichlogical reasoning and probabilistic information can be combined in order to sup-port algorithms for visual scene interpretation. The method of finding a logicallyconsistent model that is best supported by statistical information has been foundto be very effective in particular experiments on an object tracking problem. Butthe basic idea is extremely general and could be applied in many areas of com-puter vision.I have suggested that logic can provide a representation suitable for specify-ing and guiding bi-directional processing of visual information, in which bottomup construction of complex objects from their constituents can be flexibly com-bined with top down search for the constituents of an object whose existence ishypothesised. I also pointed out that the use of logic in computer vision appli-cations lacks theoretical foundations both in respect of formalising an ontologyof objects and appearances and in respect to formalising the semantics of theattribution of probabilities to propositions.In summary, the need to use logical reasoning in scene interpretation is be-coming increasingly apparent, and there are many promising approaches thatcould be taken. However, there are considerable gaps in the theory of how logicand probability should be combined, and logical theories suitable for use in visualprocessing applications are at an early stage of development.References1. Winston, P., ed.: The Psychology of Computer Vision. McGraw-Hill (1975)2. Waltz, D.L.: Understanding line drawings of scenes with shadows. In Winston, P.,ed.: The Psychology of Computer Vision. McGraw-Hill (1975) 19–923. Randell, D.A., Cui, Z., Cohn, A.G.: A spatial logic based on regions and connection.In: Proc. 3rd Int. Conf. on Knowledge Representation and Reasoning, San Mateo,Morgan Kaufmann (1992) 165–1764. Bennett, B.: Space, time, matter and things. In Welty, C., Smith, B., eds.: Pro-ceedings of the 2nd international conference on Formal Ontology in InformationSystems (FOIS’01), Ogunquit, ACM (2001) 105–116 http://www.comp.leeds.ac.uk/qsr/pub/bennett-fois01.pdf.5. Hazarika, S.M., Cohn, A.G.: Abducing qualitative spatio-temporal histories frompartial observatio ns. In Fensel, D., Guinchiglia, F., McGuinness, D., Williams,M.A., eds.: Proceedings of the Eight Conference on Principles of Knowledge Rep-resentation and Reasoning (KR 2002 ), Morgan Kaufmann (2002) 14–256. Haritaoglu, I., Harwood, D., Davis, L.: W4: Who? when? where? what? a real timesystem for detecting and tracking people. In: Proc. International Conference onAutomatic Face and Gesture Recognition. (1998) 222–227Combining Logic and Probability in Tracking and Scene Interpretation 77. Stauffer, C., Grimson, W.: Adaptive background mixture models for real-timetracking. In: Proc. Computer Vision and Pattern Recognition. (1999) 246–2528. Magee, D.R.: Tracking multiple vehicles using foreground, background and motionmodels. Image and Vision Computing 20(8) (2004) 581–5949. Bennett, B., Magee, D.R., Cohn, A.G., Hogg, D.C.: Using spatio-temporal conti-nuity constraints to enhance visual tracking of moving objects. In Saitta, L., ed.:Proceedings of the 16th European Conference on Artificial Intelligence (ECAI-04),ECCAI, IOS Press (2004) 922–92610. Bennett, B., Cohn, A.G., Magee, D.: Enforcing global spatio-temporal consistencyto enhance reliability of moving object tracking and classification. KI 19 (2005)32–3511. Bennett, B., Magee, D.R., Cohn, A.G., Hogg, D.C.: Enhanced tracking and recog-nition of moving objects by reasoning about spatio-temporal continuity. Imageand Vision Computing 26 (2008) 67–81 http://www.comp.leeds.ac.uk/qsr/pub/Bennett08imavis.pdf.12. Goodman, N.: The Structure of Appearance. Bobbs-Merill (second edition, 1966)(1951)13. Whitehead, A.N.: Process and Reality: corrected edition. The Free Press, Macmil-lan Pub. Co., New York (1978) edited by D.R. Griffin and D.W. Sherburne.14. Halpern, J.Y.: An analysis of first-order logics of probability. Artificial Intelligence46 (1990) 311–35015. Bacchus, F., Halpern, J.Y., Levesque, H.J.: Reasoning about noisy sensors andeffectors in the situation calculus. Artificial Intelligence 111 (1999) 171–208

Combining Logic and Probability in  Tracking and Scene Interpretation

White Rose Research Online

promoting access to White Rose research papers    White Rose Research Online   Universities of Leeds, Sheffield and York http://eprints.whiterose.ac.uk/    White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/7821/     Published conference paper Bennett, B. (2008) Combining logic and probability in tracking and scene interpretation. In: Cohn, A.G., Hogg, D.C., Möller, R. and Neumann, B., (eds.) Logic and probability for scene interpretation. Logic and Probability for Scene Interpretation, 25th - 29th Feb 2008, Dagstuhl, Germany. Dagstuhl Seminar Proceedings (08091). Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, Germany , Dagstuhl. http://drops.dagstuhl.de/opus/volltexte/2008/1612/ eprints@whiterose.ac.uk  Combining Logic and Probabilityin Tracking and Scene InterpretationBrandon BennettUniversity of Leeds, School of ComputingLeeds, LS2 9JT, UKbrandon@comp.leeds.ac.ukAbstract. The paper gives a high-level overview of some ways in whichlogical representations and reasoning can be used in computer visionapplications, such as tracking and scene interpretation. The combinationof logical and statistical approaches is also considered.Keywords. Vision, Tracking, Logic, Probability, Spatio-Temporal Con-tinuity1 IntroductionComputer Vision was among the first problems identified as a goal of ArtificialIntelligence. In these early days, logical reasoning was seen as a general purposemechanism for modelling intelligent behaviour. Hence, research in Computer Vi-sion often involved development of semantic representations and inference mech-anisms (see e.g. the papers in the collection [1]). Indeed the development of rea-soning techniques such as constraint propagation is closely associated with earlywork in computer vision (vis [2]).Since these early days there has been a marked divergence between the fieldsof Knowledge Representation and Computer Vision. Whereas, Knowledge Rep-resentation still uses logical representations and reasoning as one of its primarytools, Computer Vision research has moved away from this, embracing statis-tical techniques as the underpinning for most of its algorithms. Perhaps themain reason for this is that many problems in visual processing turned out tobe much harder than had originally been anticipated. Interpretation of visualscenes tends to be highly unreliable (except in highly constrained artificial set-ting). Hence, statistics are employed to find the most likely interpretations outof many possibilities.However, it is now becoming apparent that statistical methods alone havelimitations. In particular, it is very difficult to combine localised probabilisticinformation into a coherent overall description without taking account seman-tic constraints between separate pieces of information. Logical reasoning canprovide a powerful mechanism for determining consistent possibilities. Also, bydescribing the conceptual structure of possible situations, semantic knowledgemay be used to guide the search for plausible interpretations of incomplete orambiguous data.2 B. Bennett2 Enhancing Tracking by Enforcing Spatio-TemporalContinuityAs an illustration of how logic may be used in visual interpretation, I first con-sider a problem of object tracking and recognition, which was tackled at Leeds,as part of the European Cognitive Vision project.1 This work involved collabo-ration between the vision research group, whose work has resulted in a numberof tracking and object recognition systems, and the Knowledge representationand Reasoning group, whose focus has been particularly on spatial and spatio-temporal representation and reasoning [3,4,5].The goal of tracking is to extract information about the positions of movingobjects from a dynamic visual scene. A common approach is to first identifymoving objects by use of so-called ‘blob-tracker’, which uses techniques such asbackground subtraction [6,7,8]. Once areas of an image have been identified ascontaining a moving object, particular objects within these areas can be identi-fied by the same methods as for static images.With real video data, such an approach will normally give very poor results.Because the objects are moving, they will be seen from many different angles.Moreover occlusions may occur, where one object overlaps or perhaps completelyhides another. Under such conditions, it is not surprising that recognition algo-rithms developed for static images are extremely unreliable.But it is evident that by separating the tracking and recognition tasks, muchuseful information concerning the objects has been thrown away. In particular,the constraints of basic physics mean that the tracked objects must move contin-uously in time and space. Thus a possible labelling of the objects correspondingto the moving blobs detected by the tracker must be consistent with this conti-nuity constraint. Handling such constraints in terms of joint probabilities wouldrequire very sophisticated statistical modelling and intensive computation. How-ever, within a logical framework, such constraints are relatively easy to state andreason with.The tracker used in this research generates a sequence of boxes indicatingthe approximate locations of moving objects. This output can be simplified so1 COGVIS: http://www.comp.leeds.ac.uk/vision/cogvis/.Fig. 1. Two possible object labellings of tracker box output.Combining Logic and Probability in Tracking and Scene Interpretation 3that frame sequences over which there is no change in the number of boxes aretreated as a single state. The result is a series of states where between successivestates is a merge or split event. These occur when objects move together (pos-sibly occluding each other) and when clustered objects separate, as illustratedin Figure 1. This representation allows continuity to be captured by very simpleconstraints: when two boxes merge, the objects that were in those boxes mustbe present in the merged box; and, when a box splits, the objects in the originalbox must be distributed between the two resulting boxes.Figure 1 shows how an output of this kind can be given different labellingsconsistent with spatio-temporal continuity. Since the logical constraints cannotdecide between these, it is here that statistical information can be used to choosethe most likely labelling. Whereas both the statistical and logical methods usedseparately cannot decide reliably between different interpretations, our experi-mental results indicate that their combination greatly improves performance. Bytaking the logically consistent model with the best statistical support, we canachieve very high accuracy. Moreover, enforcing spatio-temporal consistency en-ables us to give correct labellings even when objects are occluded or completelyhidden.More details of how logical and statistical knowledge are combined and theresults obtained are given in the following publications: [9,10,11].3 Bi-Directionality of Visual Information ProcessingOne of the primary goals of machine vision is to extract high level descriptionsfrom low level data. Partial success has been achieved by methods that are either:– Bottom-Up — where high level entities/relationships are synthesised fromlow level data.– Top-Down — where high level models are used to guide the search for evi-dence in low level data.Improved performance is often achieved by systems that employ a combina-tion of bottom up and top down mechanisms. But it seems to be increasinglyapparent (especially in much of the work reported at this Dagstuhl workshop)that visual interpretation requires information processing in a style that is in-trinsically bi-directional.By its nature, logic is undirected in the way it manipulates information.Axioms may involve concepts and relations at different levels of abstraction.For example, consider the following formula, which characterises the high-levelconcept Pillar in terms of low-level geometrical properties:D1) ∀x[Pillar(x) ↔ (Solid(x) ∧ Cylinder(x) ∧ Vertical(princ axis(x)))]This equivalence could be exploited in either direction: we may have reasonto believe a pillar is present (perhaps because of an even higher level descriptionof a building) and use it to guide the search for this object within the geometry4 B. Bennettof a scene; or we may have an arbitrary scene which we process in terms of itslow-level geometrical structure and then use definitions such as this to identifyhigh-level objects.More generally, the pillar definition could function in both directions withina single scene interpretation algorithm. To illustrate this, I first give a furtheraxiom defining (roughly) the concept of Arch as consisting of two (or more)pillars supporting a lintel.D2) Arch(x) ↔ ∃yzw[¬(x = y) ∧ Part(y, x) ∧ Part(z, x) ∧ Part(w, x)∧ Pillar(y) ∧ Pillar(z) ∧ Lintel(w) ∧ Supports(y, w) ∧ Supports(z, w)]Given these logical definitions, an algorithm could work as follows. First ituses definitions such as D1 to look for basic objects like pillars based on theirgeometrical definitions. At this stage one would probably require a relativelyhigh degree of conformity to the geometrical specification to avoid excessivegeneration of objects that may not actually be present. Then using definitionssuch as D2 the possible existence of higher level composite objects such as anarch can be inferred, even if certain components of these composites have notbeen identified. In this case, having found one pillar, one can infer that an archmay be present if a second pillar and a lintel can be found in a suitable geometricconfiguration. The geometrical definitions of pillar and arch could then be re-applied to try to find these components to complete the arch. At this stage, aweaker level of geometrical conformity could be required, since, having previouslyidentified a pillar, one already has some evidence of the existence of an arch.General purpose logical reasoners do not constrain the directionality of infor-mation manipulation and hence may provide a natural vehicle for implementingbi-directional visual processing architectures. However, the non-directedness ofgeneral proof procedures quickly leads to intractability, so it is likely that amore task-oriented control mechanism would be required to achieve reasonableperformance. One way in which logical theories might be made more amenableto the kinds of reasoning relevant to scene interpretation is to emphasise def-initional structure within the logical theory. Definitional axioms make explicittwo of the most significant directions for inference in scene interpretation: infer-ring instances of higher level concepts from information expressed in terms oflower-level concepts; and conversely, inference from the hypothesised existence ofinstances of high-level concepts to requirements on low-level information whichcan support the hypothesis. Distinguishing the lowest level primitives from de-fined concepts also facilitates grounding and model-based reasoning techniques.4 Issues in Logical ModellingWe have seen that certain semantic constraints relevant to tracking can be cap-tured by relatively simple logical rules. But scene interpretation in general in-volves an extremely rich semantics and to cover all aspects would require acomplex and extensive theory. Moreover this domain involves many subtletieswhich present significant challenges for logical modelling.Combining Logic and Probability in Tracking and Scene Interpretation 5One aspect of visual scenes that must be taken into account is the ontologicaldistinction between actual objects and their appearances. Most formal theoriesof the physical world directly represent objects and their attributes, implicitlyassuming that the domain of objects and the conditions under which they possessthese attributes are clear. But in most visual scene analysis problems theseassumptions cannot be made. On the one hand, the domain of objects may beunknown or open-ended, and on the other we do not have completely reliablemethods for detecting attributes of objects.Although the relation between objects and appearances is a major topic inphilosophical meta-physics it is seldom covered in modern logic or AI representa-tions. The following papers may provide a staring point for developing a suitablelogical formalism: [12,13].Another important issue is the relationship between the logical theory andinformation of a statistical nature. In the tracking work described above a rathersimple approach was used, we identified a model that was consistent with thelogical theory and best supported by the statistical information. To evaluatethe statistical support we simply summed over a number of localised probabilitymeasures. Although this gave good results, it lacked any rigorous theoreticalbasis.In order to develop a more principled approach, we need to employ a muchmore sophisticated logical treatment. One possibility is to use a fully fledgedlogic of probability such as that proposed by Halpern [14]. Use of such a logicwould make explicit certain aspects of probability which are often glossed overin its application to computer vision applications. One such aspect is the distinc-tion between various ways in which probabilities may be applied to propositionalstatements. Halpern identifies the following distinct modes in which logical state-ments may be given probabilities:– Prob(φ) — probability of φ relative to a probability distribution over possibleworlds.– Probx(φ(x)) — probability of φ(x) relative to a probability distribution overthe domain of individuals.In the context of scene interpretation, these two types of probability areoften compounded together in subtle ways. For instance, a particular type ofobject, T, may have a certain probability p of possessing an attribute φ; and agiven feature detection attribute may have a certain probability q of detectingφ if present. The first figure p is is a probability distribution over individualsof the domain, whereas q is a probability relative to the distribution of possibleworlds. Hence, the overall probability that an object of type T will be detectedas possessing attribute φ involves a complex combination of statistics evaluatedover the domains of both individuals and possible worlds.In the context of reasoning about actions and changes a rich theory of how toreason about noisy sensor has been proposed by Bacchus et al. [15]. With somemodification such a theory might also be applied as a basis for integrating logicand probability. However, due to the complexity of reasoning with formal theories6 B. Bennettand the large amount of data that vision systems must deal with, it is likely thatconsiderable simplification would need to be made to achieve effectiveness in realvision applications.5 ConclusionThe aim of this short article has been to indicate some promising ways in whichlogical reasoning and probabilistic information can be combined in order to sup-port algorithms for visual scene interpretation. The method of finding a logicallyconsistent model that is best supported by statistical information has been foundto be very effective in particular experiments on an object tracking problem. Butthe basic idea is extremely general and could be applied in many areas of com-puter vision.I have suggested that logic can provide a representation suitable for specify-ing and guiding bi-directional processing of visual information, in which bottomup construction of complex objects from their constituents can be flexibly com-bined with top down search for the constituents of an object whose existence ishypothesised. I also pointed out that the use of logic in computer vision appli-cations lacks theoretical foundations both in respect of formalising an ontologyof objects and appearances and in respect to formalising the semantics of theattribution of probabilities to propositions.In summary, the need to use logical reasoning in scene interpretation is be-coming increasingly apparent, and there are many promising approaches thatcould be taken. However, there are considerable gaps in the theory of how logicand probability should be combined, and logical theories suitable for use in visualprocessing applications are at an early stage of development.References1. Winston, P., ed.: The Psychology of Computer Vision. McGraw-Hill (1975)2. Waltz, D.L.: Understanding line drawings of scenes with shadows. In Winston, P.,ed.: The Psychology of Computer Vision. McGraw-Hill (1975) 19–923. Randell, D.A., Cui, Z., Cohn, A.G.: A spatial logic based on regions and connection.In: Proc. 3rd Int. Conf. on Knowledge Representation and Reasoning, San Mateo,Morgan Kaufmann (1992) 165–1764. Bennett, B.: Space, time, matter and things. In Welty, C., Smith, B., eds.: Pro-ceedings of the 2nd international conference on Formal Ontology in InformationSystems (FOIS’01), Ogunquit, ACM (2001) 105–116 http://www.comp.leeds.ac.uk/qsr/pub/bennett-fois01.pdf.5. Hazarika, S.M., Cohn, A.G.: Abducing qualitative spatio-temporal histories frompartial observatio ns. In Fensel, D., Guinchiglia, F., McGuinness, D., Williams,M.A., eds.: Proceedings of the Eight Conference on Principles of Knowledge Rep-resentation and Reasoning (KR 2002 ), Morgan Kaufmann (2002) 14–256. Haritaoglu, I., Harwood, D., Davis, L.: W4: Who? when? where? what? a real timesystem for detecting and tracking people. In: Proc. International Conference onAutomatic Face and Gesture Recognition. (1998) 222–227Combining Logic and Probability in Tracking and Scene Interpretation 77. Stauffer, C., Grimson, W.: Adaptive background mixture models for real-timetracking. In: Proc. Computer Vision and Pattern Recognition. (1999) 246–2528. Magee, D.R.: Tracking multiple vehicles using foreground, background and motionmodels. Image and Vision Computing 20(8) (2004) 581–5949. Bennett, B., Magee, D.R., Cohn, A.G., Hogg, D.C.: Using spatio-temporal conti-nuity constraints to enhance visual tracking of moving objects. In Saitta, L., ed.:Proceedings of the 16th European Conference on Artificial Intelligence (ECAI-04),ECCAI, IOS Press (2004) 922–92610. Bennett, B., Cohn, A.G., Magee, D.: Enforcing global spatio-temporal consistencyto enhance reliability of moving object tracking and classification. KI 19 (2005)32–3511. Bennett, B., Magee, D.R., Cohn, A.G., Hogg, D.C.: Enhanced tracking and recog-nition of moving objects by reasoning about spatio-temporal continuity. Imageand Vision Computing 26 (2008) 67–81 http://www.comp.leeds.ac.uk/qsr/pub/Bennett08imavis.pdf.12. Goodman, N.: The Structure of Appearance. Bobbs-Merill (second edition, 1966)(1951)13. Whitehead, A.N.: Process and Reality: corrected edition. The Free Press, Macmil-lan Pub. Co., New York (1978) edited by D.R. Griffin and D.W. Sherburne.14. Halpern, J.Y.: An analysis of first-order logics of probability. Artificial Intelligence46 (1990) 311–35015. Bacchus, F., Halpern, J.Y., Levesque, H.J.: Reasoning about noisy sensors andeffectors in the situation calculus. Artificial Intelligence 111 (1999) 171–208

Combining logic and probability in tracking and scene interpretation

https://eprints.whiterose.ac.uk/7821/1/Bennett308dagstuhl.pdf

Combining logic and probability in tracking and scene interpretation

Abstract

Similar works

Full text

Available Versions

Dagstuhl Research Online Publication Server

White Rose Research Online