Computer vision researchers working in mobile robotics and other real-time domains are forced to con- front issues not normally addressed in the computer vision literature. Among these are communication, or how to get data from one process to another, data storage and retrieval, primarily for transient, image- based data, and database management, for maps, ob- ject models and other permanent (typically 3D) data. This paper reviews eorts at CMU, SRI and UMass to build real-time computer vision systems for mobile robotics, and presents a new tool, called ISR3, for com- munication, data storage/retrieval and database man- agement on the UMass Mobile Perception Laboratory (MPL), a NAVLAB-like autonomous vehicle

Draper, Bruce A.

Hanson, Allen R.

Kutlu, Gökhan

Riseman, Edward M.

English

Computer vision researchers working in mobile robotics and other real-time domains are forced to confront issues not normally addressed in the computer vision literature. Among these are: communications or how to get data from one process to another; data storage and retrieval (primarily for transient image-based data); and database management for maps, object model and other permanent (typically 3D) data. This paper reviews efforts at CMU, SRI and UMass to build real-time computer vision systems for mobile robotics, and presents a new tool, called ISR3, for communications, data storage/retrieval and database management on the UMass Mobile Perception Laboratory (MPL), a NAVLAB-like autonomous vehicle

ScholarWorks@UMass Amherst

University of Massachusetts AmherstScholarWorks@UMass AmherstComputer Science Department Faculty PublicationSeries Computer Science1994ISR3: Communication and Data Storage for anUnmanned Ground Vehicle*Bruce A. DraperUniversity of Massachusetts - AmherstGökhan KutluUniversity of Massachusetts - AmherstEdward M. RisemanUniversity of Massachusetts - AmherstAllen R. HansonUniversity of Massachusetts - AmherstFollow this and additional works at: https://scholarworks.umass.edu/cs_faculty_pubsPart of the Computer Sciences CommonsThis Article is brought to you for free and open access by the Computer Science at ScholarWorks@UMass Amherst. It has been accepted for inclusionin Computer Science Department Faculty Publication Series by an authorized administrator of ScholarWorks@UMass Amherst. For more information,please contact scholarworks@library.umass.edu.Recommended CitationDraper, Bruce A.; Kutlu, Gökhan; Riseman, Edward M.; and Hanson, Allen R., "ISR3: Communication and Data Storage for anUnmanned Ground Vehicle*" (1994). Computer Science Department Faculty Publication Series. 47.Retrieved from https://scholarworks.umass.edu/cs_faculty_pubs/47ISR3: Communication and Data Storage for an Unmanned GroundVehicleBruce A. Draper Gokhan Kutlu Edward M. Riseman Allen R. HansonComputer Vision Laboratory, Dept. of Computer ScienceUniversity of Massachusetts, Amherst, MA 01003AbstractComputer vision researchers working in mobilerobotics and other real-time domains are forced to con-front issues not normally addressed in the computervision literature. Among these are communication,or how to get data from one process to another, datastorage and retrieval, primarily for transient, image-based data, and database management, for maps, ob-ject models and other permanent (typically 3D) data.This paper reviews eorts at CMU, SRI and UMassto build real-time computer vision systems for mobilerobotics, and presents a new tool, called ISR3, for com-munication, data storage/retrieval and database man-agement on the UMass Mobile Perception Laboratory(MPL), a NAVLAB-like autonomous vehicle.1 IntroductionAs computer vision technology matures, researchersare forced to divert some of their attention from sub-problems (such as line extraction, model matchingand shape from shading) to complete systems. Al-though many open subproblems remain, the need todemonstrate success on real-world applications andthe growing belief that vision is inherently task-oriented [1, 2, 5] are spurring research into the practi-cal issues of building working systems.Much of this work has focused on supporting al-gorithm development and integration. In particu-lar, both university researchers and commercial ven-dors have produced software environments that com-bine vision-oriented development tools with librariesof standard algorithms and representations.This work was supported by ARPA through U.S. ArmyTACOM under contract DAAE07-91-C-RO35, by ARPAthrough U.S. Army TEC under contract DACA76-92-C-0041,and by ARPA through Rome Labs under contract F30602-94-C-0042.None of these environments, however, are de-signed for real-time applications. Researchers who ap-ply computer vision to applications such as mobilerobotics need support for inter-process communica-tion, short-term data storage and retrieval, and long-term data management. This paper reviews some ofthe approaches that have been tried for solving theseproblems, such as ISR [4], CODGER [9] and the CoreKnowledge System (CKS; [10]). We then present afurther development of ISR, called ISR3, for manag-ing these issues on board an unmanned vehicle (MPL).2 Real-time Integration2.1 IssuesWhen algorithms developed o-line are to be com-bined for a real-time application such as an unmannedvehicle, several basic problems need to be solved.Among them are:Communication. Data must be passed from oneprocess|and often one processor|to the next. Thelatency of the communication channel can be critical.When data is shared among processes, concurrencycontrol becomes important to protect the integrity ofthe data.Data Storage/Retrieval. Vision algorithms of-ten produce large amount of temporary data. A lineextraction algorithm, for example, may produce hun-dreds (or thousands) of line segments from an image.Although not persistent, such data may be repeat-edly accessed by other modules (e.g. line grouping ormodel matching algorithms), so the eciency of dataretrieval is critical.Database Management. Maps, models andother forms of a-priori knowledge make up a perma-nent data base of information that visual algorithmsrepeatedly access and occasionally alter. Althoughless voluminous than the temporary data mentionedabove, this data is persistent and must be managedover time by ecient storage and access mechanismswhich are geared to the nature (e.g. spatial, temporal,3D) of the data.2.2 Current TechnologyOne of the rst tools to support real-time imageunderstanding was CODGER, a blackboard-style datastore developed for CMU's NAVLAB [9]. The idea be-hind CODGER was that its \whiteboard" would serveas the central data store, with all other processes read-ing data from the white board and posting results to it.CODGER was also an information fusion mechanismthat merged data by computing coordinate transfor-mations and adjusting for time delays. Its authorscalled it a whiteboard, rather than a blackboard, be-cause it had xed-length message buers, so that newmessages forced old messaged to be purged. In ef-fect, this was a primitive memory management sys-tem. Since CODGER's \whiteboard" was managedby a single process, it was an example of a (client-server model) data store.Unfortunately, the central blackboard of CODGERbecame a bottleneck since all data messagespassed from one process to another had to gothrough CODGER. This problem was exacerbated byCODGER's role as a data fusion mechanism: Klugereports that CODGER spent most of its time comput-ing coordinate transformations [6], slowing the centralcommunication process down even further.After CODGER was abandoned, a point-to-pointcommunication package called EDDIE [12] was de-veloped for NAVLAB. EDDIE was built on standardinterprocess communication (IPC) protocols, and al-lowed processes to exchange instances of predenedmessage types. EDDIE also included simple controlmechanisms, such as the ability to \wake up" a processwhen a message arrives, and a small geometric model-ing and map system. EDDIE has since been replacedon board NAVLAB by TCX [11], a point-to-pointcommunication package, and the Annotated Map Sys-tem [12] for controlling vehicles behaviors based onlocation.At about the same time as CODGER, researchersat SRI proposed a blackboard system called the CoreKnowledge System (CKS; [10]). CKS, like CODGER,was intended as a blackboard-style data store for anautonomous vehicle. Unlike CODGER, however, itwas not intended for data fusion. The philosophy be-hind CKS was that all messages (data) are merelyhypotheses produced by knowledge sources, and thatrather than fuse data, the blackboard should recordwhich process proposed it and how much condenceit had. Memory management was handled by giv-ing each knowledge source a memory budget, whichit could allocate to messages it wrote or intended toread.Another type of data store, called ISR [4], was de-veloped at the University of Massachusetts. UnlikeCODGER and CKS, ISR was designed as a tool for o-line algorithm development. As ISR's developers, wewere interested in the knowledge-directed interpreta-tion of outdoor scenes, and the control issues thereof.ISR was built to integrate many dierent visual pro-cedures into a single system, and therefore to developcommon data structures and mechanisms for passingdata from one procedure to the next.3 ISR3Under the ARPA Unmanned Ground Vehicle(UGV) program, the University of Massachusetts wasdeveloping the Mobile Perception Laboratory (MPL),an autonomous vehicle similar to CMU's NAVLAB[11]. The goal for MPL was a system that inte-grates low-level behaviors, such as road following andobstacle detection, with high-level behaviors such aslandmark-based navigation and map building. To sup-port this project, we designed a prototype data storefor real-time vision called ISR3, which was imple-mented on board MPL.ISR (an acronym for intermediate symbolic repre-sentation) has been the name of a series of symbolicdatabases for vision developed at the University ofMassachusetts [4]. The ISR databases reect a beliefthat computer vision requires more than image-likearrays of numerical data; computer vision depends onsymbolic representations of abstract image events suchas regions, lines, and surfaces, and on mechanisms foreciently accessing data objects under various typesof constraints (such as spatial proximity). One ver-sion, ISR1.5, is now commercially available as partof KBVision [13], while the most recent version, ISR3,was used on-board MPL. Although each version of ISRis a renement of its predecessor, they all assume thatvisual procedures operate on symbolic records, calledtokens, or on groups of tokens, and that visual pro-cedures manipulate tokens both for internal computa-tions and for exchanging data with other procedures.In particular, vision applications typically needtemporary storage for large numbers of tokens, whichthey then access by name, feature value or spatial loca-tion. Most of these tokens exist for only a short dura-tion, such as the time required to process one image ora short sequence of images, and should then be deallo-cated, although a few correspond to signicant results(critical features, updated maps, etc.) and should per-sist over time. ISR3 is an in-memory database whosetokens are C++ class instances (unlike previous ver-sions of the ISR) and that provides a library of func-tions for storing and retrieving tokens, and for tokenI/O.As a rule, most visual algorithms operate on sets ofdata rather than on individual data instances. Match-ing algorithms, for example, compare a set of modeldata instances to image data instances. Thereforemost of ISR3's storage and retrieval commands arein the form of set operations, such as a request toaccess all long, straight lines in the upper corner ofan image. Special facilities for optimizing spatial re-trieval over arbitrary data sets are also provided, asare macros for iterating over the instances of a set,and functions for taking the union, intersection anddierences of sets.4 System DemonstrationISR3 was the data management system used dur-ing demonstrations of the Mobile Perception Labora-tory (MPL) in the fall of 1993. These demonstrationsshowed the integration of low-level, reactive behaviorssuch as road following and obstacle avoidance withhigh-level perceptual capabilities such as landmarkrecognition, as described in [8]. ISR3 was thereforeforced to support a wide variety of image processingneeds.At one extreme were groups of processes that onlyused ISR3 as a low-latency communications channel.For example, ALVINN [7] (a neural network road fol-lower), its video preprocessor, and the MPL's steer-ing/throttle controller used ISR3 in this way, withthe video preprocessor passing reduced (32x32 inten-sity) images to ALVINN at approximately 10Hz, andALVINN sending steering commands to the controllerat the same rate. These processes did not requireany complex data storage or retrieval capabilities, butthe speed of the communication channel was criti-cal, since the latency of road following is one of twofactors (along with the latency of obstacle detection)that determines a vehicle's speed. (In these exper-iments, obstacle detection was slower than road fol-lowing and limited MPL to 5 mph, so slower commu-nication could have been tolerated.) The stereo corre-spondence and reexive avoidance modules also usedISR3 as a simple communications channel, this timefor passing 255240 disparity images at about 2.5Hz.At the other extreme were processes that relied pri-marily on ISR3's data storage and retrieval capabili-ties. The landmark recognition behavior, for exam-ple, was composed of three processes, one of which re-trieved models of landmarks from the model base (or\map") based on the vehicle's estimated position, onewhich extracted 2D line segments from images (afterusing color and texture to focus attention), and one ofwhich matched image lines to model lines, determin-ing the vehicle's position and orientation relative tothe map [3]. In this case, the speed of communicationbetween these processes was less critical; the vehiclewas stopped during landmark recognition, so delayswere not a hazard. Moreover, the model matching al-gorithm took on the order of a minute to execute, soany communication delays were comparatively small.Rapid spatial access to the data was critical, how-ever; the model matcher only ran as quickly as it didbecause it used the ISR3's spatial access routines toquickly retrieve sets of data lines near the projectionsof model lines. Data storage and retrieval eciencywas therefore the critical item for these processes.5 Future WorkIn general, ISR3 has been very successful when usedas a data store for the structured storage and retrievalof temporary image data, such as regions of interest,points and line segments. We underestimated, how-ever, how often it would be used as a simple commu-nications channel and how often we would need to portit to other machines, and are making improvements inthese areas, as discussed below.5.1 Point-to-Point CommunicationsThe version of ISR3 used on MPL in the fall of1993 used the physically shared memory of a SiliconGraphics Iris 4D to implement interprocess communi-cation. This had very low overhead, but meant thatISR3 could not be ported to machines (or networks ofmachines) without shared memory.We are currently replacing the shared memorymodel with an explicit virtual point-to-point commu-nications layer developed on top of TCX, to makeISR3 portable to machines without shared memoryand to provide the simplest possible communicationmechanism for processes which do not require any-thing more. Processes will pass data between them-selves by opening a communications channel and writ-ing to it. The channel will eectively copy data dur-ing transmission, resulting in more data copying butno synchronization overhead (note that the hardesttiming constraints are on low-level processes like roadfollowing that pass small amounts of data and there-fore will require very little data copying). This vir-tual communications layer will be designed to use thefastest means of communication available between anytwo processes, whether through shared memory oracross a network. In all cases, however, it will pro-vide a simple point-to-point interface to the systemdesigner.5.2 Long-term Data ManagementAlthough most visual data is temporary, some typesof data such as maps and object models are persistentand need to be managed. Currently, long-term andshort-term data are not distinguished from each otherin ISR3; long-term data is simply short-term data thatis never purged, and all data is potentially shared be-tween processes. With the change to a point-to-pointsystem, this approach will no longer be possible. In-stead, there will be a special database process thatmanages long-term data. Processes that need to ac-cess this data open a communications channel to thedatabase, which acts as a server. The database willmanage simultaneous updates, record the history ofhow maps and models are changed, and be able torecover data in the event of a systems failure. Thedatabase will also provide 3D spatial organization andaccess mechanisms, as opposed to the current 2D ones,since long-term data is generally 3D.The reason we believe we can provide a centralclient-server database without it becoming a bottle-neck (as happened with CODGER) is that 1) mostdata is temporary and will not go through the centraldatabase, and 2) the processes that do use permanentdata are typically high-level processes such as land-mark navigation that do not have the most demand-ing timing constraints. Also, although there is someoverhead involved in opening a communication chan-nel to the central database, no process has to do thismore than once, usually during start-up.6 ConclusionReal-time computer vision applications requiretools to support communication, short-term data stor-age and retrieval, and long-term database manage-ment. ISR3 is a tool that supports communicationand data storage (and will soon be extended to man-age persistent data) that allows computer vision algo-rithms developed o-line to be integrated into practi-cal real-time applications.References[1] J. Aloimonos. Purposive and Qualitative ActiveVision. In IUW, Pittsburgh, PA, pages 816{828,Sept. 1990.[2] D. Ballard. Animate Vision. Articial Intelligence,48:57{86, 1991.[3] R. Beveridge. Local Search Algorithms for Geomet-ric Object Recognition: Optimal Correspondenceand Pose. PhD thesis, Univ. of Massachusetts,1993.[4] J. Brolio, B. Draper, R. Beveridge, and A. Han-son. The ISR: an Intermediate Symbolic Repre-sentation for Computer Vision. IEEE Computer,22(12):22{30, 1989.[5] K. Ikeuchi and M. Hebert. Task Oriented Vision.Conf. on Intelligent Robots and Systems, Raleigh,NC, pages 2187{2194, July 1992.[6] K. Kluge. Multisensor System Integration for Au-tonomous Navigation Tasks. To appear in Intelli-gent Vehicles.[7] D. Pomerleau. Neural Network Perception for Mo-bile Robot Guidance. PhD thesis, Carnegie MellonUniversity, 1992.[8] B. Rochwerger, C. Fennema, B. Draper, A. Han-son and E. Riseman. An Architecture for ReactiveBehavior. ICPR (this volume).[9] A. Stentz. The CODGER System for MobileRobot Navigation. In [11], 1990.[10] T. Strat and G. Smith. Core Knowledge System:Storage and Retrieval of Inconsistent Information.IUW, pages 660-665, 1988.[11] C. Thorpe. Vision and Navigation: The CarnegieMellon Navlab. Kluwer Academic Publishers,1990.[12] C. Thorpe and J. Gowdy. Annotated Maps forAutonomous Land Vehicles. In IUW, Pittsburgh,PA, pages 765{771, Sept. 1990.[13] T. Williams. Image Understanding Tools. InICPR, Atlantic City, N.J., pages 606{610, June1990.

ISR3: Communication and Data Storage for an Unmanned Ground Vehicle*

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.29.436

ISR3: Communication and Data Storage for an Unmanned Ground Vehicle*

Abstract

Similar works

Full text

Available Versions

ScholarWorks@UMass Amherst