A new audio-visual analysis approach and tools for parsing colonoscopy videos

Colonoscopy is an important screening tool for colorectal cancer. During a colonoscopic procedure, a tiny video camera at the tip of the endoscope generates a video signal of the internal mucosa of the colon. The video data are displayed on a monitor for real-time analysis by the endoscopist. We call videos captured from colonoscopic procedures colonoscopy videos. Because these videos possess unique characteristics, new types of semantic units and parsing techniques are required. In this paper, we introduce a new analysis approach that includes (a) a new definition of semantic unit - scene (a segment of visual and audio data that correspond to an endoscopic segment of the colon); (b) a novel scene segmentation algorithm using audio and visual analysis to recognize scene boundaries. We design a prototype system to implement the proposed approach. This system also provides the tools for video/image browsing. The tools enable the users to quickly locate and browse scenes of interest. Experiments on real colonoscopy videos show the effectiveness of our algorithms. The proposed techniques and software are useful (1) for post-procedure reviews, (2) for developing an effective content-based retrieval system for colonoscopy videos to facilitate endoscopic research and education, and (3) for development of a systematic approach to assess endoscopists\u27 procedural skills

AVIDENSE: Advanced Video Analysis System for Colonoscopy Semantics

Colonoscopy is an important screening tool for colorectal cancer. During a colonoscopic procedure, a tiny video camera at the tip of the endoscope generates a video signal of the internal mucosa of the colon. The video data are displayed on a monitor for real-time analysis by the endoscopist. We call videos captured from colonoscopic procedures colonoscopy videos . To the best of our knowledge, they are not captured for post procedural review or analysis in the current practice. Because of the unique characteristics of colonoscopy videos, new types of semantic units and new image/video analyzing techniques are required. In this dissertation, we aim to develop new image/video analysis techniques for these videos to extract important semantic units, such as colonoscopic scenes, operation shots, and appendix images. Our contributions include two parts: (a) new definitions of semantic units (colonoscopic scene, operation shot, and appendix image); and (b) novel image/video analysis algorithms, including novel scene segmentation algorithms using audio and visual information to recognize scene boundaries, new computer-aided detection approaches for operation shot detection, and new image analysis methods for appendix image classification. The new image processing and content-based video analysis algorithms can be extended to videos from other endoscopic procedures, such as upper gastrointestinal endoscopy, EGD, enteroscopy, bronchoscopy, cystoscopy, and laparoscopy. Our research is very useful for the following platforms and resources: (a) platforms for new methods to discover unknown patterns of diseases and cancers; (b) platforms for improving and assessing endoscopists procedural skills; and (c) education resources for endoscopic research