Image Analysis via Applied Harmonic Analysis : Perceptual Image Quality Assessment, Visual Servoing, and Feature Detection

Certain systems of analyzing functions developed in the field of applied harmonic analysis are specifically designed to yield efficient representations of structures which are characteristic of common classes of two-dimensional signals, like images. In particular, functions in these systems are typically sensitive to features that define the geometry of a signal, like edges and curves in the case of images. These properties make them ideal candidates for a wide variety of tasks in image processing and image analysis. This thesis discusses three recently developed approaches to utilizing systems of wavelets, shearlets, and alpha-molecules in specific image analysis tasks. First, a perceptual image similarity measure is introduced that is solely based on the coefficients obtained from six discrete Haar wavelet filters but yields state of the art correlations with human opinion scores on large benchmark databases. The second application concerns visual servoing, which is a technique for controlling the motion of a robot by using feedback from a visual sensor. In particular, it will be investigated how the coefficients yielded by discrete wavelet and shearlet transforms can be used as the visual features that control the motion of a robot with six degrees of freedom. Finally, a novel framework for the detection and characterization of features such as edges, ridges, and blobs in two-dimensional images is presented and evaluated in extensive numerical experiments. Here, versatile and robust feature detectors are obtained by exploiting the special symmetry properties of directionally sensitive analyzing functions in systems created within the recently introduced alpha-molecule framework

Image Analysis via Applied Harmonic Analysis : Perceptual Image Quality Assessment, Visual Servoing, and Feature Detection

Certain systems of analyzing functions developed in the field of applied harmonic analysis are specifically designed to yield efficient representations of structures which are characteristic of common classes of two-dimensional signals, like images. In particular, functions in these systems are typically sensitive to features that define the geometry of a signal, like edges and curves in the case of images. These properties make them ideal candidates for a wide variety of tasks in image processing and image analysis. This thesis discusses three recently developed approaches to utilizing systems of wavelets, shearlets, and alpha-molecules in specific image analysis tasks. First, a perceptual image similarity measure is introduced that is solely based on the coefficients obtained from six discrete Haar wavelet filters but yields state of the art correlations with human opinion scores on large benchmark databases. The second application concerns visual servoing, which is a technique for controlling the motion of a robot by using feedback from a visual sensor. In particular, it will be investigated how the coefficients yielded by discrete wavelet and shearlet transforms can be used as the visual features that control the motion of a robot with six degrees of freedom. Finally, a novel framework for the detection and characterization of features such as edges, ridges, and blobs in two-dimensional images is presented and evaluated in extensive numerical experiments. Here, versatile and robust feature detectors are obtained by exploiting the special symmetry properties of directionally sensitive analyzing functions in systems created within the recently introduced alpha-molecule framework

Robotic Ultrasound Imaging: State-of-the-Art and Future Perspectives

Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques.Comment: Accepted by Medical Image Analysi

Vision-based methods for state estimation and control of robotic systems with application to mobile and surgical robots

For autonomous systems that need to perceive the surrounding environment for the accomplishment of a given task, vision is a highly informative exteroceptive sensory source. When gathering information from the available sensors, in fact, the richness of visual data allows to provide a complete description of the environment, collecting geometrical and semantic information (e.g., object pose, distances, shapes, colors, lights). The huge amount of collected data allows to consider both methods exploiting the totality of the data (dense approaches), or a reduced set obtained from feature extraction procedures (sparse approaches). This manuscript presents dense and sparse vision-based methods for control and sensing of robotic systems. First, a safe navigation scheme for mobile robots, moving in unknown environments populated by obstacles, is presented. For this task, dense visual information is used to perceive the environment (i.e., detect ground plane and obstacles) and, in combination with other sensory sources, provide an estimation of the robot motion with a linear observer. On the other hand, sparse visual data are extrapolated in terms of geometric primitives, in order to implement a visual servoing control scheme satisfying proper navigation behaviours. This controller relies on visual estimated information and is designed in order to guarantee safety during navigation. In addition, redundant structures are taken into account to re-arrange the internal configuration of the robot and reduce its encumbrance when the workspace is highly cluttered. Vision-based estimation methods are relevant also in other contexts. In the field of surgical robotics, having reliable data about unmeasurable quantities is of great importance and critical at the same time. In this manuscript, we present a Kalman-based observer to estimate the 3D pose of a suturing needle held by a surgical manipulator for robot-assisted suturing. The method exploits images acquired by the endoscope of the robot platform to extrapolate relevant geometrical information and get projected measurements of the tool pose. This method has also been validated with a novel simulator designed for the da Vinci robotic platform, with the purpose to ease interfacing and employment in ideal conditions for testing and validation. The Kalman-based observers mentioned above are classical passive estimators, whose system inputs used to produce the proper estimation are theoretically arbitrary. This does not provide any possibility to actively adapt input trajectories in order to optimize specific requirements on the performance of the estimation. For this purpose, active estimation paradigm is introduced and some related strategies are presented. More specifically, a novel active sensing algorithm employing visual dense information is described for a typical Structure-from-Motion (SfM) problem. The algorithm generates an optimal estimation of a scene observed by a moving camera, while minimizing the maximum uncertainty of the estimation. This approach can be applied to any robotic platforms and has been validated with a manipulator arm equipped with a monocular camera

Estimation of the Local Scour from a Cylindrical Bridge Pier Using a Compilation Wavelet Model and Artificial Neural Network

In the present study, an artificial neural network and its combination with wavelet theory are used as the computational tool to predict the depth of local scouring from the bridge pier. The five variables measured are the pier diameter of the bridge, the critical and the average velocities, the average diameter of the bed aggregates, and the flow depth. In this study, the neural wavelet method is used as a preprocessor. The data was passed through the wavelet filter and then passed to the artificial neural network. Among the various wavelet functions used for preprocessing, the dmey function results in the highest correlation coefficient and the lowest RMSE and is more efficient than other functions. In the wavelet-neural network compilation method, the neural network activator function was replaced by different wavelet functions. The results show that the neural network method with the Polywog4 wavelet activator function with a correlation coefficient of 87% is an improvement of 8.75% compared to the normal neural network model. By performing data filtering by wavelet and using the resulting coefficients in the neural network, the resulting correlation coefficient is 82%, only a 2.5% improvement compared to the normal neural network. By analyzing the results obtained from neural network methods, the wavelet-neural network predicted errors compared to experimental observations were 8.26, 1.56, and 1.24%, respectively. According to the evaluation criteria, combination of the best effective hydraulic parameters, the combination of wavelet function and neural network, and the number of neural network neurons achieved the best results

Wavelet and Shearlet-based Image Representations for Visual Servoing

International audienceA visual servoing scheme consists of a closed-loop control approach which uses visual information feedback to control the motion of a robotic system. Probably the most popular visual servoing method is image-based visual servoing (IBVS). This kind of method uses geometric visual features extracted from the image to design the control law. However, extracting, matching and tracking geometric visual features over time significantly limits the versatility of visual servoing controllers in various industrial and medical applications, in particular for " low-structured " medical images, e.g., ultrasounds and optical coherence tomography modalities. In order to overcome the limits of conventional IBVS, one can consider novel visual servoing paradigms known as " direct " or " featureless " approaches. This paper deals with the development of a new generation of direct visual servoing methods in which the signal control inputs are the coefficients of a multiscale image representation. In particular, we consider the use of multiscale image representations that are based on discrete wavelet and shearlet transforms. Up to now, one of the main obstacles in the investigation of multiscale image representations for visual ser-voing schemes was the issue of obtaining an analytical formulation of the interaction matrix that links the variation of wavelet and shearlet coefficients to the spatial velocity of the camera and the robot. In this paper, we derive four direct visual servoing controllers: two that are based on subsampled respectively non-subsampled wavelet coefficients and two that are based on the coefficients of subsampled respectively non-subsampled discrete shearlet transforms. All proposed controllers were tested in both simulation and experimental scenarios (using a 6 degrees-of-freedom (DOF) Cartesian robot in an eye-in-hand configuration). The objective of this paper is to provide an analysis of the respective strengths and weaknesses of wavelet-and shearlet-based visual servoing controllers

Wavelet and Shearlet-based Image Representations for Visual Servoing

International audienceA visual servoing scheme consists of a closed-loop control approach which uses visual information feedback to control the motion of a robotic system. Probably the most popular visual servoing method is image-based visual servoing (IBVS). This kind of method uses geometric visual features extracted from the image to design the control law. However, extracting, matching and tracking geometric visual features over time significantly limits the versatility of visual servoing controllers in various industrial and medical applications, in particular for ”low-structured” medical images, e.g., ultrasounds and opti- cal coherence tomography modalities. In order to overcome the limits of conventional IBVS, one can consider novel visual servoing paradigms known as “direct” or “featureless” approaches. This paper deals with the development of a new generation of direct visual servoing methods in which the signal control inputs are the coefficients of a multiscale image representation. In particular, we consider the use of multiscale image representations that are based on discrete wavelet and shearlet transforms. Up to now, one of the main obstacles in the investigation of multiscale image representations for visual ser- voing schemes was the issue of obtaining an analytical formulation of the interaction matrix that links the variation of wavelet and shearlet coefficients to the spatial velocity of the camera and the robot. In this paper, we derive four direct visual servoing controllers: two that are based on subsampled respec- tively non-subsampled wavelet coefficients and two that are based on the coefficients of subsampled respectively non-subsampled discrete shearlet transforms. All proposed controllers were tested in both simulation and experimental scenarios (using a 6 degrees-of-freedom (DOF) Cartesian robot in an eye- in-hand configuration). The objective of this paper is to provide an analysis of the respective strengths and weaknesses of wavelet- and shearlet-based visual servoing controllers

Asservissement visuel direct utilisant les décompositions en shearlets et en ondelettes de l'image

A visual servoing scheme consists of a closed-loop control approach which uses visual information feedback to control the movement of a robotic system. This data, called visual features, can be 2D or 3D. This thesis deals with the development of a new generation of 2D direct visual servoing methods in which the signal control inputs are the coefficients of a multiscale image representation. Specially, we consider the use of multiscale image representations that are based on discrete wavelet and shearlet transformations. This kind of representations allows us to obtain several descriptions of the image based on either low or high frequencies levels. Indeed, high coefficients in the wavelet or in the shearlet transformation of the image correspond to image singularities. This thesis has begun with the development of a shearlet-based visual servoing for ultrasound imaging that has performed well in precision and robustness for this medical application. Nevertheless, the main contribution is a framework allowing us to use several multi-scale representations of the image. It was then tested with conventional white light camera and with an optical coherence tomography imaging system with nominal and unfavorable conditions. Then, the wavelet and the shearlet based methods showed their accuracy and their robustness in several conditions and led to the use of both visual servoing and compressed sensing as the main perspective of this work.L'asservissement visuel est un procédé consistant à utiliser l'information visuelle obtenue par un capteur afin de commander un système robotique. Ces informations, appelées primitives visuelles peuvent être d'ordre 2D ou 3D. Le travail présenté ici porte sur une nouvelle approche 2D utilisant des primitives directes : les décompositions de l'image en ondelettes ou en shearlets. Ces représentations présentent en effet l'avantage de décrire l'image sous différentes formes, mettant l'accent soit sur les basses fréquences de l'image, soit sur les hautes fréquences selon plusieurs directions. Les zones de l'image contenant beaucoup d'information, comme les contours ou les points singuliers, possèdent alors de forts coefficients dans la transformée en ondelettes ou en shearlets de l'image, tandis que les zones uniformes possèdent des coefficients proches de zéro. Les travaux de cette thèse montrent la précision et la robustesse de l'approche utilisant la décomposition en shearlets dans le cadre de l'imagerie échographique. Néanmoins, sa contribution majeure est l'élaboration d'une commande permettant d'utiliser au choix les ondelettes ou les shearlets ainsi que la validation de cette méthode sur caméra monoculaire et sur capteur de type tomographie par cohérence optique dans différentes conditions d'utilisation. Cette méthode présente des performances significatives en termes de précision et de robustesse et ouvre la porte vers une utilisation couplée de l'asservissement visuel et de l'acquisition comprimée

Asservissement visuel direct utilisant les décompositions en shearlets et en ondelettes de l'image

A visual servoing scheme consists of a closed-loop control approach which uses visual information feedback to control the movement of a robotic system. This data, called visual features, can be 2D or 3D. This thesis deals with the development of a new generation of 2D direct visual servoing methods in which the signal control inputs are the coefficients of a multiscale image representation. Specially, we consider the use of multiscale image representations that are based on discrete wavelet and shearlet transformations. This kind of representations allows us to obtain several descriptions of the image based on either low or high frequencies levels. Indeed, high coefficients in the wavelet or in the shearlet transformation of the image correspond to image singularities. This thesis has begun with the development of a shearlet-based visual servoing for ultrasound imaging that has performed well in precision and robustness for this medical application. Nevertheless, the main contribution is a framework allowing us to use several multi-scale representations of the image. It was then tested with conventional white light camera and with an optical coherence tomography imaging system with nominal and unfavorable conditions. Then, the wavelet and the shearlet based methods showed their accuracy and their robustness in several conditions and led to the use of both visual servoing and compressed sensing as the main perspective of this work.L'asservissement visuel est un procédé consistant à utiliser l'information visuelle obtenue par un capteur afin de commander un système robotique. Ces informations, appelées primitives visuelles peuvent être d'ordre 2D ou 3D. Le travail présenté ici porte sur une nouvelle approche 2D utilisant des primitives directes : les décompositions de l'image en ondelettes ou en shearlets. Ces représentations présentent en effet l'avantage de décrire l'image sous différentes formes, mettant l'accent soit sur les basses fréquences de l'image, soit sur les hautes fréquences selon plusieurs directions. Les zones de l'image contenant beaucoup d'information, comme les contours ou les points singuliers, possèdent alors de forts coefficients dans la transformée en ondelettes ou en shearlets de l'image, tandis que les zones uniformes possèdent des coefficients proches de zéro. Les travaux de cette thèse montrent la précision et la robustesse de l'approche utilisant la décomposition en shearlets dans le cadre de l'imagerie échographique. Néanmoins, sa contribution majeure est l'élaboration d'une commande permettant d'utiliser au choix les ondelettes ou les shearlets ainsi que la validation de cette méthode sur caméra monoculaire et sur capteur de type tomographie par cohérence optique dans différentes conditions d'utilisation. Cette méthode présente des performances significatives en termes de précision et de robustesse et ouvre la porte vers une utilisation couplée de l'asservissement visuel et de l'acquisition comprimée