Performance evaluation of a information fusion systems

Les systèmes de fusion d'informations sont principalement composés, d'outils mathématiques permettant de réaliser la représentation et la combinaison des données. L'objectif de ces systèmes peut s'exprimer comme un problème de décision sur la vérité ou la vraisemblance d'une proposition étant donné une ou plusieurs informations issues de différentes sources. Les systèmes de fusion cherchent à exploiter au mieux les propriétés des sources de données en tenant compte de l'imperfection de l'information (imprécis, incomplet, ambigu, incertain, etc.) ainsi que l'aspect redondant, complémentaire et conflictuel des informations. Le système de fusion concerné par cette thèse a la capacité d'intégrer dans ses traitements de la connaissance experte. On le nomme système de fusion coopératif. Puisque ce système cherche à intégrer pleinement les experts dans son fonctionnement, il est important de mettre à disposition des utilisateurs des informations aidant à mieux comprendre la fusion réalisée. De tels systèmes disposent de nombreux paramètres à régler et ces derniers ont un impact important sur la qualité des résultats obtenus. Une des grandes problématiques liées à ces systèmes de fusion d'informations porte sur l'évaluation de leurs performances. L'évaluation doit permettre d'améliorer la qualité de la fusion, d'améliorer l'interaction expert/système et d'aider à mieux ajuster les paramètres du système. En général, l'évaluation de tels systèmes est réalisée en fin de chaîne par une évaluation globale du résultat. Mais, celle-ci ne permet pas de savoir précisément l'endroit de la chaîne qui nécessite une intervention. Une autre difficulté réside dans le fait qu'une vérité terrain complète sur le résultat n'est pas toujours disponible, ce qui complique la tâche d'évaluation de performances de ce type de systèmes. Le contexte applicatif de ce travail est l'interprétation d'images tridimensionnelles (images tomographiques, images sismiques, images synthétiques, ...). Dans ce contexte une évaluation locale des systèmes de fusion d'informations, a été mise en place. L'approche a montré son intérêt dans l'ajustement efficace des paramètres et dans la coopération avec l'expert.Information fusion systems are mainly composed from mathematical tools allowing to realize data representation and combination. The aim of these systems can be expressed as a decision problem on the truth or plausibility of a proposition based on several information coming from different sources. Fusion try to manage the characteristics of the sources taking into account the information imperfection (inaccurate, incomplete, ambiguous, uncertain, etc.) and the redundant aspect, the complement and the conflictual aspect of information. Fusion systems concerned by this thesis have the ability to integrate the expert knowledge in their treatments. They are called cooperative fusion systems. Since these systems are trying to associate experts, it is important to provide to the users some informations that help them to better understand the fusion process. Such systems have many parameters that must be adjusted. These parameters have an important impact on the quality of the obtained results. One of the major problems associated to information fusion systems concerns the evaluation of their performance. A pertinent evaluation will allow to improve the quality of the fusion, to improve expert/system interaction and to better adjust the parameters of the system. Generally, the evaluation of such systems is made in the ouput of the processing chain by a global evaluation of the results. But it does not allow to know the precise subpart of the treatement chain that requires an adjustment of its parameters. Another difficulty releases in the fact that a complete ground truth of the result is not always available, which complicates the performance evaluation task. The application context of this work is the interpretation of 3D images (tomographic images, seismic images, synthetic images, ...). In this context, a local evaluation of the information fusion systems has been implemented. The approach has shown its interest in the efficient adjustment of parameters and the cooperation with expert

Évaluation de performance d'un système de fusion d'informations : application au traitement d'images 3D

Information fusion systems are mainly composed from mathematical tools allowing to realize data representation and combination. The aim of these systems can be expressed as a decision problem on the truth or plausibility of a proposition based on several information coming from different sources. Fusion try to manage the characteristics of the sources taking into account the information imperfection (inaccurate, incomplete, ambiguous, uncertain, etc.) and the redundant aspect, the complement and the conflictual aspect of information. Fusion systems concerned by this thesis have the ability to integrate the expert knowledge in their treatments. They are called cooperative fusion systems. Since these systems are trying to associate experts, it is important to provide to the users some informations that help them to better understand the fusion process. Such systems have many parameters that must be adjusted. These parameters have an important impact on the quality of the obtained results. One of the major problems associated to information fusion systems concerns the evaluation of their performance. A pertinent evaluation will allow to improve the quality of the fusion, to improve expert/system interaction and to better adjust the parameters of the system. Generally, the evaluation of such systems is made in the ouput of the processing chain by a global evaluation of the results. But it does not allow to know the precise subpart of the treatement chain that requires an adjustment of its parameters. Another difficulty releases in the fact that a complete ground truth of the result is not always available, which complicates the performance evaluation task. The application context of this work is the interpretation of 3D images (tomographic images, seismic images, synthetic images, ...). In this context, a local evaluation of the information fusion systems has been implemented. The approach has shown its interest in the efficient adjustment of parameters and the cooperation with expert.Les systèmes de fusion d'informations sont principalement composés, d'outils mathématiques permettant de réaliser la représentation et la combinaison des données. L'objectif de ces systèmes peut s'exprimer comme un problème de décision sur la vérité ou la vraisemblance d'une proposition étant donné une ou plusieurs informations issues de différentes sources. Les systèmes de fusion cherchent à exploiter au mieux les propriétés des sources de données en tenant compte de l'imperfection de l'information (imprécis, incomplet, ambigu, incertain, etc.) ainsi que l'aspect redondant, complémentaire et conflictuel des informations. Le système de fusion concerné par cette thèse a la capacité d'intégrer dans ses traitements de la connaissance experte. On le nomme système de fusion coopératif. Puisque ce système cherche à intégrer pleinement les experts dans son fonctionnement, il est important de mettre à disposition des utilisateurs des informations aidant à mieux comprendre la fusion réalisée. De tels systèmes disposent de nombreux paramètres à régler et ces derniers ont un impact important sur la qualité des résultats obtenus. Une des grandes problématiques liées à ces systèmes de fusion d'informations porte sur l'évaluation de leurs performances. L'évaluation doit permettre d'améliorer la qualité de la fusion, d'améliorer l'interaction expert/système et d'aider à mieux ajuster les paramètres du système. En général, l'évaluation de tels systèmes est réalisée en fin de chaîne par une évaluation globale du résultat. Mais, celle-ci ne permet pas de savoir précisément l'endroit de la chaîne qui nécessite une intervention. Une autre difficulté réside dans le fait qu'une vérité terrain complète sur le résultat n'est pas toujours disponible, ce qui complique la tâche d'évaluation de performances de ce type de systèmes. Le contexte applicatif de ce travail est l'interprétation d'images tridimensionnelles (images tomographiques, images sismiques, images synthétiques, ...). Dans ce contexte une évaluation locale des systèmes de fusion d'informations, a été mise en place. L'approche a montré son intérêt dans l'ajustement efficace des paramètres et dans la coopération avec l'expert

Aggregation Evaluation of a Fusion System Devoted to Image Interpretation

International audienceInformation fusion has been studied in various domains of computer sciences and engineering, and the use of these techniques has known a large increase. Fusion systems have become over time complex systems integrating information extraction, their representations in an appropriate space, their combinations and their interpretations. The performance evaluation of such systems has become a real problem. The choices of methods and parameter values have a significant impact on the quality of the results. A global evaluation of the fused result does not allow the end-users to adjust the numerous parameters. We propose a local approach to evaluate the mission completeness of the subparts of the fusion system. In this paper, we focuse on the formulation of the mission of fusion subparts and we measure their achievement degree. The aim is to provide to the end-users which subpart do not completely achieve the functionality they were designed for

Local versus global evaluation of a cooperative fusion system for 3D image interpretation

International audienceInformation fusion approaches are more and more used in complex applications in which there is a real need to take into account several kinds of information simultaneously. Fusion systems become complex because they involve all the information treatment chain steps (from the extraction to the decision). A global evaluation of the fused result does not allow the end-users to adjust the numerous parameters and to efficiency interact with the system. This paper proposes a local approach to evaluate the mission completeness of each subpart of the fusion systems. For this, the main mission of each subpart needs to be well formulated and then, a mission achievement measurement will allow to quantify the performance of the subpart according to its objective and independently to the method used inside the subparts. The proposed measurement is based on an histograms comparison and the approach is then illustrated on a real cooperative fusion system devoted to 3D tomographic image interpretation

A multi level evaluation for fusion system interaction improvement

International audienceFusion systems for image interpretation are complex systems that involve a complete information treatment chain (from the information extraction to the decision). Local evaluation of all the sub-parts that composed the system is an interesting way to better characterize its behaviour but it generates many numerical indicators. This paper proposes two intermediate evaluations based on the construction of symbolic indicators from the numerical separability indexes. All the available quality information are then used into a progressive dashboard that allows to better interact with the system

Evaluation locale de la performance basée sur un critère Mission/Qualité

National audienceThe development of an image processing chain remains a delicate task and requires many adjustments difficult to achieve for a non-specialist. The processing chain is cut into four sub-systems and a method is proposed to guide the user to the sub-system which requires the tuning of its parameters to improve the quality of the final result. To evaluate local performance, we developed a test mission/quality associated with each sub-systems. A measure based on a criterion of separability between regions by calculating the Manhattan distance allows a macroscopic adjustment. This same criterion calculated with the Earth Mover's Distance(EMD) allows then a finer adjustment. The feasibility of this approach is used on 3D tomographic images for the quality control of industrial parts

Performance Evaluation of a Fusion System Devoted to Image Interpretation

International audienceThe use of fusion systems has known a wide growth and they now need reliable ways to evaluate their performance. Fusion systems are complex because they involve a complete information treatment chain (from the information extraction to the decision). This paper studies the different approaches used for system evaluation and proposes a local evaluation method for the evaluation of each subpart of the fusion system. The approach is then illustrated on cooperative fusion system devoted to 3D image interpretation

Towards the Supervision of a Fusion System for 3D Image Interpretation

International audienceThe use of cooperative fusion systems for image interpretation according to expert knowledge has known a wide growth and they now need reliable ways to evaluate their performance. Local evaluation is an interesting way to better characterize the system behaviour and consequently to have information on which subpart needs to be adjusted. A previously proposed local evaluation based on separability indexes, generates many numerical indicators. In this paper we proposes a symbolic representation of these separability index easier to understand by the experts. To overcome the application of a threshold, the uncertainty of the numeric measures is represented thanks to possibility distributions

Local evaluation of a fusion system for 3D tomographic image interpretation

International audienceInformation fusion has been studied in various domains of computer sciences and engineering, and these techniques have increasingly been used. Such systems aim to build new interesting information from much information. Over the time, fusion systems have become complex systems integrating information extraction, representations, combinations and interpretations. The performance evaluation of such systems has become a real problem. The choices of methods and parameter values have a significant impact on the quality of the results. A global evaluation of the fused results does not allow the end-users to adjust the numerous parameters. We propose a local approach to evaluate the mission completeness of the fusion system subparts. In this paper, we focus on the formulation of the mission of extraction subparts and we measure their degree of achievement. We aim at showing the end-users which subparts do not completely achieve the functionality they were designed for

A Separability Index Based on Earth Mover's Distance for Local Evaluation of Fusion Systems

International audienceThis paper deals with the uses of the Earth Mover's Distance for local evaluation of fusion system. Local evaluation is an interesting way to better characterize the system behavior and consequently to have information on which subpart needs to be adjusted. The definition of the subpart mission and its measurement are two key points in this problematic. This paper shows that the use of the Earth Mover's Distance for measuring the mission achievement is a more complete mathematical tool. The efficiency of the approach is illustrated on an information fusion system devoted to 3D image analysis