Evaluating Visual Data Analysis Systems: A Discussion Report

International audienceVisual data analysis is a key tool for helping people to make sense of and interact with massive data sets. However, existing evaluation methods (e.g., database benchmarks, individual user studies) fail to capture the key points that make systems for visual data analysis (or visual data systems) challenging to design. In November 2017, members of both the Database and Visualization communities came together in a Dagstuhl seminar to discuss the grand challenges in the intersection of data analysis and interactive visualization. In this paper, we report on the discussions of the working group on the evaluation of visual data systems, which addressed questions centered around developing better evaluation methods, such as " How do the different communities evaluate visual data systems? " and " What we could learn from each other to develop evaluation techniques that cut across areas? ". In their discussions, the group brainstormed initial steps towards new joint evaluation methods and developed a first concrete initiative — a trace repository of various real-world workloads and visual data systems — that enables researchers to derive evaluation setups (e.g., performance benchmarks, user studies) under more realistic assumptions, and enables new evaluation perspectives (e.g., broader meta analysis across analysis contexts, reproducibility and comparability across systems)

A Monitoring Language for Run Time and Post-Mortem Behavior Analysis and Visualization

UFO is a new implementation of FORMAN, a declarative monitoring language, in which rules are compiled into execution monitors that run on a virtual machine supported by the Alamo monitor architecture.Comment: In M. Ronsse, K. De Bosschere (eds), proceedings of the Fifth International Workshop on Automated Debugging (AADEBUG 2003), September 2003, Ghent. cs.SE/030902

Enabling Realistic Cross-Layer Analysis based on Satellite Physical Layer Traces

We present a solution to evaluate the performance of transport protocols as a function of link layer reliability schemes (i.e. ARQ, FEC and Hybrid ARQ) applied to satellite physical layer traces. As modelling such traces is complex and may require approximations, the use of real traces will minimise the potential for erroneous performance evaluations resulting from imperfect models. Our Trace Manager Tool (TMT) produces the corresponding link layer output, which is then used within the ns-2 network simulator via the additionally developed ns-2 interface module. We first present the analytical models for the link layer with bursty erasure packets and for the link layer reliability mechanisms with bursty erasures. Then, we present details of the TMT tool and our validation methodology, demonstrating that the selected performance metrics (recovery delay and throughput efficiency) exhibit a good match between the theoretical results and those obtained with TMT. Finally, we present results showing the impact of different link layer reliability mechanisms on the performance of TCP Cubic transport layer protocol

PADAMOT : project overview report

Background and relevance to radioactive waste management International consensus confirms that placing radioactive wastes and spent nuclear fuel deep underground in a geological repository is the generally preferred option for their long-term management and disposal. This strategy provides a number of advantages compared to leaving it on or near the Earth’s surface. These advantages come about because, for a well chosen site, the geosphere can provide: • a physical barrier that can negate or buffer against the effects of surface dominated natural disruptive processes such as deep weathering, glaciation, river and marine erosion or flooding, asteroid/comet impact and earthquake shaking etc. • long and slow groundwater return pathways from the facility to the biosphere along which retardation, dilution and dispersion processes may operate to reduce radionuclide concentration in the groundwater. • a stable, and benign geochemical environment to maximise the longevity of the engineered barriers such as the waste containers and backfill in the facility. • a natural radiation shield around the wastes. • a mechanically stable environment in which the facility can be constructed and will afterwards be protected. • an environment which reduces the likelihood of the repository being disturbed by inadvertent human intrusion such as land use changes, construction projects, drilling, quarrying and mining etc. • protection against the effects of deliberate human activities such as vandalism, terrorism and war etc. However, safety considerations for storing and disposing of long-lived radioactive wastes must take into account various scenarios that might affect the ability of the geosphere to provide the functionality listed above. Therefore, in order to provide confidence in the ability of a repository to perform within the deep geological setting at a particular site, a demonstration of geosphere “stability” needs to be made. Stability is defined here to be the capacity of a geological and hydrogeological system to minimise the impact of external influences on the repository environment, or at least to account for them in a manner that would allow their impacts to be evaluated and accounted for in any safety assessments. A repository should be sited where the deep geosphere is a stable host in which the engineered containment can continue to perform according to design and in which the surrounding hydrogeological, geomechanical and geochemical environment will continue to operate as a natural barrier to radionuclide movement towards the biosphere. However, over the long periods of time during which long-lived radioactive wastes will pose a hazard, environmental change at the surface has the potential to disrupt the stability of the geosphere and therefore the causes of environmental change and their potential consequences need to be evaluated. As noted above, environmental change can include processes such as deep weathering, glaciation, river and marine erosion. It can also lead to changes in groundwater boundary conditions through alternating recharge/discharge relationships. One of the key drivers for environmental change is climate variability. The question then arises, how can geosphere stability be assessed with respect to changes in climate? Key issues raised in connection with this are: • What evidence is there that 'going underground' eliminates the extreme conditions that storage on the surface would be subjected to in the long term? • How can the additional stability and safety of the deep geosphere be demonstrated with evidence from the natural system? As a corollary to this, the capacity of repository sites deep underground in stable rock masses to mitigate potential impacts of future climate change on groundwater conditions therefore needs to be tested and demonstrated. To date, generic scenarios for groundwater evolution relating to climate change are currently weakly constrained by data and process understanding. Hence, the possibility of site-specific changes of groundwater conditions in the future can only be assessed and demonstrated by studying groundwater evolution in the past. Stability of groundwater conditions in the past is an indication of future stability, though both the climatic and geological contexts must be taken into account in making such an assertion

Evaluating advanced search interfaces using established information-seeking model

When users have poorly defined or complex goals search interfaces offering only keyword searching facilities provide inadequate support to help them reach their information-seeking objectives. The emergence of interfaces with more advanced capabilities such as faceted browsing and result clustering can go some way to some way toward addressing such problems. The evaluation of these interfaces, however, is challenging since they generally offer diverse and versatile search environments that introduce overwhelming amounts of independent variables to user studies; choosing the interface object as the only independent variable in a study would reveal very little about why one design out-performs another. Nonetheless if we could effectively compare these interfaces we would have a way to determine which was best for a given scenario and begin to learn why. In this article we present a formative framework for the evaluation of advanced search interfaces through the quantification of the strengths and weaknesses of the interfaces in supporting user tactics and varying user conditions. This framework combines established models of users, user needs, and user behaviours to achieve this. The framework is applied to evaluate three search interfaces and demonstrates the potential value of this approach to interactive IR evaluation