Challenges in Data Intensive Analysis at Scientific Experimental User Facilities

Today's scientific challenges such as routes to a sustainable energy future, materials by design or biological and chemical environmental remediation methods, are complex problems that require the integration of a wide range of complementary expertise to be addressed successfully. Experimental and computational science research methods can hereby offer fundamental insights for their solution. Experimental facilities in particular can contribute through a large variety of investigative methods, which can span length scales from millions of kilometers (radar) to the sub-nucleus (LHC). These methods are used to probe structure, properties, and function of objects from single elements to whole communities. Hereby direct imaging techniques are a powerful means to develop an atomistic understanding of scientific issues. For example, the identification ofmechanisms associated with chemical, material, and biological transformations requires the direct observation of the reactions to build up an understanding of the atom-by-atom structural and chemical changes. Computational science can aid the planning of such experiments, correlate results, explain or predict the phenomena as they would be observed and thus aid their interpretation. Furthermore computational science can be essential for the investigation of phenomena that are difficult to observe due to their scale, reaction time or extreme conditions. Combining experimental and computational techniques provides scientists with the ability to research structures and processes at various levels of theory, e.g. providing molecular 'movies' of complex reactions that show bond breaking and reforming in natural time scales, along with the intermediate states to understand the mechanisms that govern the chemical transformations. This chapter will discuss the critical data intensive analysis challenges faced by the experimental science community at large scale and laboratory based facilities. The chapter will highlight current solutions and lay out perspectives for the future, such as methods to achieve real time analysis capabilities and the challenges and opportunities of data integration across experimental scales, levels of theory, and varying techniques