Simplifying the Development, Use and Sustainability of HPC Software

Developing software to undertake complex, compute-intensive scientific processes requires a challenging combination of both specialist domain knowledge and software development skills to convert this knowledge into efficient code. As computational platforms become increasingly heterogeneous and newer types of platform such as Infrastructure-as-a-Service (IaaS) cloud computing become more widely accepted for HPC computations, scientists require more support from computer scientists and resource providers to develop efficient code and make optimal use of the resources available to them. As part of the libhpc stage 1 and 2 projects we are developing a framework to provide a richer means of job specification and efficient execution of complex scientific software on heterogeneous infrastructure. The use of such frameworks has implications for the sustainability of scientific software. In this paper we set out our developing understanding of these challenges based on work carried out in the libhpc project.Comment: 4 page position paper, submission to WSSSPE13 worksho

High-precision measurements of the co-polar correlation coefficient: non-Gaussian errors and retrieval of the dispersion parameter µ in rainfall

The co-polar correlation coefficient (ρhv) has many applications, including hydrometeor classification, ground clutter and melting layer identification, interpretation of ice microphysics and the retrieval of rain drop size distributions (DSDs). However, we currently lack the quantitative error estimates that are necessary if these applications are to be fully exploited. Previous error estimates of ρhv rely on knowledge of the unknown "true" ρhv and implicitly assume a Gaussian probability distribution function of ρhv samples. We show that frequency distributions of ρhv estimates are in fact highly negatively skewed. A new variable: L = -log10(1 - ρhv) is defined, which does have Gaussian error statistics, and a standard deviation depending only on the number of independent radar pulses. This is verified using observations of spherical drizzle drops, allowing, for the first time, the construction of rigorous confidence intervals in estimates of ρhv. In addition, we demonstrate how the imperfect co-location of the horizontal and vertical polarisation sample volumes may be accounted for. The possibility of using L to estimate the dispersion parameter (µ) in the gamma drop size distribution is investigated. We find that including drop oscillations is essential for this application, otherwise there could be biases in retrieved µ of up to ~8. Preliminary results in rainfall are presented. In a convective rain case study, our estimates show µ to be substantially larger than 0 (an exponential DSD). In this particular rain event, rain rate would be overestimated by up to 50% if a simple exponential DSD is assumed

A robust automated technique for operational calibration of ceilometers using the integrated backscatter from totally attenuating liquid clouds

A simple and robust method for calibrating ceilometers has been tested in an operational environment demonstrating that the calibrations are stable to better than ± 5 % over a period of a year. The method relies on using the integrated backscatter (B) from liquid clouds that totally extinguish the ceilometer signal; B is inversely proportional to the lidar ratio (S) of the backscatter to the extinction for cloud droplets. The calibration technique involves scaling the observed backscatter so that B matches the predicted value for S of 18.8 ± 0.8 sr for cloud droplets, at ceilometer wavelengths. For accurate calibration, care must be taken to exclude any profiles having targets with different values of S, such as drizzle drops and aerosol particles, profiles that do not totally extinguish the ceilometer signal, profiles with low cloud bases that saturate the receiver, and any profiles where the window transmission or the lidar pulse energy is low. A range dependent multiple scattering correction that depends on the ceilometer optics should be applied to the profile. A simple correction for water vapour attenuation for ceilometers operating at around 910 nm wavelength is applied to the signal using the vapour profiles from a forecast analysis. For a generic ceilometer in the UK the 90-day running mean of the calibration coefficient over a period of 20 months is constant to within 3 % with no detectable annual cycle, thus confirming the validity of the humidity and multiple scattering correction. For Gibraltar, where cloud cover is less prevalent than in the UK, the 90-day running mean calibration coefficient was constant to within 4 %. The more sensitive ceilometer model operating at 1064 nm is unaffected by water vapour attenuation but is more prone to saturation in liquid clouds. We show that reliable calibration is still possible, provided the clouds used are above a certain altitude. The threshold is instrument dependent but is typically around 2 km. We also identify a characteristic signature of saturation, and remove any profiles with this signature. Despite the more restricted sample of cloud profiles, a robust calibration is readily achieved, and, in the UK, the running mean 90-day calibration coefficients varied by about 4 % over a period of one year. The consistency of profiles observed by nine pairs of co-located ceilometers in the UK Met Office network operating at around 910 nm and 1064 nm provided independent validation of the calibration technique. EUMETNET is currently networking 700 European ceilometers so they can provide ceilometer profiles in near real time to European weather forecast centres and has adopted the cloud calibration technique described in this paper for ceilometers with a wavelength of around 910 nm

A Ceph S3 Object Data Store for HEP

We present a novel data format design that obviates the need for data tiers by storing individual event data products in column objects. The objects are stored and retrieved through Ceph S3 technology, with a layout designed to minimize metadata volume and maximize data processing parallelism. Performance benchmarks of data storage and retrieval are presented.Comment: CHEP2023 proceedings, to be published in EPJ Web of Conference

Taking a Breath of the Wild: are geoscientists more effective than non-geoscientists in determining whether video game world landscapes are realistic?

From the wilderness of Hyrule, the continent of Tamriel, and the geographies of Middle Earth, players of video games are exposed to wondrous, fantastic, but ultimately fake, landscapes. Given the time people may spend in these worlds compared to the time they spend being trained in geoscience, we wondered whether expert geoscientists would differ from non-geoscientists in whether they judge the landscapes in these video games to be “realistic”. Since video games present a great opportunity for tangential learning, it would be a missed opportunity if it turns out that features obviously fake to geoscientists are perceived as plausible by non-geoscientists.To satisfy our curiosity and answer this question, we conducted a survey where we asked people to judge both photos from real landscapes as well as screenshots from the recent The Legend of Zelda: Breath of the Wild video game on how likely they thought the features in the picture were to exist in the real world. Since game world screenshots are easily identified based on their rendered, pixelated nature, we pre-processed all pictures with an artistic “Van Gogh” filter that removed the rendered nature but retained the dominant landscape features.We found that there is a small but significant difference between geoscientists and non-geoscientists, with geoscientists being slightly better at judging which pictures are from the real world versus from the video game world. While significant, the effect is small enough to conclude that fantastical worlds in video games can be used for tangential learning on geoscientific subjects

Using advanced manufacturing technology for smarter construction

Productivity in the UK construction sector has historically lagged behind other industry sectors. The government is aiming to improve this through increasing the level of pre-manufactured value in built assets. Since 2001, the University of Sheffield's Advanced Manufacturing Research Centre has been developing technological innovations for the aerospace and automotive sectors. This paper shows how lessons learnt from, and technologies developed for, these sectors can be transferred into the construction supply chain through horizontal innovation. Technologies such as robotics and automation, augmented and virtual reality, discrete event simulation, large volume metrology, and improved tools and processes all have a role to play. Significant productivity increases are possible, with the benefit often driven by the digitalisation of traditionally manual paper-based processes

Optical systems integrated modeling

An integrated modeling capability that provides the tools by which entire optical systems and instruments can be simulated and optimized is a key technology development, applicable to all mission classes, especially astrophysics. Many of the future missions require optical systems that are physically much larger than anything flown before and yet must retain the characteristic sub-micron diffraction limited wavefront accuracy of their smaller precursors. It is no longer feasible to follow the path of 'cut and test' development; the sheer scale of these systems precludes many of the older techniques that rely upon ground evaluation of full size engineering units. The ability to accurately model (by computer) and optimize the entire flight system's integrated structural, thermal, and dynamic characteristics is essential. Two distinct integrated modeling capabilities are required. These are an initial design capability and a detailed design and optimization system. The content of an initial design package is shown. It would be a modular, workstation based code which allows preliminary integrated system analysis and trade studies to be carried out quickly by a single engineer or a small design team. A simple concept for a detailed design and optimization system is shown. This is a linkage of interface architecture that allows efficient interchange of information between existing large specialized optical, control, thermal, and structural design codes. The computing environment would be a network of large mainframe machines and its users would be project level design teams. More advanced concepts for detailed design systems would support interaction between modules and automated optimization of the entire system. Technology assessment and development plans for integrated package for initial design, interface development for detailed optimization, validation, and modeling research are presented