Refactoring data delivery: The case study of the new tools for census flow data at UKDS

This paper presents the API and user interface for disseminating census flow data developed as part of the UK Data Service (UKDS) strategy to modernise its supported software. It explores the development challenges and the diverse technologies employed, contextualised within the broader scope of data services. The new tools were in the final stages of development at the time of writing. We opted to construct these tools from the ground up, a decision influenced by the complex nature of the underlying data, which necessitated high levels of flexibility and adaptability. This paper will critically evaluate the decision-making process, weighing the advantages and disadvantages of developing in-house solutions versus the trend of relying on external, often commercial, platforms. Third-party solutions frequently compromise functionality and the ability to tailor to specific requirements, especially when dealing with highly complex data. The presentation will showcase some functionalities of these new tools, highlighting ongoing enhancements, including integrating AI and machine-learning technologies. We will also discuss the advantages of the design principle to separate the user interface from the backend API. This approach improves user experience and promotes better interoperability. While acknowledging that budget and overall resource constraints are a common hurdle in such initiatives, this case study provides insights into the feasible options available to data services striving to deliver robust and comprehensive data to their users. The insights and experience shared are intended to contribute to the dynamic relationship between data services and information science, especially when dealing with data related to the social sciences. By offering practical examples and lessons learned, we aim to inform those enhancing data accessibility, utility, and distribution in a rapidly evolving digital environment, where data services often face challenges in keeping pace with technological advances

UK Census 2021 Challenges and data access: data on the LGBTQ+ population and privacy

The UK Data Service (UKDS) is the flagship project in UK research infrastructure. It is a collaboration between the universities of Essex, Manchester, Southampton, UCL, Edinburgh and Jisc, and it provides a wide range of social sciences, humanities, and economic research data. Part of the data offered by UKDS is census data. The Office for National Statistics has partnered with UKDS to disseminate 2011 and 2021 census data, trusting it with securing access to safeguarded tables. The 2021 census has been unique because it took place amid the Covid-19 pandemic, which is expected to have affected the given responses. For these reasons, Scotland decided to delay its census by one year to 2022, which has made harmonising census data across the UK almost impossible. This paper will discuss the challenges and peculiarities of the 2021 census in the UK and will demonstrate how UKDS handles data management and access to census data. As the first data derived from the 2021 census began hitting the headlines in late November 2022, most data are yet to be released. The paper will present some of the privacy measures implemented to protect persons’ identities in areas with few people. In addition, it will present findings on gender identity and sexual orientation, which were introduced for the first time in the UK census history. The UKDS is currently developing new tools to modernise its census delivery services. The paper will present some of these applications, with the most emphasis on introducing the census origin and destination data API and the new user interface, both currently under heavy development. These tools will be made available after ONS releases the related data, expected in Q2 2023. The IASSIST conference will be the first opportunity to present them to the public

Meeting new challenges posed by the UK Censuses

This paper builds on and extends a presentation at IASSIST2023 which outlined the challenges faced in disseminating data from the 2021/2022 Censuses in the United Kingdom, as part of the UK Data Service (UKDS). The UKDS is a key part of UK research infrastructure, and provides a wide range of social sciences, humanities, and economic research data with census data being one of the major collections. A range of tools have been developed within UKDS to provide access to data, and these continue to be extended. New workflows exist for aggregate data and for origin-destination data – the latter being addressed in detail in a separate submission to this conference. In this paper we illustrate the full set of current tools, explain how they can be used, and showcase examples of analysis. Census data are disseminated with a variety of license arrangements: open, safeguarded, and secure, and we reflect on shifting balances of risk appetite, and what this means for researchers, including those based outside the UK. We also reflect more broadly on long-standing and new challenges faced in disseminating this data: the size and complexity of the resulting outputs; user expectations; new competitors etc. The 2021 census round in the UK was notable for a number of reasons: operational changes enforced by the covid-19 pandemic including deferment in Scotland, new questions and legal challenges to question and guidance wording. The wider context is a public consultation over the future of population and migration data collection in the UK, with the possibility that the 2021 census will also turn out to have the last such census, and thus we also reflect on how we can adapt a census archive/service to a future 'administrative based census' archive/service, with more frequent data about which researchers may be less aware