Capturing Data Provenance from Statistical Software

We have created tools that automate one of the most burdensome aspects of documenting the provenance of research data: describing data transformations performed by statistical software.  Researchers in many fields use statistical software (SPSS, Stata, SAS, R, Python) for data transformation and data management as well as analysis.  The C2Metadata ("Continuous Capture of Metadata for Statistical Data") Project creates a metadata workflow paralleling the data management process by deriving provenance information from scripts used to manage and transform data.  C2Metadata differs from most previous data provenance initiatives by documenting transformations at the variable level rather than describing a sequence of opaque programs.  Command scripts for statistical software are translated into an independent Structured Data Transformation Language (SDTL), which serves as an intermediate language for describing data transformations.   SDTL can be used to add variable-level provenance to data catalogues and codebooks and to create "variable lineages" for auditing software operations.   Better data documentation makes research more transparent and expands the discovery and re-use of research data

Automating the Capture of Data Transformation Metadata from Statistical Analysis Software

The C2Metadata (“Continuous Capture of Metadata for Statistical Data”) Project automates one of the most burdensome aspects of documenting the provenance of research data: describing data transformations performed by statistical software. Researchers in many fields use statistical software (SPSS, Stata, SAS, R, Python) for data transformation and data management as well as analysis. The C2Metadata Project creates a metadata workflow paralleling the data management process by deriving provenance information from scripts used to manage and transform data. C2Metadata differs from most previous data provenance initiatives by documenting transformations at the variable level rather than describing a sequence of opaque programs. Scripts used with statistical software are translated into an independent Structured Data Transformation Language (SDTL), which serves as an intermediate language for describing data transformations. SDTL can be used to add variable-level provenance to data catalogs and codebooks and to create “variable lineages” for auditing software operations. Better data documentation makes research more transparent and expands the discovery and re-use of research data.National Science Foundation grant ACI-1640575https://deepblue.lib.umich.edu/bitstream/2027.42/156014/3/Automating_metadata_capture_v15.pd

Provenance Metadata for Statistical Data: An Introduction to Structured Data Transformation Language (SDTL)

Structured Data Transformation Language (SDTL) provides structured, machine actionable representations of data transformation commands found in statistical analysis software. The Continuous Capture of Metadata for Statistical Data Project (C2Metadata) created SDTL as part of an automated system that captures provenance metadata from data transformation scripts and adds variable derivations to standard metadata files. SDTL also has potential for auditing scripts and for translating scripts between languages. SDTL is expressed in a set of JSON schemas, which are machine actionable and easily serialized to other formats. Statistical software languages have a number of special features that have been carried into SDTL. We explain how SDTL handles differences among statistical languages and complex operations, such as merging files and reshaping data tables from “wide” to “long”.National Science Foundation grant ACI-1640575https://deepblue.lib.umich.edu/bitstream/2027.42/156015/1/SDTL_Intro_v14.pdfDescription of SDTL_Intro_v14.pdf : Main articl

Recovery, rehabilitation and follow-up services following critical illness: an updated UK national cross-sectional survey and progress report

Objective: To comprehensively update and survey the current provision of recovery, rehabilitation and follow-up services for adult critical care patients across the UK. Design: Cross-sectional, self-administered, predominantly closed-question, electronic, online survey. Setting: Institutions providing adult critical care services identified from national databases. Participants: Multiprofessional critical care clinicians delivering services at each site. Results: Responses from 176 UK hospital sites were included (176/242, 72.7%). Inpatient recovery and follow-up services were present at 127/176 (72.2%) sites, adopting multiple formats of delivery and primarily delivered by nurses (n=115/127, 90.6%). Outpatient services ran at 130 sites (73.9%), predominantly as outpatient clinics. Most services (n=108/130, 83.1%) were co-delivered by two or more healthcare professionals, typically nurse/intensive care unit (ICU) physician (n=29/130, 22.3%) or nurse/ICU physician/physiotherapist (n=19/130, 14.6%) teams. Clinical psychology was most frequently lacking from inpatient or outpatient services. Lack of funding was consistently the primary barrier to service provision, with other barriers including logistical and service prioritisation factors indicating that infrastructure and profile for services remain inadequate. Posthospital discharge physical rehabilitation programmes were relatively few (n=31/176, 17.6%), but peer support services were available in nearly half of responding institutions (n=85/176, 48.3%). The effects of the COVID-19 pandemic resulted in either increasing, decreasing or reformatting service provision. Future plans for long-term service transformation focus on expansion of current, and establishment of new, outpatient services. Conclusion: Overall, these data demonstrate a proliferation of recovery, follow-up and rehabilitation services for critically ill adults in the past decade across the UK, although service gaps remain suggesting further work is required for guideline implementation. Findings can be used to enhance survivorship for critically ill adults, inform policymakers and commissioners, and provide comparative data and experiential insights for clinicians designing models of care in international healthcare jurisdictions

C2Metadata: Continuous Capture of Metadata

This poster presentation, given at the 2017 Society of American Archivists (SAA) Research Forum on July 25, 2017 in Portland, Oregon, describes the C2Metadata project to develop new tools that will work with common statistical packages to automate the capture of metadata at the granularity of individual data transformations.Supported by the Data Infrastructure Building Blocks (DIBBs) program of the National Science Foundation through grant NSF ACI-1640575.https://deepblue.lib.umich.edu/bitstream/2027.42/145473/1/Lyle_C2Metadata-updated3.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/145473/3/Lyle_ResearchForumAbstractBio2017_1.pdfDescription of Lyle_C2Metadata-updated3.pdf : Poster presentationDescription of Lyle_ResearchForumAbstractBio2017_1.pdf : Poster presentation abstrac