Presentation at the North American Data Documentation Conference (NADDI) 2013Paradata is data on study processes and the collection of study data. Here we describe the development of a Paradata Information Model
(PIM) in support of the National Children¹s Study (NCS) of the Eunice Kennedy Shriver National Institute of Child Health and Development. We propose that paradata can be recorded with accompanying metadata informed by the General Longitudinal Business Process Model (GLBPM) developed by the Data Documentation Initiative
(DDI) and the General Statistical Business Process Model (GSBPM). The PIM is be constructed in a joint top-down and bottom-up approach, appropriating broad verbs from DDI, HL7, LS-DAM, and CDISC, while incorporating study-specific processes involved in collecting NCS operational data elements (ODEs). 
The hope of paradata in longitudinal studies is that the collection of paradata will ensure that future researchers can integrate disparate data sets collected by a variety of technologies, especially in rapidly-evolving fields like genomics. Additionally, by giving PIM elements preconditions and postconditions, we can develop software agents which use paradata metadata as well as other information to assist humans in conducting biomedical research, ultimately facilitating more rapid collection and analysis of information and enabling a broader subset of researchers to discover and extract relevant information from study data sets.Institute for Policy & Social Research, University of Kansas; University of Kansas Libraries; Alfred P. Sloan Foundation; Data Documentation Initiative Alliance, Booz Allen Hamilto

Carpenter, Danielle

Greenfield, Jay

English

KU ScholarWorks

The Paradata Information Model NADDI April 2013 Lawrence, KS  1 Agenda • Motivation for the Paradata Information Model (PIM) – The National Children’s Study (NCS) – The National Science Digital Library (NSDL) • Model Building – Bricoleur and Bricolage – Meet the Strawman: Generic Longitudinal Business Process Model (GLBPM) – Specializing GLBPM – The PIM Formalism • Sequencing Data Collection, Data Processing and Data Understanding Activities – The Microarray Experiment Use Case – Understanding Sequences – The Gamification of GSBPM • Next Steps 2 PARADATA IN THE NATIONAL CHILDREN’S STUDY Use Case 3 In the NCS so-called operational data elements were defined and designed to assist in the assessment of feasibility, acceptability and cost From Lepkowski and Kirgis based on work by Mosher and Groves  4 Instrument and Operational Data Elements 5 Paradata in the NCS is a DataCollectionType without any instruments… 6 In DDI our ODEs are variables…  7 …that we locate in logical records 8 There is in fact a better way… • What we haven’t done is a best practice described in Documenting a Wider Variety of Data Using the Data Documentation Initiative 3.1 – Here it is suggested that we first use <Instrument> to document a biospecimen data collection – Then it is suggested that we use a <ProcessingEvent> and its <GenerationInstruction> to tie blood draw with a <LogicalProduct>  – The <LogicalProduct> contains assay findings from the blood draw – certain cardiovascular variables  • This is useful and a great best practice • But the cardiovascular variables are findings – Findings come at the end of a sequence in which first a procedure is performed at a visit and then the material collected during the procedure is handled and processed 9 Putting findings in context… • What we need is for our <ProcessingEvent> with its <GenerationInstruction> to account not just for the finding, i.e. the data • Additionally, it needs to describe the process that makes the finding occur. i.e. the paradata • We specifically need the paradata to qualify the data – How does one go and reproduce a finding? – What did it cost? – What value does it have?    Study What Finding Condition How Perform Procedure Perform Material Handing Is it a keeper? Portfolio  Management Cost Benefit We perform activities in the service of collecting, processing and evaluating data. We define paradata variables. There is paradata metadata. Should the design and collection of paradata be in this picture? This is GLBPM… 10 Let’s also talk about how paradata figures into retrospective evaluation more specifically. This is perhaps a “reach” because the idea of considering the “buzz” surrounding a finding as paradata may seem novel or even farfetched 11 12 A retrospective evaluation of cow’s milk… What was the data quality of this question’s variable? Given the question’s intent what chatter should we be listening for in the research community? What is the chatter? 13 Cow’s Milk We can search PubMed using our MeSH terms 14 What is the chatter? 15 We return to our domain model and look for asthma under diseases and disorders Domain Model Exposure Stressor See Exposure Stressors Receptor Built Environment Human Individual See Human Attributes Outcome Biologic Response Phenotypic Outcome Function See NIH Toolbox Outcomes Disease Disorder Finding Intervention 16 Documenting the chatter • There is  perhaps an association between cow’s milk ingested by children and asthma under certain circumstances • The type of association is a “pathway” • Pathways make our data hypothesis aware  Question 17 A retrospective evaluation of cow’s milk… What was the data quality of this question’s variable? Given the question’s intent what chatter should we be listening for in the research community? What is the chatter? data collection paradata question metadata research community chatter is retrospective evaluation data Together they participate in a workflow. Data quality is a precondition for listening to research community chatter. The question’s intent is an input that informs the listening activity 18 Lessons Learned: Paradata doesn’t seem to know any boundaries  From NSDL: In creating the concept of the STEM Exchange, we needed to distinguish between traditional, relatively static metadata that describes a digital learning object and the dynamic information about digital learning objects that is generated as they are used, reused, adapted, contextualized, favorited, tweeted, retweeted, shared, and all the other social media style ways in which educational users interact with resources. In this context, paradata captures the user activity related to the resource that helps to elucidate its potential educational utility.  We first presented paradata at a March 3, 2010 presentation as... – a complement to metadata, not a replacement – separate layer of information from metadata – a means to automate information generation about resource use by using social networking tools – a means to create an open source and open access data space around resources – emphasizes dissemination rather than description – accommodates expert and user-generated knowledge  19 Lessons Learned: Research is a practice with steps. Paradata can be about one step at a time like call records, question navigation or question acceptability but in the end we all walk around with very long checklists and paradata is a multi-step process • Has an EHR record for this participant been received (paradata)? • If not, what is the next visit in the protocol for this participant (paradata + metadata)? – Does it include performing a procedure (metadata)? – If so, what preparations have been undertaken (paradata)?  • When was the equipment last calibrated (paradata)?  • Is a kit required (metadata)?  • If so, is this kit in store (paradata)?  20 A PARTIAL PARADATA MODEL AND HOW WE BUILT IT Bricoleur and Bricolage The PIM Bricolage 21 Click green for more specificity 22 Sampling Interview 23 24 Performed Procedure Performed Material Processing Click green for more specificity 25 26 Performed Sequencing 27 Some paradata information model construction principles (1 of 2) • We don’t need to be right. We just don’t want to be wrong • We begin with the strawman – The strawman is an upper ontology or in the context of UML modeling the strawman consists of upper layer components in a conceptual model that is a work in progress – We don’t create our own strawman – We borrow a published one • We specialize a strawman by once again borrowing from other published models – We are bricoleurs constructing bricolage – In his book The Savage Mind (1962, English translation 1966), French anthropologist Claude Lévi-Strauss used 'bricolage' to describe the characteristic patterns of mythological thought. In his description it is opposed to the engineers' creative thinking, which proceeds from goals to means. Mythical thought, according to Levi-Strauss, attempts to re-use available materials in order to solve new problems.[3]  28 Some paradata information model construction principles (2 of 2) – In information systems, bricolage is used by Claudio Ciborra to describe the way in which strategic information systems (SIS) can be built in order to maintain successful competitive advantage over a longer period of time than standard SIS. By valuing tinkering and allowing SIS to evolve from the bottom-up, rather than implementing it from the top-down, the firm will end up with something that is deeply rooted in the organisational culture that is specific to that firm and is much less easily imitated.[6] – In her book Life on the Screen (1995), Sherry Turkle discusses the concept of bricolage as it applies to problem solving in code projects and workspace productivity. She advocates the "bricoleur style" of programming as a valid and underexamined alternative to what she describes as the conventional structured "planner" approach. In this style of coding, the programmer works without an exhaustive preliminary specification, opting instead for a step-by-step growth and re-evaluation process. In her essay "Epistemological Pluralism", Turkle writes: "The bricoleur resembles the painter who stands back between brushstrokes, looks at the canvas, and only after this contemplation, decides what to do next."[7] • Our strawman is GLBPM…  29 Specializing the GLBPM (1 of 3) • PIM is a taxonomy of the activities performed in the service of research data collection, processing and understanding. These activities are described with increasing specificity as we traverse the PIM where activities go from broad to narrow: Visit Perform Procedure Perform Specimen Collection Kit Preparation Device Calibration Interim Storage Preparation Material Inspection (QC) 30 Specializing the GLBPM (2 of 3) • We have taken the broad from first DDI Lifecycle and HL7 and then from the DDI GLBPM   • Next we specialized parts of the model further so it could describe biomedical research paradata  – Here we have been guided by the caBIG Life Sciences Domain Analysis Model  (LS-DAM) which is itself based on the CDISC Biomedical Research Integrated Domain Group (BRIDG) Domain Analysis Model  • LS-DAM provided specificity with its Specimen Core.  – The Specimen Core identifies both the players and the acts these players engage in to handle and process biologic specimens once they are collected: DDI  Lifecycle GLBPM BRIDG LS-DAM HL7 RIM Specializing the GLBPM (3 of 3) • We have in turn specialized the LS-DAM by observing practices in the National Children’s Study (NCS) – By way of example LS-DAM includes a PerformedSpecimenCollectionStep which has remain largely unspecialized because the focus of the Specimen Core is more on the PerformedMaterialProcessStep. – However, the NCS Operational Data Elements include a number of activities that detail the PerformedSpecimenCollectionStep including:  • Precision Thermometer Calibration and Verification • Preparation of refrigerators and freezers • Preparation of a breast milk collection kit  • The PIM, like all bricoleur, is nothing but opportunism with a dash of good judgment 31 32 PIM has a formalism (1 of 3) • Actually it is a blend of two formalisms. • The first formalism is HL7 RIM • In the RIM entities play a role and participate in acts. In the predicate there can be multiple acts, and multiple actors so in the same act there can be a doctor and a patient, a researcher and a participant and so forth 33 PIM has a formalism (2 of 3) • The second model of an activity is in widespread use across the semantic web.  – In this model there is an ACTOR, a VERB and an OBJECT or, again, a SUBJECT, PREDICATE and an OBJECT.  – In the second model an activity is a triple. Here is an example from JSON. JSON is JavaScript Object Notation:  PIM has a formalism (3 of 3) • We might represent the PIM as a UML model or as an ontology using OWL – In either case PIM is on a path that supports an RDF layer • That being said, this formalism is lacking • What it lacks is a layer that groups and sequences data collection, data processing and data understanding activities • Our next section is a detailed use case that will provide the requirements for this process model  34 35 THE MICROARRAY EXPERIMENT USE CASE A Deep Dive into Sequencing Paradata 36 The Microarray Experiment (1 of 4) 37 The Microarray Experiment (2 of 4) 38 The Microarray Experiment (3 of 4) 39 The Microarray Experiment (4 of 4) • At each step paradata is collected to document and assure data quality – In general, a researcher should record the date on which each step of an experiment occurred and any information necessary to allow another scientist to reproduce the experiment. This is the content that is typically recorded into a lab notebook.  – This information is also important when checking the quality of data produced by the experiment.  • For example, if a researcher has too many samples in a microarray experiment to complete all at the same time, he or she may break the set of samples up into smaller batches.  • If the researcher then notices while doing later analysis that all of the microarrays with the highest fluorescent signal came from one batch, he or she would be concerned that the signal he or she observes is due to technical error, a “batch effect,” rather than true biological signal.  • In the genotyping experiment above, the majority of steps shown in parts A, B, and C are shared with many other types of biological samples, and many of these actions are already encompassed by LS-DAM.  – This suggests that much of the paradata of interest in a genomic workflow can be adopted by modifying an existing scientific domain model.  – Only Part D contains steps that specifically relate to genomic analyses.  40 PIM REDUX Lessons Learned from the Microarray Experiment Use Case 41 PIM Pong (1 of 2) Activity Activity Sequence Activity Actor Verb Object Sequence Sequence Game PIM atoms PIM building blocks aka “molecules” PIM is a composer who plugs-’n-plays sequences. PIM compositions include “the microarray experiment  game”, “the whole genome sequencing game”, etc. 42 PIM Pong (2 of 2) • To put this in words… – Imagine a log • It might be the log a case manager keeps by participant and visit • It might be the lab log a research assistant keeps who is conducting a genomic experiment with a collected biospecimen – The log is smart: it knows sequences • With case management it know the sequence of activities for scheduling a visit – Some of the activities may be conditional on the outcome(s) of other activities • In the genotyping experiment there are sequences like “investigator isolates DNA” and “investigator purifies DNA” – In order to further the conduct of research science workers assemble sequences into compositions they perform in real time • The log is a record of what happens in the course of these compositions   43 PUTTING IT ALL TOGETHER DDI Study Model From Wendy’s Core Common Business Core Occurrent Model Actionable Paradata Business Core  Continuant Model Descriptive Metadata Business Core Continuant Model Actors Each continuant model is an inventory of all entities  existing at a time. Each occurrent model is an inventory (processory) of all the processes unfolding through a given interval of time. See Basic Formal Ontology for a discussion of dependencies. One type of dependency of interest here is realizable/realized by where a dependent continuant is realized by an occurrent 45 Continuants, Occurrents and their Dependencies Input • Actor(s) • DDI Study Object(s) Output DDI Study Object(s) 46 Continuants, Occurrents and their Dependencies: Automatic Data Linking • Today CDISC has launched its Healthcare Link that aims to leverage EHR information programmatically in clinical trial systems.  – In the Healthcare Link there is a Retrieve Form for Data Capture (RFD).  – An RFD Profile that negotiates with a host EHR system fuels the RFD.  – The RFD works like a questionnaire only the EHR is the respondent and the interviewer is the RFD Profile. RFD is enabled to mark partial completes, breakoffs and comment on data quality in a way that conforms to the 21CFR11 quality tracking standard.  47  Data Linking Instance Input • EHR • RFD Profile Output Linked Data Audit Trail 48 Continuants, Occurrents and their Dependencies: Automatic Comparisons (1 of 3) • DDI provides a metadata framework for represents the code lists that are used in connection with single select and multi-select questions: 49 Continuants, Occurrents and their Dependencies: Automatic Comparisons (2 of 3) • Using this framework it is possible to perform comparisons between questions with code lists. In this example, a code list has changed over time in a longitudinal study: 50 Continuants, Occurrents and their Dependencies: Automatic Comparisons (3 of 3) • Using processing steps with preconditions and postconditions, comparisons like this may be performed by software agents informed by business rules and a decision tree that guides the workflow. 51 Comparison Instance Input • Software Agent • Code List #1 • Code List #2 • Mapping Algorithm Output Map Harmonized Code List 52 Continuants, Occurrents and their Dependencies: Automatic Biospecimen Management • Today in the area of PerformMaterialProcessing much biospecimen handling is performed by machines with complex control systems that humans can configure.  53  Biospecimen QC Instance Input • Brooks BioStoreTM • Biospecimens • BioStoreTM Program Instructions Output Quality Reports Audit Trails 54 Continuants, Occurrents and their Dependencies: Automatic Experiments (1 of 2) • Today high-throughput technologies are used to gather large quantities of biological data.  • Microarrays and next-generation sequencing allow investigators to observe an individual’s genome and epigenome • And mass spectrometry provides access to the proteome (the complete set of proteins of an organism) and metabolome (the complete set of metabolites of an organism).   • Robotic equipment is available to automate sample processing for these technologies, and relies on vendor- and user-provided workflows of steps having preconditions and postconditions.  55 Continuants, Occurrents and their Dependencies: Automatic Experiments (2 of 2) • Users frequently turn to software which allows custom combination of building blocks for data analysis and visualization.  – These pipelines allow scientific investigators with biological questions but minimal computational expertise to extract meaningful information from these large data sets. – Additionally, output information from these experiments can be electronically annotated with links to ontologies containing biological processes, cellular components and molecular functions to provide further biological insight.  – Linking to standardized ontologies allows investigators to summarize large, complex quantities of information with simpler, well-documented terminology.  56  Automatic Genomic Experiment Instance Input • Robot • Annotator • Biospecimens • Standardized Ontologies • Experiment Instructions  Output Annotated Findings Audit Trails 57 Questions? 

The Paradata Information Model

https://kuscholarworks.ku.edu/bitstream/handle/1808/11065/Greenfield_paradata_NADDI2013.pdf?sequence=2&isAllowed=y

The Paradata Information Model

Abstract

Similar works

Full text

Available Versions

KU ScholarWorks