Notches on the dial: a call to action to develop plain language communication with the public about users and uses of health data

Population data science [1] researchers are not alone in recognizing the value of health and health-related data. In the era of big data, and with advent of machine learning and other artificial intelligence methods, organizations around the world are actively working to turn data into knowledge, and, in some cases, profit. The media and members of the public have taken notice, with high profile news stories about data breaches and privacy concerns [2-4] alongside some stories that call for increased use of data [5,6]. In response, public and private sector data-holding organizations and jurisdictions are turning their attention to policies, processes and regulations intended to ensure that personal data are used in ways that that the public supports. In some cases, these efforts include involving “publics” in decisions about data, such as using patient and lay person advice and other inputs to help shape policies [7-10]

Approaches to Capacity Building for Machine Learning and Artificial Intelligence Applications in Health

Many health systems and research institutes are interested in supplementing their traditional analyses of linked data with machine learning (ML) and other artificial intelligence (AI) methods and tools. However, the availability of individuals who have the required skills to develop and/or implement ML/AI is a constraint, as there is high demand for ML/AI talent in many sectors. The three organizations presenting are all actively involved in training and capacity building for ML/AI broadly, and each has a focus on, and/or discrete initiatives for, particular trainees. P. Alison Paprica, Vector Institute for artificial intelligence, Institute for Clinical Evaluative Sciences, University of Toronto, Canada. Alison is VP, Health Strategy and Partnerships at Vector, responsible for health strategy and also playing a lead role in “1000AIMs” – a Vector-led initiative in support of the Province of Ontario’s \$30 million investment to increase the number of AI-related master’s program graduates to 1,000 per year within five years. Frank Sullivan, University of St Andrews Scotland. Frank is a family physician and an associate director of HDRUK@Scotland. Health Data Research UK \url{https://hdruk.ac.uk/} has recently provided funding to six sites across the UK to address challenging healthcare issues through use of data science. A 50 PhD student Doctoral Training Scheme in AI has also been announced. Each site works in close partnership with National Health Service bodies and the public to translate research findings into benefits for patients and populations. Yin Aphinyanaphongs – INTREPID NYU clinical training program for incoming clinical fellows. Yin is the Director of the Clinical Informatics Training Program at NYU Langone Health. He is deeply interested in the intersection of computer science and health care and as a physician and a scientist, he has a unique perspective on how to train medical professionals for a data drive world. One version of this teaching process is demonstrated in the INTREPID clinical training program. Yin teaches clinicians to work with large scale data within the R environment and generate hypothesis and insights. The session will begin with three brief presentations followed by a facilitated session where all participants share their insights about the essential skills and competencies required for different kinds of ML/AI application and contributions. Live polling and voting will be used at the end of the session to capture participants’ view on the key learnings and take away points. The intended outputs and outcomes of the session are: • Participants will have a better understanding of the skills and competencies required for individuals to contribute to AI applications in health in various ways • Participants will gain knowledge about different options for capacity building from targeted enhancement of the skills of clinical fellows, to producing large number of applied master’s graduates, to doctoral-level training After the session, the co-leads will work together to create a resource that summarizes the learnings from the session and make them public (though publication in a peer-reviewed journal and/or through the IPDLN website

The Ontario Data Safe Haven: Bringing High Performance Computing to Population-wide Data Assets

Introduction Canadian provincial health systems have a data advantage – longitudinal population-wide data for publicly funded health services, in many cases going back 20 years or more. With the addition of high performance computing (HPC), these data can serve as the foundation for leading-edge research using machine learning and artificial intelligence. Objectives and Approach The Institute for Clinical Evaluative Sciences (ICES) and HPC4Health are creating the Ontario Data Safe Haven (ODSH) – a secure HPC cloud located within the HPC4Health physical environment at the Hospital for Sick Children in Toronto. The ODSH will allow research teams to post, access and analyze individual datasets over which they have authority, and enable linkage to Ontario administrative and other data. To start, the ODSH is focused on creating a private cloud meeting ICES’ legislated privacy and security requirements to support HPC-intensive analyses of ICES data. The first ODSH projects are partnerships between ICES scientists and machine learning. Results As of March 2018, the technological build of the ODSH was tested and completed and the privacy and security policy framework and documentation were completed. We will present the structure of the ODSH, including the architectural choices made when designing the environment, and planned functionality in the future. We will describe the experience to-date for the very first analysis done using the ODSH: the automatic mining of clinical terminology in primary care electronic medical records using deep neural networks. We will also present the plans for a high-cost user Risk Dashboard program of research, co-designed by ICES scientists and health faculty from the Vector Institute for artificial intelligence, that will make use of the ODSH beginning May 2018. Conclusion/Implications Through a partnership of ICES, HPC4Health and the Vector Institute, a secure private cloud ODSH has been created as is starting to be used in leading edge machine learning research studies that make use of Ontario’s population-wide data assets

Integrating population-wide laboratory testing data with physician audit-and-feedback reports to improve glycemic and cholesterol control among Ontarians with diabetes

Introduction Improving the care and management of patients with diabetes, particularly those with extreme blood glucose and/or cholesterol levels, has been identified as a key priority area for healthcare in Ontario. A multi-organizational collaboration produces audit-and-feedback reports distributed to consenting primary care physicians across the province for quality improvement purposes. Objectives and Approach We examined the feasibility of linking the Ontario Laboratory Information System (OLIS), a large and nearly population-wide database of laboratory test results in Ontario, with the existing provincial audit-and-feedback reporting structure to integrate aggregated, physician-level measures of glycemic and cholesterol control among patients with diabetes. All Ontario residents alive on March 31, 2014, attached to a primary care physician, and diagnosed with diabetes for at least two years were included. These patients were linked to OLIS to extract laboratory test orders and results for glycated hemoglobin (HbA1C) and low-density lipoproteins (LDL) between April 1, 2013 and March 31, 2014. Results There were 1,108,530 diabetes patients included who were assigned to 10,085 primary care physicians. During fiscal year (FY) 2013, 70%, 64%, and 59% of diabetes patients were tested for HbA1C, LDL, and both measures, respectively. Among the 648,238 diabetes patients with at least one of each test in FY2013, 13% had a HbA1C test exceeding a threshold of 9%, 4% had a LDL test exceeding a threshold of 4 mmol/L, and 0.8% exceeded both thresholds. At the physician-level, the median (Interquartile Range) proportions of diabetes patients exceeding the testing thresholds were 12% (9%-16%) for HbA1c and 4% (2%-6%) for LDL. In a multilevel logistic regression model, there was significant between-physician variability in the proportions of diabetes patients exceeding the HbA1C (p Conclusion/Implications We developed a mechanism for integrating population-wide, clinical laboratory test results into physician audit-and-feedback reports to improve diabetes care in Ontario. Significant variation observed in the aggregated, physician-level proportions of diabetes patients testing above clinical thresholds for HbA1C and LDL highlights the importance of reporting such information to physicians

General Public Views on Uses and Users of Administrative Health Data

ABSTRACT Objectives High profile initiatives and reports highlight the potential benefits that could be realized by increasing access to health data, but do members of the general public share this view? The objective was to gain insight into the general public’s attitudes toward users and uses of administrative health data. Approach In fall 2015, four professionally-moderated focus groups with a total of 31 Ontario participants were conducted; two in Thunder Bay, two in Toronto. Participants were asked to review and comment on: general information about research based on linked administrative health data, a case study and models through which various users might use administrative health data. Results Support for research based on linked administrative health data was strongest when people agreed with the purposes for which studies were conducted. The main concerns related to the security of personal data generally (e.g., Canada Revenue Agency hacking incidents were noted) and potentially inappropriate uses of health data, particularly by the private sector (e.g., strong reservations about studies done solely or primarily with a profit motive). Participants were reassured when provided with information about the process for removing or coding identifying information from health data, and about the oversight provided by the Information and Privacy Commissioner of Ontario. However, even when fully informed of privacy and security safeguards, participants still felt that risks unavoidably increase when there are more people and organizations accessing data. Conclusions Members of general public were generally supportive of research based on linked administrative health data but with conditions, particularly when the possibility of private sector research was discussed. Notably, and citing security concerns, focus group participants preferred models that had a limited number of individuals or organizations accessing data

Institute for Clinical Evaluative Sciences (ICES) Exploratory Data & Analytic Services Private Sector Pilot Project

ABSTRACT Objectives Prior to the launch of ICES Data & Analytic Services (DAS) in March 2014, only ICES scientists and analysts could access ICES data, and data could only be accessed at physical ICES locations. The DAS infrastructure, which allows public sector researchers to work with coded record level data remotely through a secure virtual environment, together with broader trends including high profile reports that call for increased access to data and the Ontario government’s Open Data initiative, prompted ICES to launch a pilot project to explore potential DAS work with the private sector. Approach Three mandatory principles were established for all work with the private sector: (i) alignment with ICES’ mission, vision and values; (ii) transparency; (iii) private sector work must not detract from ICES’ research institute work. The pilot included: a jurisdictional scan; informal conversations with private sector organizations to determine potential services/studies of interest; extensive discussions with data partners; the selection and conduct of two pilot studies; focus groups with members of the general public and scientists; external advice on business model options; and an external evaluation of the pilot. No changes to data sharing agreements or ICES processes were required as work with the private sector and public sector are equally allowed under Ontario law. Results The two pilot studies were successfully completed. The first study “The disease burden of gout in Ontario: A real world data retrospective study” was performed by researchers at IMS Brogan (a healthcare analytic services provider) who were provided with access to coded record-level data using the DAS iDAVE environment and performed their own analyses. In the second pilot study, “The impact of adherence to biologics on healthcare resource utilization in rheumatoid arthritis”, Janssen researchers established the research question and study design, and DAS staff and scientists provided advice about data holdings, performed the analyses, and provided Janssen and three government-funded decision making bodies with results tables. Research Ethics Board approval was required for both studies, and both private sector organizations are in the process of publishing findings. Conclusions ICES was able to work with private sector organizations without compromising the three principles. Based on the evaluation of the private sector pilot, and the findings from the focus groups, ICES will begin offering limited analytic services to private sector researchers beginning June 2016 under ICES’ existing corporate structure, and bring recommendations regarding ongoing operations to the ICES Board in June 2017

Answering questions posed by health system stakeholders using linked administrative health data at the Institute for Clinical Evaluative Sciences (ICES)

ABSTRACT Objectives There is a growing need to broaden access to administrative health data in order to support decision making and planning by health system stakeholders. An initiative funded by the Ontario Ministry of Health and Long-Term Care, the Applied Health Research Question (AHRQ) portfolio leverages the linked administrative health data holdings and the scientific and clinical expertise at ICES to answer questions generated by stakeholders that will have a direct impact on health care policy, planning or practice. Approach Eligible requesters include government ministries, health care providers and planners. Requests detail the purpose of the research question, the related scientific literature, and the planned use and intended impact of the research findings. An internal review team meets monthly to adjudicate; requests demonstrably needing research findings rapidly are adjudicated on an ad hoc basis. Eligible requests are those that aim to inform evidence-based decision making, do not advocate for a particular answer and are feasible in terms of data availability. All projects are reviewed by the internal privacy office to ensure that use of the administrative health data is in accordance with both data sharing agreements and legislation governing use of personal health information. At no cost to the requesting organization, ICES scientists and research staff formulate the analysis plan, conduct the analysis and prepare the research product (data tables, a slide deck and/or a written report); and, may opt to publish noteworthy findings. All research products must be cleared for risk of re-identification prior to being shared externally. Results Requests have steadily increased from 43 submissions in fiscal year 2012/13, to 59 in 2014/15 and 74 to date in 2015/16. In fiscal year 2014/15, provincial government and government agencies were the most frequent requesters (39%), followed by hospitals and other health care providers (19%), disease advocacy groups (12%) and professional associations (10%). Requests include assessment of health care utilization; health system performance and evaluation; and chronic disease prevalence and treatment. Time to complete reports varies from 5 days to 24 months, depending on project complexity and requirements. Requesters report that AHRQ research findings have influenced decision-making, policy development and health care practice; and have inspired future research. Conclusion This initiative demonstrates the value and feasibility of using the linked administrative health data to answer questions to meet the unique needs of health planners and policymakers, and presents an opportunity for collaboration beyond the academic research community