Bourov, S.

Garcia, A.O.

Hammad, A.

Jejkal, T.

Neumair, B.

Otte, J.C.

Pfeiffer, S.

Schenker, T.

Schmidt, C.

Streit, A.

van Wezel, J.

Garcia, A. O.

Otte, J. C.

Wezel, J. van

KITopen

KARLSRUHE INSTITUTE OF TECHNOLOGYData Management and Analysisat the Large Scale Data FacilityA. García1 , S. Bourov1 , A. Hammad1 , T. Jejkal2 , J. Otte3 , S. Pfeiffer4 , T. Schenker4 , C. Schmidt4 , J. van Wezel1 , B. Neumair1 , A. Streit11 Steinbuch Centre for Computing, KIT2 Institute for Data Processing and Electronics, KIT3 Institute of Toxicology and Genetics, KIT4 Institute for Applied Computer Science, KIT ICDIM 2011, Melbourne | September 27, 2011KIT – University of the State of Baden-Wuerttemberg andNational Laboratory of the Helmholtz Associationwww.kit.eduExecutive summaryMany experiments have a data-management problem!This project aims at improving the situation, with adapted infrastructureand servicesData Intensive Computing workflows are critical for the value of the dataWe present a concrete data intensive analysis use case at the LSDF2 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhat is the data challenge?Experiments getting exponentially more data⇒ experiments need storage⇒ need sophisticated data analysis workflowsScience produces data!Old data is very valuableto analyse change in timefor reprocessingfor analysis by other scientists, in other contextsInvisible (not-found, no-metadata) data is lost data⇒ adequate meta data greatly increases data value⇒ single big scientific DB is more valuable thanmany small onesData is used by large virtual communities!⇒ access to data is critical3 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhat is the data challenge?Experiments getting exponentially more data⇒ experiments need storage⇒ need sophisticated data analysis workflowsScience produces data!Old data is very valuableto analyse change in timefor reprocessingfor analysis by other scientists, in other contextsInvisible (not-found, no-metadata) data is lost data⇒ adequate meta data greatly increases data value⇒ single big scientific DB is more valuable thanmany small onesData is used by large virtual communities!⇒ access to data is critical3 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhat is the data challenge?Experiments getting exponentially more data⇒ experiments need storage⇒ need sophisticated data analysis workflowsScience produces data!Old data is very valuableto analyse change in timefor reprocessingfor analysis by other scientists, in other contextsInvisible (not-found, no-metadata) data is lost data⇒ adequate meta data greatly increases data value⇒ single big scientific DB is more valuable thanmany small onesData is used by large virtual communities!⇒ access to data is critical3 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhat is the data challenge?Experiments getting exponentially more data⇒ experiments need storage⇒ need sophisticated data analysis workflowsScience produces data!Old data is very valuableto analyse change in timefor reprocessingfor analysis by other scientists, in other contextsInvisible (not-found, no-metadata) data is lost data⇒ adequate meta data greatly increases data value⇒ single big scientific DB is more valuable thanmany small onesData is used by large virtual communities!⇒ access to data is critical3 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhy do experiments produce so much data?Zebrafish embryo, raw picture, 4MB (24 per fish)4 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhy do experiments produce so much data?Institute of Toxicology and Genetics @ KIT⇒ Zebra fishes’ embryonal development reconstruction⇒ Toxicological studies of drugsHigh Throughput Microscopyfully automated microscopesrobot moves object to microscopecan potentially run 24*7produce high resolution images (4 MB each)over varying parameters (focus point, cell markers, ...)≈200k images per day,≈ 2 TB/dayEstimated: 1+ PB/year in 2012,6 PB/year in 2014Raw data must be heavily analysed5 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhy do experiments produce so much data?Institute of Toxicology and Genetics @ KIT⇒ Zebra fishes’ embryonal development reconstruction⇒ Toxicological studies of drugsHigh Throughput Microscopyfully automated microscopesrobot moves object to microscopecan potentially run 24*7produce high resolution images (4 MB each)over varying parameters (focus point, cell markers, ...)≈200k images per day,≈ 2 TB/dayEstimated: 1+ PB/year in 2012,6 PB/year in 2014Raw data must be heavily analysed5 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookThe Large Scale Data Facility Projectinvolving several institutesin cooperation with BioQuant of Univ. HeidelbergLSDF started end of 2009 at KITto address the needs of Data Intensive Science:providing large scale storage and data processingopen protocols and APIs for access to data and metadataadded value services for community specific needs (multi-disciplinary)6 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookThe Large Scale Data Facility Projectinvolving several institutesin cooperation with BioQuant of Univ. HeidelbergLSDF started end of 2009 at KITto address the needs of Data Intensive Science:providing large scale storage and data processingopen protocols and APIs for access to data and metadataadded value services for community specific needs (multi-disciplinary)6 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhat infrastructure are we talking about?Currently 2 PB in 2 storage systemsTape for archive and backupDedicated 10 GE network backboneDirect network connection to some institutes7 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookThe current architecture8 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookHow to deal with so much data?Metadata is essentialNeeds to be stored and kept up to date with dataMetadata schema is highly project-dependent⇒ use a project metadata DB Example:9 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookHow to deal with so much data?Metadata is essentialNeeds to be stored and kept up to date with dataMetadata schema is highly project-dependent⇒ use a project metadata DB Example:ExperimentDATA- write once, read many- persistentBASIC METADATA- write once, read many- persistentMETADATA 1- basic metadata- processing A metadata- results A METADATA 2- basic metadata- processing B1 metadata- results B1- processing B2 metadata- results B2METADATA N- basic metadata- processing N metadata- results N9 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookThe current architecture10 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhich access APIs and tools?Different components accessible through different protocols and APIs⇒ need a unified access layerAbstract Data Access Layer, low-level interface to LSDF⇒ extensible to support new backends, authentication mechanismsFor end-users: DataBrowsergraphical tool for exploring and managing the LSDF dataalso library for meta-data accessbased on ADALAPIconnects to the meta-data repositorywill be available as web GUI11 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookR. Stotzka et. al., Perspective of the Large Scale Data Facilitysupporting nuclear fusion applications, Proceedings PDP 2011.Which access APIs and tools?Different components accessible through different protocols and APIs⇒ need a unified access layerAbstract Data Access Layer, low-level interface to LSDF⇒ extensible to support new backends, authentication mechanismsFor end-users: DataBrowsergraphical tool for exploring and managing the LSDF dataalso library for meta-data accessbased on ADALAPIconnects to the meta-data repositorywill be available as web GUI11 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookR. Stotzka et. al., Perspective of the Large Scale Data Facilitysupporting nuclear fusion applications, Proceedings PDP 2011.The current architecture12 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookCan we process the data?Data has to be processed!Exascale⇒ bring computing to the data!!(15 days to transfer 1 PB over ideal 10Gb/s link)⇒ dedicated 60 nodes clusterHadoop environment+ 110 TB Hadoop filesystemextreme scalabilityCloud environment OpenNebulausers can deploy own dedicateddata-processing VMsreliable, highly flexible,and very fast to deploy13 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookCan we process the data?Data has to be processed!Exascale⇒ bring computing to the data!!(15 days to transfer 1 PB over ideal 10Gb/s link)⇒ dedicated 60 nodes clusterHadoop environment+ 110 TB Hadoop filesystemextreme scalabilityCloud environment OpenNebulausers can deploy own dedicateddata-processing VMsreliable, highly flexible,and very fast to deploy13 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookCan we process the data?Data has to be processed!Exascale⇒ bring computing to the data!!(15 days to transfer 1 PB over ideal 10Gb/s link)⇒ dedicated 60 nodes clusterHadoop environment+ 110 TB Hadoop filesystemextreme scalabilityCloud environment OpenNebulausers can deploy own dedicateddata-processing VMsreliable, highly flexible,and very fast to deploy13 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookHow to deal with data?With Cloud instances, if customized SW environment is requiredIntegrated with the Kepler workflow orquestratoruser-friendly interfaceWith workflows in the Hadoop cluster14 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookHow to deal with data?With Cloud instances, if customized SW environment is requiredIntegrated with the Kepler workflow orquestratoruser-friendly interfaceWith workflows in the Hadoop cluster14 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookHow to deal with data?With Cloud instances, if customized SW environment is requiredIntegrated with the Kepler workflow orquestratoruser-friendly interfaceWith workflows in the Hadoop cluster14 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookProcessing data at LSDFExperiments should be able to process data locally⇒ help the users automate the workflowsAllow tagging data and triggeringexecution via DataBrowserData from finished workflows storedand tagged in DBused for zebrafish microscopy data15 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookT. Jejkal et al., LAMBDA – The LSDF Execution Framework forData Intensive Applications, Proceedings PDP 2012.Processing data at LSDFExperiments should be able to process data locally⇒ help the users automate the workflowsAllow tagging data and triggeringexecution via DataBrowserData from finished workflows storedand tagged in DBused for zebrafish microscopy data15 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookT. Jejkal et al., LAMBDA – The LSDF Execution Framework forData Intensive Applications, Proceedings PDP 2012.Microscopy data analysisZebra-fish data setsData imported with DataBrowser, meta-data addedImagingProjectGUI to define image analysis chainbased on operators from different libraries (VTK, ITK, Matlab,...)creates sequence-XML, XML description of operators’ chainUser tags data for processingusing DataBrowserassociating sequence-XMLCron job starts execution based on tagsprojectProcessorExecuted in Hadoop clusterresult-XML storedSpeedup of 60-120 depending on operatorsI/O bound analysis16 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookS. Pfeiffer, M. Mai, and J. Calliess, On the Computational Benefit of Tensor Separation forHigh-Dimensional Discrete Convolutions, Multidimensional Systems and Signal Processing 31, 2010, pp. 1-25.J.C. Otte, S. Pfeiffer, T. Schenker, and C. Schmidt, Concept of multidimensional image analysis withflexible operators and cluster-computing, 6th European Molecular Imaging Meeting, Leiden, The Netherlands, 2011.Microscopy data analysisZebra-fish data setsData imported with DataBrowser, meta-data addedImagingProjectGUI to define image analysis chainbased on operators from different libraries (VTK, ITK, Matlab,...)creates sequence-XML, XML description of operators’ chainUser tags data for processingusing DataBrowserassociating sequence-XMLCron job starts execution based on tagsprojectProcessorExecuted in Hadoop clusterresult-XML storedSpeedup of 60-120 depending on operatorsI/O bound analysis16 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookS. Pfeiffer, M. Mai, and J. Calliess, On the Computational Benefit of Tensor Separation forHigh-Dimensional Discrete Convolutions, Multidimensional Systems and Signal Processing 31, 2010, pp. 1-25.J.C. Otte, S. Pfeiffer, T. Schenker, and C. Schmidt, Concept of multidimensional image analysis withflexible operators and cluster-computing, 6th European Molecular Imaging Meeting, Leiden, The Netherlands, 2011.What’s ahead?Improved storage, network capacity: 6 PB in 2012Data management system iRODSObject StorageInvestigate and deploy new technologiesAdditional communities being integratedMeteorology and climate research (“archival” quality)Geophysics, seismology dataKATRIN experiment, neutrino massDARIAH, arts and humanitiesto offer more community tailored supportProject expansion17 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhat’s ahead?Improved storage, network capacity: 6 PB in 2012Data management system iRODSObject StorageInvestigate and deploy new technologiesAdditional communities being integratedMeteorology and climate research (“archival” quality)Geophysics, seismology dataKATRIN experiment, neutrino massDARIAH, arts and humanitiesto offer more community tailored supportProject expansion17 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookWhat’s ahead?Improved storage, network capacity: 6 PB in 2012Data management system iRODSObject StorageInvestigate and deploy new technologiesAdditional communities being integratedMeteorology and climate research (“archival” quality)Geophysics, seismology dataKATRIN experiment, neutrino massDARIAH, arts and humanitiesto offer more community tailored supportProject expansion17 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookConcluding remarksInfrastructure, 2 PB and storage services up and runningFirst software tools availableExperimental data being stored and processedMany scientific communities interested and getting involvedCan’t just “store files”Training for new tools, data management workflowsFocus on users, added value servicesSame problem at most (all?) research institutions⇒ Open for new partnerships, international collaborations18 ICDIM 2011 Ariel García – LSDFIntroduction The facility Software DIC OutlookThanks for listening!Questions?19 ICDIM 2011 Ariel García – LSDF

Data management and analysis at the large scale data facility

A. O. Garcia

S. Bourov

A. Hammad

T. Jejkal

J.C. Otte

S. Pfeiffer

T. Schenker

C. Schmidt

J. van Wezel

B. Neumair

A. Streit

Crossref

Data management and analysis at the Large Scale Data Facility

https://publikationen.bibliothek.kit.edu/240082518/3815825

Data management and analysis at the large scale data facility

Abstract

Similar works

Full text

Available Versions

KITopen

Crossref