Hartmann, V.

Jejkal, T.

Stotzka, R.

Sutter, M.

Trunov, A.

van Wezel, J.

English

Wezel, J. van

KITopen

Universität Karlsruhe ( )THResearch University founded 1825Forschungszentrum Karlsruhein der Helmholtz-GemeinschaftContact:Architecture to fulfill the requirementsof Data Intensive ApplicationsMichael SutterEmail: michael.sutter@ipe.fzk.dePhone: +49 7247 82-5676AbstractThe high throughput and high content microscopy facility at KIT will produce data in the order of PB’s per year. The data is stored and archived at the Large Scale Data Facility (LSDF) at KIT, a new storage infrastructure developed to fulfill the requirements of data intensive applications. The computational requirements for analyzing the data are covered by Grid, HPC and Cloud computing resources in combination with seamlessly integrated software environments, allowing the access to the LSDF. All components can be controlled via a simple interface: The Data Browser, allowing scientists to focus on the evaluation and processing of data within near real-time.Border conditionsCurrent data rates up to 240 MB/sData set size ~ 150 – 300 GBHandling of thousands of data setsAimsPersistent storage and archiving of data including Meta DataWorld-wide secure access Temporal and spatial analysis in 3D in near real-timeEasy-to-useEasy to extend with new algorithms and workflowsRobotic microscopesImage stacksMeta DataScientistControlsSoftware, service andworkflow repositoryData analysisImage processingPattern recognitionStatisticsData miningVisualizationetc.Analysis MachineComputing infrastructureCloud GridHPCSpecialhardwareControlsFetches data + Meta DataUses software components and web servicesProposed architectureSoftware componentsData Browser:Easy-to-use access to Grid storage elements within the LSDF Extensible GUI integrating security, data organization, visualization and access to the Analysis MachineData and meta data models designed and implemented by an operative serviceAnalysis Machine:Software service environment to run interactively, to develop and to debug data analysis workflowsIntegrates computing infrastructures, e.g. Grid and Cloud resources and hides their complexity from the userIntegrates complex software modulesProvides interfaces to scientific problem solving environmentsM. Sutter1, R. Stotzka1, V. Hartmann1, T. Jejkal1, A. Trunov2, J. van Wezel21 Forschungszentrum Karlsruhe GmbH, Institute for Data Processing and Electronics, Karlsruhe, Germany2 Forschungszentrum Karlsruhe GmbH, Steinbuch Centre for Computing, Karlsruhe, GermanyData BrowserLarge Scale Data FacilityStorage elementStorage elementStorage elementStorage elementTapeStorageMeta dataMeta dataMeta dataMeta dataMeta DataStorage elementStorage elementStorage elementStorage elementStorage ElementControls

Architecture to fulfill the requirements of data intensive applications

https://publikationen.bibliothek.kit.edu/220077188/3815557

Architecture to fulfill the requirements of data intensive applications

Abstract

Similar works

Full text

Available Versions

KITopen