16,695 research outputs found
Deploying Jupyter Notebooks at scale on XSEDE resources for Science Gateways and workshops
Jupyter Notebooks have become a mainstream tool for interactive computing in
every field of science. Jupyter Notebooks are suitable as companion
applications for Science Gateways, providing more flexibility and
post-processing capability to the users. Moreover they are often used in
training events and workshops to provide immediate access to a pre-configured
interactive computing environment. The Jupyter team released the JupyterHub web
application to provide a platform where multiple users can login and access a
Jupyter Notebook environment. When the number of users and memory requirements
are low, it is easy to setup JupyterHub on a single server. However, setup
becomes more complicated when we need to serve Jupyter Notebooks at scale to
tens or hundreds of users. In this paper we will present three strategies for
deploying JupyterHub at scale on XSEDE resources. All options share the
deployment of JupyterHub on a Virtual Machine on XSEDE Jetstream. In the first
scenario, JupyterHub connects to a supercomputer and launches a single node job
on behalf of each user and proxies back the Notebook from the computing node
back to the user's browser. In the second scenario, implemented in the context
of a XSEDE consultation for the IRIS consortium for Seismology, we deploy
Docker in Swarm mode to coordinate many XSEDE Jetstream virtual machines to
provide Notebooks with persistent storage and quota. In the last scenario we
install the Kubernetes containers orchestration framework on Jetstream to
provide a fault-tolerant JupyterHub deployment with a distributed filesystem
and capability to scale to thousands of users. In the conclusion section we
provide a link to step-by-step tutorials complete with all the necessary
commands and configuration files to replicate these deployments.Comment: 7 pages, 3 figures, PEARC '18: Practice and Experience in Advanced
Research Computing, July 22--26, 2018, Pittsburgh, PA, US
Ten Simple Rules for Reproducible Research in Jupyter Notebooks
Reproducibility of computational studies is a hallmark of scientific
methodology. It enables researchers to build with confidence on the methods and
findings of others, reuse and extend computational pipelines, and thereby drive
scientific progress. Since many experimental studies rely on computational
analyses, biologists need guidance on how to set up and document reproducible
data analyses or simulations.
In this paper, we address several questions about reproducibility. For
example, what are the technical and non-technical barriers to reproducible
computational studies? What opportunities and challenges do computational
notebooks offer to overcome some of these barriers? What tools are available
and how can they be used effectively?
We have developed a set of rules to serve as a guide to scientists with a
specific focus on computational notebook systems, such as Jupyter Notebooks,
which have become a tool of choice for many applications. Notebooks combine
detailed workflows with narrative text and visualization of results. Combined
with software repositories and open source licensing, notebooks are powerful
tools for transparent, collaborative, reproducible, and reusable data analyses
Enabling Interactive Analytics of Secure Data using Cloud Kotta
Research, especially in the social sciences and humanities, is increasingly
reliant on the application of data science methods to analyze large amounts of
(often private) data. Secure data enclaves provide a solution for managing and
analyzing private data. However, such enclaves do not readily support discovery
science---a form of exploratory or interactive analysis by which researchers
execute a range of (sometimes large) analyses in an iterative and collaborative
manner. The batch computing model offered by many data enclaves is well suited
to executing large compute tasks; however it is far from ideal for day-to-day
discovery science. As researchers must submit jobs to queues and wait for
results, the high latencies inherent in queue-based, batch computing systems
hinder interactive analysis. In this paper we describe how we have augmented
the Cloud Kotta secure data enclave to support collaborative and interactive
analysis of sensitive data. Our model uses Jupyter notebooks as a flexible
analysis environment and Python language constructs to support the execution of
arbitrary functions on private data within this secure framework.Comment: To appear in Proceedings of Workshop on Scientific Cloud Computing,
Washington, DC USA, June 2017 (ScienceCloud 2017), 7 page
Teaching photonic integrated circuits with Jupyter notebooks : design, simulation, fabrication
At Ghent University, we have built a course curriculum on integrated photonics, and in particular silicon photonics, based on interactive Jupyter Notebooks. This has been used in short workshops, specialization courses at PhD level, as well as the M.Sc. Photonics Engineering program at Ghent University and the Free University of Brussels. The course material teaches the concepts of on-chip waveguides, basic building blocks, circuits, the design process, fabrication and measurements. The Jupyter notebook environment provides an interface where static didactic content (text, figures, movies, formulas) is mixed with Python code that the user can modify and execute, and interactive plots and widgets to explore the effect of changes in circuits or components. The Python environment supplies a host of scientific and engineering libraries, while the photonic capabilities are based on IPKISS, a commercial design framework for photonic integrated circuits by Luceda Photonics. The IPKISS framework allows scripting of layout and simulation directly from the Jupyter notebooks, so the teaching modules contain live circuit simulation, as well as integration with electromagnetic solvers. Because this is a complete design framework, students can also use it to tape out a small chip design which is fabricated through a rapid prototyping service and then measured, allowing the students to validate the actual performance of their design against the original simulation. The scripting in Jupyter notebooks also provides a self-documenting design flow, and the use of an established design tool guarantees that the acquired skills can be transferred to larger, real-world design projects
Teaching Data Science
We describe an introductory data science course, entitled Introduction to
Data Science, offered at the University of Illinois at Urbana-Champaign. The
course introduced general programming concepts by using the Python programming
language with an emphasis on data preparation, processing, and presentation.
The course had no prerequisites, and students were not expected to have any
programming experience. This introductory course was designed to cover a wide
range of topics, from the nature of data, to storage, to visualization, to
probability and statistical analysis, to cloud and high performance computing,
without becoming overly focused on any one subject. We conclude this article
with a discussion of lessons learned and our plans to develop new data science
courses.Comment: 10 pages, 4 figures, International Conference on Computational
Science (ICCS 2016
- …