3,291 research outputs found
Deep Learning for Automated Experimentation in Scanning Transmission Electron Microscopy
Machine learning (ML) has become critical for post-acquisition data analysis
in (scanning) transmission electron microscopy, (S)TEM, imaging and
spectroscopy. An emerging trend is the transition to real-time analysis and
closed-loop microscope operation. The effective use of ML in electron
microscopy now requires the development of strategies for microscopy-centered
experiment workflow design and optimization. Here, we discuss the associated
challenges with the transition to active ML, including sequential data analysis
and out-of-distribution drift effects, the requirements for the edge operation,
local and cloud data storage, and theory in the loop operations. Specifically,
we discuss the relative contributions of human scientists and ML agents in the
ideation, orchestration, and execution of experimental workflows and the need
to develop universal hyper languages that can apply across multiple platforms.
These considerations will collectively inform the operationalization of ML in
next-generation experimentation.Comment: Review Articl
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Much of the previous work towards digital agents for graphical user
interfaces (GUIs) has relied on text-based representations (derived from HTML
or other structured data sources), which are not always readily available.
These input representations have been often coupled with custom, task-specific
action spaces. This paper focuses on creating agents that interact with the
digital world using the same conceptual interface that humans commonly use --
via pixel-based screenshots and a generic action space corresponding to
keyboard and mouse actions. Building upon recent progress in pixel-based
pretraining, we show, for the first time, that it is possible for such agents
to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based
instruction following tasks
GriddlyJS: A Web IDE for Reinforcement Learning
Progress in reinforcement learning (RL) research is often driven by the design of new, challenging environments-a costly undertaking requiring skills orthogonal to that of a typical machine learning researcher. The complexity of environment development has only increased with the rise of procedural-content generation (PCG) as the prevailing paradigm for producing varied environments capable of testing the robustness and generalization of RL agents. Moreover, existing environments often require complex build processes, making reproducing results difficult. To address these issues, we introduce GriddlyJS, a web-based Integrated Development Environment (IDE) based on the Griddly engine. GriddlyJS allows researchers to visually design and debug arbitrary, complex PCG grid-world environments using a convenient graphical interface, as well as visualize, evaluate, and record the performance of trained agent models. By connecting the RL workflow to the advanced functionality enabled by modern web standards, GriddlyJS allows publishing interactive agent-environment demos that reproduce experimental results directly to the web. To demonstrate the versatility of GriddlyJS, we use it to quickly develop a complex compositional puzzle-solving environment alongside arbitrary human-designed environment configurations and their solutions for use in automatic curriculum learning and offline RL. The GriddlyJS IDE is open source and freely available at https://griddly.ai
- …