71 research outputs found
FP-IRL: Fokker-Planck-based Inverse Reinforcement Learning -- A Physics-Constrained Approach to Markov Decision Processes
Inverse Reinforcement Learning (IRL) is a compelling technique for revealing
the rationale underlying the behavior of autonomous agents. IRL seeks to
estimate the unknown reward function of a Markov decision process (MDP) from
observed agent trajectories. However, IRL needs a transition function, and most
algorithms assume it is known or can be estimated in advance from data. It
therefore becomes even more challenging when such transition dynamics is not
known a-priori, since it enters the estimation of the policy in addition to
determining the system's evolution. When the dynamics of these agents in the
state-action space is described by stochastic differential equations (SDE) in
It^{o} calculus, these transitions can be inferred from the mean-field theory
described by the Fokker-Planck (FP) equation. We conjecture there exists an
isomorphism between the time-discrete FP and MDP that extends beyond the
minimization of free energy (in FP) and maximization of the reward (in MDP). We
identify specific manifestations of this isomorphism and use them to create a
novel physics-aware IRL algorithm, FP-IRL, which can simultaneously infer the
transition and reward functions using only observed trajectories. We employ
variational system identification to infer the potential function in FP, which
consequently allows the evaluation of reward, transition, and policy by
leveraging the conjecture. We demonstrate the effectiveness of FP-IRL by
applying it to a synthetic benchmark and a biological problem of cancer cell
dynamics, where the transition function is inaccessible
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts
For years, researchers have been devoted to generalizable object perception
and manipulation, where cross-category generalizability is highly desired yet
underexplored. In this work, we propose to learn such cross-category skills via
Generalizable and Actionable Parts (GAParts). By identifying and defining 9
GAPart classes (lids, handles, etc.) in 27 object categories, we construct a
large-scale part-centric interactive dataset, GAPartNet, where we provide rich,
part-level annotations (semantics, poses) for 8,489 part instances on 1,166
objects. Based on GAPartNet, we investigate three cross-category tasks: part
segmentation, part pose estimation, and part-based object manipulation. Given
the significant domain gaps between seen and unseen object categories, we
propose a robust 3D segmentation method from the perspective of domain
generalization by integrating adversarial learning techniques. Our method
outperforms all existing methods by a large margin, no matter on seen or unseen
categories. Furthermore, with part segmentation and pose estimation results, we
leverage the GAPart pose definition to design part-based manipulation
heuristics that can generalize well to unseen object categories in both the
simulator and the real world. Our dataset, code, and demos are available on our
project page.Comment: To appear in CVPR 2023 (Highlight
Recommended from our members
PHF8 and REST/NRSF co-occupy gene promoters to regulate proximal gene expression
Chromatin regulators play an important role in the development of human diseases. In this study, we focused on Plant Homeo Domain Finger protein 8 (PHF8), a chromatin regulator that has attracted special concern recently. PHF8 is a histone lysine demethylase ubiquitously expressed in nuclei. Mutations of PHF8 are associated with X-linked mental retardation. It usually functions as a transcriptional co-activator by associating with H3K4me3 and RNA polymerase II. We found that PHF8 may associate with another regulator, REST/NRSF, predominately at promoter regions via studying several published PHF8 chromatin immunoprecipitation-sequencing (ChIP-Seq) datasets. Our analysis suggested that PHF8 not only activates but may also repress gene expression
Integrating inverse reinforcement learning into data-driven mechanistic computational models: a novel paradigm to decode cancer cell heterogeneity
Cellular heterogeneity is a ubiquitous aspect of biology and a major obstacle to successful cancer treatment. Several techniques have emerged to quantify heterogeneity in live cells along axes including cellular migration, morphology, growth, and signaling. Crucially, these studies reveal that cellular heterogeneity is not a result of randomness or a failure in cellular control systems, but instead is a predictable aspect of multicellular systems. We hypothesize that individual cells in complex tissues can behave as reward-maximizing agents and that differences in reward perception can explain heterogeneity. In this perspective, we introduce inverse reinforcement learning as a novel approach for analyzing cellular heterogeneity. We briefly detail experimental approaches for measuring cellular heterogeneity over time and how these experiments can generate datasets consisting of cellular states and actions. Next, we show how inverse reinforcement learning can be applied to these datasets to infer how individual cells choose different actions based on heterogeneous states. Finally, we introduce potential applications of inverse reinforcement learning to three cell biology problems. Overall, we expect inverse reinforcement learning to reveal why cells behave heterogeneously and enable identification of novel treatments based on this new understanding
Arbetsflöden som kan efterfrÄgas: Utökning av dataflödesströmning med dynamisk begÀran/Äterkopplingskommunikation
Stream processing systems have been widely adopted in applications such as recommendation systems, anomaly detection, and system monitoring due to their real-time capabilities. Improving observability in stream processing systems can further expand their application scenarios, including the implementation of stateful serverless applications. Stateful serverless applications are an emerging model in serverless computing that focuses on addressing the challenges of state management, enabling developers to build distributed applications in a simpler way. One possible implementation of stateful serverless applications is based on stream processing engines. However, the current approaches for observability in stream processing engines suffer from issues such as efficiency, consistency, and functionality, resulting in limited practical use cases. To address these challenges, we propose Queryable Workflow, an extension to stream processing engines. This extension allows users to access or modify the state within stream processing engines with transactional semantics using a SQL interface, enabling use cases such as ad-hoc querying, serializable updates, or even stateful serverless applications. We implemented our system on stream processing engines such as Portals and Apache Flink, and evaluated their performance. The result showed that our system has achieved 4.33x throughput improvement and 30% latency reduction compared to a baseline implemented with Apache Flink and Apache Kafka. With hand-crafted optimizations, our system achieved to process over 29,000 queries per second with a 99th percentile latency of 8.58 ms under a single-threaded runtime. Our proposed system provides a viable option for implementing stateful serverless applications that require transactional guarantees, while also expanding the potential application scenarios for stream processing engines.Strömbehandlingssystem har pÄ grund av sina realtidsegenskaper fÄtt stor spridning i tillÀmpningar som rekommendationssystem, anomalidetektering och systemövervakning. FörbÀttrad observerbarhet i stream processing-system kan ytterligare utöka deras tillÀmpningsscenarier, inklusive implementeringen av stateful serverless-applikationer. Stateful serverless-applikationer Àr en framvÀxande modell inom serverless computing som fokuserar pÄ att hantera utmaningarna med tillstÄndshantering, vilket gör det möjligt för utvecklare att bygga distribuerade applikationer pÄ ett enklare sÀtt. En möjlig implementering av stateful serverless-applikationer Àr baserad pÄ stream processing-motorer. De nuvarande metoderna för observerbarhet i strömbehandlingsmotorer lider dock av problem som effektivitet, konsistens och funktionalitet, vilket resulterar i begrÀnsade praktiska anvÀndningsfall. För att ta itu med dessa utmaningar föreslog vi Queryable Workflow, ett tillÀgg till stream processing-motorer. Med detta tillÀgg kan anvÀndare komma Ät eller Àndra tillstÄndet i strömbehandlingsmotorer med transaktionssemantik med hjÀlp av ett SQL-grÀnssnitt, vilket möjliggör anvÀndningsfall som ad hoc-förfrÄgningar, serialiserbara uppdateringar eller till och med serverlösa applikationer med tillstÄnd. Vi implementerade vÄrt system pÄ stream processing-motorer som Portals och Apache Flink, och utvÀrderade deras prestanda. Resultatet visade att vÄrt system har förbÀttrat genomströmningen 4,33 gÄnger och minskat latensen med 30% jÀmfört med en baslinje som implementerats med Apache Flink och Apache Kafka. Med handgjorda optimeringar lyckades vÄrt system bearbeta över 29 000 frÄgor per sekund med en 99:e percentil latens pÄ 8,58 ms under en enkeltrÄdad körtid. VÄrt föreslagna system har gett ett hÄllbart alternativ för att implementera stateful serverless-applikationer som krÀver transaktionsgarantier, samtidigt som det ocksÄ utökat de potentiella applikationsscenarierna för stream processing-motorer
Arbetsflöden som kan efterfrÄgas: Utökning av dataflödesströmning med dynamisk begÀran/Äterkopplingskommunikation
Stream processing systems have been widely adopted in applications such as recommendation systems, anomaly detection, and system monitoring due to their real-time capabilities. Improving observability in stream processing systems can further expand their application scenarios, including the implementation of stateful serverless applications. Stateful serverless applications are an emerging model in serverless computing that focuses on addressing the challenges of state management, enabling developers to build distributed applications in a simpler way. One possible implementation of stateful serverless applications is based on stream processing engines. However, the current approaches for observability in stream processing engines suffer from issues such as efficiency, consistency, and functionality, resulting in limited practical use cases. To address these challenges, we propose Queryable Workflow, an extension to stream processing engines. This extension allows users to access or modify the state within stream processing engines with transactional semantics using a SQL interface, enabling use cases such as ad-hoc querying, serializable updates, or even stateful serverless applications. We implemented our system on stream processing engines such as Portals and Apache Flink, and evaluated their performance. The result showed that our system has achieved 4.33x throughput improvement and 30% latency reduction compared to a baseline implemented with Apache Flink and Apache Kafka. With hand-crafted optimizations, our system achieved to process over 29,000 queries per second with a 99th percentile latency of 8.58 ms under a single-threaded runtime. Our proposed system provides a viable option for implementing stateful serverless applications that require transactional guarantees, while also expanding the potential application scenarios for stream processing engines.Strömbehandlingssystem har pÄ grund av sina realtidsegenskaper fÄtt stor spridning i tillÀmpningar som rekommendationssystem, anomalidetektering och systemövervakning. FörbÀttrad observerbarhet i stream processing-system kan ytterligare utöka deras tillÀmpningsscenarier, inklusive implementeringen av stateful serverless-applikationer. Stateful serverless-applikationer Àr en framvÀxande modell inom serverless computing som fokuserar pÄ att hantera utmaningarna med tillstÄndshantering, vilket gör det möjligt för utvecklare att bygga distribuerade applikationer pÄ ett enklare sÀtt. En möjlig implementering av stateful serverless-applikationer Àr baserad pÄ stream processing-motorer. De nuvarande metoderna för observerbarhet i strömbehandlingsmotorer lider dock av problem som effektivitet, konsistens och funktionalitet, vilket resulterar i begrÀnsade praktiska anvÀndningsfall. För att ta itu med dessa utmaningar föreslog vi Queryable Workflow, ett tillÀgg till stream processing-motorer. Med detta tillÀgg kan anvÀndare komma Ät eller Àndra tillstÄndet i strömbehandlingsmotorer med transaktionssemantik med hjÀlp av ett SQL-grÀnssnitt, vilket möjliggör anvÀndningsfall som ad hoc-förfrÄgningar, serialiserbara uppdateringar eller till och med serverlösa applikationer med tillstÄnd. Vi implementerade vÄrt system pÄ stream processing-motorer som Portals och Apache Flink, och utvÀrderade deras prestanda. Resultatet visade att vÄrt system har förbÀttrat genomströmningen 4,33 gÄnger och minskat latensen med 30% jÀmfört med en baslinje som implementerats med Apache Flink och Apache Kafka. Med handgjorda optimeringar lyckades vÄrt system bearbeta över 29 000 frÄgor per sekund med en 99:e percentil latens pÄ 8,58 ms under en enkeltrÄdad körtid. VÄrt föreslagna system har gett ett hÄllbart alternativ för att implementera stateful serverless-applikationer som krÀver transaktionsgarantier, samtidigt som det ocksÄ utökat de potentiella applikationsscenarierna för stream processing-motorer
- âŠ