FP-IRL: Fokker-Planck-based Inverse Reinforcement Learning -- A Physics-Constrained Approach to Markov Decision Processes

Inverse Reinforcement Learning (IRL) is a compelling technique for revealing the rationale underlying the behavior of autonomous agents. IRL seeks to estimate the unknown reward function of a Markov decision process (MDP) from observed agent trajectories. However, IRL needs a transition function, and most algorithms assume it is known or can be estimated in advance from data. It therefore becomes even more challenging when such transition dynamics is not known a-priori, since it enters the estimation of the policy in addition to determining the system's evolution. When the dynamics of these agents in the state-action space is described by stochastic differential equations (SDE) in It^{o} calculus, these transitions can be inferred from the mean-field theory described by the Fokker-Planck (FP) equation. We conjecture there exists an isomorphism between the time-discrete FP and MDP that extends beyond the minimization of free energy (in FP) and maximization of the reward (in MDP). We identify specific manifestations of this isomorphism and use them to create a novel physics-aware IRL algorithm, FP-IRL, which can simultaneously infer the transition and reward functions using only observed trajectories. We employ variational system identification to infer the potential function in FP, which consequently allows the evaluation of reward, transition, and policy by leveraging the conjecture. We demonstrate the effectiveness of FP-IRL by applying it to a synthetic benchmark and a biological problem of cancer cell dynamics, where the transition function is inaccessible

GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts

For years, researchers have been devoted to generalizable object perception and manipulation, where cross-category generalizability is highly desired yet underexplored. In this work, we propose to learn such cross-category skills via Generalizable and Actionable Parts (GAParts). By identifying and defining 9 GAPart classes (lids, handles, etc.) in 27 object categories, we construct a large-scale part-centric interactive dataset, GAPartNet, where we provide rich, part-level annotations (semantics, poses) for 8,489 part instances on 1,166 objects. Based on GAPartNet, we investigate three cross-category tasks: part segmentation, part pose estimation, and part-based object manipulation. Given the significant domain gaps between seen and unseen object categories, we propose a robust 3D segmentation method from the perspective of domain generalization by integrating adversarial learning techniques. Our method outperforms all existing methods by a large margin, no matter on seen or unseen categories. Furthermore, with part segmentation and pose estimation results, we leverage the GAPart pose definition to design part-based manipulation heuristics that can generalize well to unseen object categories in both the simulator and the real world. Our dataset, code, and demos are available on our project page.Comment: To appear in CVPR 2023 (Highlight

PHF8 and REST/NRSF co-occupy gene promoters to regulate proximal gene expression

Chromatin regulators play an important role in the development of human diseases. In this study, we focused on Plant Homeo Domain Finger protein 8 (PHF8), a chromatin regulator that has attracted special concern recently. PHF8 is a histone lysine demethylase ubiquitously expressed in nuclei. Mutations of PHF8 are associated with X-linked mental retardation. It usually functions as a transcriptional co-activator by associating with H3K4me3 and RNA polymerase II. We found that PHF8 may associate with another regulator, REST/NRSF, predominately at promoter regions via studying several published PHF8 chromatin immunoprecipitation-sequencing (ChIP-Seq) datasets. Our analysis suggested that PHF8 not only activates but may also repress gene expression

Integrating inverse reinforcement learning into data-driven mechanistic computational models: a novel paradigm to decode cancer cell heterogeneity

Cellular heterogeneity is a ubiquitous aspect of biology and a major obstacle to successful cancer treatment. Several techniques have emerged to quantify heterogeneity in live cells along axes including cellular migration, morphology, growth, and signaling. Crucially, these studies reveal that cellular heterogeneity is not a result of randomness or a failure in cellular control systems, but instead is a predictable aspect of multicellular systems. We hypothesize that individual cells in complex tissues can behave as reward-maximizing agents and that differences in reward perception can explain heterogeneity. In this perspective, we introduce inverse reinforcement learning as a novel approach for analyzing cellular heterogeneity. We briefly detail experimental approaches for measuring cellular heterogeneity over time and how these experiments can generate datasets consisting of cellular states and actions. Next, we show how inverse reinforcement learning can be applied to these datasets to infer how individual cells choose different actions based on heterogeneous states. Finally, we introduce potential applications of inverse reinforcement learning to three cell biology problems. Overall, we expect inverse reinforcement learning to reveal why cells behave heterogeneously and enable identification of novel treatments based on this new understanding

Arbetsflöden som kan efterfrågas: Utökning av dataflödesströmning med dynamisk begäran/återkopplingskommunikation

Stream processing systems have been widely adopted in applications such as recommendation systems, anomaly detection, and system monitoring due to their real-time capabilities. Improving observability in stream processing systems can further expand their application scenarios, including the implementation of stateful serverless applications. Stateful serverless applications are an emerging model in serverless computing that focuses on addressing the challenges of state management, enabling developers to build distributed applications in a simpler way. One possible implementation of stateful serverless applications is based on stream processing engines. However, the current approaches for observability in stream processing engines suffer from issues such as efficiency, consistency, and functionality, resulting in limited practical use cases. To address these challenges, we propose Queryable Workflow, an extension to stream processing engines. This extension allows users to access or modify the state within stream processing engines with transactional semantics using a SQL interface, enabling use cases such as ad-hoc querying, serializable updates, or even stateful serverless applications. We implemented our system on stream processing engines such as Portals and Apache Flink, and evaluated their performance. The result showed that our system has achieved 4.33x throughput improvement and 30% latency reduction compared to a baseline implemented with Apache Flink and Apache Kafka. With hand-crafted optimizations, our system achieved to process over 29,000 queries per second with a 99th percentile latency of 8.58 ms under a single-threaded runtime. Our proposed system provides a viable option for implementing stateful serverless applications that require transactional guarantees, while also expanding the potential application scenarios for stream processing engines.Strömbehandlingssystem har på grund av sina realtidsegenskaper fått stor spridning i tillämpningar som rekommendationssystem, anomalidetektering och systemövervakning. Förbättrad observerbarhet i stream processing-system kan ytterligare utöka deras tillämpningsscenarier, inklusive implementeringen av stateful serverless-applikationer. Stateful serverless-applikationer är en framväxande modell inom serverless computing som fokuserar på att hantera utmaningarna med tillståndshantering, vilket gör det möjligt för utvecklare att bygga distribuerade applikationer på ett enklare sätt. En möjlig implementering av stateful serverless-applikationer är baserad på stream processing-motorer. De nuvarande metoderna för observerbarhet i strömbehandlingsmotorer lider dock av problem som effektivitet, konsistens och funktionalitet, vilket resulterar i begränsade praktiska användningsfall. För att ta itu med dessa utmaningar föreslog vi Queryable Workflow, ett tillägg till stream processing-motorer. Med detta tillägg kan användare komma åt eller ändra tillståndet i strömbehandlingsmotorer med transaktionssemantik med hjälp av ett SQL-gränssnitt, vilket möjliggör användningsfall som ad hoc-förfrågningar, serialiserbara uppdateringar eller till och med serverlösa applikationer med tillstånd. Vi implementerade vårt system på stream processing-motorer som Portals och Apache Flink, och utvärderade deras prestanda. Resultatet visade att vårt system har förbättrat genomströmningen 4,33 gånger och minskat latensen med 30% jämfört med en baslinje som implementerats med Apache Flink och Apache Kafka. Med handgjorda optimeringar lyckades vårt system bearbeta över 29 000 frågor per sekund med en 99:e percentil latens på 8,58 ms under en enkeltrådad körtid. Vårt föreslagna system har gett ett hållbart alternativ för att implementera stateful serverless-applikationer som kräver transaktionsgarantier, samtidigt som det också utökat de potentiella applikationsscenarierna för stream processing-motorer

Arbetsflöden som kan efterfrågas: Utökning av dataflödesströmning med dynamisk begäran/återkopplingskommunikation

Stream processing systems have been widely adopted in applications such as recommendation systems, anomaly detection, and system monitoring due to their real-time capabilities. Improving observability in stream processing systems can further expand their application scenarios, including the implementation of stateful serverless applications. Stateful serverless applications are an emerging model in serverless computing that focuses on addressing the challenges of state management, enabling developers to build distributed applications in a simpler way. One possible implementation of stateful serverless applications is based on stream processing engines. However, the current approaches for observability in stream processing engines suffer from issues such as efficiency, consistency, and functionality, resulting in limited practical use cases. To address these challenges, we propose Queryable Workflow, an extension to stream processing engines. This extension allows users to access or modify the state within stream processing engines with transactional semantics using a SQL interface, enabling use cases such as ad-hoc querying, serializable updates, or even stateful serverless applications. We implemented our system on stream processing engines such as Portals and Apache Flink, and evaluated their performance. The result showed that our system has achieved 4.33x throughput improvement and 30% latency reduction compared to a baseline implemented with Apache Flink and Apache Kafka. With hand-crafted optimizations, our system achieved to process over 29,000 queries per second with a 99th percentile latency of 8.58 ms under a single-threaded runtime. Our proposed system provides a viable option for implementing stateful serverless applications that require transactional guarantees, while also expanding the potential application scenarios for stream processing engines.Strömbehandlingssystem har på grund av sina realtidsegenskaper fått stor spridning i tillämpningar som rekommendationssystem, anomalidetektering och systemövervakning. Förbättrad observerbarhet i stream processing-system kan ytterligare utöka deras tillämpningsscenarier, inklusive implementeringen av stateful serverless-applikationer. Stateful serverless-applikationer är en framväxande modell inom serverless computing som fokuserar på att hantera utmaningarna med tillståndshantering, vilket gör det möjligt för utvecklare att bygga distribuerade applikationer på ett enklare sätt. En möjlig implementering av stateful serverless-applikationer är baserad på stream processing-motorer. De nuvarande metoderna för observerbarhet i strömbehandlingsmotorer lider dock av problem som effektivitet, konsistens och funktionalitet, vilket resulterar i begränsade praktiska användningsfall. För att ta itu med dessa utmaningar föreslog vi Queryable Workflow, ett tillägg till stream processing-motorer. Med detta tillägg kan användare komma åt eller ändra tillståndet i strömbehandlingsmotorer med transaktionssemantik med hjälp av ett SQL-gränssnitt, vilket möjliggör användningsfall som ad hoc-förfrågningar, serialiserbara uppdateringar eller till och med serverlösa applikationer med tillstånd. Vi implementerade vårt system på stream processing-motorer som Portals och Apache Flink, och utvärderade deras prestanda. Resultatet visade att vårt system har förbättrat genomströmningen 4,33 gånger och minskat latensen med 30% jämfört med en baslinje som implementerats med Apache Flink och Apache Kafka. Med handgjorda optimeringar lyckades vårt system bearbeta över 29 000 frågor per sekund med en 99:e percentil latens på 8,58 ms under en enkeltrådad körtid. Vårt föreslagna system har gett ett hållbart alternativ för att implementera stateful serverless-applikationer som kräver transaktionsgarantier, samtidigt som det också utökat de potentiella applikationsscenarierna för stream processing-motorer