Long N-step Surrogate Stage Reward to Reduce Variances of Deep Reinforcement Learning in Complex Problems

High variances in reinforcement learning have shown impeding successful convergence and hurting task performance. As reward signal plays an important role in learning behavior, multi-step methods have been considered to mitigate the problem, and are believed to be more effective than single step methods. However, there is a lack of comprehensive and systematic study on this important aspect to demonstrate the effectiveness of multi-step methods in solving highly complex continuous control problems. In this study, we introduce a new long $N$-step surrogate stage (LNSS) reward approach to effectively account for complex environment dynamics while previous methods are usually feasible for limited number of steps. The LNSS method is simple, low computational cost, and applicable to value based or policy gradient reinforcement learning. We systematically evaluate LNSS in OpenAI Gym and DeepMind Control Suite to address some complex benchmark environments that have been challenging to obtain good results by DRL in general. We demonstrate performance improvement in terms of total reward, convergence speed, and coefficient of variation (CV) by LNSS. We also provide analytical insights on how LNSS exponentially reduces the upper bound on the variances of Q value from a respective single step metho

Zero-full law for well approximable sets in generalized Cantor sets

Let $t \geq 2$ be an integer and $C(b,D)$ be a generalized Cantor set with $b \geq 3$. We study how close can numbers in $C(b,D)$ be approximated by rational numbers with denominators $t^n$. For any function $\psi : \mathbb{N} \to (0,\infty)$, let $W_{t} (\psi)$ be the set of numbers in $[0,1]$ such that $|x - p/t^n| < \psi(n)$ for infinitely many $(p,n) \in \mathbb{N}^2$. We correct an error in a result of Levesley, Salp and Velani (Math. Ann., 338:97-118, 2007) on the Hausdorff measure of $W_{t} (\psi) \cap C(b,D)$ when $b=t$, and also prove a generalization when $b$ and $t$ are multiplicatively dependent. If $b$ and $t$ are multiplicatively independent but have the same prime divisors, we obtain a partial result on the Hausdorff measure and bounds for the Hausdorff dimension, which are close to the multiplicatively dependent case. Based on these results, several conjectures are proposed

Automatic cell segmentation by adaptive thresholding (ACSAT) for large-scale calcium imaging datasets

Advances in calcium imaging have made it possible to record from an increasingly larger number of neurons simultaneously. Neuroscientists can now routinely image hundreds to thousands of individual neurons. An emerging technical challenge that parallels the advancement in imaging a large number of individual neurons is the processing of correspondingly large datasets. One important step is the identification of individual neurons. Traditional methods rely mainly on manual or semimanual inspection, which cannot be scaled for processing large datasets. To address this challenge, we focused on developing an automated segmentation method, which we refer to as automated cell segmentation by adaptive thresholding (ACSAT). ACSAT works with a time-collapsed image and includes an iterative procedure that automatically calculates global and local threshold values during successive iterations based on the distribution of image pixel intensities. Thus, the algorithm is capable of handling variations in morphological details and in fluorescence intensities in different calcium imaging datasets. In this paper, we demonstrate the utility of ACSAT by testing it on 500 simulated datasets, two wide-field hippocampus datasets, a wide-field striatum dataset, a wide-field cell culture dataset, and a two-photon hippocampus dataset. For the simulated datasets with truth, ACSAT achieved >80% recall and precision when the signal-to-noise ratio was no less than ∼24 dB.DP2 NS082126 - NINDS NIH HHSPublished versio

Scaling Up, Scaling Deep: Blockwise Graph Contrastive Learning

Oversmoothing is a common phenomenon in graph neural networks (GNNs), in which an increase in the network depth leads to a deterioration in their performance. Graph contrastive learning (GCL) is emerging as a promising way of leveraging vast unlabeled graph data. As a marriage between GNNs and contrastive learning, it remains unclear whether GCL inherits the same oversmoothing defect from GNNs. This work undertakes a fundamental analysis of GCL from the perspective of oversmoothing on the first hand. We demonstrate empirically that increasing network depth in GCL also leads to oversmoothing in their deep representations, and surprisingly, the shallow ones. We refer to this phenomenon in GCL as long-range starvation', wherein lower layers in deep networks suffer from degradation due to the lack of sufficient guidance from supervision (e.g., loss computing). Based on our findings, we present BlockGCL, a remarkably simple yet effective blockwise training framework to prevent GCL from notorious oversmoothing. Without bells and whistles, BlockGCL consistently improves robustness and stability for well-established GCL methods with increasing numbers of layers on real-world graph benchmarks. We believe our work will provide insights for future improvements of scalable and deep GCL frameworks.Comment: Preprint; Code is available at https://github.com/EdisonLeeeee/BlockGC

eMotions: A Large-Scale Dataset for Emotion Recognition in Short Videos

Nowadays, short videos (SVs) are essential to information acquisition and sharing in our life. The prevailing use of SVs to spread emotions leads to the necessity of emotion recognition in SVs. Considering the lack of SVs emotion data, we introduce a large-scale dataset named eMotions, comprising 27,996 videos. Meanwhile, we alleviate the impact of subjectivities on labeling quality by emphasizing better personnel allocations and multi-stage annotations. In addition, we provide the category-balanced and test-oriented variants through targeted data sampling. Some commonly used videos (e.g., facial expressions and postures) have been well studied. However, it is still challenging to understand the emotions in SVs. Since the enhanced content diversity brings more distinct semantic gaps and difficulties in learning emotion-related features, and there exists information gaps caused by the emotion incompleteness under the prevalently audio-visual co-expressions. To tackle these problems, we present an end-to-end baseline method AV-CPNet that employs the video transformer to better learn semantically relevant representations. We further design the two-stage cross-modal fusion module to complementarily model the correlations of audio-visual features. The EP-CE Loss, incorporating three emotion polarities, is then applied to guide model optimization. Extensive experimental results on nine datasets verify the effectiveness of AV-CPNet. Datasets and code will be open on https://github.com/XuecWu/eMotions

Privacy-preserving design of graph neural networks with applications to vertical federated learning

The paradigm of vertical federated learning (VFL), where institutions collaboratively train machine learning models via combining each other's local feature or label information, has achieved great success in applications to financial risk management (FRM). The surging developments of graph representation learning (GRL) have opened up new opportunities for FRM applications under FL via efficiently utilizing the graph-structured data generated from underlying transaction networks. Meanwhile, transaction information is often considered highly sensitive. To prevent data leakage during training, it is critical to develop FL protocols with formal privacy guarantees. In this paper, we present an end-to-end GRL framework in the VFL setting called VESPER, which is built upon a general privatization scheme termed perturbed message passing (PMP) that allows the privatization of many popular graph neural architectures.Based on PMP, we discuss the strengths and weaknesses of specific design choices of concrete graph neural architectures and provide solutions and improvements for both dense and sparse graphs. Extensive empirical evaluations over both public datasets and an industry dataset demonstrate that VESPER is capable of training high-performance GNN models over both sparse and dense graphs under reasonable privacy budgets

What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders

The last years have witnessed the emergence of a promising self-supervised learning strategy, referred to as masked autoencoding. However, there is a lack of theoretical understanding of how masking matters on graph autoencoders (GAEs). In this work, we present masked graph autoencoder (MaskGAE), a self-supervised learning framework for graph-structured data. Different from standard GAEs, MaskGAE adopts masked graph modeling (MGM) as a principled pretext task - masking a portion of edges and attempting to reconstruct the missing part with partially visible, unmasked graph structure. To understand whether MGM can help GAEs learn better representations, we provide both theoretical and empirical evidence to comprehensively justify the benefits of this pretext task. Theoretically, we establish close connections between GAEs and contrastive learning, showing that MGM significantly improves the self-supervised learning scheme of GAEs. Empirically, we conduct extensive experiments on a variety of graph benchmarks, demonstrating the superiority of MaskGAE over several state-of-the-arts on both link prediction and node classification tasks.Comment: KDD 2023 research track. Code available at https://github.com/EdisonLeeeee/MaskGA

Efficacy and safety of intranasal insulin on postoperative cognitive dysfunction in elderly patients after laparoscopic radical resection of colorectal cancer: a double-blind pilot study

ObjectiveTo evaluate the efficacy and safety of intranasal insulin on postoperative cognitive dysfunction (POCD) in elderly patients after laparoscopic radical resection of colorectal cancer.MethodsOlder patients scheduled for laparoscopic radical resection of colorectal cancer at Beijing Luhe Hospital, Capital Medical University, between August 2023 and November 2023, were enrolled in this double-blind pilot study. Patients were randomized to the control and insulin groups at a 1:1 ratio. The primary outcome was the rate of POCD at postoperative 7 days.ResultsA total of 61 patients (30 in the insulin group) were analyzed. The insulin group had a significantly lower POCD rate compared with the control group at postoperative day 7 [4(13.3%) vs. 12 (38.7%), p = 0.024]. The serum levels of IL-6, TNF-α and S100β at T2-5 in the insulin group were significantly lower than those of the control group (IL-6: mean difference at T2, −4.14, p = 0.036; T3, −3.84, p = 0.039; T4, −3.37, p = 0.013; T5, −2.57, p = 0.042; TNF-α: mean difference at T2, −3.19, p = 0.002; T3, −2.35, p = 0.028; T4, −2.30, p = 0.019; T5, −1.96, p = 0.0181; S100β: mean difference at T2, −8.30, p = 0.019; T3, −23.95, p = 0.020; T4, −20.01, p = 0.023; T5, −17.67, p = 0.010). No insulin allergic reactions, nasal irritation, or hypoglycemic reactions were observed in either of the groups.ConclusionIntranasal insulin may decrease the risk of POCD and inhibit the elevated serum IL-6, TNF-α, and S100β levels in elderly patients after laparoscopic radical resection of colorectal cancer, which proves that intranasal insulin may be a promising therapeutic option for POCD.Clinical trial registrationIdentifier, ChiCTR2300074423