15 research outputs found
FocusFlow: Boosting Key-Points Optical Flow Estimation for Autonomous Driving
Key-point-based scene understanding is fundamental for autonomous driving
applications. At the same time, optical flow plays an important role in many
vision tasks. However, due to the implicit bias of equal attention on all
points, classic data-driven optical flow estimation methods yield less
satisfactory performance on key points, limiting their implementations in
key-point-critical safety-relevant scenarios. To address these issues, we
introduce a points-based modeling method that requires the model to learn
key-point-related priors explicitly. Based on the modeling method, we present
FocusFlow, a framework consisting of 1) a mix loss function combined with a
classic photometric loss function and our proposed Conditional Point Control
Loss (CPCL) function for diverse point-wise supervision; 2) a conditioned
controlling model which substitutes the conventional feature encoder by our
proposed Condition Control Encoder (CCE). CCE incorporates a Frame Feature
Encoder (FFE) that extracts features from frames, a Condition Feature Encoder
(CFE) that learns to control the feature extraction behavior of FFE from input
masks containing information of key points, and fusion modules that transfer
the controlling information between FFE and CFE. Our FocusFlow framework shows
outstanding performance with up to +44.5% precision improvement on various key
points such as ORB, SIFT, and even learning-based SiLK, along with exceptional
scalability for most existing data-driven optical flow methods like PWC-Net,
RAFT, and FlowFormer. Notably, FocusFlow yields competitive or superior
performances rivaling the original models on the whole frame. The source code
will be available at https://github.com/ZhonghuaYi/FocusFlow_official.Comment: The source code of FocusFlow will be available at
https://github.com/ZhonghuaYi/FocusFlow_officia
Towards Anytime Optical Flow Estimation with Event Cameras
Event cameras are capable of responding to log-brightness changes in
microseconds. Its characteristic of producing responses only to the changing
region is particularly suitable for optical flow estimation. In contrast to the
super low-latency response speed of event cameras, existing datasets collected
via event cameras, however, only provide limited frame rate optical flow ground
truth, (e.g., at 10Hz), greatly restricting the potential of event-driven
optical flow. To address this challenge, we put forward a high-frame-rate,
low-latency event representation Unified Voxel Grid, sequentially fed into the
network bin by bin. We then propose EVA-Flow, an EVent-based Anytime Flow
estimation network to produce high-frame-rate event optical flow with only
low-frame-rate optical flow ground truth for supervision. The key component of
our EVA-Flow is the stacked Spatiotemporal Motion Refinement (SMR) module,
which predicts temporally-dense optical flow and enhances the accuracy via
spatial-temporal motion refinement. The time-dense feature warping utilized in
the SMR module provides implicit supervision for the intermediate optical flow.
Additionally, we introduce the Rectified Flow Warp Loss (RFWL) for the
unsupervised evaluation of intermediate optical flow in the absence of ground
truth. This is, to the best of our knowledge, the first work focusing on
anytime optical flow estimation via event cameras. A comprehensive variety of
experiments on MVSEC, DESC, and our EVA-FlowSet demonstrates that EVA-Flow
achieves competitive performance, super-low-latency (5ms), fastest inference
(9.2ms), time-dense motion estimation (200Hz), and strong generalization. Our
code will be available at https://github.com/Yaozhuwa/EVA-Flow.Comment: Code will be available at https://github.com/Yaozhuwa/EVA-Flo
Event-Based Fusion for Motion Deblurring with Cross-modal Attention
Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution, providing valid image degradation information within the exposure time. In this paper, we rethink the event-based image deblurring problem and unfold it into an end-to-end two-stage image restoration network. To effectively fuse event and image features, we design an event-image cross-modal attention module applied at multiple levels of our network, which allows to focus on relevant features from the event branch and filter out noise. We also introduce a novel symmetric cumulative event representation specifically for image deblurring as well as an event mask gated connection between the two stages of our network which helps avoid information loss. At the dataset level, to foster event-based motion deblurring and to facilitate evaluation on challenging real-world images, we introduce the Real Event Blur (REBlur) dataset, captured with an event camera in an illumination controlled optical laboratory. Our Event Fusion Network (EFNet) sets the new state of the art in motion deblurring, surpassing both the prior best-performing image-based method and all event-based methods with public implementations on the GoPro dataset (by up to 2.47dB) and on our REBlur dataset, even in extreme blurry conditions. The code and our REBlur dataset will be made publicly available
Event-Based Fusion for Motion Deblurring with Cross-modal Attention
Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution, providing valid image degradation information within the exposure time. In this paper, we rethink the event-based image deblurring problem and unfold it into an end-to-end two-stage image restoration network. To effectively fuse event and image features, we design an event-image cross-modal attention module applied at multiple levels of our network, which allows to focus on relevant features from the event branch and filter out noise. We also introduce a novel symmetric cumulative event representation specifically for image deblurring as well as an event mask gated connection between the two stages of our network which helps avoid information loss. At the dataset level, to foster event-based motion deblurring and to facilitate evaluation on challenging real-world images, we introduce the Real Event Blur (REBlur) dataset, captured with an event camera in an illumination controlled optical laboratory. Our Event Fusion Network (EFNet) sets the new state of the art in motion deblurring, surpassing both the prior best-performing image-based method and all event-based methods with public implementations on the GoPro dataset (by up to 2.47dB) and on our REBlur dataset, even in extreme blurry conditions. The code and our REBlur dataset will be made publicly available
Nitration of Hsp90 induces cell death
Oxidative stress is a widely recognized cause of cell death associated with neurodegeneration, inflammation, and aging. Tyrosine nitration in these conditions has been reported extensively, but whether tyrosine nitration is a marker or plays a role in the cell-death processes was unknown. Here, we show that nitration of a single tyrosine residue on a small proportion of 90-kDa heat-shock protein (Hsp90), is sufficient to induce motor neuron death by the P2X7 receptor-dependent activation of the Fas pathway. Nitrotyrosine at position 33 or 56 stimulates a toxic gain of function that turns Hsp90 into a toxic protein. Using an antibody that recognizes the nitrated Hsp90, we found immunoreactivity in motor neurons of patients with amyotrophic lateral sclerosis, in an animal model of amyotrophic lateral sclerosis, and after experimental spinal cord injury. Our findings reveal that cell death can be triggered by nitration of a single protein and highlight nitrated Hsp90 as a potential target for the development of effective therapies for a large number of pathologies
Prevention of diabetic nephropathy in Ins2+/āAkitaJ mice by the mitochondria-targeted therapy MitoQ
Mitochondrial production of ROS (reactive oxygen species) is thought to be associated with the cellular damage resulting from chronic exposure to high glucose in long-term diabetic patients. We hypothesized that a mitochondria-targeted antioxidant would prevent kidney damage in the Ins2+/āAkitaJ mouse model (Akita mice) of TypeĀ 1 diabetes. To test this we orally administered a mitochondria-targeted ubiquinone (MitoQ) over a 12-week period and assessed tubular and glomerular function. Fibrosis and pro-fibrotic signalling pathways were determined by immunohistochemical analysis, and mitochondria were isolated from the kidney for functional assessment. MitoQ treatment improved tubular and glomerular function in the Ins2+/āAkitaJ mice. MitoQ did not have a significant effect on plasma creatinine levels, but decreased urinary albumin levels to the same level as non-diabetic controls. Consistent with previous studies, renal mitochondrial function showed no significant change between any of the diabetic or wild-type groups. Importantly, interstitial fibrosis and glomerular damage were significantly reduced in the treated animals. The pro-fibrotic transcription factors phospho-Smad2/3 and Ī²-catenin showed a nuclear accumulation in the Ins2+/āAkitaJ mice, which was prevented by MitoQ treatment. These results support the hypothesis that mitochondrially targeted therapies may be beneficial in the treatment of diabetic nephropathy. They also highlight a relatively unexplored aspect of mitochondrial ROS signalling in the control of fibrosis
Efficient Human Pose Estimation via 3D Event Point Cloud
Human Pose Estimation (HPE) based on RGB images has experienced a rapid
development benefiting from deep learning. However, event-based HPE has not
been fully studied, which remains great potential for applications in extreme
scenes and efficiency-critical conditions. In this paper, we are the first to
estimate 2D human pose directly from 3D event point cloud. We propose a novel
representation of events, the rasterized event point cloud, aggregating events
on the same position of a small time slice. It maintains the 3D features from
multiple statistical cues and significantly reduces memory consumption and
computation complexity, proved to be efficient in our work. We then leverage
the rasterized event point cloud as input to three different backbones,
PointNet, DGCNN, and Point Transformer, with two linear layer decoders to
predict the location of human keypoints. We find that based on our method,
PointNet achieves promising results with much faster speed, whereas Point
Transfomer reaches much higher accuracy, even close to previous
event-frame-based methods. A comprehensive set of results demonstrates that our
proposed method is consistently effective for these 3D backbone models in
event-driven human pose estimation. Our method based on PointNet with 2048
points input achieves 82.46mm in MPJPE3D on the DHP19 dataset, while only has a
latency of 12.29ms on an NVIDIA Jetson Xavier NX edge computing platform, which
is ideally suitable for real-time detection with event cameras. Code will be
made publicly at https://github.com/MasterHow/EventPointPose.Comment: Code will be made publicly at
https://github.com/MasterHow/EventPointPos
PanoFlow: Learning 360{\deg} Optical Flow for Surrounding Temporal Understanding
Optical flow estimation is a basic task in self-driving and robotics systems,
which enables to temporally interpret traffic scenes. Autonomous vehicles
clearly benefit from the ultra-wide Field of View (FoV) offered by 360{\deg}
panoramic sensors. However, due to the unique imaging process of panoramic
cameras, models designed for pinhole images do not directly generalize
satisfactorily to 360{\deg} panoramic images. In this paper, we put forward a
novel network framework--PanoFlow, to learn optical flow for panoramic images.
To overcome the distortions introduced by equirectangular projection in
panoramic transformation, we design a Flow Distortion Augmentation (FDA)
method, which contains radial flow distortion (FDA-R) or equirectangular flow
distortion (FDA-E). We further look into the definition and properties of
cyclic optical flow for panoramic videos, and hereby propose a Cyclic Flow
Estimation (CFE) method by leveraging the cyclicity of spherical images to
infer 360{\deg} optical flow and converting large displacement to relatively
small displacement. PanoFlow is applicable to any existing flow estimation
method and benefits from the progress of narrow-FoV flow estimation. In
addition, we create and release a synthetic panoramic dataset Flow360 based on
CARLA to facilitate training and quantitative analysis. PanoFlow achieves
state-of-the-art performance on the public OmniFlowNet and the established
Flow360 benchmarks. Our proposed approach reduces the End-Point-Error (EPE) on
Flow360 by 27.3%. On OmniFlowNet, PanoFlow achieves an EPE of 3.17 pixels, a
55.5% error reduction from the best published result. We also qualitatively
validate our method via a collection vehicle and a public real-world OmniPhotos
dataset, indicating strong potential and robustness for real-world navigation
applications. Code and dataset are publicly available at
https://github.com/MasterHow/PanoFlow.Comment: Code and dataset are publicly available at
https://github.com/MasterHow/PanoFlo