Learning Second-Order Attentive Context for Efficient Correspondence Pruning

Correspondence pruning aims to search consistent correspondences (inliers) from a set of putative correspondences. It is challenging because of the disorganized spatial distribution of numerous outliers, especially when putative correspondences are largely dominated by outliers. It's more challenging to ensure effectiveness while maintaining efficiency. In this paper, we propose an effective and efficient method for correspondence pruning. Inspired by the success of attentive context in correspondence problems, we first extend the attentive context to the first-order attentive context and then introduce the idea of attention in attention (ANA) to model second-order attentive context for correspondence pruning. Compared with first-order attention that focuses on feature-consistent context, second-order attention dedicates to attention weights itself and provides an additional source to encode consistent context from the attention map. For efficiency, we derive two approximate formulations for the naive implementation of second-order attention to optimize the cubic complexity to linear complexity, such that second-order attention can be used with negligible computational overheads. We further implement our formulations in a second-order context layer and then incorporate the layer in an ANA block. Extensive experiments demonstrate that our method is effective and efficient in pruning outliers, especially in high-outlier-ratio cases. Compared with the state-of-the-art correspondence pruning approach LMCNet, our method runs 14 times faster while maintaining a competitive accuracy.Comment: 9 pages, 8 figures; Accepted to AAAI 2023 (Oral

Learning Probabilistic Coordinate Fields for Robust Correspondences

We introduce Probabilistic Coordinate Fields (PCFs), a novel geometric-invariant coordinate representation for image correspondence problems. In contrast to standard Cartesian coordinates, PCFs encode coordinates in correspondence-specific barycentric coordinate systems (BCS) with affine invariance. To know \textit{when and where to trust} the encoded coordinates, we implement PCFs in a probabilistic network termed PCF-Net, which parameterizes the distribution of coordinate fields as Gaussian mixture models. By jointly optimizing coordinate fields and their confidence conditioned on dense flows, PCF-Net can work with various feature descriptors when quantifying the reliability of PCFs by confidence maps. An interesting observation of this work is that the learned confidence map converges to geometrically coherent and semantically consistent regions, which facilitates robust coordinate representation. By delivering the confident coordinates to keypoint/feature descriptors, we show that PCF-Net can be used as a plug-in to existing correspondence-dependent approaches. Extensive experiments on both indoor and outdoor datasets suggest that accurate geometric invariant coordinates help to achieve the state of the art in several correspondence problems, such as sparse feature matching, dense image registration, camera pose estimation, and consistency filtering. Further, the interpretable confidence map predicted by PCF-Net can also be leveraged to other novel applications from texture transfer to multi-homography classification.Comment: Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligenc

Point-and-Shoot All-in-Focus Photo Synthesis from Smartphone Camera Pair

All-in-Focus (AIF) photography is expected to be a commercial selling point for modern smartphones. Standard AIF synthesis requires manual, time-consuming operations such as focal stack compositing, which is unfriendly to ordinary people. To achieve point-and-shoot AIF photography with a smartphone, we expect that an AIF photo can be generated from one shot of the scene, instead of from multiple photos captured by the same camera. Benefiting from the multi-camera module in modern smartphones, we introduce a new task of AIF synthesis from main (wide) and ultra-wide cameras. The goal is to recover sharp details from defocused regions in the main-camera photo with the help of the ultra-wide-camera one. The camera setting poses new challenges such as parallax-induced occlusions and inconsistent color between cameras. To overcome the challenges, we introduce a predict-and-refine network to mitigate occlusions and propose dynamic frequency-domain alignment for color correction. To enable effective training and evaluation, we also build an AIF dataset with 2686 unique scenes. Each scene includes two photos captured by the main camera, one photo captured by the ultrawide camera, and a synthesized AIF photo. Results show that our solution, termed EasyAIF, can produce high-quality AIF photos and outperforms strong baselines quantitatively and qualitatively. For the first time, we demonstrate point-and-shoot AIF photo synthesis successfully from main and ultra-wide cameras.Comment: Early Access by IEEE Transactions on Circuits and Systems for Video Technology 202

Fast Full-frame Video Stabilization with Iterative Optimization

Video stabilization refers to the problem of transforming a shaky video into a visually pleasing one. The question of how to strike a good trade-off between visual quality and computational speed has remained one of the open challenges in video stabilization. Inspired by the analogy between wobbly frames and jigsaw puzzles, we propose an iterative optimization-based learning approach using synthetic datasets for video stabilization, which consists of two interacting submodules: motion trajectory smoothing and full-frame outpainting. First, we develop a two-level (coarse-to-fine) stabilizing algorithm based on the probabilistic flow field. The confidence map associated with the estimated optical flow is exploited to guide the search for shared regions through backpropagation. Second, we take a divide-and-conquer approach and propose a novel multiframe fusion strategy to render full-frame stabilized views. An important new insight brought about by our iterative optimization approach is that the target video can be interpreted as the fixed point of nonlinear mapping for video stabilization. We formulate video stabilization as a problem of minimizing the amount of jerkiness in motion trajectories, which guarantees convergence with the help of fixed-point theory. Extensive experimental results are reported to demonstrate the superiority of the proposed approach in terms of computational speed and visual quality. The code will be available on GitHub.Comment: Accepted by ICCV202

Anthropogenic Activities Generate High-Refractory Black Carbon along the Yangtze River Continuum

12 pages, 7 figuresCombustion-driven particulate black carbon (PBC) is a crucial slow-cycling pool in the organic carbon flux from rivers to oceans. Since the refractoriness of PBC stems from the association of non-homologous char and soot, the composition and source of char and soot must be considered when investigating riverine PBC. Samples along the Yangtze River continuum during different hydrological periods were collected in this study to investigate the association and asynchronous combustion drive of char and soot in PBC. The results revealed that PBC in the Yangtze River, with higher refractory nature, accounts for 13.73 ± 6.89% of particulate organic carbon, and soot occupies 37.53 ± 11.00% of PBC. The preponderant contribution of fossil fuel combustion to soot (92.57 ± 3.20%) compared to char (27.55 ± 5.92%), suggested that fossil fuel combustion is a crucial driver for PBC with high soot percentage. Redundancy analysis and structural equation modeling confirmed that the fossil fuel energy used by anthropogenic activities promoting soot is the crucial reason for high-refractory PBC. We estimated that the Yangtze River transported 0.15–0.23 Tg of soot and 0.15–0.25 Tg of char to the ocean annually, and the export of large higher refractory PBC to the ocean can form a long-term sink and prolong the residence time of terrigenous carbonThis study was supported by grants from the National Natural Science Foundation of China (nos. 42277214, 42207256, and 41971286), major programs of the National Social Science Foundation of China (grant nos. 22&ZD136), the Special Science and Technology Innovation Program for Carbon Peak and Carbon Neutralization of Jiangsu Province (grant no. BE2022612)Peer reviewe

Data for Elastically-mediated Positioning of 2D Colloids

Data for publication “Elastically-mediated Positioning of 2D Colloids” by Weiyue Xin, Hao Wu, Gregory Grason, Maria Santore, 2021.https://scholarworks.umass.edu/data/1124/thumbnail.jp

Maternal mortality surveillance in an inland Chinese province

Objectives: To evaluate the Chinese maternal mortality surveillance system in an inland province and identify how it can be improved. Methods: The review process and Chinese Maternal Deaths Reporting Form were compared with standards recommended by the UK Confidential Enquiry into Maternal and Child Health using interviews with key personnel, field observations, and reports and audits from 2003-2005. Results: The Chinese Maternal Deaths Reporting Form does not provide anonymity for the deceased woman, the health workers, or hospitals. The information collected is often insufficient to identify substandard care. The Review Committee was not multidisciplinary and the review was not confidential. The review findings were only available to the Review Committee. Conclusion: Confidentiality should be a requirement in the maternal mortality surveillance system. The anonymous findings should be available to health workers, and be used to improve the system and inform the community about performance

Switchable Positioning of Plate-Like Inclusions in Lipid Membranes: Elastically Mediated Interactions of Planar Colloids in 2D Fluids

We demonstrate how manipulating curvature in an elastic fluid lamella enables the reversible relative positioning of flat, rigid, plate-like micrometer-scale inclusions, with spacings from about a micrometer to tens of micrometers. In an experimental model comprising giant unilamellar vesicles containing solid domain pairs coexisting in a fluid membrane, we adjusted vesicle inflation to manipulate membrane curvature and mapped the inter-domain separation. A two-dimensional model of the pair potential predicts the salient experimental observations and reveals both attractions and repulsions, producing a potential minimum entirely a result of the solid domain rigidity and bending energy in the fluid membrane. The impact of vesicle inflation on domain separation in vesicles containing two solid domains was qualitatively consistent with observations in vesicles containing many domains. The behavior differs qualitatively from the pure repulsions between fluid membrane domains or interactions between nanoscopic inclusions whose repulsive or attractive character is not switchable

GIS Partial Discharge Pattern Recognition Based on Time-Frequency Features and Improved Convolutional Neural Network

Different types of partial discharge (PD) in gas-insulated switchgear (GIS) cause different damage to GIS insulation, correctly identifying the PD type is very important for evaluating the insulation status of GIS. This paper proposes a PD pattern recognition method based on an improved feature fusion convolutional neural network (IFCNN) to fully use the time-frequency features of PD pulses to realize PD pattern recognition. Firstly, the one-dimensional time-domain feature sequence of the PD pulse and the corresponding wavelet time-frequency diagram are applied as inputs. Secondly, the convolutional neural network (CNN) with two parallel channels is used for feature extraction, the extracted fault information is fused, and the shallow features of the wavelet time-frequency diagram are fused to prevent feature loss caused by pooling operation. Finally, the extracted features are sent to the classifier to recognize different types of PD. The discharge data of different types of PD are obtained for testing by experiments and simulation. Compared with 1-D CNN and 2-D CNN under the same specification, the proposed method can mine more potential local features of discharge pulses by fusing the time-frequency features of PD pulses in different dimensions, and improves the recognition accuracy to 95.8%

GIS Partial Discharge Pattern Recognition Based on Time-Frequency Features and Improved Convolutional Neural Network

Different types of partial discharge (PD) in gas-insulated switchgear (GIS) cause different damage to GIS insulation, correctly identifying the PD type is very important for evaluating the insulation status of GIS. This paper proposes a PD pattern recognition method based on an improved feature fusion convolutional neural network (IFCNN) to fully use the time-frequency features of PD pulses to realize PD pattern recognition. Firstly, the one-dimensional time-domain feature sequence of the PD pulse and the corresponding wavelet time-frequency diagram are applied as inputs. Secondly, the convolutional neural network (CNN) with two parallel channels is used for feature extraction, the extracted fault information is fused, and the shallow features of the wavelet time-frequency diagram are fused to prevent feature loss caused by pooling operation. Finally, the extracted features are sent to the classifier to recognize different types of PD. The discharge data of different types of PD are obtained for testing by experiments and simulation. Compared with 1-D CNN and 2-D CNN under the same specification, the proposed method can mine more potential local features of discharge pulses by fusing the time-frequency features of PD pulses in different dimensions, and improves the recognition accuracy to 95.8%