CRF Learning with CNN Features for Image Segmentation

Conditional Random Rields (CRF) have been widely applied in image segmentations. While most studies rely on hand-crafted features, we here propose to exploit a pre-trained large convolutional neural network (CNN) to generate deep features for CRF learning. The deep CNN is trained on the ImageNet dataset and transferred to image segmentations here for constructing potentials of superpixels. Then the CRF parameters are learnt using a structured support vector machine (SSVM). To fully exploit context information in inference, we construct spatially related co-occurrence pairwise potentials and incorporate them into the energy function. This prefers labelling of object pairs that frequently co-occur in a certain spatial layout and at the same time avoids implausible labellings during the inference. Extensive experiments on binary and multi-class segmentation benchmarks demonstrate the promise of the proposed method. We thus provide new baselines for the segmentation performance on the Weizmann horse, Graz-02, MSRC-21, Stanford Background and PASCAL VOC 2011 datasets

Deep Convolutional Neural Fields for Depth Estimation from a Single Image

We consider the problem of depth estimation from a single monocular image in this work. It is a challenging task as no reliable depth cues are available, e.g., stereo correspondences, motions, etc. Previous efforts have been focusing on exploiting geometric priors or additional sources of information, with all using hand-crafted features. Recently, there is mounting evidence that features from deep convolutional neural networks (CNN) are setting new records for various vision applications. On the other hand, considering the continuous characteristic of the depth values, depth estimations can be naturally formulated into a continuous conditional random field (CRF) learning problem. Therefore, we in this paper present a deep convolutional neural field model for estimating depths from a single image, aiming to jointly explore the capacity of deep CNN and continuous CRF. Specifically, we propose a deep structured learning scheme which learns the unary and pairwise potentials of continuous CRF in a unified deep CNN framework. The proposed method can be used for depth estimations of general scenes with no geometric priors nor any extra information injected. In our case, the integral of the partition function can be analytically calculated, thus we can exactly solve the log-likelihood optimization. Moreover, solving the MAP problem for predicting depths of a new image is highly efficient as closed-form solutions exist. We experimentally demonstrate that the proposed method outperforms state-of-the-art depth estimation methods on both indoor and outdoor scene datasets.Comment: fixed some typos. in CVPR15 proceeding

Discriminative Training of Deep Fully-connected Continuous CRF with Task-specific Loss

Recent works on deep conditional random fields (CRF) have set new records on many vision tasks involving structured predictions. Here we propose a fully-connected deep continuous CRF model for both discrete and continuous labelling problems. We exemplify the usefulness of the proposed model on multi-class semantic labelling (discrete) and the robust depth estimation (continuous) problems. In our framework, we model both the unary and the pairwise potential functions as deep convolutional neural networks (CNN), which are jointly learned in an end-to-end fashion. The proposed method possesses the main advantage of continuously-valued CRF, which is a closed-form solution for the Maximum a posteriori (MAP) inference. To better adapt to different tasks, instead of using the commonly employed maximum likelihood CRF parameter learning protocol, we propose task-specific loss functions for learning the CRF parameters. It enables direct optimization of the quality of the MAP estimates during the course of learning. Specifically, we optimize the multi-class classification loss for the semantic labelling task and the Turkey's biweight loss for the robust depth estimation problem. Experimental results on the semantic labelling and robust depth estimation tasks demonstrate that the proposed method compare favorably against both baseline and state-of-the-art methods. In particular, we show that although the proposed deep CRF model is continuously valued, with the equipment of task-specific loss, it achieves impressive results even on discrete labelling tasks

Structured Learning of Tree Potentials in CRF for Image Segmentation

We propose a new approach to image segmentation, which exploits the advantages of both conditional random fields (CRFs) and decision trees. In the literature, the potential functions of CRFs are mostly defined as a linear combination of some pre-defined parametric models, and then methods like structured support vector machines (SSVMs) are applied to learn those linear coefficients. We instead formulate the unary and pairwise potentials as nonparametric forests---ensembles of decision trees, and learn the ensemble parameters and the trees in a unified optimization problem within the large-margin framework. In this fashion, we easily achieve nonlinear learning of potential functions on both unary and pairwise terms in CRFs. Moreover, we learn class-wise decision trees for each object that appears in the image. Due to the rich structure and flexibility of decision trees, our approach is powerful in modelling complex data likelihoods and label relationships. The resulting optimization problem is very challenging because it can have exponentially many variables and constraints. We show that this challenging optimization can be efficiently solved by combining a modified column generation and cutting-planes techniques. Experimental results on both binary (Graz-02, Weizmann horse, Oxford flower) and multi-class (MSRC-21, PASCAL VOC 2012) segmentation datasets demonstrate the power of the learned nonlinear nonparametric potentials.Comment: 10 pages. Appearing in IEEE Transactions on Neural Networks and Learning System

A database of microwave single-scattering properties for nonspherical ice particles

Capsule Summary A database containing microwave single-scattering properties for 11 ice particle shapes have been produced using discrete dipole approximation and is now publicly available. Corresponding Author: Guosheng Liu Department of Meteorology Florida State University Tallahassee, FL 32306-4520 (850) 644-6298 (850) 644-9642 (fax) [email protected] 1 Abstract As satellite observations at high microwave frequencies have recently become available, there is an increasing demand for methods that accurately evaluate the single-scattering properties of nonspherical ice particles at these frequencies. Algorithm developers can use the single-scattering data sets in the retrievals of cloud ice water content and snowfall rate. However, the methods that correctly handle the scattering of complex nonspherical particles are computationally inefficient and impractical for physical retrieval algorithms, in which scattering needs to be evaluated many times for particles with various sizes and shapes. As a remedy, during the past several years we have computed the scattering properties -scattering and absorption cross sections, phase functions, asymmetric parameters and backscattering cross sections -using an accurate discrete dipole approximation method and arranged the results into an easy-to-access database. The database contains the scattering properties at frequencies from 15 to 340 GHz, with temperatures from 0 to -40 ºC, of particle sizes (maximum dimension) from 50 to 12,500 µm, and for 11 particle shapes. The database along with an easy-to-use reading program is now made available to interested investigators. This article explains how this database is derived and how it can be used. 2 Despite their great significance in modulating the Earth radiation budget and the global hydrological cycle (e.g., Since then for the past several years, the author has been conducting DDA simulations for more particles shapes, broader particle size ranges, and over a wider range of frequencies. The results of these simulations has been arranged into a database containing single-scattering properties at frequencies from 15 to 340 GHz, with temperatures from 0 to -40°C, of particle sizes (maximum dimension) from 50 to 12,500 µm, and for 11 particle shapes. The database along with an easy-to-use reading program is now made available to interested investigators. A similar database that covers near-to far-infrared spectral regions has been reported by Yang et al. (2005). The current database covers microwave spectral region, complementary to the Yang et al. database. MODELING THE SCATTERING PROPERTY OF ICE PARTICLES. The DDA model developed by Draine and Flatau We have performed DDA modeling for eleven types of ice particle shapes. In where β, θ, and ϕ are the 3 angles to describe the orientation of the ice particle in the Draine and Flatau DDA model. All DDA model-calculated quantities presented in the database are orientationally averaged using calculations at 16 βs, 17 θs and 16 ϕs. The characteristics of the ice particles are listed in Columns and plates. Five types of columns and plates are included Rosettes. Rosettes shape, respectively. The 5-or 6-bullet rosette is constructed by adding one or two columns to a 4-bullet rosette in the direction perpendicular to the other 4 bullets. Columns that make up the same rosette have the same length and aspect ratio. But the aspect ratios for rosettes with different number of bullets are different depending on the area ratio -maximum dimension relations given below. To determine the aspect ratio of the columns, we use a relationship between the maximum dimension, D max , and the area ratio, A r , derived by Heymsfield and Miloshevich Snowflakes. Two types of snowflakes are considered. The first type is a sector-like particle It should be noted that for all the particles designed in this study, ice portion of the particle is made of pure ice with a density of 0.916 g cm -3 , not ice mixed with air bubbles. Accuracy. There are two possible causes for an inaccuracy of the computed results: Currently, calculations for 16x17x16 particle orientations are performed to represent random orientation. To examine whether these many orientations are "random" enough, we further conducted calculations for sector snowflakes at doubled numbers of βs, θs and ϕs, i.e., 32x33x32 orientations. Results show that doubling the number of orientational angles does not cause measurable change (<1%) in absorption and scattering 8 cross sections and asymmetry parameters. However, for backscatter cross sections (therefore, for phase function at 180º direction as well), the difference between current results and those of doubled orientational angles increases dramatically with frequency and particle size. At frequencies below 150 GHz, the difference is less than 5% for all particle sizes; but it increases to over 20% for particle size of 10,000 µm at 340 GHz. Therefore, readers are cautioned that backscatter cross sections at frequencies over 150 GHz, particularly for particle size larger than 5,000 µm, contain significant error (up to 20%) in the current version due to the lack of "randomness". We plan to conduct computations using more orientations in the future and announce the new results at the website which hosts the database. Fortunately, backscatter cross sections are used only for computing radar reflectivities; and currently available high-frequency radar is the Wband CloudSat radar with a frequency of 94 GHz. For this frequency, our comparison results show that the error due to this lack of randomness is less than 1.5%. Comparison has also been performed with the exact solution of T-matrix (Mishchenko et al., 2002) for circular cylinder and plate. While these shapes are not included in our database, it is assumed that the comparison gives an indication of the accuracy for the hexagonal column and plate in our database. The results showed that for "overall averaged" parameters (i.e., absorption and scattering cross sections, and asymmetry parameter), the relative errors for the two particle shapes are within ~2% when using |m|ks<0.5 and particle orientation angles greater than 8x9x8, a result being in-line with WHAT ARE IN THE DATABASE? Given a particle shape, the single-scattering properties of ice particles also depend on particle size, frequency and temperature. To build the database, the single-scattering properties have been computed at many "anchor" points defined by particle shape, size, frequency and temperature. These "anchor" points cover a broad range of particle sizes, frequencies and temperatures at reasonable increments, so that users can derive the scattering properties for any size, frequency and temperature through interpolation. In The following single-scattering properties are computed: the absorption cross section (C abs ), the scattering cross section (C sca ), the backscatter cross section (C b ), the 10 asymmetry parameter (g), and the phase function [P(cosΘ), where Θ is referred to as scattering angle that is the angle between incident and observing directions]. Absorption and scattering cross sections, in unit of area, are quantities describing how much incident radiation is absorbed or scattered to all directions by the particle. Backscatter cross section (also in unit of area) describes the scattered energy to the opposite direction of the incident radiation, a necessary quantity to assess radar reflectivity. The phase function is a physical quantity that describes the angular distribution of scattered energy, while the asymmetry parameter describes the degree of symmetry of scattered energy distributed with respect to the plane dividing forward and backward hemispheres. The phase function archived in this database is normalized, so that 1 cos ) (cos , where λ is wavelength. For easy viewing purpose, the absorption, scattering and backscatter cross sections are normalized by πr eff 2 in these plots. The above parameters of corresponding to spheres of the same mass but with radii of r eff (i.e., solid sphere) are also shown in the diagrams, which clearly illustrates the inaccuracy of using spheres as a shortcut to represent nonspherical ice particles. The database is downloadable from: http://cirrus.met.fsu.edu/research/scatdb.html. Along with a data file containing the computed results, a subroutine program (scatdb.c, 11 which can be called by either C or Fortran programs) is provided to read and interpolate (if needed) the scattering properties at required frequencies, temperatures, and particle sizes. In many applications, the scattering properties required by users may not match exactly with those calculated at the "anchor" points. In these cases, the subroutine will perform a linear interpolation using values at the nearby "anchor" points. SUMMARY. This article serves as an announcement of the availability of a useful database that contains the single-scattering properties at microwave frequencies of ice particles with a variety of shapes. The database contains the scattering properties at frequencies from 15 to 340 GHz, with temperatures from 0 to -40°C, of particle sizes (maximum dimension) from 50 to 12,500 µm, and for 11 particle shapes. The database along with an easy-to-use reading program is now made available to interested investigators, so that they can perform radiative transfer modeling without repeating the lengthy computation of nonspherical scattering. This paper explains how the database is derived and how it can be accessed. The address of the website that hosts this database is http://cirrus.met.fsu.edu/research/scatdb.html. Limitations and future improvements. There are several limitations of this database. First, the eleven types are only a fraction of ice particle shapes observable in nature. As our knowledge based on in situ measurements increases, results for additional ice particle shapes will be added to this database. Second, many ice particles have preferential orientations when falling due to aerodynamic balance. The preferential orientation is particularly important for backscatter cross sections that radars observe. Future addition to the database should include single-scattering properties for preferential orientated 12 particles, particularly at radar frequencies. In relation to this addition, many modern radars observe polarized signatures; therefore, inclusion of depolarization properties will be also beneficial. Finally, improving the accuracy of the DDA results by reducing interdipole spacing and increasing orientational angles is needed for frequencies higher than 150 GHz to ensure the backscatter cross sections with an accuracy of a few percent. We will keep readers informed by posting any improvements made to the database at the aforementioned website. ACKNOWLEDGEMENTS. The author is very grateful to Drs. B. T. Draine and P. J

A database of microwave single-scattering properties for nonspherical ice particles

Capsule Summary A database containing microwave single-scattering properties for 11 ice particle shapes have been produced using discrete dipole approximation and is now publicly available. Corresponding Author: Guosheng Liu Department of Meteorology Florida State University Tallahassee, FL 32306-4520 (850) 644-6298 (850) 644-9642 (fax) [email protected] 1 Abstract As satellite observations at high microwave frequencies have recently become available, there is an increasing demand for methods that accurately evaluate the single-scattering properties of nonspherical ice particles at these frequencies. Algorithm developers can use the single-scattering data sets in the retrievals of cloud ice water content and snowfall rate. However, the methods that correctly handle the scattering of complex nonspherical particles are computationally inefficient and impractical for physical retrieval algorithms, in which scattering needs to be evaluated many times for particles with various sizes and shapes. As a remedy, during the past several years we have computed the scattering properties -scattering and absorption cross sections, phase functions, asymmetric parameters and backscattering cross sections -using an accurate discrete dipole approximation method and arranged the results into an easy-to-access database. The database contains the scattering properties at frequencies from 15 to 340 GHz, with temperatures from 0 to -40 ºC, of particle sizes (maximum dimension) from 50 to 12,500 m, and for 11 particle shapes. The database along with an easy-to-use reading program is now made available to interested investigators. This article explains how this database is derived and how it can be used. 2 Despite their great significance in modulating the Earth radiation budget and the global hydrological cycle (e.g., We have performed DDA modeling for eleven types of ice particle shapes. In where , , and are the 3 angles to describe the orientation of the ice particle in the Draine and Flatau DDA model. All DDA model-calculated quantities presented in the database are orientationally averaged using calculations at 16 s, 17 s and 16 s. The characteristics of the ice particles are listed in 5 Columns and plates. Five types of columns and plates are included Rosettes. Rosettes shape, respectively. The 5-or 6-bullet rosette is constructed by adding one or two columns to a 4-bullet rosette in the direction perpendicular to the other 4 bullets. Columns that make up the same rosette have the same length and aspect ratio. But the aspect ratios for rosettes with different number of bullets are different depending on the area ratio -maximum dimension relations given below. To determine the aspect ratio of the columns, we use a relationship between the maximum dimension, D max , and the area ratio, A r , derived by 6 Snowflakes. Two types of snowflakes are considered. The first type is a sector-like particle It should be noted that for all the particles designed in this study, ice portion of the particle is made of pure ice with a density of 0.916 g cm -3 , not ice mixed with air bubbles. Accuracy. There are two possible causes for an inaccuracy of the computed results: Currently, calculations for 16x17x16 particle orientations are performed to represent random orientation. To examine whether these many orientations are "random" enough, we further conducted calculations for sector snowflakes at doubled numbers of s, s and s, i.e., 32x33x32 orientations. Results show that doubling the number of orientational angles does not cause measurable change (<1%) in absorption and scattering cross sections and asymmetry parameters. However, for backscatter cross sections 8 (therefore, for phase function at 180º direction as well), the difference between current results and those of doubled orientational angles increases dramatically with frequency and particle size. At frequencies below 150 GHz, the difference is less than 5% for all particle sizes; but it increases to over 20% for particle size of 10,000 m at 340 GHz. Therefore, readers are cautioned that backscatter cross sections at frequencies over 150 GHz, particularly for particle size larger than 5,000 m, contain significant error (up to 20%) in the current version due to the lack of "randomness". We plan to conduct computations using more orientations in the future and announce the new results at the website which hosts the database. Fortunately, backscatter cross sections are used only for computing radar reflectivities; and currently available high-frequency radar is the Wband CloudSat radar with a frequency of 94 GHz. For this frequency, our comparison results show that the error due to this lack of randomness is less than 1.5%. WHAT ARE IN THE DATABASE? Given a particle shape, the single-scattering properties of ice particles also depend on particle size, frequency and temperature. To build the database, the single-scattering properties have been computed at many "anchor" points defined by particle shape, size, frequency and temperature. These "anchor" points cover a broad range of particle sizes, frequencies and temperatures at reasonable increments, so that users can derive the scattering properties for any size, frequency and temperature through interpolation. In The following single-scattering properties are computed: the absorption cross section (C abs ), the scattering cross section (C sca ), the backscatter cross section The database is downloadable from: http://cirrus.met.fsu.edu/research/scatdb.html. Along with a data file containing the computed results, a subroutine program (scatdb.c, which can be called by either C or Fortran programs) is provided to read and interpolate (if needed) the scattering properties at required frequencies, temperatures, and particle sizes. In many applications, the scattering properties required by users may not match exactly with those calculated at the "anchor" points. In these cases, the subroutine will perform a linear interpolation using values at the nearby "anchor" points. SUMMARY. This article serves as an announcement of the availability of a useful database that contains the single-scattering properties at microwave frequencies of ice particles with a variety of shapes. The database contains the scattering properties at frequencies from 15 to 340 GHz, with temperatures from 0 to -40C, of particle sizes (maximum dimension) from 50 to 12,500 m, and for 11 particle shapes. The database along with an easy-to-use reading program is now made available to interested investigators, so that they can perform radiative transfer modeling without repeating the lengthy computation of nonspherical scattering. This paper explains how the database is derived and how it can be accessed. The address of the website that hosts this database is http://cirrus.met.fsu.edu/research/scatdb.html. Limitations and future improvements. There are several limitations of this database. First, the eleven types are only a fraction of ice particle shapes observable in nature. As our knowledge based on in situ measurements increases, results for additional ice particle 11 shapes will be added to this database. Second, many ice particles have preferential orientations when falling due to aerodynamic balance. The preferential orientation is particularly important for backscatter cross sections that radars observe. Future addition to the database should include single-scattering properties for preferential orientated particles, particularly at radar frequencies. In relation to this addition, many modern radars observe polarized signatures; therefore, inclusion of depolarization properties will be also beneficial. Finally, improving the accuracy of the DDA results by reducing interdipole spacing and increasing orientational angles is needed for frequencies higher than 150 GHz to ensure the backscatter cross sections with an accuracy of a few percent. We will keep readers informed by posting any improvements made to the database at the aforementioned website. ACKNOWLEDGEMENTS. The author is very grateful to Drs. B. T. Draine and P. J

High-order variational Lagrangian schemes for compressible fluids

We present high-order variational Lagrangian finite element methods for compressible fluids using a discrete energetic variational approach. Our spatial discretization is mass/momentum/energy conserving and entropy stable. Fully implicit time stepping is used for the temporal discretization, which allows for a much larger time step size for stability compared to explicit methods, especially for low-Mach number flows and/or on highly distorted meshes. Ample numerical results are presented to showcase the good performance of our proposed scheme.Comment: 24 page

Towards Robust Curve Text Detection with Conditional Spatial Expansion

It is challenging to detect curve texts due to their irregular shapes and varying sizes. In this paper, we first investigate the deficiency of the existing curve detection methods and then propose a novel Conditional Spatial Expansion (CSE) mechanism to improve the performance of curve text detection. Instead of regarding the curve text detection as a polygon regression or a segmentation problem, we treat it as a region expansion process. Our CSE starts with a seed arbitrarily initialized within a text region and progressively merges neighborhood regions based on the extracted local features by a CNN and contextual information of merged regions. The CSE is highly parameterized and can be seamlessly integrated into existing object detection frameworks. Enhanced by the data-dependent CSE mechanism, our curve text detection system provides robust instance-level text region extraction with minimal post-processing. The analysis experiment shows that our CSE can handle texts with various shapes, sizes, and orientations, and can effectively suppress the false-positives coming from text-like textures or unexpected texts included in the same RoI. Compared with the existing curve text detection algorithms, our method is more robust and enjoys a simpler processing flow. It also creates a new state-of-art performance on curve text benchmarks with F-score of up to 78.4$\%$.Comment: This paper has been accepted by IEEE International Conference on Computer Vision and Pattern Recognition (CVPR 2019