Object Counting and Localization: A Statistical Approach

Scene understanding is fundamental to many computer vision applications such as autonomous driving, robot navigation and human-machine interaction; visual object counting and localization are important building blocks of scene understanding. In this dissertation, we present: (1) a framework that employs doubly stochastic Poisson (Cox) processes to estimate the number of instances of an object in an image and (2) a Bayesian model that localizes multiple instances of an object using counts from image sub-regions. Poisson processes are well-suited for modeling events that occur randomly in space, such as the location of objects in an image or the enumeration of objects in a scene. The proposed algorithm selects a subset of bounding boxes in the image domain, then queries them for the presence of the object of interest by running a pre-trained convolutional neural net (CNN) classifier. The resulting observations are then aggregated, and a posterior distribution over the intensity of a Cox process is computed. This intensity function is summed up, providing an estimator of the number of instances of the object over the entire image. Despite the flexibility and versatility of Poisson processes, their application to large datasets is limited, as their computational complexity and storage requirements do not easily scale with image size, typically requiring $O(n^3)$ computation time and $O(n^2)$ storage, where $n$ is the number of observations. To mitigate this problem, we employ the Kronecker algebra, which takes advantage of the tensor product structure of covariance matrices. As the likelihood is non-Gaussian, the Laplace approximation is used for inference, employing the conjugate gradient and Newton's method. Our approach has then close to linear performance, requiring only $O(n^{3/2})$ computation time and $O(n)$ memory. We demonstrate the counting results on both simulated data and real-world datasets, comparing the results with state-of-the-art counting methods. We then extend this framework by noting that most object detection and classification systems rely upon the use of region proposal networks or upon classifying the ``objectness'' of specific sub-windows to help detect potential object locations within an image. We use our Cox model to convert such region proposals to a well-defined Poisson intensity. This output can be used as-is to directly estimate object counts, or can be plugged into pre-existing object detection frameworks to improve their counting and detection performance. This remapping does not require the original network to be re-trained: the parameters of the model can be estimated analytically from the training data. Furthermore, we consider the problem of quickly localizing multiple instances of an object by asking questions of the form ``How many instances are there in this set?", while obtaining noisy answers. We evaluate the performance of the partitioning \textit{policy} using the expected entropy of the posterior distribution after a fixed number of questions with noisy answers. We derive a lower bound for the value of this problem and study a specific policy, named the \textit{dyadic policy}. We show that this policy achieves a value which is no more than twice this lower bound when answers are noise-free, and show a more general constant factor approximation guarantee for the noisy setting. We present an empirical evaluation of this policy on simulated data for the problem of detecting multiple instances of the same object in an image. Finally, we present experiments on localizing multiple objects simultaneously on real images

Exploring the limits of approximate number perception

How do we perceive the number of objects in a visual scene? Are there limits to how finely we can perceive magnitudes like number? In a series of three empirical chapters, I explore the limits of our perceptual abilities and the perceptual basis of approximate number thoughts. First, I investigate laypeople’s intuitive beliefs about perceptual limits. Additionally, I empirically test whether there is evidence for discrimination limits; that is, can people actually tell which is the larger group between 50 and 51 dots? I find that the JND is highly consistent with naïve theories about perception, yet discrimination data indicates that there is no limit to how small of differences we are able to perceive. Next, I propose a novel modeling approach to quantify the contribution of non-task features during magnitude discrimination task. In contrast to previous modeling attempts, I make use of separate parameters for reliance (how much are you “listening” to this feature?) and precision (how precise is your representation of that feature?). Fitting this model to adult discrimination performance on three magnitude comparison tasks, I find that subjects are using distinct representations for different magnitude comparison tasks, and that precision and reliance should be considered separable contributors to number performance. Finally, I dive into the stimuli themselves to test the widely made claim that non- numerical continuous features pattern with number in the real world. Looking at photographs as well as the illustrations of children’s counting books, I find variability in the extent to which continuous features explain significant variability in number. These features perform comparatively better in photographs, but interestingly, I find that both adult and child estimation based on counting book illustrations is superior to their performance in photographs, which indicates that continuous features are not the primary perceptual basis used to extract number from visual scenes

Cox Processes for Counting by Detection

In this work, doubly stochastic Poisson (Cox) processes and convolutional neural net (CNN) classifiers are used to estimate the number of instances of an object in an image. Poisson processes are well suited to model events that occur randomly in space, such as the location of objects in an image or the enumeration of objects in a scene. The proposed algorithm selects a subset of bounding boxes in the image domain, then queries them for the presence of the object of interest by running a pre-trained CNN classifier. The resulting observations are then aggregated, and a posterior distribution over the intensity of a Cox process is computed. This intensity function is summed up, providing an estimator of the number of instances of the object over the entire image. Despite the flexibility and versatility of Cox processes, their application to large datasets is limited as their computational complexity and storage requirements do not easily scale with image size, typically requiring O(n3) computation time and O(n2) storage, where n is the number of observations. To mitigate this problem, we employ the Kronecker algebra, which takes advantage of direct product structures. As the likelihood is non-Gaussian, the Laplace approximation is used for inference, employing the conjugate gradient and Newton’s method. Our approach has then close to linear performance, requiring only O(n3/2) computation time and O(n) memory. Results are presented on simulated data and on images from the publicly available MS COCO dataset. We compare our counting results with the state-of-the-art detection method, Faster RCNN, and demonstrate superior performance