Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10x or 100x? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between `enormous data' and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pre-training) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-the-art results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.Comment: ICCV 2017 camera read

Finding communities in sparse networks

Spectral algorithms based on matrix representations of networks are often used to detect communities but classic spectral methods based on the adjacency matrix and its variants fail to detect communities in sparse networks. New spectral methods based on non-backtracking random walks have recently been introduced that successfully detect communities in many sparse networks. However, the spectrum of non-backtracking random walks ignores hanging trees in networks that can contain information about the community structure of networks. We introduce the reluctant backtracking operators that explicitly account for hanging trees as they admit a small probability of returning to the immediately previous node unlike the non-backtracking operators that forbid an immediate return. We show that the reluctant backtracking operators can detect communities in certain sparse networks where the non-backtracking operators cannot while performing comparably on benchmark stochastic block model networks and real world networks. We also show that the spectrum of the reluctant backtracking operator approximately optimises the standard modularity function similar to the flow matrix. Interestingly, for this family of non- and reluctant-backtracking operators the main determinant of performance on real-world networks is whether or not they are normalised to conserve probability at each node.Comment: 11 pages, 4 figure

Limit Sets for Natural Extensions of Schelling's Segregation Model

Thomas Schelling developed an influential demographic model that illustrated how, even with relatively mild assumptions on each individual's nearest neighbor preferences, an integrated city would likely unravel to a segregated city, even if all individuals prefer integration. Individuals in Schelling's model cities are divided into two groups of equal number and each individual is 'happy' or 'unhappy' when the number of similar neighbors cross a simple threshold. In this manuscript we consider natural extensions of Schelling's original model to allow the two groups have different sizes and to allow different notions of happiness of an individual. We observe that differences in aggregation patterns of majority and minority groups are highly sensitive to the happiness threshold; for low threshold, the differences are small, and when the threshold is raised, striking new patterns emerge. We also observe that when individuals strongly prefer to live integrated neighborhoods, the final states exhibit a new tessellated-like structure.Comment: 19 pages, 10 figure

Schelling's Segregation Model: Parameters, scaling, and aggregation

Thomas Schelling proposed a simple spatial model to illustrate how, even with relatively mild assumptions on each individual's nearest neighbor preferences, an integrated city would likely unravel to a segregated city, even if all individuals prefer integration. This agent based lattice model has become quite influential amongst social scientists, demographers, and economists. Aggregation relates to individuals coming together to form groups and Schelling equated global aggregation with segregation. Many authors assumed that the segregation which Schelling observed in simulations on very small cities persists for larger, realistic size cities. We describe how different measures could be used to quantify the segregation and unlock its dependence on city size, disparate neighbor comfortability threshold, and population density. We identify distinct scales of global aggregation, and show that the striking global aggregation Schelling observed is strictly a small city phenomenon. We also discover several scaling laws for the aggregation measures. Along the way we prove that as the Schelling model evolves, the total perimeter of the interface between the different agents decreases, which provides a useful analytical tool to study the evolution.clusters, segregation, simulation, statistics