F. Lattari

F. Visin

J. Masci

M. Ciccone

M. Matteucci

Archivio istituzionale della ricerca - Politecnico di Milano

ReConvNet: Video Object Segmentationwith Spatio-Temporal Features ModulationFrancesco Lattari1∗, Marco Ciccone1∗, Matteo Matteucci1, Jonathan Masci3, Francesco Visin1,2†1Politecnico di Milano, 2Google DeepMind, 3NNAISENSE{francesco.lattari, marco.ciccone, matteo.matteucci}@polimi.it,visin@google.com, jonathan@nnaisense.comAbstractWe introduce ReConvNet, a recurrent convolutional ar-chitecture for semi-supervised video object segmentationthat is able to fast adapt its features to focus on any spe-cific object of interest at inference time. Generalizationto new objects never observed during training is knownto be a hard task for supervised approaches that wouldneed to be retrained. To tackle this problem, we propose amore efficient solution that learns spatio-temporal featuresself-adapting to the object of interest via conditional affinetransformations. This approach is simple, can be trainedend-to-end and does not necessarily require extra train-ing steps at inference time. Our method shows competitiveresults on DAVIS2016 with respect to state-of-the art ap-proaches that use online fine-tuning, and outperforms themon DAVIS2017. ReConvNet shows also promising results onthe DAVIS-Challenge 2018 winning the 10-th position.1. IntroductionSemi-Supervised Video Object Segmentation is the taskof segmenting specific objects of interest in a video se-quence, given their segmentation in the first frame. Thisposes an interesting challenge for standard supervisedmethods, as the model cannot be trained to discriminate be-tween a fixed set of classes based on semantics [13], butrather has to learn to segment unseen objects based on asingle instance. Classic supervised techniques fail to gener-alize easily to new objects whose traits are potentially verydifferent from the training data. This problem is known inthe literature as “domain adaptation”.Most of the proposed approaches cast the problem as aone-shot learning task: after a first general pre-training onthe entire dataset, at inference time the generic model is∗The authors equally contributed.†Most of the work has been carried out while at Politecnico di Milanoadapted into an object-specific one by fine-tuning on trans-formations of the first frame for each test sequence [2, 15].This procedure is computationally expensive, requiring sev-eral extra steps of back-propagation for each sequence. Fur-thermore, it heavily depends on the ability of the data aug-mentation procedure to produce realistic sequences. Indeed,generating high-quality sequences is clearly a very hard andill-posed task that adds an extra layer of complexity to theproblem.In this work we propose a system that can adapt to thespecific object of interest at inference time without the needof an expensive fine-tuning stage. This mechanism is in-spired by fast-weights [10]. In this framework a slow net-work generates, on-line, the weights of a second one, calledfast network, to fast adapt to a new task or to a change in theenvironment. More recently HyperNetworks [3] proposeto use a network to infer a transformation of the weightsrather than the weights themselves. Similarly, FiLM [8]proposes an extension of conditional batch-normalizationwhere a module produces an affine transformation that isapplied to the features of each layer of the main neural net-work in order to impose a more explicit conditioning.In the context of object segmentation, features mod-ulation has been investigated on the DAVIS dataset byOSMN [17], that extends OSVOS [2] by specializing a partof the architecture to condition the predictions on the tar-get object. Specifically, this is achieved by modulating theactivations of the Segmentation Network (SN) via a singleforward pass of a Modulating Network (MN) that outputsa scale parameter for each channel based on the object fea-tures, and a shift parameter for each location that acts as aspatial attention mechanism similar to [12]. Here the SNlearns to perform generic segmentation and the MN fast-adapts it to attend to the object of interest.In this paper we introduce ReConvNet, a recurrent convo-lutional architecture for semi-supervised video object seg-mentation. Our architecture self-adapts to segment unseen1arXiv:1806.05510v2  [cs.CV]  18 Jun 2018framet VGG1 VGG2 VGG3 VGG4 VGG5mt blobtCNN 1x1β3 β4 β5VGGv1 VGGv2 VGGv3 frame0   m0 cframe0 MLPv6up and  fuseɣ ɣb3 ɣb4 ɣb5MLPv7 MLPv8cLSTM3 cLSTM4 cLSTM5VGGv4 VGGv5ɣc3 ɣc4 ɣc5CNN 1x1CNN 1x1MLPCNNconvLSTMDownscalecropFigure 1. ReConvNet architecture. The Segmentation Network (yellow) is a VGG-16-based architecture enhanced with convLSTMlayers. The Visual Modulator (blue) biases the SN to be selective toward the object of interest via a per-channel multiplicative interaction,while the Spatial Modulator (red) enforces a spatial prior via an additive per-location modulation.object without the need of extra fine-tuning steps of supervi-sion. The main contributions of this work are the following:• We extend the successful OSMN architecture with thepossibility to model highly non-linear intrinsic tempo-ral correlations between consecutive frames via con-vLSTM units. This modification outperforms the base-line.• On DAVIS2016 we show comparable performance tomethods that make use of online fine-tuning and weoutperform them on the more challenging DAVIS2017.• We place 10-th in the DAVIS challenge 2018 withoutresorting to online fine-tuning or other post-processingsteps.• We show that features modulation is orthogonal to on-line fine-tuning and that indeed combining the two re-sults in a further performance boost.2. ArchitectureReConvNet is composed of three main components: theSegmentation Network (SN), the Visual Modulator (VM)and the Spatial Modulator (SM).Segmentation Network The original OSMN Segmenta-tion Network is a Fully Convolutional Network (FCN),based on VGG-16 [11] layers that are propagated to the de-coder after upsampling in a hyper-column fashion [4] to re-cover details at multiple scales. Hence, OSMN processeseach frame independently, often resulting in segmentationmasks that lack temporal consistency and exhibit high vari-ance across the sequence.A natural way to incorporate temporal structure into themodel is to add recurrent units. Here we use convLSTMlayers, an adaptation of the original LSTM [5] cell that takesinto account the spatial structure of its input. As proposedby [16], the LSTM cell can be modified by replacing thematrix multiplications of each transformation with convo-lution operations. The convLSTM blocks are interleavedto the last three VGG-16 layers to endow the network withmulti-scale spatio-temporal processing capability.Visual Modulator The visual modulator is in charge ofbiasing the activations of the segmentation network to targetthe object of interest. This strong conditioning is achievedby a VGG-16 network that takes as input the first framecropped around the target object and resized to 224 × 224,and produces a set Γ of vectors of scaling coefficients – onefor each of the last three convolutional layers of the SN. Inaddition to the coefficients computed by OSMN, we alsocompute those for the convLSTM modules. All the visualmodulation coefficients are multiplied to the feature mapsfi = γi  fi, where  indicates channel-wise multipli-cation. This has the effect of enhancing the maps relatedto the target object and suppressing the least useful, poten-tially distracting ones, allowing the segmentation networkto quickly adapt to the object of interest without resortingto expensive steps of back-propagation at inference time.Spatial Modulator To help discriminating between mul-tiple instances of the same object and, more generally, toprovide a loose prior on the location of the target object, thenetwork is also enriched with a spatial attention mechanism.A rough estimate of the position of the target object can beobtained by fitting a “gaussian blob” on the segmentationpredicted at time t and fed to a Spatial Modulator compo-nent. This, in turn, produces a set of shift coefficients β, onefor each of the last three VGG layers, via a 1×1 convolutionapplied on the blob downsampled to the layer’s resolution.Note that, as opposed to the VM case, we do not generatemodulation coefficients for the convLSTM layers. The spa-tial coefficients are summed pixel-wise to the activations ofthe corresponding layers, therefore shifting the focus on theparts of the image where the object is more likely located.The combination of VM and SM, fi = γi  fi + βi, is thenapplied to the features.3. ExperimentsIn this section we describe the experimental settings usedto evaluate the ReConvNet model.Experimental setting To ensure a fair comparison be-tween ReConvNet and the OSMN baseline we initialize thecomponents of the OSMN model in our architecture withthe pretrained weights as provided by the authors. This isdone on both DAVIS2016 and DAVIS2017, to ensure thatany improvement can be clearly attributed to the introduc-tion of a recurrent architecture.For the initialization of the remaining modules, in an ef-fort to minimize the factors of variations with respect to thebaseline, the extra channels of the visual modulator are ini-tialized as in [17] and, similarly, the input-to-hidden con-volutions in the convLSTM layers use the same initializa-tion as the convolutional layers in the baseline. Lastly, thehidden-to-hidden convolutions are initialized to be orthog-onal [9].The model is trained with a lower learning rate for thenon-recurrent than for the recurrent component, namely10−6 and 10−5 respectively. We found it beneficial to trainwith the Lovasz loss [1] that directly optimizes the IoUmeasure. Finally, to prevent overfitting we employed early-stopping and data augmentation [14] of the inputs of the vi-sual and spatial modulators with random shift, scale, and ro-tation transformations. When online fine-tuning is used, themodel is trained on each test sequence with random trans-formation of the first frame for 300 iterations and learningrate 10−6 for all components.Pretraining In order to make use of the relative abun-dance of segmented static images for pre-training, we splitthe training process in two phases. First, we train the non-recurrent components of the model on MSCOCO [6] tolearn segmentation coupled with modulation, then we trainthe full network on DAVIS2017 to account for the spatio-temporal recurrent component as well as to make use of themodulation to focus on the target object throughout the se-quence.The single frame pre-training procedure proved to bean essential proxy to bootstrap the temporal-consistent seg-mentation. In fact, DAVIS contains only a few examples formost semantic classes, making it very easy for the networkto overfit on the training examples.As it can be expected, the problem is exacerbated bymodels with high capacity, and even more by those thatexploit a visual modulator to tackle semi-supervised seg-mentation. As shown in [17] the set of parameters Γ, pro-duced by the visual modulator, pushes the model to learna semantic mapping in an embedding space where visuallysimilar objects are close in `2 distance. Learning this map-ping requires a large enough amount of diverse examples.We choose the MSCOCO [6] dataset for its wide range ofclasses and intra-class variations.3.1. Single Object SegmentationWe first evaluate our model on DAVIS 2016, that focuseson single objects. This is a hard task that allows us to val-idate the model and to compare with the OSMN baseline.As shown in Table 1, thanks to the combination of spatio-temporal consistency given by the convLSTM units andtheir features modulation, ReConvNet outperforms OSMNby 5.4 points on the mean IoU (J -mean) metric and is 4-thin the leaderboard of the semi-supervised approaches1 whencomparing on the average between the J and F scores.It is important to highlight that OSVOS [2], OSVOS-S[7] and onAVOS [15] perform online fine-tuning on the firstframe of the video sequence at inference time. Moreover,OSVOS utilizes a boundary snapping approach, onAVOSmakes use of a CRF post-processing step, and OSVOS-Sincorporates instance-aware semantic information from astate-of-the-art instance segmentation method to further im-prove the performance.Most of these methods introduce expensive computationsteps at inference time that are normally not needed whenresorting to features modulation. Nothing prevents thoughto pair this technique with online fine-tuning or CRF post-processing to improve performance. Indeed, with a fewsteps of fine-tuning at inference time ReConvNet gains 6.9points on the J&F-mean placing itself 0.5 points belowonAVOS, which scored 2nd in the public leaderboard.3.2. Multiple Objects SegmentationDAVIS2017 The most recent version of DAVIS intro-duces the challenging task of multiple objects segmenta-tion. On this dataset ReConvNet has been trained by feedingthe visual modulator with one randomly picked object fromthe scene at a time and using the segmentation of the sameobject in the current frame as target. Table 1 shows thatReConvNet adapts very well to the multiobject task outper-forming the baseline OSMN by 10.9 points on the J&F-mean metric. Remarkably, our method also outperformsthe state-of-the-art OSVOS and onAVOS by 5.4 and 0.3points, respectively, without the need of expensive onlinefine-tuning. Introducing online fine-tuning, theJ&F-meanimproves by 4.5, that is 2.2 points more than OSVOS-S,the current state of the art in the public leaderboard on theDAVIS2017 validation set.DAVIS Challenge 2018 We participated to the DAVISChallenge 2018 retraining on the training set augmentedwith the validation set. Our preliminary evaluation on thetest-dev set scored 52.7 and 62.9 on J&F-mean withoutand with online fine-tuning respectively, placing us 8-th inthe test-dev public leaderboard.1https://davischallenge.org/davis2016/soa_compare.htmlDAVIS2016 DAVIS2017Method FT J&F J F J&F J FM ↑ M ↑ R ↑ D ↓ M ↑ R ↑ D ↓ M ↑ M ↑ R ↑ D ↓ M ↑ R ↑ D ↓OSMN (2nd) [17] 7 - 74.0 - - - - - 54.8 52.5 60.9 21.5 57.1 66.1 24.3ReConvNet (ours) 7 78.1 79.4 89.6 7.7 76.8 86.6 7.7 65.7 62.7 70.5 21.6 68.7 77.3 21.6OSVOS [2] 3 80.2 79.8 93.6 14.9 80.6 92.6 15.0 60.3 56.6 63.8 26.1 63.9 73.8 27.0onAVOS [15] 3 85.5 86.1 96.1 5.2 84.9 89.7 5.8 65.4 61.6 67.4 27.9 69.1 75.4 26.6OSVOS-S [7] 3 86.6 85.6 96.8 5.5 87.5 95.9 8.2 68.0 64.7 74.2 15.1 71.3 80.7 18.5ReConvNet (ours) 3 85.0 85.4 95.9 8.5 84.6 93.9 12.1 70.2 66.6 75.4 28.1 73.7 83.1 29.6Table 1. Comparisons of our approach vs OSMN baseline and top-3 state-of-the-art algorithms on DAVIS 2016 and 2017 validation sets.Legend. FT: Online fine-tuning on the first frame; M: Mean; R: Recall; D: Decay.On the test-challenge set ReConvNet scored 54.5 J&F-mean, and 51.8 and 57.2 J -mean and F-mean, respec-tively, ranking 10-th in the final DAVIS Challenge 2018evaluation. This is an encouraging result considering thatno online fine-tuning was employed: by adding gradientsteps at inference time it is reasonable to expect a perfor-mance boost similar to the one consistently witnessed in theprevious experiments.4. Conclusions and Future workWe presented ReConvNet, a powerful and efficient recur-rent convolutional model to perform semi-supervised videoobject segmentation. The model is able to learn spatio-temporal features that self-adapt to focus on the object ofinterest without the need of extra fine-tuning at inferencetime. ReConvNet outperforms the baseline by a consider-able margin, proving the effectiveness of incorporating tem-poral consistency into the model. Our results reinforce theconjecture that features modulation is a valid approach tosemi-supervised video object segmentation. We plan to per-form a more in-depth analysis of the interaction between thetemporal components and the features modulation, since webelieve it is crucial to better understand the potential of theproposed model.AcknowledgementsWe thank Jürgen Schmidhuber and Imanol Schlag forhelpful discussions on fast weights and Razvan Pascanu forinsightful comments on the model. We are also grateful toAGS SpA for providing the NVIDIA 1080Ti machine to runall the experiments. Finally, our thoughts go to Aaron, Adri-ana and Michal, for giving the initial thrust to this work.References[1] M. Berman, A. R. Triki, and M. B. Blaschko. The lovász-softmax loss: A tractable surrogate for the optimizationof the intersection-over-union measure in neural networks.arXiv preprint arXiv:1705.08790, 2017. 3[2] S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taixé,D. Cremers, and L. Van Gool. One-shot video object seg-mentation. In CVPR, 2017. 1, 3, 4[3] D. Ha, A. Dai, and Q. V. Le. Hypernetworks. arXiv preprintarXiv:1609.09106, 2016. 1[4] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik. Hyper-columns for object segmentation and fine-grained localiza-tion. In CVPR, pages 447–456, 2015. 2[5] S. Hochreiter and J. Schmidhuber. Long short-term memory.Neural computation, 9(8):1735–1780, 1997. 2[6] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra-manan, P. Dollár, and C. L. Zitnick. Microsoft coco: Com-mon objects in context. In ECCV, 2014. 3[7] K.-K. Maninis, S. Caelles, Y. Chen, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool. Video object segmen-tation without temporal information. TPAMI, 2018. 3, 4[8] E. Perez, F. Strub, H. De Vries, V. Dumoulin, andA. Courville. Film: Visual reasoning with a general con-ditioning layer. arXiv preprint arXiv:1709.07871, 2017. 1[9] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact so-lutions to the nonlinear dynamics of learning in deep linearneural networks. arXiv preprint arXiv:1312.6120, 2013. 3[10] J. Schmidhuber. Learning to control fast-weight memories:An alternative to dynamic recurrent networks. Neural Com-putation, 4(1):131–139, 1992. 1[11] K. Simonyan and A. Zisserman. Very deep convolutionalnetworks for large-scale image recognition. arXiv preprintarXiv:1409.1556, 2014. 2[12] M. F. Stollenga, J. Masci, F. Gomez, and J. Schmidhuber.Deep networks with internal selective attention through feed-back connections. In Advances in Neural Information Pro-cessing Systems 27. 2014. 1[13] F. Visin, M. Ciccone, A. Romero, K. Kastner, K. Cho,Y. Bengio, M. Matteucci, and A. Courville. Reseg: A recur-rent neural network-based model for semantic segmentation.In CVPR Workshops, June 2016. 1[14] F. Visin and A. Romero. Dataset loaders: a python library toload and preprocess datasets. https://github.com/fvisin/dataset_loaders, 2017. 3[15] P. Voigtlaender and B. Leibe. Online adaptation of convo-lutional neural networks for video object segmentation. InBMVC, 2017. 1, 3, 4[16] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong,and W.-c. Woo. Convolutional lstm network: A machinelearning approach for precipitation nowcasting. In NIPS,2015. 2[17] L. Yang, Y. Wang, X. Xiong, J. Yang, and A. K. Katsaggelos.Efficient video object segmentation via network modulation.arXiv preprint arXiv:1802.01218, 2018. 1, 3, 4

ReConvNet: Video Object Segmentation with Spatio-Temporal Features Modulation

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Politecnico di Milano