Collaborative Feature Learning from Social Media

Image feature representation plays an essential role in image recognition and related tasks. The current state-of-the-art feature learning paradigm is supervised learning from labeled data. However, this paradigm requires large-scale category labels, which limits its applicability to domains where labels are hard to obtain. In this paper, we propose a new data-driven feature learning paradigm which does not rely on category labels. Instead, we learn from user behavior data collected on social media. Concretely, we use the image relationship discovered in the latent space from the user behavior data to guide the image feature learning. We collect a large-scale image and user behavior dataset from Behance.net. The dataset consists of 1.9 million images and over 300 million view records from 1.9 million users. We validate our feature learning paradigm on this dataset and find that the learned feature significantly outperforms the state-of-the-art image features in learning better image similarities. We also show that the learned feature performs competitively on various recognition benchmarks

Love Thy Neighbors: Image Annotation by Exploiting Image Metadata

Some images that are difficult to recognize on their own may become more clear in the context of a neighborhood of related images with similar social-network metadata. We build on this intuition to improve multilabel image annotation. Our model uses image metadata nonparametrically to generate neighborhoods of related images using Jaccard similarities, then uses a deep neural network to blend visual information from the image and its neighbors. Prior work typically models image metadata parametrically, in contrast, our nonparametric treatment allows our model to perform well even when the vocabulary of metadata changes between training and testing. We perform comprehensive experiments on the NUS-WIDE dataset, where we show that our model outperforms state-of-the-art methods for multilabel image annotation even when our model is forced to generalize to new types of metadata.Comment: Accepted to ICCV 201

Fine-grained artworks classification

In this thesis, we apply deep convolutional neural networks to ne-grained artwork classification on the large-scale painting collection, WikiArt. We propose a new architecture that aggregates features from different convolutional layers to exploit earlier layer features. The new architecture is evaluated on the challenging fine-grained artist and year classification. We also propose a regularization method that penalizes correlations of convolutional feature maps. With the decorrelation regularization, we further improve the classification accuracy of the proposed architecture

Open-source intelligence em sistemas SIEM

Tese de mestrado, Segurança Informática, Universidade de Lisboa, Faculdade de Ciências, 2015A OSINT é uma interminável fonte de informação valiosa, em qualquer que seja o contexto, no qual exista a necessidade de lidar com ameaças humanas e imprevisíveis. A segurança informática não é excepção a esta regra e o uso de informação proveniente de canais OSINT tem-se, como temos vindo a observar com o advento da Threat Intelligence, firmado como um componente fundamental. Propomo-nos, com este trabalho, a integrar este canal de valioso conhecimento no SIEM (um paradigma também indispensável da área) de uma forma automatizada, através de uma ferramenta/framework que visa estabelecer a fundação de um instrumento extensível para recolher e reduzir grandes quantidades de informação a conjuntos, utilizáveis e úteis, de valiosos dados e conhecimentos sobre ameaças. Essa ferramenta irá recolher dados e, servindo-se de uma técnica simplista de aprendizagem de máquinas supervisionada, refiná-los, garantindo que ao SIEM apenas é passada informação relevante. Por forma a validar os nossos esforços, providenciamos provas empíricas da aplicabilidade da nossa solução, em contexto prático e real, demonstrando, efectivamente, o poder de síntese, com base em feedback do utilizador, da nossa solução. Os nossos resultados apresentam bons indicadores de que a nossa abordagem é viável e que o nosso componente é capaz de reduzir e filtrar volumes significativos de informação de redes sociais a conjuntos, manuseáveis, de informação estratégica.OSINT is a source of endlessly valuable information for all contexts that have to deal with unpredictable human threats. Computer Security is no exception to this idea and the use of OSINT for Threat Intelligence has been widely established as a fundamental component. We propose to integrate this channel of knowledge into a SIEM platform, a widely employed paradigm in the sector. We also aim to do it in an automated fashion, through a tool that tries to lay the groundwork of an extendable instrument to collect and reduce vast amounts of information to usable amounts of threat data. This tool retrieves data and, leveraging a simplistic supervised machine learning technique, refines it ensuring that the SIEM platform is to receive only relevant information. In order to validate our efforts we provide empirical evidence of the applicability of our solution, demonstrating, in practical context, its power for synthesizing information based on user-provided feedback. Our results reveal good evidence that our approach is a viable one and that our prototype is capable of reducing and filtering large volumes of social networking data, to manageable sets of intelligence