Pose estimation system based on monocular cameras

Our world is full of wonders. It is filled with mysteries and challenges, which through the ages inspired and called for the human civilization to grow itself, either philosophically or sociologically. In time, humans reached their own physical limitations; nevertheless, we created technology to help us overcome it. Like the ancient uncovered land, we are pulled into the discovery and innovation of our time. All of this is possible due to a very human characteristic - our imagination. The world that surrounds us is mostly already discovered, but with the power of computer vision (CV) and augmented reality (AR), we are able to live in multiple hidden universes alongside our own. With the increasing performance and capabilities of the current mobile devices, AR is what we dream it can be. There are still many obstacles, but this future is already our reality, and with the evolving technologies closing the gap between the real and the virtual world, soon it will be possible for us to surround ourselves into other dimensions, or fuse them with our own. This thesis focuses on the development of a system to predict the camera’s pose estimation in the real-world regarding to the virtual world axis. The work was developed as a sub-module integrated on the M5SAR project: Mobile Five Senses Augmented Reality System for Museums, aiming to a more immerse experience with the total or partial replacement of the environments’ surroundings. It is based mainly on man-made buildings indoors and their typical rectangular cuboid shape. With the possibility of knowing the user’s camera direction, we can then superimpose dynamic AR content, inviting the user to explore the hidden worlds. The M5SAR project introduced a new way to explore the existent historical museums by exploring the human’s five senses: hearing, smell, taste, touch, vision. With this innovative technology, the user is able to enhance their visitation and immerse themselves into a virtual world blended with our reality. A mobile device application was built containing an innovating framework: MIRAR - Mobile Image Recognition based Augmented Reality - containing object recognition, navigation, and additional AR information projection in order to enrich the users’ visit, providing an intuitive and compelling information regarding the available artworks, exploring the hearing and vision senses. A device specially designed was built to explore the additional three senses: smell, taste and touch which, when attached to a mobile device, either smartphone or tablet, would pair with it and automatically react in with the offered narrative related to the artwork, immersing the user with a sensorial experience. As mentioned above, the work presented on this thesis is relative to a sub-module of the MIRAR regarding environment detection and the superimposition of AR content. With the main goal being the full replacement of the walls’ contents, and with the possibility of keeping the artwork visible or not, it presented an additional challenge with the limitation of using only monocular cameras. Without the depth information, any 2D image of an environment, to a computer doesn’t represent the tridimensional layout of the real-world dimensions. Nevertheless, man-based building tends to follow a rectangular approach to divisions’ constructions, which allows for a prediction to where the vanishing point on any environment image may point, allowing the reconstruction of an environment’s layout from a 2D image. Furthermore, combining this information with an initial localization through an improved image recognition to retrieve the camera’s spatial position regarding to the real-world coordinates and the virtual-world, alas, pose estimation, allowed for the possibility of superimposing specific localized AR content over the user’s mobile device frame, in order to immerse, i.e., a museum’s visitor into another era correlated to the present artworks’ historical period. Through the work developed for this thesis, it was also presented a better planar surface in space rectification and retrieval, a hybrid and scalable multiple images matching system, a more stabilized outlier filtration applied to the camera’s axis, and a continuous tracking system that works with uncalibrated cameras and is able to achieve particularly obtuse angles and still maintain the surface superimposition. Furthermore, a novelty method using deep learning models for semantic segmentation was introduced for indoor layout estimation based on monocular images. Contrary to the previous developed methods, there is no need to perform geometric calculations to achieve a near state of the art performance with a fraction of the parameters required by similar methods. Contrary to the previous work presented on this thesis, this method performs well even in unseen and cluttered rooms if they follow the Manhattan assumption. An additional lightweight application to retrieve the camera pose estimation is presented using the proposed method.O nosso mundo está repleto de maravilhas. Está cheio de mistérios e desafios, os quais, ao longo das eras, inspiraram e impulsionaram a civilização humana a evoluir, seja filosófica ou sociologicamente. Eventualmente, os humanos foram confrontados com os seus limites físicos; desta forma, criaram tecnologias que permitiram superá-los. Assim como as terras antigas por descobrir, somos impulsionados à descoberta e inovação da nossa era, e tudo isso é possível graças a uma característica marcadamente humana: a nossa imaginação. O mundo que nos rodeia está praticamente todo descoberto, mas com o poder da visão computacional (VC) e da realidade aumentada (RA), podemos viver em múltiplos universos ocultos dentro do nosso. Com o aumento da performance e das capacidades dos dispositivos móveis da atualidade, a RA pode ser exatamente aquilo que sonhamos. Continuam a existir muitos obstáculos, mas este futuro já é o nosso presente, e com a evolução das tecnologias a fechar o fosso entre o mundo real e o mundo virtual, em breve será possível cercarmo-nos de outras dimensões, ou fundi-las dentro da nossa. Esta tese foca-se no desenvolvimento de um sistema de predição para a estimação da pose da câmara no mundo real em relação ao eixo virtual do mundo. Este trabalho foi desenvolvido como um sub-módulo integrado no projeto M5SAR: Mobile Five Senses Augmented Reality System for Museums, com o objetivo de alcançar uma experiência mais imersiva com a substituição total ou parcial dos limites do ambiente. Dedica-se ao interior de edifícios de arquitetura humana e a sua típica forma de retângulo cuboide. Com a possibilidade de saber a direção da câmara do dispositivo, podemos então sobrepor conteúdo dinâmico de RA, num convite ao utilizador para explorar os mundos ocultos. O projeto M5SAR introduziu uma nova forma de explorar os museus históricos existentes através da exploração dos cinco sentidos humanos: a audição, o cheiro, o paladar, o toque e a visão. Com essa tecnologia inovadora, o utilizador pode engrandecer a sua visita e mergulhar num mundo virtual mesclado com a nossa realidade. Uma aplicação para dispositivo móvel foi criada, contendo uma estrutura inovadora: MIRAR - Mobile Image Recognition based Augmented Reality - a possuir o reconhecimento de objetos, navegação e projeção de informação de RA adicional, de forma a enriquecer a visita do utilizador, a fornecer informação intuitiva e interessante em relação às obras de arte disponíveis, a explorar os sentidos da audição e da visão. Foi também desenhado um dispositivo para exploração em particular dos três outros sentidos adicionais: o cheiro, o toque e o sabor. Este dispositivo, quando afixado a um dispositivo móvel, como um smartphone ou tablet, emparelha e reage com este automaticamente com a narrativa relacionada à obra de arte, a imergir o utilizador numa experiência sensorial. Como já referido, o trabalho apresentado nesta tese é relativo a um sub-módulo do MIRAR, relativamente à deteção do ambiente e a sobreposição de conteúdo de RA. Sendo o objetivo principal a substituição completa dos conteúdos das paredes, e com a possibilidade de manter as obras de arte visíveis ou não, foi apresentado um desafio adicional com a limitação do uso de apenas câmaras monoculares. Sem a informação relativa à profundidade, qualquer imagem bidimensional de um ambiente, para um computador isso não se traduz na dimensão tridimensional das dimensões do mundo real. No entanto, as construções de origem humana tendem a seguir uma abordagem retangular às divisões dos edifícios, o que permite uma predição de onde poderá apontar o ponto de fuga de qualquer ambiente, a permitir a reconstrução da disposição de uma divisão através de uma imagem bidimensional. Adicionalmente, ao combinar esta informação com uma localização inicial através de um reconhecimento por imagem refinado, para obter a posição espacial da câmara em relação às coordenadas do mundo real e do mundo virtual, ou seja, uma estimativa da pose, foi possível alcançar a possibilidade de sobrepor conteúdo de RA especificamente localizado sobre a moldura do dispositivo móvel, de maneira a imergir, ou seja, colocar o visitante do museu dentro de outra era, relativa ao período histórico da obra de arte em questão. Ao longo do trabalho desenvolvido para esta tese, também foi apresentada uma melhor superfície planar na recolha e retificação espacial, um sistema de comparação de múltiplas imagens híbrido e escalável, um filtro de outliers mais estabilizado, aplicado ao eixo da câmara, e um sistema de tracking contínuo que funciona com câmaras não calibradas e que consegue obter ângulos particularmente obtusos, continuando a manter a sobreposição da superfície. Adicionalmente, um algoritmo inovador baseado num modelo de deep learning para a segmentação semântica foi introduzido na estimativa do traçado com base em imagens monoculares. Ao contrário de métodos previamente desenvolvidos, não é necessário realizar cálculos geométricos para obter um desempenho próximo ao state of the art e ao mesmo tempo usar uma fração dos parâmetros requeridos para métodos semelhantes. Inversamente ao trabalho previamente apresentado nesta tese, este método apresenta um bom desempenho mesmo em divisões sem vista ou obstruídas, caso sigam a mesma premissa Manhattan. Uma leve aplicação adicional para obter a posição da câmara é apresentada usando o método proposto

Decision Support Based on Bio-PEPA Modeling and Decision Tree Induction: A New Approach, Applied to a Tuberculosis Case Study

The problem of selecting determinant features generating appropriate model structure is a challenge in epidemiological modelling. Disease spread is highly complex, and experts develop their understanding of its dynamic over years. There is an increasing variety and volume of epidemiological data which adds to the potential confusion. We propose here to make use of that data to better understand disease systems. Decision tree techniques have been extensively used to extract pertinent information and improve decision making. In this paper, we propose an innovative structured approach combining decision tree induction with Bio-PEPA computational modelling, and illustrate the approach through application to tuberculosis. By using decision tree induction, the enhanced Bio-PEPA model shows considerable improvement over the initial model with regard to the simulated results matching observed data. The key finding is that the developer expresses a realistic predictive model using relevant features, thus considering this approach as decision support, empowers the epidemiologist in his policy decision making

POISED: Spotting Twitter Spam Off the Beaten Paths

Cybercriminals have found in online social networks a propitious medium to spread spam and malicious content. Existing techniques for detecting spam include predicting the trustworthiness of accounts and analyzing the content of these messages. However, advanced attackers can still successfully evade these defenses. Online social networks bring people who have personal connections or share common interests to form communities. In this paper, we first show that users within a networked community share some topics of interest. Moreover, content shared on these social network tend to propagate according to the interests of people. Dissemination paths may emerge where some communities post similar messages, based on the interests of those communities. Spam and other malicious content, on the other hand, follow different spreading patterns. In this paper, we follow this insight and present POISED, a system that leverages the differences in propagation between benign and malicious messages on social networks to identify spam and other unwanted content. We test our system on a dataset of 1.3M tweets collected from 64K users, and we show that our approach is effective in detecting malicious messages, reaching 91% precision and 93% recall. We also show that POISED's detection is more comprehensive than previous systems, by comparing it to three state-of-the-art spam detection systems that have been proposed by the research community in the past. POISED significantly outperforms each of these systems. Moreover, through simulations, we show how POISED is effective in the early detection of spam messages and how it is resilient against two well-known adversarial machine learning attacks

Sediment Dynamics and Channel Connectivity on Hillslopes

The pattern, magnitude, and frequency of hillslope erosion and deposition are spatially varied under the influence of micro-topography and channel geometry. This research investigates the interrelationships between erosion/deposition, micro-topography, and channel connectivity on a hillslope in Loudon, Tennessee using the centimeter (cm) level temporal Digital Elevation Models collected using laser scanning. This research addressed (1) the effect of spatial resolution on the erosion/deposition quantification, and rill delineation; (2) the influences of micro-topographic factors (e.g. slope, roughness, aspect) on erosion and deposition; (3) the relationship between the structural connectivity -- depressions and confluence of rills -- and the sedimentological connectivity. I conducted (1) visual and quantitative assessments for the erosion and deposition, and the revised automated proximity and conformity analysis for the rill network; (2) quantile regression for micro-topographic factors using segmented rill basins; and (3) cross-correlation analysis using erosion and deposition series along the channels.Overall, rills are sedimentologically more dynamic than the interrill areas. A larger grid size reduces the detectable changes in both areal and volumetric quantities, and also decreases the total length and number of rills. The offset between delineated rills and the reference increases with larger grid sizes. A larger rill basin has higher erosion and deposition with the magnitude of erosion greater than deposition. The slope has a positive influence on erosion and a negative one on deposition; roughness has a positive influence on deposition and a negative one on erosion. Areas that are more north-facing experience higher erosion and lower deposition. Rill length explains 46% of the variability for erosion and 24% for deposition. The depressions are associated with higher erosion in the downslope direction. The correlations between the erosion and the confluence are positive; the correlation between the deposition and the sink is positive. Overall, the influence of structural connectivity on the sedimentological connectivity is within 25 cm in both upstream and downstream directions. This research contributes to the understanding in how the sediment movement on hillslopes is governed by topographic variations and channel connectivity, and future work may explore hillslope channels at broader geographical and temporal scales

Gyr cows (Bos taurus indicus) in the peripartum period: assessment of calving prediction devices and factors affecting the maternal behavior and defensiveness

The aim of this study was to evaluate the potential use of the reticulo-rumen temperature and activity pattern of Gyr heifers as calving predictors and characterize the defensiveness and maternal care of primiparous and multiparous Gyr cows, evaluating the possible effects of parity and training protocol to the first milking prior calving toward these behaviors. Fifty-two Gir Leiteiro cows from the Empresa de Pesauissa Agropecuária de Minas Gerais (Epamig Oeste, Uberaba, Brazil) were used. All samples came from the same herd divided into three experimental groups, one for each chapter. (Chapter I): Forty pregnant Gir heifers received an intra-ruminal bolus that recorded reticulo-rumen temperature (Trr) and activity (Act). The animals had Trr and Act monitored during the prepartum period. We observed a decrease in Trr and an increase in Act in the days before calving. Differences in Trr and Act were most evident during the last 21 and 11 hours before parturition, respectively. There was a drop of 0.20°C in Trr. The analyzes revealed that both characteristics have the potential to predict parturition, however particularities in the thermal physiology of Zebu cattle must be considered when using devices validated only for European breeds. (Chapter II): Thirty-one Gir cows, among primiparous (n = 16) and multiparous (n = 15) were allocated in a maternity paddock monitored by video cameras. The behavior of the animals was collected in four periods: Pre-calving, Post-calving, First handling of calf and Post-handling. Primiparous cows showed a longer duration of standing with an arched spine and tended to move more than multiparous cows in the pre-calving period, which can be considered an indicator of pain and discomfort in these animals. Both primiparous and multiparous cows were protective of their calves, but only multiparous females were aggressive towards the handlers in the first calf handling. Furthermore, more protective cows spent more time eating before calving, while less attentive cows spent more time lying down before calving. (Chapter III): Thirty-seven primiparous dairy Gyr cows were allocated into two groups: Training Group (n = 16) was submitted to a protocol for the first milking, involving tactile stimulation; Control group (n = 21) was submitted to the common management of the farm, without interactions and/or additional handling. Animal behavior was recorded in three periods: Post-calving, First handling of calf and Post-handling. Calf latency to stand up, weight, and sex influenced cow-calf interactions, whereas training group cows touched less and spent more time not interacting with their calves. Both Training and Control groups had protective dams, but a higher percentage of Trained group dams were calmer toward calf handling. In conclusion, Trr and Act had potential to calving prediction in Gyr Heifers; Multiparous Gyr cows tended be more aggressive with their calves’ defense than primiparous; Training protocol to the first milking involving tactile stimulation reduced maternal care and defensiveness in primiparous Gyr cows.O objetivo desta tese foi avaliar o potencial uso dos padrões de temperatura retículo-ruminal e atividade de novilhas Gir como preditores de parto e caracterizar a defesa e cuidado materno de vacas Gir, avaliando os possíveis efeitos da paridade e do protocolo de treinamento para a primeira ordenha nestes comportamentos. Foram utilizadas cinquenta e duas vacas Gir Leiteiro da Empresa de Pesquisa Agropecuária d Minas Gerais (Epamig Oeste, Uberaba, Brasil). Todas as amostras foram oriundas de um mesmo rebanho dividido em três grupos experimentais, um para cada capítulo. (Capítulo I): Quarenta novilhas Gir prenhes receberam um bolus intra-ruminal que registrou a temperatura retículo-rúmen (Trr) e atividade (Act). Os animais tiveram Trr e Act monitorados durante o período pré-parto. Observamos diminuição do Trr e aumento do Act nos dias que antecederam o parto. As diferenças em Trr e Act foram mais evidentes durante as últimas 21 e 11 horas antes do parto, respectivamente. Houve queda de 0,20°C na Trr. As análises revelaram que ambas as características têm potencial para predizer o parto, porém particularidades na fisiologia térmica de bovinos zebuínos devem ser consideradas quando se utilizam dispositivos validados apenas para raças europeias. (Capítulo II): Trinta e uma vacas Gir, dentre primíparas (n = 16) e multíparas (n = 15) foram alocadas em um piquete de maternidade monitorado por câmeras de vídeo. Os comportamentos dos animais foram coletados em quatro períodos: Pré-parto, Pós-parto, Primeiro manejo do bezerro e Pós-manejo. As vacas primíparas apresentaram maior duração dos comportamentos de ficar em pé com a coluna arqueada e tenderam a se movimentar mais do que as multíparas no período pré-parto, o que pode ser considerado indicador de dor e desconforto nesses animais. Tanto as primíparas quanto as multíparas foram protetoras de seus bezerros, mas apenas as multíparas foram agressivas com os tratadores no primeiro manejo do bezerro. Além disso, vacas mais protetoras passaram mais tempo comendo antes do parto, enquanto vacas menos atentas passaram mais tempo deitadas antes do parto. (Capítulo III): Trinta e sete vacas Gir leiteiras primíparas foram alocadas em dois grupos: O Grupo Treinamento (n = 16) foi submetido a um protocolo para a primeira ordenha, envolvendo estimulação tátil; O Grupo de controle (n = 21) foi submetido ao manejo comum da fazenda, sem interações e/ou manejos adicionais. Os comportamentos dos animais foram registrados em três períodos: pós-parto, primeiro manejo do bezerro e pós-manejo. A latência do bezerro para se levantar, o peso e o sexo influenciaram as interações vaca-bezerro. Vacas do Grupo Treinamento tocaram menos e passaram mais tempo sem interagir com seus bezerros. Ambos os grupos de treinamento e controle tinham mães protetoras, mas uma porcentagem maior de mães do Grupo Treinamento foram mais calmas em relação ao manejo dos bezerros. Em conclusão, Trr e Act apresentaram potencial para predição de parto em Novilhas Gir; Vacas Gir multíparas tendem a ser mais agressivas na proteção de seus bezerros do que primíparas; Protocolo de treinamento para a primeira ordenha reduziu o cuidado e defesa materna nas vacas Gir primíparas