First impressions: A survey on vision-based apparent personality trait analysis

© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Personality analysis has been widely studied in psychology, neuropsychology, and signal processing fields, among others. From the past few years, it also became an attractive research area in visual computing. From the computational point of view, by far speech and text have been the most considered cues of information for analyzing personality. However, recently there has been an increasing interest from the computer vision community in analyzing personality from visual data. Recent computer vision approaches are able to accurately analyze human faces, body postures and behaviors, and use these information to infer apparent personality traits. Because of the overwhelming research interest in this topic, and of the potential impact that this sort of methods could have in society, we present in this paper an up-to-date review of existing vision-based approaches for apparent personality trait recognition. We describe seminal and cutting edge works on the subject, discussing and comparing their distinctive features and limitations. Future venues of research in the field are identified and discussed. Furthermore, aspects on the subjectivity in data labeling/evaluation, as well as current datasets and challenges organized to push the research on the field are reviewed.Peer ReviewedPostprint (author's final draft

Multimodal interface for an intelligent wheelchair

Tese de mestrado integrado. Engenharia Informática e Computação. Universidade do Porto. Faculdade de Engenharia. 201

Probabilistic Human-Robot Information Fusion

This thesis is concerned with combining the perceptual abilities of mobile robots and human operators to execute tasks cooperatively. It is generally agreed that a synergy of human and robotic skills offers an opportunity to enhance the capabilities of today’s robotic systems, while also increasing their robustness and reliability. Systems which incorporate both human and robotic information sources have the potential to build complex world models, essential for both automated and human decision making. In this work, humans and robots are regarded as equal team members who interact and communicate on a peer-to-peer basis. Human-robot communication is addressed using probabilistic representations common in robotics. While communication can in general be bidirectional, this work focuses primarily on human-to-robot information flow. More specifically, the approach advocated in this thesis is to let robots fuse their sensor observations with observations obtained from human operators. While robotic perception is well-suited for lower level world descriptions such as geometric properties, humans are able to contribute perceptual information on higher abstraction levels. Human input is translated into the machine representation via Human Sensor Models. A common mathematical framework for humans and robots reinforces the notion of true peer-to-peer interaction. Human-robot information fusion is demonstrated in two application domains: (1) scalable information gathering, and (2) cooperative decision making. Scalable information gathering is experimentally demonstrated on a system comprised of a ground vehicle, an unmanned air vehicle, and two human operators in a natural environment. Information from humans and robots was fused in a fully decentralised manner to build a shared environment representation on multiple abstraction levels. Results are presented in the form of information exchange patterns, qualitatively demonstrating the benefits of human-robot information fusion. The second application domain adds decision making to the human-robot task. Rational decisions are made based on the robots’ current beliefs which are generated by fusing human and robotic observations. Since humans are considered a valuable resource in this context, operators are only queried for input when the expected benefit of an observation exceeds the cost of obtaining it. The system can be seen as adjusting its autonomy at run-time based on the uncertainty in the robots’ beliefs. A navigation task is used to demonstrate the adjustable autonomy system experimentally. Results from two experiments are reported: a quantitative evaluation of human-robot team effectiveness, and a user study to compare the system to classical teleoperation. Results show the superiority of the system with respect to performance, operator workload, and usability

Creative Writing Pedagogy: The Autobiographical Narrative in Hybrid Projects as a Means to Explore Intersectionality

My thesis addresses the role of creative writing methods in fostering close observation, attention to detail, critical thinking and a keener awareness of intersectionalities in writing classrooms across disciplines, but most especially the humanities and social sciences. I contend that the real work of the academy is critical thinking. Further, using creative writing, specifically autobiographical narrative in FYC, anticipates multimodal projects and digital storytelling, all of which fosters creative and critical thinking

Creative Writing Pedagogy: The Autobiographical Narrative in Hybrid Projects as a Means to Explore Intersectionality

My thesis addresses the role of creative writing methods in fostering close observation, attention to detail, critical thinking and a keener awareness of intersectionalities in writing classrooms across disciplines, but most especially the humanities and social sciences. I contend that the real work of the academy is critical thinking. Further, using creative writing, specifically autobiographical narrative in FYC, anticipates multimodal projects and digital storytelling, all of which fosters creative and critical thinking

Organising a photograph collection based on human appearance

This thesis describes a complete framework for organising digital photographs in an unsupervised manner, based on the appearance of people captured in the photographs. Organising a collection of photographs manually, especially providing the identities of people captured in photographs, is a time consuming task. Unsupervised grouping of images containing similar persons makes annotating names easier (as a group of images can be named at once) and enables quick search based on query by example. The full process of unsupervised clustering is discussed in this thesis. Methods for locating facial components are discussed and a technique based on colour image segmentation is proposed and tested. Additionally a method based on the Principal Component Analysis template is tested, too. These provide eye locations required for acquiring a normalised facial image. This image is then preprocessed by a histogram equalisation and feathering, and the features of MPEG-7 face recognition descriptor are extracted. A distance measure proposed in the MPEG-7 standard is used as a similarity measure. Three approaches to grouping that use only face recognition features for clustering are analysed. These are modified k-means, single-link and a method based on a nearest neighbour classifier. The nearest neighbour-based technique is chosen for further experiments with fusing information from several sources. These sources are context-based such as events (party, trip, holidays), the ownership of photographs, and content-based such as information about the colour and texture of the bodies of humans appearing in photographs. Two techniques are proposed for fusing event and ownership (user) information with the face recognition features: a Transferable Belief Model (TBM) and three level clustering. The three level clustering is carried out at “event” level, “user” level and “collection” level. The latter technique proves to be most efficient. For combining body information with the face recognition features, three probabilistic fusion methods are tested. These are the average sum, the generalised product and the maximum rule. Combinations are tested within events and within user collections. This work concludes with a brief discussion on extraction of key images for a representation of each cluster

Interfaces de fala silenciosa multimodais para português europeu com base na articulação

Doutoramento conjunto MAPi em InformáticaThe concept of silent speech, when applied to Human-Computer Interaction (HCI), describes a system which allows for speech communication in the absence of an acoustic signal. By analyzing data gathered during different parts of the human speech production process, Silent Speech Interfaces (SSI) allow users with speech impairments to communicate with a system. SSI can also be used in the presence of environmental noise, and in situations in which privacy, confidentiality, or non-disturbance are important. Nonetheless, despite recent advances, performance and usability of Silent Speech systems still have much room for improvement. A better performance of such systems would enable their application in relevant areas, such as Ambient Assisted Living. Therefore, it is necessary to extend our understanding of the capabilities and limitations of silent speech modalities and to enhance their joint exploration. Thus, in this thesis, we have established several goals: (1) SSI language expansion to support European Portuguese; (2) overcome identified limitations of current SSI techniques to detect EP nasality (3) develop a Multimodal HCI approach for SSI based on non-invasive modalities; and (4) explore more direct measures in the Multimodal SSI for EP acquired from more invasive/obtrusive modalities, to be used as ground truth in articulation processes, enhancing our comprehension of other modalities. In order to achieve these goals and to support our research in this area, we have created a multimodal SSI framework that fosters leveraging modalities and combining information, supporting research in multimodal SSI. The proposed framework goes beyond the data acquisition process itself, including methods for online and offline synchronization, multimodal data processing, feature extraction, feature selection, analysis, classification and prototyping. Examples of applicability are provided for each stage of the framework. These include articulatory studies for HCI, the development of a multimodal SSI based on less invasive modalities and the use of ground truth information coming from more invasive/obtrusive modalities to overcome the limitations of other modalities. In the work here presented, we also apply existing methods in the area of SSI to EP for the first time, noting that nasal sounds may cause an inferior performance in some modalities. In this context, we propose a non-invasive solution for the detection of nasality based on a single Surface Electromyography sensor, conceivable of being included in a multimodal SSI.O conceito de fala silenciosa, quando aplicado a interação humano-computador, permite a comunicação na ausência de um sinal acústico. Através da análise de dados, recolhidos no processo de produção de fala humana, uma interface de fala silenciosa (referida como SSI, do inglês Silent Speech Interface) permite a utilizadores com deficiências ao nível da fala comunicar com um sistema. As SSI podem também ser usadas na presença de ruído ambiente, e em situações em que privacidade, confidencialidade, ou não perturbar, é importante. Contudo, apesar da evolução verificada recentemente, o desempenho e usabilidade de sistemas de fala silenciosa tem ainda uma grande margem de progressão. O aumento de desempenho destes sistemas possibilitaria assim a sua aplicação a áreas como Ambientes Assistidos. É desta forma fundamental alargar o nosso conhecimento sobre as capacidades e limitações das modalidades utilizadas para fala silenciosa e fomentar a sua exploração conjunta. Assim, foram estabelecidos vários objetivos para esta tese: (1) Expansão das linguagens suportadas por SSI com o Português Europeu; (2) Superar as limitações de técnicas de SSI atuais na deteção de nasalidade; (3) Desenvolver uma abordagem SSI multimodal para interação humano-computador, com base em modalidades não invasivas; (4) Explorar o uso de medidas diretas e complementares, adquiridas através de modalidades mais invasivas/intrusivas em configurações multimodais, que fornecem informação exata da articulação e permitem aumentar a nosso entendimento de outras modalidades. Para atingir os objetivos supramencionados e suportar a investigação nesta área procedeu-se à criação de uma plataforma SSI multimodal que potencia os meios para a exploração conjunta de modalidades. A plataforma proposta vai muito para além da simples aquisição de dados, incluindo também métodos para sincronização de modalidades, processamento de dados multimodais, extração e seleção de características, análise, classificação e prototipagem. Exemplos de aplicação para cada fase da plataforma incluem: estudos articulatórios para interação humano-computador, desenvolvimento de uma SSI multimodal com base em modalidades não invasivas, e o uso de informação exata com origem em modalidades invasivas/intrusivas para superar limitações de outras modalidades. No trabalho apresentado aplica-se ainda, pela primeira vez, métodos retirados do estado da arte ao Português Europeu, verificando-se que sons nasais podem causar um desempenho inferior de um sistema de fala silenciosa. Neste contexto, é proposta uma solução para a deteção de vogais nasais baseada num único sensor de eletromiografia, passível de ser integrada numa interface de fala silenciosa multimodal

Click fraud : how to spot it, how to stop it?

Online search advertising is currently the greatest source of revenue for many Internet giants such as Google™, Yahoo!™, and Bing™. The increased number of specialized websites and modern profiling techniques have all contributed to an explosion of the income of ad brokers from online advertising. The single biggest threat to this growth is however click fraud. Trained botnets and even individuals are hired by click-fraud specialists in order to maximize the revenue of certain users from the ads they publish on their websites, or to launch an attack between competing businesses. Most academics and consultants who study online advertising estimate that 15% to 35% of ads in pay per click (PPC) online advertising systems are not authentic. In the first two quarters of 2010, US marketers alone spent $5.7 billion on PPC ads, where PPC ads are between 45 and 50 percent of all online ad spending. On average about$1.5 billion is wasted due to click-fraud. These fraudulent clicks are believed to be initiated by users in poor countries, or botnets, who are trained to click on specific ads. For example, according to a 2010 study from Information Warfare Monitor, the operators of Koobface, a program that installed malicious software to participate in click fraud, made over $2 million in just over a year. The process of making such illegitimate clicks to generate revenue is called click-fraud. Search engines claim they filter out most questionable clicks and either not charge for them or reimburse advertisers that have been wrongly billed. However this is a hard task, despite the claims that brokers\u27 efforts are satisfactory. In the simplest scenario, a publisher continuously clicks on the ads displayed on his own website in order to make revenue. In a more complicated scenario. a travel agent may hire a large, globally distributed, botnet to click on its competitor\u27s ads, hence depleting their daily budget. We analyzed those different types of click fraud methods and proposed new methodologies to detect and prevent them real time. While traditional commercial approaches detect only some specific types of click fraud, Collaborative Click Fraud Detection and Prevention (CCFDP) system, an architecture that we have implemented based on the proposed methodologies, can detect and prevents all major types of click fraud. The proposed solution analyzes the detailed user activities on both, the server side and client side collaboratively to better describe the intention of the click. Data fusion techniques are developed to combine evidences from several data mining models and to obtain a better estimation of the quality of the click traffic. Our ideas are experimented through the development of the Collaborative Click Fraud Detection and Prevention (CCFDP) system. Experimental results show that the CCFDP system is better than the existing commercial click fraud solution in three major aspects: 1) detecting more click fraud especially clicks generated by software; 2) providing prevention ability; 3) proposing the concept of click quality score for click quality estimation. In the CCFDP initial version, we analyzed the performances of the click fraud detection and prediction model by using a rule base algorithm, which is similar to most of the existing systems. We have assigned a quality score for each click instead of classifying the click as fraud or genuine, because it is hard to get solid evidence of click fraud just based on the data collected, and it is difficult to determine the real intention of users who make the clicks. Results from initial version revealed that the diversity of CF attack Results from initial version revealed that the diversity of CF attack types makes it hard for a single counter measure to prevent click fraud. Therefore, it is important to be able to combine multiple measures capable of effective protection from click fraud. Therefore, in the CCFDP improved version, we provide the traffic quality score as a combination of evidence from several data mining algorithms. We have tested the system with a data from an actual ad campaign in 2007 and 2008. We have compared the results with Google Adwords reports for the same campaign. Results show that a higher percentage of click fraud present even with the most popular search engine. The multiple model based CCFDP always estimated less valid traffic compare to Google. Sometimes the difference is as high as 53%. Detection of duplicates, fast and efficient, is one of the most important requirement in any click fraud solution. Usually duplicate detection algorithms run in real time. In order to provide real time results, solution providers should utilize data structures that can be updated in real time. In addition, space requirement to hold data should be minimum. In this dissertation, we also addressed the problem of detecting duplicate clicks in pay-per-click streams. We proposed a simple data structure, Temporal Stateful Bloom Filter (TSBF), an extension to the regular Bloom Filter and Counting Bloom Filter. The bit vector in the Bloom Filter was replaced with a status vector. Duplicate detection results of TSBF method is compared with Buffering, FPBuffering, and CBF methods. False positive rate of TSBF is less than 1% and it does not have false negatives. Space requirement of TSBF is minimal among other solutions. Even though Buffering does not have either false positives or false negatives its space requirement increases exponentially with the size of the stream data size. When the false positive rate of the FPBuffering is set to 1% its false negative rate jumps to around 5%, which will not be tolerated by most of the streaming data applications. We also compared the TSBF results with CBF. TSBF uses only half the space or less than standard CBF with the same false positive probability. One of the biggest successes with CCFDP is the discovery of new mercantile click bot, the Smart ClickBot. We presented a Bayesian approach for detecting the Smart ClickBot type clicks. The system combines evidence extracted from web server sessions to determine the final class of each click. Some of these evidences can be used alone, while some can be used in combination with other features for the click bot detection. During training and testing we also addressed the class imbalance problem. Our best classifier shows recall of 94%. and precision of 89%, with F1 measure calculated as 92%. The high accuracy of our system proves the effectiveness of the proposed methodology. Since the Smart ClickBot is a sophisticated click bot that manipulate every possible parameters to go undetected, the techniques that we discussed here can lead to detection of other types of software bots too. Despite the enormous capabilities of modern machine learning and data mining techniques in modeling complicated problems, most of the available click fraud detection systems are rule-based. Click fraud solution providers keep the rules as a secret weapon and bargain with others to prove their superiority. We proposed validation framework to acquire another model of the clicks data that is not rule dependent, a model that learns the inherent statistical regularities of the data. Then the output of both models is compared. Due to the uniqueness of the CCFDP system architecture, it is better than current commercial solution and search engine/ISP solution. The system protects Pay-Per-Click advertisers from click fraud and improves their Return on Investment (ROI). The system can also provide an arbitration system for advertiser and PPC publisher whenever the click fraud argument arises. Advertisers can gain their confidence on PPC advertisement by having a channel to argue the traffic quality with big search engine publishers. The results of this system will booster the internet economy by eliminating the shortcoming of PPC business model. General consumer will gain their confidence on internet business model by reducing fraudulent activities which are numerous in current virtual internet world

Engaging socially-vulnerable communities in science: exploring Science&Art approaches

Social inclusion in science is a complex issue. During the past decades, research centres, science centres, museums and other institutions invested in science communication aiming to promote cultural activities to diverse audiences. Despite this investment, science communicators from all over the world face the same challenge: how to reach citizens that are not interested in science? The main goals for this project were to explore innovative techniques to engage socially-vulnerable communities with science, and propose a model of science communication built on this practice-based research. The project, named “Embodying Memories”, was developed in a collaborative way between science partners (IGC - Instituto Gulbenkian de Ciência, iNOVA Media Lab), art partners (museum from FCG – Fundação Calouste Gulbenkian) and administrative partners (Câmara Municipal de Oeiras). The target audience, a senior community of women, most illiterate and migrant from Africa, was involved on the project plan since early stages, starting with the topic choice - Memory. The project implementation consisted of eight sessions that took place over a period of more than two months in 2018, covering several themes related to memory and brain. Diverse formats were used for the session’s activities, from scientific presentations, neuroscience stories or study cases, community memories sharing, to more interactive activities stimulating body movement, abstraction and self-expression. Besides in-door sessions at the migrant support centre, a visit to the FCG museum and a visit to IGC laboratories were organized, and a project public presentation was performed. The project was qualitatively evaluated to identify changes in awareness, knowledge, engagement, attitude and social inclusion, which was made by the analysis of field notes, attendance record, pre/post assessment focus group, community project evaluation, project narrative, and public presentation content. Overall, it was considered that the project had a moderate achievement, from a balance between very high attendance and willingness to participate in new cultural experiences, high engagement with the project, moderate increase in knowledge about neuroscience, and some increase in awareness and engagement with science, stimulation of curiosity, abstraction and self-expression. To achieve a high level of engagement, a dynamic equilibrium was constantly in a trial between the six axes of the project (science education, art education, cultural entertainment, social inclusion, mental health promotion, institutional advertising), and respective institutions. The most important project achievements were the fluidity and fruition of the project itself, and the opportunity given to participants to engage with Science & Art, to visit the museum and laboratories, to meet scientists and science instruments. A relevant asset of the project, was the existence of the boundary spanner, which was developed along pre- and during sessions by taking actions, visits, share experiences and events to inhabit the laboratory sphere, the museum sphere, and the community world. The role of the boundary spanner was crucial, yet challenging to balance between how much would be desirable for each partner to stay in and out of their comfort zones and territories. Based on insights gained from the project development and evaluation, a model was proposed to guide science communication projects using Science & Art approaches to promote social inclusion. The model entails the following phases: Phase 1. Design, plan and collaboration; Phase 2. Implementation; and Phase 3. Evaluation.A inclusão social em ciência é um tema complexo. Todavia tem-se assistido nas últimas décadas a um esforço crescente por parte das instituições de investigação científica, dos centros de ciência, museus, e outras organizações, na promoção de atividades culturais dirigidas a públicos diversificados. Apesar deste investimento, por todo o mundo os comunicadores de ciência deparam-se com o mesmo desafio: como chegar a cidadão que não estão interessados em ciência. Os objetivos principais deste projeto foram a exploração de técnicas inovadoras de envolvimento de comunidades socialmente vulneráveis em ciência e a proposta de um modelo de comunicação de ciência decorrente desta investigação de base-prática. O projeto, denominado “Dar Corpo às Memórias”, desenrolou-se de forma colaborativa entre os parceiros científicos (IGC - Instituto Gulbenkian de Ciência, iNOVA Media Lab), artísticos (museu da FCG – Fundação Calouste Gulbenkian) e administrativos (Câmara Municipal de Oeiras). O público-alvo, uma comunidade sénior de mulheres maioritariamente iletradas e migrantes de África, foi envolvido no projeto desde as fases iniciais, começando na própria escolha do tema – Memória. A fase de implementação do projeto consistiu num conjunto de oito sessões, ao longo de mais de dois meses, durante as quais foram abordados vários temas ligados à memória e ao cérebro. As atividades tiveram natureza diversa desde a apresentação de informação científica, narrativa de histórias da neurociência ou casos de estudo interessantes, partilha de memórias das participantes, até atividades mais interativas de estímulo ao movimento, à abstração e autoexpressão. Além das sessões que decorreram no centro de apoio a migrantes, foram também efetuadas duas visitas (ao museu da FCG e aos laboratórios do IGC) e uma apresentação pública do projeto. O projeto foi qualitativamente avaliado para identificar mudanças de consciencialização, conhecimento, envolvimento, atitude e inclusão social, com recurso à análise das notas de campo, registo de assiduidade, pré/pós grupos de foco, avaliação qualitativa feita pela comunidade, narrativa do projeto feita pela comunidade e conteúdo da apresentação pública. De forma global, considerou-se que o impacto do projeto foi moderado, com níveis de participação e abertura a novas experiências culturais muito elevados, elevado envolvimento com o projeto, moderado aumento de conhecimentos nas áreas das neurociências, e algum aumento de consciencialização, envolvimento com a ciência, estímulo da curiosidade, abstração e autoexpressão. Para atingir elevados níveis de envolvimento, foram efetuadas constantes tentativas de equilíbrio dinâmico entre os seis eixos do projeto (educação científica, educação artística, animação cultural, inclusão social, promoção da saúde mental, publicidade institucional) e respetivas instituições. Os maiores sucessos do projeto foram a sua própria fluidez e fruição e a oportunidade dada às participantes de envolvimento com a ciência e a arte, participação numa visita ao museu e noutra aos laboratórios, o encontro com cientistas e os instrumentos da ciência. Uma mais-valia relevante do projeto foi a existência de uma “boundary spanner” – uma pessoa facilitadora de várias valências - , que se foi desenvolvendo durante as fases pré-sessões e durante as sessões, através de ações, visitas, partilha de experiências e eventos para habitar a esfera do laboratório, a esfera do museu e o universo da comunidade. O papel do “boundary spanner” foi crucial, mas também desafiante, na medida em que requereu uma avaliação do quão cada parceiro estava disponível para permanecer ou sair da sua zona de conforto e territórios. Com base nos conhecimentos ganhos durante o desenvolvimento e avaliação deste projeto, foi proposto um modelo para projetos de comunicação de ciência para a promoção da inclusão social com recurso a abordagens de ciência & arte. O modelo consiste nas seguintes fases: Fase 1. Conceção, planeamento e colaboração; Fase 2. Implementação, e Fase 3. Avaliação