CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Socket Intents: OS Support for Using Multiple Access Networks and its Benefits for Web Browsing

In today's Internet, mobile devices are connected to multiple access networks, e.g., WiFi/DSL and LTE. To take advantage of the networks' diverse paths characteristics (delay, bandwidth, and reliability) and aggregate bandwidth, we need smart strategies for choosing which interface(s) to use for what traffic. In this paper, we present an approach how to tackle this challenge as part of the Operating System (OS): With the concept of Socket Intents, applications can express what they know about their communication pattern and their preferences. Using our Socket Intents Prototype and our modified BSD Socket Interface, this information is used to choose the most appropriate path or path combination on a per message or per connection basis. We evaluate our system based on the use case of Web browsing: Using our prototype and a client-side proxy, we show the feasibility and benefits of our design. Using a flow-based simulator and a full factorial experimental design, we study a broad range of access network combinations (based on typical DSL and LTE scenarios) and real workloads (Alexa Top 100 and Top 1000 Web Sites). Our policies achieve performance benefits in more than 50% of the cases and speedups of more than factor two in 20% of the cases without adding overhead in the other cases.Comment: Submitted to IEEE/ACM Transactions on Networkin

Predictive Analytics of E-Commerce Search Behavior for Conversion

This study explores online customer search behavior on a large e-commerce website—Walmart.com. In order to more accurately predict customer purchase conversion based on their search behavior, we adopt a modern machine-learning technique, random forest, as well as logistic regression to develop two computational models. We also integrate information retrieval literature to propose metrics to quantify online consumers’ search behavior. Results show that the random forest model performs better with a very high accuracy rate (76%) in predicting customers who will purchase the item they browsed. Among all the predictors, page and session dwell time, user type, click entropy, and click position are the strongest influential factors for the conversion behavior. The findings suggest that, with the enhanced metrics and modeling approaches, search behavior could offer strong cues about customers’ purchasing decision. Additionally, the findings also suggest operational implications about how to accommodate and induce the desired search behavior with the e-commerce website

Towards enabling cross-layer information sharing to improve today's content delivery systems

Content is omnipresent and without content the Internet would not be what it is today. End users consume content throughout the day, from checking the latest news on Twitter in the morning, to streaming music in the background (while working), to streaming movies or playing online games in the evening, and to using apps (e.g., sleep trackers) even while we sleep in the night. All of these different kinds of content have very specific and different requirements on a transport—on one end, online gaming often requires a low latency connection but needs little throughput, and, on the other, streaming a video requires high throughput, but it performs quite poorly under packet loss. Yet, all content is transferred opaquely over the same transport, adhering to a strict separation of network layers. Even a modern transport protocol such as Multi-Path TCP, which is capable of utilizing multiple paths, cannot take the (above) requirements or needs of that content into account for its path selection. In this work we challenge the layer separation and show that sharing information across the layers is beneficial for consuming web and video content. To this end, we created an event-based simulator for evaluating how applications can make informed decisions about which interfaces to use delivering different content based on a set of pre-defined policies that encode the (performance) requirements or needs of that content. Our policies achieve speedups of a factor of two in 20% of our cases, have benefits in more than 50%, and create no overhead in any of the cases. For video content we created a full streaming system that allows an even finer grained information sharing between the transport and the application. Our streaming system, called VOXEL, enables applications to select dynamically and on a frame granularity which video data to transfer based on the current network conditions. VOXEL drastically reduces video stalls in the 90th-percentile by up to 97% while not sacrificing the stream's visual fidelity. We confirmed our performance improvements in a real-user study where 84% of the participants clearly preferred watching videos streamed with VOXEL over the state-of-the-art.Inhalte sind allgegenwärtig und ohne Inhalte wäre das Internet nicht das, was es heute ist. Endbenutzer konsumieren Inhalte von früh bis spät - es beginnt am Morgen mit dem Lesen der neusten Nachrichten auf Twitter, dem online hören von Musik während der Arbeit, wird fortgeführt mit dem Schauen von Filmen über Online-Streaming Dienste oder dem spielen von Mehrspieler Online Spielen am Abend, und sogar dem, mit dem Internet synchronisierten, Überwachens des eigenen Schlafes in der Nacht. All diese verschiedenen Arten von Inhalten haben sehr spezifische und unterschiedliche Ansprüche an den Transport über das Internet - auf der einen Seite sind es Online Spiele, die eine sehr geringe Latenz, aber kaum Durchsatz benötigen, auf der Anderen gibt es Video-Streaming Dienste, die einen sehr hohen Datendurchsatz benötigen, aber, sehr nur schlecht mit Paketverlust umgehen können. Jedoch werden all diese Inhalte über den selben, undurchsichtigen, Transportweg übertragen, weil an eine strikte Unterteilung der Netzwerk- und Transportschicht festgehalten wird. Sogar ein modernes Übertragungsprotokoll wie MPTCP, welches es ermöglicht mehrere Netzwerkpfade zu nutzen, kann die (oben genannten) Anforderungen oder Bedürfnisse des Inhaltes, nicht für die Pfadselektierung, in Betracht ziehen. In dieser Arbeit fordern wir die Trennung der Schichten heraus und zeigen, dass ein Informationsaustausch zwischen den Netzwerkschichten von großem Vorteil für das Konsumieren von Webseiten und Video Inhalten sein kann. Hierzu haben wir einen Ereignisorientierten Simulator entwickelt, mit dem wir untersuchten wie Applikationen eine informierte Entscheidung darüber treffen können, welche Netzwerkschnittstellen für verschiedene Inhalte, basierend auf vordefinierten Regeln, welche die Leistungsvorgaben oder Bedürfnisse eines Inhalts kodieren, benutzt werden sollen. Unsere Regeln erreichen eine Verbesserung um einen Faktor von Zwei in 20% unserer Testfälle, haben einen Vorteil in mehr als 50% der Fälle und erzeugen in keinem Fall einen Mehraufwand. Für Video Inhalte haben wir ein komplettes Video-Streaming System entwickelt, welches einen noch feingranulareren Informationsaustausch zwischen der Applikation und des Transportes ermöglicht. Unser, VOXEL genanntes, System ermöglicht es Applikationen dynamisch und auf Videobild Granularität zu bestimmen welche Videodaten, entsprechend der aktuellen Netzwerksituation, übertragen werden sollen. VOXEL kann das stehenbleiben von Videos im 90%-Perzentil drastisch, um bis zu 97%, reduzieren, ohne dabei die visuelle Qualität des übertragenen Videos zu beeinträchtigen. Wir haben unsere Leistungsverbesserung in einer Studie mit echten Benutzern bestätigt, bei der 84% der Befragten es, im vergleich zum aktuellen Stand der Technik, klar bevorzugten Videos zu schauen, die über VOXEL übertragen wurden

Sensing, interpreting, and anticipating human social behaviour in the real world

Low-level nonverbal social signals like glances, utterances, facial expressions and body language are central to human communicative situations and have been shown to be connected to important high-level constructs, such as emotions, turn-taking, rapport, or leadership. A prerequisite for the creation of social machines that are able to support humans in e.g. education, psychotherapy, or human resources is the ability to automatically sense, interpret, and anticipate human nonverbal behaviour. While promising results have been shown in controlled settings, automatically analysing unconstrained situations, e.g. in daily-life settings, remains challenging. Furthermore, anticipation of nonverbal behaviour in social situations is still largely unexplored. The goal of this thesis is to move closer to the vision of social machines in the real world. It makes fundamental contributions along the three dimensions of sensing, interpreting and anticipating nonverbal behaviour in social interactions. First, robust recognition of low-level nonverbal behaviour lays the groundwork for all further analysis steps. Advancing human visual behaviour sensing is especially relevant as the current state of the art is still not satisfactory in many daily-life situations. While many social interactions take place in groups, current methods for unsupervised eye contact detection can only handle dyadic interactions. We propose a novel unsupervised method for multi-person eye contact detection by exploiting the connection between gaze and speaking turns. Furthermore, we make use of mobile device engagement to address the problem of calibration drift that occurs in daily-life usage of mobile eye trackers. Second, we improve the interpretation of social signals in terms of higher level social behaviours. In particular, we propose the first dataset and method for emotion recognition from bodily expressions of freely moving, unaugmented dyads. Furthermore, we are the first to study low rapport detection in group interactions, as well as investigating a cross-dataset evaluation setting for the emergent leadership detection task. Third, human visual behaviour is special because it functions as a social signal and also determines what a person is seeing at a given moment in time. Being able to anticipate human gaze opens up the possibility for machines to more seamlessly share attention with humans, or to intervene in a timely manner if humans are about to overlook important aspects of the environment. We are the first to propose methods for the anticipation of eye contact in dyadic conversations, as well as in the context of mobile device interactions during daily life, thereby paving the way for interfaces that are able to proactively intervene and support interacting humans.Blick, Gesichtsausdrücke, Körpersprache, oder Prosodie spielen als nonverbale Signale eine zentrale Rolle in menschlicher Kommunikation. Sie wurden durch vielzählige Studien mit wichtigen Konzepten wie Emotionen, Sprecherwechsel, Führung, oder der Qualität des Verhältnisses zwischen zwei Personen in Verbindung gebracht. Damit Menschen effektiv während ihres täglichen sozialen Lebens von Maschinen unterstützt werden können, sind automatische Methoden zur Erkennung, Interpretation, und Antizipation von nonverbalem Verhalten notwendig. Obwohl die bisherige Forschung in kontrollierten Studien zu ermutigenden Ergebnissen gekommen ist, bleibt die automatische Analyse nonverbalen Verhaltens in weniger kontrollierten Situationen eine Herausforderung. Darüber hinaus existieren kaum Untersuchungen zur Antizipation von nonverbalem Verhalten in sozialen Situationen. Das Ziel dieser Arbeit ist, die Vision vom automatischen Verstehen sozialer Situationen ein Stück weit mehr Realität werden zu lassen. Diese Arbeit liefert wichtige Beiträge zur autmatischen Erkennung menschlichen Blickverhaltens in alltäglichen Situationen. Obwohl viele soziale Interaktionen in Gruppen stattfinden, existieren unüberwachte Methoden zur Augenkontakterkennung bisher lediglich für dyadische Interaktionen. Wir stellen einen neuen Ansatz zur Augenkontakterkennung in Gruppen vor, welcher ohne manuelle Annotationen auskommt, indem er sich den statistischen Zusammenhang zwischen Blick- und Sprechverhalten zu Nutze macht. Tägliche Aktivitäten sind eine Herausforderung für Geräte zur mobile Augenbewegungsmessung, da Verschiebungen dieser Geräte zur Verschlechterung ihrer Kalibrierung führen können. In dieser Arbeit verwenden wir Nutzerverhalten an mobilen Endgeräten, um den Effekt solcher Verschiebungen zu korrigieren. Neben der Erkennung verbessert diese Arbeit auch die Interpretation sozialer Signale. Wir veröffentlichen den ersten Datensatz sowie die erste Methode zur Emotionserkennung in dyadischen Interaktionen ohne den Einsatz spezialisierter Ausrüstung. Außerdem stellen wir die erste Studie zur automatischen Erkennung mangelnder Verbundenheit in Gruppeninteraktionen vor, und führen die erste datensatzübergreifende Evaluierung zur Detektion von sich entwickelndem Führungsverhalten durch. Zum Abschluss der Arbeit präsentieren wir die ersten Ansätze zur Antizipation von Blickverhalten in sozialen Interaktionen. Blickverhalten hat die besondere Eigenschaft, dass es sowohl als soziales Signal als auch der Ausrichtung der visuellen Wahrnehmung dient. Somit eröffnet die Fähigkeit zur Antizipation von Blickverhalten Maschinen die Möglichkeit, sich sowohl nahtloser in soziale Interaktionen einzufügen, als auch Menschen zu warnen, wenn diese Gefahr laufen wichtige Aspekte der Umgebung zu übersehen. Wir präsentieren Methoden zur Antizipation von Blickverhalten im Kontext der Interaktion mit mobilen Endgeräten während täglicher Aktivitäten, als auch während dyadischer Interaktionen mittels Videotelefonie

Sistema de recomendação para alívio de stress

Stress is considered to be a normal part of our lives, especially when taken into account that people are constantly trying to push their limits and the limits of others around them. Whether at home or at their jobs, the idea that to be successful one must work harder is deeply rooted within society because such behavior has shown positive results in the past. Stressful events can work as a reactor for people to feel the necessary motivation to move on with their tasks. However, if uncontrolled, may lead to health-related consequences, such as cardiovascular diseases, sleep deprivation, and anxiety. Therefore, it is important to not only recognize that stress has serious negative impacts in the lives of people but also to find mechanisms to cope with it. This document presents a solution in the form of a web application capable of providing stresseasing and health improving recommendations that adapt to the users of the application by considering their ratings from past recommendation, as well as their profiles. Through an engaging and interactive graphical user interface, users can receive personalized recommendations via system notifications. These notifications are composed of a demonstrative card and a description, as well as a collection of documents with insightful information regarding the recommendations being provided. This web application is supported by three major components, designed to operate both synchronously and asynchronously in a microservices-oriented architecture, promoting the flexibility and scalability of the solution. Furthermore, for the solution to provide recommendations, it was necessary to implement a filtering technique. Among the most common ones, the content-based filtering is the most advantageous, meaning that a content-based recommender system was developed as part of the solution. Lastly, it was concluded that the web application satisfactorily meets the established requirements. However, due to lack of user-generated data, the randomly generated data used to demonstrate how a proper evaluation would be conducted cannot be subject for interpretation.O stress é considerado parte integrante nas nossas vidas, especialmente quando consideramos que as pessoas estão constantemente a tentar superar os seus limites, muitas vezes delegando essas mesmas expectativas àqueles que as rodeiam. Tanto em casa, com os seus parceiros e familiares, como no seu local de trabalho, a ideia de que para se obter sucesso tem de se trabalhar mais está enraizada na sociedade, sendo que comportamentos semelhantes no passado comprovaram resultados positivos nesse sentido. Os eventos de stress podem funcionar como um sentimento de motivação para que as pessoas consigam desempenhar as suas funções diariamente. Contudo, caso não sejam controlados, poderão trazer consequências graves relacionadas com a saúde, tais como doenças cardiovasculares, privação do sono, e problemas de ansiedade. Assim, é importante não apenas reconhecer que o stress tem diversos impactos sérios na vida das pessoas, mas também encontrar formas de o gerir apropriadamente. Este documento apresenta uma solução na forma de uma aplicação web capaz de fornecer recomendações para redução do stress e melhoria de saúde, que se adaptam aos utilizadores da aplicação considerando as suas avaliações a recomendações no passado. Através de uma interface gráfica envolvente e interativa, os utilizadores podem receber recomendações personalizadas por meio de notificações do sistema. Essas notificações são compostas por um cartão demonstrativo e uma descrição, assim como de um conjunto de documentos com informações detalhadas acerca das recomendações fornecidas. Esta aplicação é composta por três componentes principais, desenhados para interagirem entre si tanto de forma síncrona como assíncrona numa arquitetura orientada a microserviços, promovendo a flexibilidade e escalabilidade da solução. Para além disso, para que a solução seja capaz de fornecer recomendações, foi preciso implementar uma técnica de filtragem. Entre as mais comuns, a filtragem baseada em conteúdo demonstrou ser a mais vantajosa, significando que foi desenvolvido um sistema de recomendação baseado em conteúdo como parte da solução. Finalmente, concluiu-se que a aplicação web vai satisfatoriamente de encontro aos requisitos estabelecidos. Contudo, devido à falta de dados gerados pelos utilizadores da aplicação, os dados gerados aleatoriamente para demonstrar de que forma uma avaliação adequada seria realizada, não estão sujeitos a interpretação

Evaluating and improving web performance using free-to-use tools

Abstract. Fast website loading speeds can increase conversion rates and search engine rankings as well as encourage users to explore the site further, among other positive things. The purpose of the study was to find and compare free-to-use tools that can both evaluate the performance (loading and rendering speed) of a website and give suggestions how the performance could be improved. In addition, three tools were used to evaluate the performance of an existing WordPress site. Some of the performance improvement suggestions given by the tools were then acted upon, and the performance of the website was re-evaluated using the same tools. The research method used in the study was experimental research, and the research question was “How to evaluate and improve web performance using free-to-use tools?” There were also five sub-questions, of which the first two related to the tools and their features, and the last three to the case website. Eight free-to-use web performance evaluation tools were compared focusing on what performance metrics they evaluate, what performance improvement suggestions they can give, and six other features that can be useful to know in practice. In alphabetical order, the tools were: GTmetrix, Lighthouse, PageSpeed Insights, Pingdom Tools, Test My Site, WebPageTest, Website Speed Test (by Dotcom-Tools) and Website Speed Test (by Uptrends). The amounts of metrics evaluated by the tools ranged from one to fifteen. The performance improvement suggestions given by the tools could be put into three categories, meaning that the suggestions largely overlapped between the tools. All tools except Lighthouse were web-based tools. The performance of the case website was evaluated using GTmetrix, PageSpeed Insights and WebPageTest. On desktop, the performance was in the high-end range though varying between the three tools, and on mobile, the performance was noticeably slower due to the challenges of mobile devices (e.g. lower processing power compared to desktop computers) and mobile networks (e.g. higher latency compared to broadband connections). The common bottlenecks based on the suggestions given by the three tools seemed to be lack of using a CDN (Content Delivery Network), serving unoptimized images and serving large amounts of JavaScript. The results of the performance re-evaluation were mixed, highlighting the importance of carefully considering each performance improvement suggestion. The main takeaways of the study for practitioners are to use multiple tools to get a wide variety of performance metrics and suggestions, and to regard the suggestions and relative performance scores given by the tools only as guidelines with the main goal being to improve the time-based performance metrics