Konzepte und Modelle für die Erstellung verteilter Multimedia-Anwendungen und -Inhalte in einer Multiscreen-Umgebung

The continuing trend towards consuming media content on multiple screens such as smart TVs and smartphones is growing steadily. The key enabler for the adoption of multiscreen is the consumption of multimedia content on almost any device with a screen. It is becoming even more significant with the introduction of new media formats such as 360° videos featuring new device categories like head-mounted displays. While traditional application models focus on individual screens, investigations into concepts and models for the provisioning of multiscreen applications and multimedia content across different devices and platforms are only partially addressed. The lack of methods for modeling and conceptualizing multiscreen applications, the requirement for interoperable APIs and protocols, and the need for techniques to deliver high-quality multimedia content to devices with restricted resources are currently the main limitations for a unique multiscreen experience. This dissertation tackles these limitations and introduces a unified multiscreen application model and runtime environment targeting devices with varying characteristics and capabilities. The proposed approach applies the Separation of Concerns design principle to the multiscreen domain. It enables the composition of modular, reusable, atomic, and self-adapting components that can be dynamically migrated between devices within a multiscreen application. Thereby, different communication and distribution paradigms of application components are examined and evaluated. This work focuses primarily on multimedia applications and presents new techniques for the preparation, distribution, and playback of multimedia content in a multiscreen environment. It proposes a novel approach that enables the playback of processing-intensive content on constrained devices such as the playback of 360° videos on TV sets. The foundation of this approach is the partial pre-rendering of multimedia content and the distribution of the processing load across devices on which the application is running. This also results in a reduction of the required bitrate by up to 80% with the same image quality. We investigated open web standards as the foundation for the introduced solutions, as the web has quickly developed towards a platform for multimedia applications characterized by rich graphical interfaces and a high level of interactivity across multiple devices and platforms. Some results of this work have been published at international conferences and contributed to the W3C Second Screen Working Group, which defines specifications for multiscreen-related APIs and protocols. Parts of the work have also been patented.Der anhaltende Trend, Medieninhalte auf mehreren Bildschirmen wie Smart TVs und Smartphones zu konsumieren, nimmt stetig zu. Der Hauptfaktor für den Einsatz von Multiscreen ist der Konsum von Multimedia-Inhalten auf fast jedem Gerät mit einem Screen. Sie gewinnt mit der Einführung neuer Medienformate wie 360° Videos und neuen Gerätekategorien wie Head Mounted Displays noch mehr an Bedeutung. Während traditionelle Applikationsmodelle sich auf einzelne Screens beschränken, wurden Konzepte und Modelle zur Bereitstellung von Multiscreen Anwendungen und multimedialen Inhalten über verschiedene Geräte und Plattformen hinweg nur teilweise untersucht. Das Fehlen von Methoden zur Modellierung und Konzeption von Multiscreen Anwendungen, die Anforderungen an interoperable APIs und Protokolle, sowie die Notwendigkeit, hochwertige Multimedia-Inhalte für Geräte mit begrenzten Ressourcen bereitzustellen, sind derzeit die wichtigsten Einschränkungen für ein durchgängiges Multiscreen-Erlebnis. Diese Dissertation befasst sich mit diesen Einschränkungen und stellt ein einheitliches Multiscreen-Anwendungsmodell und eine Laufzeitumgebung für Geräte mit unterschiedlichen Eigenschaften und Fähigkeiten vor. Der vorgeschlagene Ansatz wendet das Separation of Concerns Design-Prinzip auf die Multiscreen-Domäne an. Es ermöglicht die Verwendung modularer, wiederverwendbarer, atomarer und sich selbstanpassender Komponenten, die innerhalb einer Multiscreen-Anwendung dynamisch zwischen Geräten migriert werden können. Dabei werden verschiedene Kommunikations- und Distributionsparadigmen von Anwendungskomponenten erforscht und bewertet. Diese Arbeit konzentriert sich in erster Linie auf Multimedia Anwendungen und stellt neue Techniken zur Aufbereitung, Verteilung und Wiedergabe von MultimediaInhalten in einer Multiscreen-Umgebung vor. Sie schlägt einen innovativen Ansatz vor, der die Wiedergabe von rechenintensiven Inhalten auf Geräten mit eingeschränkten Ressourcen wie der Wiedergabe von 360° Videos auf TV Geräten ermöglicht. Grundlage dieses Ansatzes ist die partielle Prerendering von Multimedia-Inhalten und die Verteilung der Verarbeitungslast auf die Geräte, auf denen die Anwendung läuft. Dies führt auch zu einer Reduzierung der erforderlichen Bitrate um bis zu 80% bei gleicher Bildqualität. Dazu werden offene Webstandards als Grundlage für die vorgestellten Lösungsansätze untersucht, da sich das Web schnell zu einer Plattform für Multimedia Anwendungen entwickelt hat, die sich durch vielfältige grafische Oberflächen und ein hohes Maß an Interaktivität über mehrere Plattformen hinweg auszeichnet. Einige Ergebnisse dieser Arbeit wurden auf internationalen Konferenzen veröffentlicht und in die W3C Second Screen Working Group eingebracht, die Spezifikationen für Multiscreen-bezogene APIs und Protokolle definiert. Teile der Arbeit wurden patentiert

Streaming and playback of 16k 360° videos on the web

Major platforms such as YouTube and Facebook have introduced 360° video streaming. We shortly introduce state-of-the art technology to deliver 360° video and limitations that come with it. Currently, most 360° videos offer SD (standard definition) Field of View (FOV), which significantly limits the immersive experience for the user. Bandwidth limitations, end device constraints and lack of higher resolution 360° cameras prevent FOV with better quality to be delivered. In this paper, we explain the content generation process for a 16k 360° equirectangular video, which enables a 4k FOV. We leverage MPEG DASH tools for efficient and interoperable playback of 360° videos across devices. We show that with this solution, 4k FOV can be efficiently delivered to various end devices already today such as TVs, smartphones and tablets

Towards a remote launch mechanism of TV companion applications using iBeacon

TV sets and companion devices (Smartphones, Tablets, etc.) have outgrown their original purpose and are now playing together an important role to offer the best user experience on multiple screens. However, the collaboration between TV and companion applications faces challenges that go beyond traditional single screen applications. These range from discovery, wake-up and pairing of devices, to application launch, communication, synchronization and adaptation to target device and screen size. In this position paper, we will limit ourselves to the Remote Launch aspect and introduce an idea for a new mechanism for launching companion applications using iBeacon technology

Interactive 360° narrative for TV use

Given appropriate playout technologies, 360° video can be shown on regular SmartTVs (HbbTV), even as live video. However, limitations of TVs restrict the amount of interactivity that can be provided on top of the 360° navigation. The paper utilizes examples from the Hyper360 project to provide suggestions for the design of interactive 360° TV applications. One of the topics addressed is the need to consider the requirements of interactive 360° videos in pre-production and during filming, providing content that will provide a viewer experience without disruptive visual and technical effects. The paper is based on the lessons learned from the steps of building an interactive application for web and HbbTV consumption, from initial technical capability tests to the finished interactive 360° experience

Empowering Civic Participation in the Policy Making Process through Social Media

The Web as medium has a high significance in everyday life of the digital society. The circumstance of growing visibility of social media covers a high potential in the range of a more citizen-centric and socially-rooted policy making. These potentials call for novel tools with the capability to analyze society’s input and predict the possible impact of policies. The paper describes a prototype tool set for policy makers that utilizes social media technologies and methods to empower public engagement, enable cross-media platform publishing, feedback tracking / analysis and provide decision support

High quality 360° video rendering and streaming: Paper presented at the 9th NEM Summit 2016, 23-25 November 2016, Porto

This paper introduces a new solution that facilitates the deployment and consumption of immersive and interactive media by ensuring optimal network delivery and media playback. We will address in this paper two main challenges a) the efficient streaming of high quality 360° video content using existing content delivery networks (CDNs) and without the need for additional bandwidth comparing to traditional video streaming and b) the playback of 360° content even on devices with limited processing resources and programmatic capabilities