Real-time WebRTC-based design for a telepresence wheelchair

© 2017 IEEE. This paper presents a novel approach to the telepresence wheelchair system which is capable of real-time video communication and remote interaction. The investigation of this emerging technology aims at providing a low-cost and efficient way for assisted-living of people with disabilities. The proposed system has been designed and developed by deploying the JavaScript with Hyper Text Markup Language 5 (HTML5) and Web Real-time Communication (WebRTC) in which the adaptive rate control algorithm for video transmission is invoked. We conducted experiments in real-world environments, and the wheelchair was controlled from a distance using the Internet browser to compare with existing methods. The results show that the adaptively encoded video streaming rate matches the available bandwidth. The video streaming is high-quality with approximately 30 frames per second (fps) and round trip time less than 20 milliseconds (ms). These performance results confirm that the WebRTC approach is a potential method for developing a telepresence wheelchair system

A telepresence wheelchair with 360-degree vision using WebRTC

© 2020 The Author(s). This paper presents an innovative approach to develop an advanced 360-degree vision telepresence wheelchair for healthcare applications. The study aims at improving a wide field of view surrounding the wheelchair to provide safe wheelchair navigation and efficient assistance for wheelchair users. A dual-fisheye camera is mounted in front of the wheelchair to capture images which can be then streamed over the Internet. Aweb real-time communication (WebRTC) protocol was implemented to provide efficient video and data streaming. An estimation model based on artificial neural networks was developed to evaluate the quality of experience (QoE) of video streaming. Experimental results confirmed that the proposed telepresence wheelchair system was able to stream a 360-degree video surrounding the wheelchair smoothly in real-time. The average streaming rate of the entire 360-degree video was 25.83 frames per second (fps), and the average peak signal to noise ratio (PSNR) was 29.06 dB. Simulation results of the proposed QoE estimation scheme provided a prediction accuracy of 94%. Furthermore, the results showed that the designed system could be controlled remotely via the wireless Internet to follow the desired path with high accuracy. The overall results demonstrate the effectiveness of our proposed approach for the 360-degree vision telepresence wheelchair for assistive technology applications

U-DiVE: Design and evaluation of a distributed photorealistic virtual reality environment

This dissertation presents a framework that allows low-cost devices to visualize and interact with photorealistic scenes. To accomplish this task, the framework makes use of Unity’s high-definition rendering pipeline, which has a proprietary Ray Tracing algorithm, and Unity’s streaming package, which allows an application to be streamed within its editor. The framework allows the composition of a realistic scene using a Ray Tracing algorithm, and a virtual reality camera with barrel shaders, to correct the lens distortion needed for the use on an inexpensive cardboard. It also includes a method to collect the mobile device’s spatial orientation through a web browser to control the user’s view, delivered via WebRTC. The proposed framework can produce low-latency, realistic and immersive environments to be accessed through low-cost HMDs and mobile devices. To evaluate the structure, this work includes the verification of the frame rate achieved by the server and mobile device, which should be higher than 30 FPS for a smooth experience. In addition, it discusses whether the overall quality of experience is acceptable by evaluating the delay of image delivery from the server up to the mobile device, in face of user’s movement. Our tests showed that the framework reaches a mean latency around 177 (ms) with household Wi-Fi equipment and a maximum latency variation of 77.9 (ms), among the 8 scenes tested.Esta dissertação apresenta um framework que permite que dispositivos de baixo custo visualizem e interajam com cenas fotorrealísticas. Para realizar essa tarefa, o framework faz uso do pipeline de renderização de alta definição do Unity, que tem um algoritmo de rastreamento de raio proprietário, e o pacote de streaming do Unity, que permite o streaming de um aplicativo em seu editor. O framework permite a composição de uma cena realista usando um algoritmo de Ray Tracing, e uma câmera de realidade virtual com shaders de barril, para corrigir a distorção da lente necessária para usar um cardboard de baixo custo. Inclui também um método para coletar a orientação espacial do dispositivo móvel por meio de um navegador Web para controlar a visão do usuário, entregue via WebRTC. O framework proposto pode produzir ambientes de baixa latência, realistas e imersivos para serem acessados por meio de HMDs e dispositivos móveis de baixo custo. Para avaliar a estrutura, este trabalho considera a verificação da taxa de quadros alcançada pelo servidor e pelo dispositivo móvel, que deve ser superior a 30 FPS para uma experiência fluida. Além disso, discute se a qualidade geral da experiência é aceitável, ao avaliar o atraso da entrega das imagens desde o servidor até o dispositivo móvel, em face da movimentação do usuário. Nossos testes mostraram que o framework atinge uma latência média em torno dos 177 (ms) com equipamentos wi-fi de uso doméstico e uma variação máxima das latências igual a 77.9 (ms), entre as 8 cenas testadas

Investigating the Effects of Network Dynamics on Quality of Delivery Prediction and Monitoring for Video Delivery Networks

Video streaming over the Internet requires an optimized delivery system given the advances in network architecture, for example, Software Defined Networks. Machine Learning (ML) models have been deployed in an attempt to predict the quality of the video streams. Some of these efforts have considered the prediction of Quality of Delivery (QoD) metrics of the video stream in an effort to measure the quality of the video stream from the network perspective. In most cases, these models have either treated the ML algorithms as black-boxes or failed to capture the network dynamics of the associated video streams. This PhD investigates the effects of network dynamics in QoD prediction using ML techniques. The hypothesis that this thesis investigates is that ML techniques that model the underlying network dynamics achieve accurate QoD and video quality predictions and measurements. The thesis results demonstrate that the proposed techniques offer performance gains over approaches that fail to consider network dynamics. This thesis results highlight that adopting the correct model by modelling the dynamics of the network infrastructure is crucial to the accuracy of the ML predictions. These results are significant as they demonstrate that improved performance is achieved at no additional computational or storage cost. These techniques can help the network manager, data center operatives and video service providers take proactive and corrective actions for improved network efficiency and effectiveness

Designing and prototyping WebRTC and IMS integration using open source tools

WebRTC, or Web Real-time Communications, is a collection of web standards that detail the mechanisms, architectures and protocols that work together to deliver real-time multimedia services to the web browser. It represents a significant shift from the historical approach of using browser plugins, which over time, have proven cumbersome and problematic. Furthermore, it adopts various Internet standards in areas such as identity management, peer-to-peer connectivity, data exchange and media encoding, to provide a system that is truly open and interoperable. Given that WebRTC enables the delivery of multimedia content to any Internet Protocol (IP)-enabled device capable of hosting a web browser, this technology could potentially be used and deployed over millions of smartphones, tablets and personal computers worldwide. This service and device convergence remains an important goal of telecommunication network operators who seek to enable it through a converged network that is based on the IP Multimedia Subsystem (IMS). IMS is an IP-based subsystem that sits at the core of a modern telecommunication network and acts as the main routing substrate for media services and applications such as those that WebRTC realises. The combination of WebRTC and IMS represents an attractive coupling, and as such, a protracted investigation could help to answer important questions around the technical challenges that are involved in their integration, and the merits of various design alternatives that present themselves. This thesis is the result of such an investigation and culminates in the presentation of a detailed architectural model that is validated with a prototypical implementation in an open source testbed. The model is built on six requirements which emerge from an analysis of the literature, including previous interventions in IMS networks and a key technical report on design alternatives. Furthermore, this thesis argues that the client architecture requires support for web-oriented signalling, identity and call handling techniques leading to a potential for IMS networks to natively support these techniques as operator networks continue to grow and develop. The proposed model advocates the use of SIP over WebSockets for signalling and DTLS-SRTP for media to enable one-to-one communication and can be extended through additional functions resulting in a modular architecture. The model was implemented using open source tools which were assembled to create an experimental network testbed, and tests were conducted demonstrating successful cross domain communications under various conditions. The thesis has a strong focus on enabling ordinary software developers to assemble a prototypical network such as the one that was assembled and aims to enable experimentation in application use cases for integrated environments

A REVIEW STUDY OF EUROPEAN R&D PROJECTS FOR SATELLITE COMMUNICATIONS IN 5G/6G ERA

Κατά τις τελευταίες δεκαετίες τα δορυφορικά συστήματα τηλεπικοινωνιών έχουν προσφέρει μια γκάμα από πολυμεσικές υπηρεσίες όπως δορυφορική τηλεόραση, δορυφορική τηλεφωνία και ευρυζωνική πρόσβαση στο διαδίκτυο. Οι μακροπρόθεσμες τεχνολογικές αναβαθμίσεις σε συνδυασμό με την προσθήκη νέων δορυφορικών συστημάτων γεωστατικής και ελλειπτικής τροχιάς και με την ενσωμάτωση τεχνολογιών πληροφορικής έχουν ωθήσει την αύξηση του μέγιστου εύρους των δορυφόρων στο 1Gbps σε μεμονωμένους δορυφόρους ενώ σε διάταξη αστερισμού μπορούν να ξεπεράσουν το 1 Tbps. Σε συνδυασμό με την μείωση του χρόνου απόκρισης σε ρυθμούς ανταγωνιστικούς με τις χερσαίες υποδομές ανοίγουν νέες ευκαιρίες και νέους ρόλους εντός ενός οικοσυστήματος ετερογενούς δικτύων 5ης γενιάς. Σε αυτήν την διατριβή, αξιολογούμε επιδοτούμενα επιστημονικά προγράμματα έρευνας και ανάπτυξης της Ευρωπαϊκής Επιτροπής Διαστήματος (ESA) και του προγράμματος επιδότησης Horizon 2020 της Ευρωπαϊκής Ένωσης, προκειμένου να εξηγήσουμε τις δυνατότητες των δορυφόρων εντός ενός ετερογενούς δικτύου 5ης γενιάς, αναφέρουμε συγκεκριμένα αυτά που αφορούν την εξέλιξη των δορυφορικών ψηφιακών συστημάτων και την ικανότητα ενσωμάτωσης τους σε τωρινές αλλά και μελλοντικές υποδομές χερσαίων τηλεπικοινωνιακών δικτύων μέσω της εμφάνισης νέων τεχνολογιών στις ηλεκτρονικές και οπτικές επικοινωνίες αέρος μαζί με την εμφάνιση τεχνολογιών πληροφορικής όπως της δικτύωσης βασισμένης στο λογισμικό και της εικονικοποίησης λειτουργιών δικτύου. Αναφερόμαστε στους στόχους του κάθε project ξεχωριστά και κατηγοριοποιημένα στους ακόλουθους τομείς έρευνας: -Συσσωμάτωση των δορυφόρων με τα επίγεια δίκτυα 5ης γενιάς με οργανωμένες μελέτες και στρατηγικές -Ενσωμάτωση των τεχνολογιών δικτύωσης βασισμένης στο λογισμικό και εικονικοποίησης λειτουργιών δικτύου στο δορυφορικών τμήμα των δικτύων 5ης γενιάς -Ο ρόλος των δορυφόρων σε εφαρμογές του διαδικτύου των πραγμάτων σε συνάφεια με τα χερσαία δίκτυα 5ης γενιάς -Ο ρόλος των δορυφόρων στην δίκτυα διανομής πολυμεσικού περιεχομένου & η επιρροή των πρωτοκόλλων διαδικτύου στην ποιότητα υπηρεσίας χρήστη κατά την διάρκεια μιας δορυφορικής σύνδεσης. -Μελλοντικές βελτιώσεις και εφαρμογές στα δορυφορικά συστήματα με έμφαση στα μελλοντικά πρότυπα του φυσικό επιπέδου Στο τέλος διαθέτουμε ένα παράρτημα που αφορά τεχνικές αναλύσεις στην εξέλιξη του φυσικού επιπέδου των δορυφορικών συστημάτων, συνοδευόμενο με την συσχετιζόμενη βιβλιογραφία για περαιτέρω μελέτη.Over the last decades satellite telecommunication systems offer many types of multimedia services like Satellite TV, telephony and broadband internet access. The long-term technological evolutions occurred into state-of-the-art satellite systems altogether with the addition of new high throughput geostatic and non-geostatic systems, individual satellites can now achieve a peak bandwidth of up to Gbps, and with possible extension into satellite constellation systems the total capacity can reach up to Tbps. Supplementary, with systems latency being comparable to terrestrial infrastructures and with integration of several computer science technologies, satellite systems can achieve new & more advanced roles inside a heterogeneous 5G network’s ecosystem. In this thesis, we have studied European Space Agency (ESA’s) and European Union’s (EU) Horizon 2020 Research and Development (R&D) funded projects in order to describe the satellite capabilities within a 5G heterogeneous network, mentioning the impact of the evolution of digital satellite communications and furthermore the integration with the state-of the art & future terrain telecommunication systems by new technologies occurred through the evolution of electronic & free space optical communications alongside with the integration of computer science’s technologies like Software Defined Networking (SDN) and Network Function Virtualization (NFV). In order to describe this evolution we have studied the concepts of each individual project, categorized chronically and individual by its scientific field of research. Our main scientific trends for this thesis are: -Satellite Integration studies & strategies into the 5G terrestrial networks -Integration of SDN and NFV technologies on 5G satellite component -Satellite’s role in the Internet of Things applications over 5G terrestrial networks -Satellite’s role in Content Distribution Networks & internet protocols impact over user’s Quality of Experience (QoE) over a satellite link -The future proposals upon the evolution of Satellite systems by upcoming improvements and corresponding standards Finally, we have created an Annex for technical details upon the evolution of physical layer of the satellite systems with the corresponding bibliography of this thesis for future study

Networked Music Composition and Performance

This thesis investigates the potential of computer networks to transform existing music-making relationships within the context of contemporary music composition and performance practice. It consists of two primary components: the development of the ZScore networked music-making environment and a portfolio of compositions written specifically for the ZScore system. The technical capabilities of the system and creative compositional intentions have been evaluated through a series of workshops and performances with musicians and live audiences. This study provides detailed information on the project’s technical and creative objectives, their implementation, and research outcomes. ZScore is a collection of third-party and newly developed components aiming to provide solutions for complex notation authoring, the reliable distribution of interactive score representations to heterogeneous clients over a computer network, precise performance scheduling, and the rendering of dynamic stave-based scores. Furthermore, it aims to facilitate distributed decision-making agency for all participants and provide immersive audience participation through mobile device connectivity. The specifications of the ZScore system have been derived from an analysis of existing networked notation systems and state-of-the-art solutions from other industries where high-throughput, low latency systems have been successfully deployed. The accompanying composition portfolio outlines the journey towards a new paradigm of music-making, starting from a traditional static score that is fully defined by a composer and faithfully performed by musicians, and concluding with an interactive networked composition model that provides music-making agency to all participants. The compositional approach deploys an interplay between objective and subjective metamodern methods and explores the entire spectrum between fully composed and freely improvised music. Each composition in the portfolio addresses a set of new challenges, both technical and aesthetic. The composition commentary provides insight into the specific challenges for each score, the strategies employed to address related issues, and the outcomes of the applied solutions

Collision Avoidance on Unmanned Aerial Vehicles using Deep Neural Networks

Unmanned Aerial Vehicles (UAVs), although hardly a new technology, have recently gained a prominent role in many industries, being widely used not only among enthusiastic consumers but also in high demanding professional situations, and will have a massive societal impact over the coming years. However, the operation of UAVs is full of serious safety risks, such as collisions with dynamic obstacles (birds, other UAVs, or randomly thrown objects). These collision scenarios are complex to analyze in real-time, sometimes being computationally impossible to solve with existing State of the Art (SoA) algorithms, making the use of UAVs an operational hazard and therefore significantly reducing their commercial applicability in urban environments. In this work, a conceptual framework for both stand-alone and swarm (networked) UAVs is introduced, focusing on the architectural requirements of the collision avoidance subsystem to achieve acceptable levels of safety and reliability. First, the SoA principles for collision avoidance against stationary objects are reviewed. Afterward, a novel image processing approach that uses deep learning and optical flow is presented. This approach is capable of detecting and generating escape trajectories against potential collisions with dynamic objects. Finally, novel models and algorithms combinations were tested, providing a new approach for the collision avoidance of UAVs using Deep Neural Networks. The feasibility of the proposed approach was demonstrated through experimental tests using a UAV, created from scratch using the framework developed.Os veículos aéreos não tripulados (VANTs), embora dificilmente considerados uma nova tecnologia, ganharam recentemente um papel de destaque em muitas indústrias, sendo amplamente utilizados não apenas por amadores, mas também em situações profissionais de alta exigência, sendo expectável um impacto social massivo nos próximos anos. No entanto, a operação de VANTs está repleta de sérios riscos de segurança, como colisões com obstáculos dinâmicos (pássaros, outros VANTs ou objetos arremessados). Estes cenários de colisão são complexos para analisar em tempo real, às vezes sendo computacionalmente impossível de resolver com os algoritmos existentes, tornando o uso de VANTs um risco operacional e, portanto, reduzindo significativamente a sua aplicabilidade comercial em ambientes citadinos. Neste trabalho, uma arquitectura conceptual para VANTs autônomos e em rede é apresentada, com foco nos requisitos arquitetônicos do subsistema de prevenção de colisão para atingir níveis aceitáveis de segurança e confiabilidade. Os estudos presentes na literatura para prevenção de colisão contra objectos estacionários são revistos e uma nova abordagem é descrita. Esta tecnica usa técnicas de aprendizagem profunda e processamento de imagem, para realizar a prevenção de colisões em tempo real com objetos móveis. Por fim, novos modelos e combinações de algoritmos são propostos, fornecendo uma nova abordagem para evitar colisões de VANTs usando Redes Neurais Profundas. A viabilidade da abordagem foi demonstrada através de testes experimentais utilizando um VANT, desenvolvido a partir da arquitectura apresentada

Computer Music and Digital Media Art Through a Web-Based Collaborative Interface

A dissertação apresentada é resultado de uma investigação em interfaces colaborativas usando tecnologias web, feita no contexto do Braga Media Arts. Como resultado é apresentado um ambiente audiovisual em rede como desenvolvimento prático, o Akson. O Akson foi inicialmente concebido como uma exploração do que poderia ser construído aproveitando a infraestrutura global da Internet, bem como a reprodução musical e visual em vários dispositivos. Este sistema foi feito pensando em seu uso em performance ao vivo e é capaz de interagir com os dispositivos do público.The dissertation presented is the result of an investigation into collaborative interfaces using web technologies, done in the context of Braga Media Arts. As a result an audiovisual environment is presented as a practical development, Akson. Akson was initially conceived of as an exploration into what could be built by leveraging global internet infrastructure as well as musical and visual playback across multiple devices. This system was done thinking about its use in live performance and is able to interact with the devices of the public