Live media production: multicast optimization and visibility for clos fabric in media data centers

Media production data centers are undergoing a major architectural shift to introduce digitization concepts to media creation and media processing workflows. Content companies such as NBC Universal, CBS/Viacom and Disney are modernizing their workflows to take advantage of the flexibility of IP and virtualization. In these new environments, multicast is utilized to provide point-to-multi-point communications. In order to build point-to-multi-point trees, Multicast has an established set of control protocols such as IGMP and PIM. The existing multicast protocols do not optimize multicast tree formation for maximizing network throughput which lead to decreased fabric utilization and decreased total number of admitted flows. In addition, existing multicast protocols are not bandwidth-aware and could cause links to over-subscribe leading to packet loss and lower video quality. TV production traffic patterns are unique due to ultra high bandwidth requirements and high sensitivity to packet loss that leads to video impairments. In such environments, operators need monitoring tools that are able to proactively monitor video flows and provide actionable alerts. Existing network monitoring tools are inadequate because they are reactive by design and perform generic monitoring of flows with no insights into video domain. The first part of this dissertation includes a design and implementation of a novel Intelligent Rendezvous Point algorithm iRP for bandwidth-aware multicast routing in media DC fabrics. iRP utilizes a controller-based architecture to optimize multicast tree formation and to increase bandwidth availability in the fabric. The system offers up to 50\% increase in fabric capacity to handle multicast flows passing through the fabric. In the second part of this dissertation, DiRP algorithm is presented. DiRP is based on a distributed decision-making approach to achieve multicast tree capacity optimization while maintaining low multicast tree setup time. DiRP algorithm is tested using commercially available data center switches. DiRP algorithm offers substantially lower path setup time compared to centralized systems while maintaining bandwidth awareness when setting up the fabric. The third part of this dissertation studies the utilization of machine learning algorithms to improve on multicast efficiency in the fabric. The work includes implementation and testing of LiRP algorithm to increase iRP\u27s fabric efficiency by implementing k-fold cross validation method to predict future multicast group memberships for time-series analysis. Testing results confirm that LiRP system increases the efficiency of iRP by up to 40\% through prediction of multicast group memberships with online arrival. In the fourth part of this dissertation, The problem of live video monitoring is studied. Existing network monitoring tools are either reactive by design or perform generic monitoring of flows with no insights into video domain. MediaFlow is a robust system for active network monitoring and reporting of video quality for thousands of flows simultaneously using a fraction of the cost of traditional monitoring solutions. MediaFlow is able to detect and report on integrity of video flows at a granularity of 100 mSec at line rate for thousands of flows. The system increases video monitoring scale by a thousand-fold compared to edge monitoring solutions

Diseño centrado en calidad para la difusión Peer-to-Peer de video en vivo

El uso de redes Peer-to-Peer (P2P) es una forma escalable para ofrecer servicios de video sobre Internet. Este documento hace foco en la definición, desarrollo y evaluación de una arquitectura P2P para distribuir video en vivo. El diseño global de la red es guiado por la calidad de experiencia (Quality of Experience - QoE), cuyo principal componente en este caso es la calidad del video percibida por los usuarios finales, en lugar del tradicional diseño basado en la calidad de servicio (Quality of Service - QoE) de la mayoría de los sistemas. Para medir la calidad percibida del video, en tiempo real y automáticamente, extendimos la recientemente propuesta metodología Pseudo-Subjective Quality Assessment (PSQA). Dos grandes líneas de investigación son desarrolladas. Primero, proponemos una técnica de distribución de video desde múltiples fuentes con las características de poder ser optimizada para maximizar la calidad percibida en contextos de muchas fallas y de poseer muy baja señalización (a diferencia de los sistemas existentes). Desarrollamos una metodología, basada en PSQA, que nos permite un control fino sobre la forma en que la señal de video es dividida en partes y la cantidad de redundancia agregada, como una función de la dinámica de los usuarios de la red. De esta forma es posible mejorar la robustez del sistema tanto como sea deseado, contemplando el límite de capacidad en la comunicación. En segundo lugar, presentamos un mecanismo estructurado para controlar la topología de la red. La selección de que usuarios servirán a que otros es importante para la robustez de la red, especialmente cuando los usuarios son heterogéneos en sus capacidades y en sus tiempos de conexión.Nuestro diseño maximiza la calidad global esperada (evaluada usando PSQA), seleccionado una topología que mejora la robustez del sistema. Además estudiamos como extender la red con dos servicios complementarios: el video bajo demanda (Video on Demand - VoD) y el servicio MyTV. El desafío en estos servicios es como realizar búsquedas eficientes sobre la librería de videos, dado al alto dinamismo del contenido. Presentamos una estrategia de "caching" para las búsquedas en estos servicios, que maximiza el número total de respuestas correctas a las consultas, considerando una dinámica particular en los contenidos y restricciones de ancho de banda. Nuestro diseño global considera escenarios reales, donde los casos de prueba y los parámetros de configuración surgen de datos reales de un servicio de referencia en producción. Nuestro prototipo es completamente funcional, de uso gratuito, y basado en tecnologías bien probadas de código abierto

Support Efficient, Scalable, and Online Social Spam Detection in System

The broad success of online social networks (OSNs) has created fertile soil for the emergence and fast spread of social spam. Fake news, malicious URL links, fraudulent advertisements, fake reviews, and biased propaganda are bringing serious consequences for both virtual social networks and human life in the real world. Effectively detecting social spam is a hot topic in both academia and industry. However, traditional social spam detection techniques are limited to centralized processing on top of one specific data source but ignore the social spam correlations of distributed data sources. Moreover, a few research efforts are conducting in integrating the stream system (e.g., Storm, Spark) with the large-scale social spam detection, but they typically ignore the specific details in managing and recovering interim states during the social stream data processing. We observed that social spammers who aim to advertise their products or post victim links are more frequently spreading malicious posts during a very short period of time. They are quite smart to adapt themselves to old models that were trained based on historical records. Therefore, these bring a question: how can we uncover and defend against these online spam activities in an online and scalable manner? In this dissertation, we present there systems that support scalable and online social spam detection from streaming social data: (1) the first part introduces Oases, a scalable system that can support large-scale online social spam detection, (2) the second part introduces a system named SpamHunter, a novel system that supports efficient online scalable spam detection in social networks. The system gives novel insights in guaranteeing the efficiency of the modern stream applications by leveraging the spam correlations at scale, and (3) the third part refers to the state recovery during social spam detection, it introduces a customizable state recovery framework that provides fast and scalable state recovery mechanisms for protecting large distributed states in social spam detection applications

Detection of encrypted traffic generated by peer-to-peer live streaming applications using deep packet inspection

The number of applications using the peer-to-peer (P2P) networking paradigm and their popularity has substantially grown over the last decade. They evolved from the le-sharing applications to media streaming ones. Nowadays these applications commonly encrypt the communication contents or employ protocol obfuscation techniques. In this dissertation, it was conducted an investigation to identify encrypted traf c ows generated by three of the most popular P2P live streaming applications: TVUPlayer, Livestation and GoalBit. For this work, a test-bed that could simulate a near real scenario was created, and traf c was captured from a great variety of applications. The method proposed resort to Deep Packet Inspection (DPI), so we needed to analyse the payload of the packets in order to nd repeated patterns, that later were used to create a set of SNORT rules that can be used to detect key network packets generated by these applications. The method was evaluated experimentally on the test-bed created for that purpose, being shown that its accuracy is of 97% for GoalBit.A popularidade e o número de aplicações que usam o paradigma de redes par-a-par (P2P) têm crescido substancialmente na última década. Estas aplicações deixaram de serem usadas simplesmente para partilha de ficheiros e são agora usadas também para distribuir conteúdo multimédia. Hoje em dia, estas aplicações têm meios de cifrar o conteúdo da comunicação ou empregar técnicas de ofuscação directamente no protocolo. Nesta dissertação, foi realizada uma investigação para identificar fluxos de tráfego encriptados, que foram gerados por três aplicações populares de distribuição de conteúdo multimédia em redes P2P: TVUPlayer, Livestation e GoalBit. Para este trabalho, foi criada uma plataforma de testes que pretendia simular um cenário quase real, e o tráfego que foi capturado, continha uma grande variedade de aplicações. O método proposto nesta dissertação recorre à técnica de Inspecção Profunda de Pacotes (DPI), e por isso, foi necessário 21nalisar o conteúdo dos pacotes a fim de encontrar padrões que se repetissem, e que iriam mais tarde ser usados para criar um conjunto de regras SNORT para detecção de pacotes chave· na rede, gerados por estas aplicações, afim de se poder correctamente classificar os fluxos de tráfego. Após descobrir que a aplicação Livestation deixou de funcionar com P2P, apenas as duas regras criadas até esse momento foram usadas. Quanto à aplicação TVUPlayer, foram criadas várias regras a partir do tráfego gerado por ela mesma e que tiveram uma boa taxa de precisão. Várias regras foram também criadas para a aplicação GoalBit em que foram usados quatro cenários: com e sem encriptação usando a opção de transmissão tracker, e com e sem encriptação usando a opção de transmissão sem necessidade de tracker (aqui foi usado o protocolo Kademlia). O método foi avaliado experimentalmente na plataforma de testes criada para o efeito, sendo demonstrado que a precisão do conjunto de regras para a aplicação GoallBit é de 97%.Fundação para a Ciência e a Tecnologia (FCT

Livenet: A low-latency video transport network for large-scale live streaming

Low-latency live streaming has imposed stringent latency requirements on video transport networks. In this paper we report on the design and operation of the Alibaba low-latency video transport network, LiveNet. LiveNet builds on a flat CDN overlay with a centralized controller for global optimization. As part of this, we present our design of the global routing computation and path assignment, as well as our fast data transmission architecture with fine-grained control of video frames. The performance results obtained from three years of operation demonstrate the effectiveness of LiveNet in improving CDN performance and QoE metrics. Compared with our prior state-of-The-Art hierarchical CDN deployment, LiveNet halves the CDN delay and ensures 98% of views do not experience stalls and that 95% can start playback within 1 second. We further report our experiences of running LiveNet over the last 3 years

Service Oriented Architecture for VoIP Conferencing

Voice/Video over IP (VoIP) systems to date have been either highly centralized or dependent on the IP multicast in nature. Global Multimedia Collaboration System is a scalable, integrated and service-oriented VoIP conferencing system, based on the XGSP collaboration framework and NaradaBrokering messaging middleware. This system can provide media and session services to heterogeneous endpoints such as H.323, SIP, Access Grid, RealPlayer as well as cellular phone. In this paper, we address the challenges of scalability, interoperablity and heterogeneity in massive VoIP conferencing system. We believe that our approach opens up new opportunities for leveraging classic VoIP systems by using new technologies in service-oriented computing

Walkabout : an asynchronous messaging architecture for mobile devices

Modern mobile devices are prolific producers and consumers of digital data, and wireless networking capabilities enable them to transfer their data over the Internet while moving. Applications running on these devices may perform transfers to upload data for backup or distribution, or to download new content on demand. Unfortunately, the limited connectivity that mobile devices experience can make these transfers slow and impractical as the amount of data increases. This thesis argues that asynchronous messaging supported by local proxies can improve the transfer capabilities of mobile devices, making it practical for them to participate in large Internet transfers. The design of the Walkabout architecture follows this approach: proxies form store-and-forward overlay networks to deliver messages asynchronously across the Internet on behalf of devices. A mobile device uploads a message to a local proxy at rapid speed, and the overlay delivers it to one or more destination devices, caching the message until each one is able to retrieve it from a local proxy. A device is able to partially upload or download a message whenever it has network connectivity, and can resume this transfer at any proxy if interrupted through disconnection. Simulation results show that Walkabout provides better throughput for mobile devices than is possible under existing methods, for a range of movement patterns. Upload and end-to-end to transfer speeds are always high when the device sending the message is mobile. In the basic Walkabout model, a message sent to a mobile device that is repeatedly moving between a small selection of connection points experiences high download and end-to-end transfer speeds, but these speeds fall as the number of connection points grows. Pre-emptive message delivery extensions improve this situation, making fast end-to-end transfers and device downloads possible under any pattern of movement. This thesis describes the design and evaluation of Walkabout, with both practical implementation and extensive simulation under real-world scenarios

Recent Trends in Communication Networks

In recent years there has been many developments in communication technology. This has greatly enhanced the computing power of small handheld resource-constrained mobile devices. Different generations of communication technology have evolved. This had led to new research for communication of large volumes of data in different transmission media and the design of different communication protocols. Another direction of research concerns the secure and error-free communication between the sender and receiver despite the risk of the presence of an eavesdropper. For the communication requirement of a huge amount of multimedia streaming data, a lot of research has been carried out in the design of proper overlay networks. The book addresses new research techniques that have evolved to handle these challenges

A scheme for efficient peer-to-peer live video streaming over wireless mesh networks

Peers in a Peer-to-Peer (P2P) live video streaming system over hybrid wireless mesh networks (WMNs) enjoy high video quality when both random network coding (RNC) and an efficient hybrid routing protocol are employed. Although RNC is the most recently used method of efficient video streaming, it imposes high transmission overhead and decoding computational complexity on the network which reduces the perceived video quality. Besides that, RNC cannot guaranty a non-existence of linear dependency in the generated coefficients matrix. In WMNs, node mobility has not been efficiently addressed by current hybrid routing protocols that increase video distortion which would lead to low video quality. In addition, these protocols cannot efficiently support nodes which operate in infrastructure mode. Therefore, the purpose of this research is to propose a P2P live video streaming scheme which consists of two phases followed by the integration of these two phases known as the third phase to provide high video quality in hybrid WMNs. In the first phase, a novel coefficients matrix generation and inversion method has been proposed to address the mentioned limitations of RNC. In the second phase, the proposed enhanced hybrid routing protocol was used to efficiently route video streams among nodes using the most stable path with low routing overhead. Moreover, this protocol effectively supports mobility and nodes which operate in infrastructure mode by exploiting the advantages of the designed locator service. Results of simulations from the first phase showed that video distortion as the most important performance metric in live video streaming, had improved by 36 percent in comparison with current RNC method which employs the Gauss-Jordan Elimination (RNC-GJE) method in decoding. Other metrics including frame dependency distortion, initial start-up delay and end-to-end delay have also improved using the proposed method. Based on previous studies, although Reactive (DYMO) routing protocol provides better performance than other existing routing protocols in a hybrid WMN, the proposed protocol in the second phase had average improvements in video distortion of l86% for hybrid wireless mesh protocol (HWMP), 49% for Reactive (Dynamic MANET On-Demand-DYMO), 75% for Proactive (Optimized Link State Routing-OLSR), and 60% for Ad-hoc on-demand Distance Vector Spanning-Tree (AODV-ST). Other metrics including end-to-end delay, packet delay variation, routing overhead and number of delivered video frames have also improved using the proposed protocol. Finally, the third phase, an integration of the first two phases has proven to be an efficient scheme for high quality P2P live video streaming over hybrid WMNs. This video streaming scheme had averagely improved video distortion by 41%, frame dependency distortion by 50%, initial start-up delay by 15% and end-to-end delay by 33% in comparison with the average introduced values by three other considered integration cases which are Reactive and RNC-GJE, Reactive and the first phase, the second phase and RNC-GJE