The crowd as a cameraman : on-stage display of crowdsourced mobile video at large-scale events

Recording videos with smartphones at large-scale events such as concerts and festivals is very common nowadays. These videos register the atmosphere of the event as it is experienced by the crowd and offer a perspective that is hard to capture by the professional cameras installed throughout the venue. In this article, we present a framework to collect videos from smartphones in the public and blend these into a mosaic that can be readily mixed with professional camera footage and shown on displays during the event. The video upload is prioritized by matching requests of the event director with video metadata, while taking into account the available wireless network capacity. The proposed framework's main novelty is its scalability, supporting the real-time transmission, processing and display of videos recorded by hundreds of simultaneous users in ultra-dense Wi-Fi environments, as well as its proven integration in commercial production environments. The framework has been extensively validated in a controlled lab setting with up to 1 000 clients as well as in a field trial where 1 183 videos were collected from 135 participants recruited from an audience of 8 050 people. 90 % of those videos were uploaded within 6.8 minutes

Crowdsourced Live Streaming over the Cloud

Empowered by today's rich tools for media generation and distribution, and the convenient Internet access, crowdsourced streaming generalizes the single-source streaming paradigm by including massive contributors for a video channel. It calls a joint optimization along the path from crowdsourcers, through streaming servers, to the end-users to minimize the overall latency. The dynamics of the video sources, together with the globalized request demands and the high computation demand from each sourcer, make crowdsourced live streaming challenging even with powerful support from modern cloud computing. In this paper, we present a generic framework that facilitates a cost-effective cloud service for crowdsourced live streaming. Through adaptively leasing, the cloud servers can be provisioned in a fine granularity to accommodate geo-distributed video crowdsourcers. We present an optimal solution to deal with service migration among cloud instances of diverse lease prices. It also addresses the location impact to the streaming quality. To understand the performance of the proposed strategies in the realworld, we have built a prototype system running over the planetlab and the Amazon/Microsoft Cloud. Our extensive experiments demonstrate that the effectiveness of our solution in terms of deployment cost and streaming quality

Collaborative Uploading in Heterogeneous Networks: Optimal and Adaptive Strategies

Collaborative uploading describes a type of crowdsourcing scenario in networked environments where a device utilizes multiple paths over neighboring devices to upload content to a centralized processing entity such as a cloud service. Intermediate devices may aggregate and preprocess this data stream. Such scenarios arise in the composition and aggregation of information, e.g., from smartphones or sensors. We use a queuing theoretic description of the collaborative uploading scenario, capturing the ability to split data into chunks that are then transmitted over multiple paths, and finally merged at the destination. We analyze replication and allocation strategies that control the mapping of data to paths and provide closed-form expressions that pinpoint the optimal strategy given a description of the paths' service distributions. Finally, we provide an online path-aware adaptation of the allocation strategy that uses statistical inference to sequentially minimize the expected waiting time for the uploaded data. Numerical results show the effectiveness of the adaptive approach compared to the proportional allocation and a variant of the join-the-shortest-queue allocation, especially for bursty path conditions.Comment: 15 pages, 11 figures, extended version of a conference paper accepted for publication in the Proceedings of the IEEE International Conference on Computer Communications (INFOCOM), 201

Prioritizing Roadway Pavement Marking Maintenance Using Lane Keep Assist Sensor Data

There are over four million miles of roads in the United States, and the prioritization of locations to perform maintenance activities typically relies on human inspection or semi-automated dedicated vehicles. Pavement markings are used to delineate the boundaries of the lane the vehicle is driving within. These markings are also used by original equipment manufacturers (OEM) for implementing advanced safety features such as lane keep assist (LKA) and eventually autonomous operation. However, pavement markings deteriorate over time due to the fact of weather and wear from tires and snowplow operations. Furthermore, their performance varies depending upon lighting (day/night) as well as surface conditions (wet/dry). This paper presents a case study in Indiana where over 5000 miles of interstate were driven and LKA was used to classify pavement markings. Longitudinal comparisons between 2020 and 2021 showed that the percentage of lanes with both lines detected increased from 80.2% to 92.3%. This information can be used for various applications such as developing or updating standards for pavement marking materials (infrastructure), quantifying performance measures that can be used by automotive OEMs to warn drivers of potential problems with identifying pavement markings, and prioritizing agency pavement marking maintenance activities

Quality-aware Content Adaptation in Digital Video Streaming

User-generated video has attracted a lot of attention due to the success of Video Sharing Sites such as YouTube and Online Social Networks. Recently, a shift towards live consumption of these videos is observable. The content is captured and instantly shared over the Internet using smart mobile devices such as smartphones. Large-scale platforms arise such as YouTube.Live, YouNow or Facebook.Live which enable the smartphones of users to livestream to the public. These platforms achieve the distribution of tens of thousands of low resolution videos to remote viewers in parallel. Nonetheless, the providers are not capable to guarantee an efficient collection and distribution of high-quality video streams. As a result, the user experience is often degraded, and the needed infrastructure installments are huge. Efficient methods are required to cope with the increasing demand for these video streams; and an understanding is needed how to capture, process and distribute the videos to guarantee a high-quality experience for viewers. This thesis addresses the quality awareness of user-generated videos by leveraging the concept of content adaptation. Two types of content adaptation, the adaptive video streaming and the video composition, are discussed in this thesis. Then, a novel approach for the given scenario of a live upload from mobile devices, the processing of video streams and their distribution is presented. This thesis demonstrates that content adaptation applied to each step of this scenario, ranging from the upload to the consumption, can significantly improve the quality for the viewer. At the same time, if content adaptation is planned wisely, the data traffic can be reduced while keeping the quality for the viewers high. The first contribution of this thesis is a better understanding of the perceived quality in user-generated video and its influencing factors. Subjective studies are performed to understand what affects the human perception, leading to the first of their kind quality models. Developed quality models are used for the second contribution of this work: novel quality assessment algorithms. A unique attribute of these algorithms is the usage of multiple features from different sensors. Whereas classical video quality assessment algorithms focus on the visual information, the proposed algorithms reduce the runtime by an order of magnitude when using data from other sensors in video capturing devices. Still, the scalability for quality assessment is limited by executing algorithms on a single server. This is solved with the proposed placement and selection component. It allows the distribution of quality assessment tasks to mobile devices and thus increases the scalability of existing approaches by up to 33.71% when using the resources of only 15 mobile devices. These three contributions are required to provide a real-time understanding of the perceived quality of the video streams produced on mobile devices. The upload of video streams is the fourth contribution of this work. It relies on content and mechanism adaptation. The thesis introduces the first prototypically evaluated adaptive video upload protocol (LiViU) which transcodes multiple video representations in real-time and copes with changing network conditions. In addition, a mechanism adaptation is integrated into LiViU to react to changing application scenarios such as streaming high-quality videos to remote viewers or distributing video with a minimal delay to close-by recipients. A second type of content adaptation is discussed in the fifth contribution of this work. An automatic video composition application is presented which enables live composition from multiple user-generated video streams. The proposed application is the first of its kind, allowing the in-time composition of high-quality video streams by inspecting the quality of individual video streams, recording locations and cinematographic rules. As a last contribution, the content-aware adaptive distribution of video streams to mobile devices is introduced by the Video Adaptation Service (VAS). The VAS analyzes the video content streamed to understand which adaptations are most beneficial for a viewer. It maximizes the perceived quality for each video stream individually and at the same time tries to produce as little data traffic as possible - achieving data traffic reduction of more than 80%

Smart Road Danger Detection and Warning

Road dangers have caused numerous accidents, thus detecting them and warning users are critical to improving traffic safety. However, it is challenging to recognize road dangers from numerous normal data and warn road users due to cluttered real-world backgrounds, ever-changing road danger appearances, high intra-class differences, limited data for one party, and high privacy leakage risk of sensitive information. To address these challenges, in this thesis, three novel road danger detection and warning frameworks are proposed to improve the performance of real-time road danger prediction and notification in challenging real-world environments in four main aspects, i.e., accuracy, latency, communication efficiency, and privacy. Firstly, many existing road danger detection systems mainly process data on clouds. However, they cannot warn users timely about road dangers due to long distances. Meanwhile, supervised machine learning algorithms are usually used in these systems requiring large and precisely labeled datasets to perform well. The EcRD is proposed to improve latency and reduce labeling cost, which is an Edge-cloud-based Road Damage detection and warning framework that leverages the fast-responding advantage of edges and the large storage and computation resources advantages of the cloud. In EcRD, a simple yet efficient road segmentation algorithm is introduced for fast and accurate road area detection by filtering out noisy backgrounds. Additionally, a light-weighted road damage detector is developed based on Gray Level Co-occurrence Matrix (GLCM) features on edges for rapid hazardous road damage detection and warning. Further, a multi-types road damage detection model is proposed for long-term road management on the cloud, embedded with a novel image-label generator based on Cycle-Consistent Adversarial Networks, which automatically generates images with corresponding labels to improve road damage detection accuracy further. EcRD achieves 91.96% accuracy with only 0.0043s latency, which is around 579 times faster than cloud-based approaches without affecting users' experience while requiring very low storage and labeling cost. Secondly, although EcRD relieves the problem of high latency by edge computing techniques, road users can only achieve warnings of hazardous road damages within a small area due to the limited communication range of edges. Besides, untrusted edges might misuse users' personal information. A novel FedRD named FedRD is developed to improve the coverage range of warning information and protect data privacy. In FedRD, a new hazardous road damage detection model is proposed leveraging the advantages of feature fusion. A novel adaptive federated learning strategy is designed for high-performance model learning from different edges. A new individualized differential privacy approach with pixelization is proposed to protect users' privacy before sharing data. Simulation results show that FedRD achieves similar high detection performance (i.e., 90.32% accuracy) but with more than 1000 times wider coverage than the state-of-the-art, and works well when some edges only have limited samples; besides, it largely preserves users' privacy. Finally, despite the success of EcRD and FedRD in improving latency and protecting privacy, they are only based on a single modality (i.e., image/video) while nowadays, different modalities data becomes ubiquitous. Also, the communication cost of EcRD and FedRD are very high due to undifferentiated data transmission (both normal and dangerous data) and frequent model exchanges in its federated learning setting, respectively. A novel edge-cloud-based privacy-preserving Federated Multimodal learning framework for Road Danger detection and warning named FedMRD is introduced to leverage the multi-modality data in the real-world and reduce communication costs. In FedMRD, a novel multimodal road danger detection model considering both inter-and intra-class relations is developed. A communication-efficient federated learning strategy is proposed for collaborative model learning from edges with non-iid and imbalanced data. Further, a new multimodal differential privacy technique for high dimensional multimodal data with multiple attributes is introduced to protect data privacy directly on users' devices before uploading to edges. Experimental results demonstrate that FedMRD achieves around 96.42% higher accuracy with only 0.0351s latency and up to 250 times less communication cost compared with the state-of-the-art, and enables collaborative learning from multiple edges with non-iid and imbalanced data in different modalities while preservers users' privacy.2021-11-2

Earth Observation Open Science and Innovation

geospatial analytics; social observatory; big earth data; open data; citizen science; open innovation; earth system science; crowdsourced geospatial data; citizen science; science in society; data scienc

Internet sharing in community networks

Cotutela Universitat Politècnica de Catalunya i Instituto Superior TécnicoThe majority of the world's population does not have any or adequate Internet access. This implies that the Internet cannot provide universal service, reaching everyone without discrimination. Global access to the Internet for all requires the expansion of network infrastructures and a dramatic reduction in Internet access costs especially in less developed geographical regions. Local communities come together to build their own network infrastructures, known as Community Networks, and provide accessible and affordable local and Internet inter-networking. Sharing resources, such as infrastructure or Internet access, is encouraged at all levels, in order to lower the cost of connectivity and services. Communities can develop their own network infrastructures as a commons, using several interconnected sub-networks when the scale requires it, and sharing several Internet gateways among their participants. Shared Internet access is offered through web proxy gateways, where individuals or organisations share the full or spare capacity of their Internet connections with other participants. However, these gateway nodes may be overloaded by the demand, and their Internet capacity may degrade due to lack of regulation. This thesis investigates whether shared Internet access in community networks can be utilized to provide universal Internet access. As a first step in this direction, in this thesis we explored characteristics, limitations and usability of a concrete shared Internet Web proxy service in community networks. Based on our findings we studied and proposed mechanisms to improve the user experience and fairness of Internet sharing Web proxy services in community networks, without introducing significant overhead to the network and other services. More specifically, we proposed a scalable client-side Internet gateway selection mechanism suitable for heterogeneous environments such as community networks. Finally, we studied and proposed techniques for sharing spare Internet capacity without deteriorating the contributors' performance.La mayoría de la población mundial no tiene ningún o un adecuado acceso a Internet. Esto implica que Internet no puede prestar un servicio universal, llegando a todos sin discriminación. El acceso global a Internet para todos requiere una drástica reducción de los costos de acceso a Internet, especialmente en zonas geográficas y poblaciones menos desarrolladas. Las comunidades locales se organizan para construir sus propias infraestructuras de red, conocidas como redes comunitarias, y proporcionan interconexión local y con Internet de forma accesible y asequible. Se fomenta la compartición de recursos, como la infraestructura o el acceso a Internet, para reducir el coste de la conectividad y los servicios. Las comunidades pueden desarrollar sus propias infraestructuras de red como un recurso común, utilizando varias subredes interconectadas dado su tamaño, y compartiendo varias pasarelas de Internet entre sus participantes. El acceso compartido a Internet se ofrece a través de pasarelas que son proxy web, donde los participantes o las organizaciones comparten la capacidad total o excedente de su conexión a Internet con otros participantes. Sin embargo, estas pasarelas pueden saturarse por la demanda, y su capacidad de acceso a Internet se puede degradar debido la falta de regulación. Esta tesis investiga si las redes comunitarias se pueden utilizar para proporcionar acceso universal a Internet. Como primer paso en esta dirección, exploramos las características, limitaciones y usabilidad de un servicio concreto de acceso compartido a Internet con proxies web en una red comunitaria. Sobre la base de nuestros hallazgos, estudiamos y proponemos mecanismos para mejorar la experiencia del usuario y la equitatividad de la compartición, sin introducir una sobrecarga significativa en la red y a otros servicios. Más específicamente, proponemos un mecanismo escalable de selección de pasarela a Internet del lado del cliente, adecuado para entornos heterogéneos como las redes comunitarias. Además, estudiamos y proponemos técnicas para compartir la capacidad de Internet sin deteriorar el desempeño de los participantes que contribuyen.Postprint (published version