ISP-friendly Peer-assisted On-demand Streaming of Long Duration Content in BBC iPlayer

In search of scalable solutions, CDNs are exploring P2P support. However, the benefits of peer assistance can be limited by various obstacle factors such as ISP friendliness - requiring peers to be within the same ISP, bitrate stratification - the need to match peers with others needing similar bitrate, and partial participation - some peers choosing not to redistribute content. This work relates potential gains from peer assistance to the average number of users in a swarm, its capacity, and empirically studies the effects of these obstacle factors at scale, using a month-long trace of over 2 million users in London accessing BBC shows online. Results indicate that even when P2P swarms are localised within ISPs, up to 88% of traffic can be saved. Surprisingly, bitrate stratification results in 2 large sub-swarms and does not significantly affect savings. However, partial participation, and the need for a minimum swarm size do affect gains. We investigate improvements to gain from increasing content availability through two well-studied techniques: content bundling - combining multiple items to increase availability, and historical caching of previously watched items. Bundling proves ineffective as increased server traffic from larger bundles outweighs benefits of availability, but simple caching can considerably boost traffic gains from peer assistance.Comment: In Proceedings of IEEE INFOCOM 201

Efficient Content Distribution With Managed Swarms

Content distribution has become increasingly important as people have become more reliant on Internet services to provide large multimedia content. Efficiently distributing content is a complex and difficult problem: large content libraries are often distributed across many physical hosts, and each host has its own bandwidth and storage constraints. Peer-to-peer and peer-assisted download systems further complicate content distribution. By contributing their own bandwidth, end users can improve overall performance and reduce load on servers, but end users have their own motivations and incentives that are not necessarily aligned with those of content distributors. Consequently, existing content distributors either opt to serve content exclusively from hosts under their direct control, and thus neglect the large pool of resources that end users can offer, or they allow end users to contribute bandwidth at the expense of sacrificing complete control over available resources. This thesis introduces a new approach to content distribution that achieves high performance for distributing bulk content, based on managed swarms. Managed swarms efficiently allocate bandwidth from origin servers, in-network caches, and end users to achieve system-wide performance objectives. Managed swarming systems are characterized by the presence of a logically centralized coordinator that maintains a global view of the system and directs hosts toward an efficient use of bandwidth. The coordinator allocates bandwidth from each host based on empirical measurements of swarm behavior combined with a new model of swarm dynamics. The new model enables the coordinator to predict how swarms will respond to changes in bandwidth based on past measurements of their performance. In this thesis, we focus on the global objective of maximizing download bandwidth across end users in the system. To that end, we introduce two algorithms that the coordinator can use to compute efficient allocations of bandwidth for each host that result in high download speeds for clients. We have implemented a scalable coordinator that uses these algorithms to maximize system-wide aggregate bandwidth. The coordinator actively measures swarm dynamics and uses the data to calculate, for each host, a bandwidth allocation among the swarms competing for the host's bandwidth. Extensive simulations and a live deployment show that managed swarms significantly outperform centralized distribution services as well as completely decentralized peer-to-peer systems

BitTorrent 시스템에서 컨텐트 번들링 및 배포

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2013. 2. 최양희.BitTorrent는 컨텐트 공유에 사용되는 가장 인기있는 인터넷 소프트웨어이다. BitTorrent가 널리 사용됨에 따라, 연구자들은 BitTorrent의 처리량, 공정성, 인센티브와 같은 이슈에 대해 연구해 왔고, 이러한 연구들은 BitTorrent 성능과 관련된 가치있는 결과들을 보여주었다. 하지만 대부분의 연구에서는, BitTorrent에서의 컨텐트 번들링 및 배포 전략과 관련해서 (1) BitTorrent 배포자가 파일을 어떤 목적으로 어떻게 번들 하는지와 (2) BitTorrent의 배포자들이 그들의 목적을 성취하기 위해 어떠한 전략들을 사용하는지 등에 대해 다루고 있지 않다. 본 학위 논문에서는, 앞서 언급한 문제들을 측정된 데이터를 바탕으로 조사하기 위해서, BitTorrent 포탈중 가장 큰 규모인 The Pirate Bay (TPB)에 대한 종합적인 측정 연구를 수행하였다. 측정된 데이터셋은 12만개의 토런트와 1600만명의 사용자로 구성되었고, 컨텐트 배포자를 (i) 가짜 배포자, (ii) 이윤추구 배포자, (iii) 이타적 배포자 세가지 종류로 분류하여 연구를 진행하였다. 또한 영화, TV, 성인물, 음악, 응용프로그램, 게임, 전자책과 같은 컨텐트 카테고리에 따라 번들링과 컨테트 배포 현황이 어떻게 되는지 조사하였다. 첫번째로, 토런트의 구조적 패턴과 스왐 참여자의 행동 패턴을 파악하기 위해 컨텐트 번들링과 관련된 현황을 조사하였다. 특별히, (1) 얼마나 컨텐트 번들링이 널리 사용되는가, (2) 어떤 파일들이 어떻게 토런트로 번들되는가, (3) 왜 배포자들이 파일을 번들해서 사용하는가, (4) 사용자들이 번들된 파일들을 어떻게 다운로드 받는가에 초점을 맞추어 연구를 수행하였다. 측정결과 72% 이상의 토런트들이 여러개의 파일로 구성되어 있는 것을 알 수 있었고, 이것은 번들이 BitTorrent의 파일 공유를 위해 널리 사용되고 있음을 보여준다. 그리고 경제적인 이득을 위해 웹사이트를 광고하는 이윤추구 배포자들이 번들을 선호하여 사용하는 경향이 있음을 알 수 있었다. 또한 번들된 토런트의 대부분의 파일(94%)이 사용자들에 의해 선택되고, 번들된 토런트가 번들이 아닌 토런트보다 평균적으로 더 인기가 좋음을 알 수 있었다. 전체적으로, 토런트의 구조적 패턴과 스왐 참여자의 특징은 컨텐트의 카테고리 종류에 따라서, 그리고 번들된 토런트인지 번들되지 않은 토런트인지에 따라서 주목할만한 차이점이 있음을 발견할 수 있었다. 다음으로, 사회경제적 관점에서 BitTorrent의 컨텐트 배포 패턴을 (1) 배포자에 의해서 파일이 어떻게 배포되는가, (2) 각 배포자들은 어떤 전략들을 사용하는가, (3) 배포 전략들이 얼마나 효과가 있는가의 측면에서 조사하였다. 측정결과 상당한 양의 트래픽(61%)이 가짜 토런트를 다운받을 때 발생하고 있는 것을 알 수 있었고, 이는 많은 양의 인터넷 트래픽이 불필요하게 낭비되고 있음을 보여 주는 것이다. 따라서 본 측정 결과로부터 알 수 있는 가짜 배포자들의 배포 패턴을 고려해서 TPB의 가짜 배포자를 걸러낼 수 있는 방법을 제안하였고, 제안된 방법이 전체 다운로드 트래픽의 45% 가량을 줄일 수 있음을 보여 주었다. 또한 이윤추구 배포자들은 그들의 수익모델(예를 들어, 개인 트래커 사이트에 새로운 사용자를 영입하는 것이나 사람들이 사진과 연결된 URL 링크를 클릭하도록 하는 것)에 따라 다른 배포 전략을 이용하고 있음을 알 수 있었다.BitTorrent is one of the most popular applications for sharing contents over the Internet. The huge success of BitTorrent has attracted the research community to investigate BitTorrent's behavior in terms of throughput, fairness, and incentive issues, revealing valuable insights into the performance aspects of BitTorrent. However, most of these studies paid little attention to understand content bundling and publishing strategies in BitTorrent from the following perspectives: (1) how, and for what purposes, are constituent files bundled by BitTorrent publishers? and (2) what strategies are adopted by BitTorrent publishers to achieve their goals? To answer these questions with data from a large-scale BitTorrent system, we conduct comprehensive measurements on one of the largest BitTorrent portals: the Pirate Bay (TPB). From the datasets of the 120 K torrents and 16 M peers, we classify BitTorrent publishers into three types: (i) fake publishers, (ii) profit-driven publishers, and (iii) altruistic publishers. Throughout this dissertation, we investigate the current practice of bundling and publishing across different content categories: Movie, TV, Porn, Music, Application, Game, and E-book. We first investigate the current practice of content bundling to understand the structural patterns of torrents and the participant behaviors of swarms. In particular, we focus on: (1) how prevalent content bundling is, (2) how and what files are bundled into torrents, (3) what motivates publishers to bundle files, and (4) how peers access the bundled files. We find that over 72% of BitTorrent torrents contain multiple files, which indicates that bundling is widely used for file sharing. We reveal that profit-driven BitTorrent publishers who promote their own web sites for financial gains like advertising tend to prefer to use the bundling. We also observe that most files (94%) in a bundle torrent are selected by users and the bundle torrents are more popular than the single (or non-bundle) ones on average. Overall, there are notable differences in the structural patterns of torrents and swarm characteristics (i) across different content categories and (ii) between single and bundle torrents. We next investigate the current practice of content publishing in BitTorrent from a socio-economic point of view, by unraveling (1) how files are published by publishers, (2) what strategies are adopted by publishers, and (3) how effective those strategies are. We show that a significant amount of traffic (61%) of BitTorrent has been generated (i.e., unnecessarily wasted) to download fake torrents. Therefore, we suggest a method to filter out fake publishers on TPB by considering their distinct publishing patterns learned from our measurement study, and show that the proposed method can reduce around 45% of the total download traffic. We also reveal that profit-driven publishers adopt different publishing strategies according to their revenue models (e.g., advertising private tracker sites to attract potential new members, or exposing image URLs to make people click the URL links).Abstract i I. Introduction 1 II. Related Work 5 2.1 Multi-torrent Systems 5 2.2 Bundling in BitTorrent 6 2.3 Bundling in Economics 7 2.4 Content publishing in BitTorrent 7 III. Methodology 9 3.1 Measurement Methodology 9 3.2 Publisher Classification 11 IV. Bundling Practice in BitTorrent: What, How, and Why 14 4.1 Introduction 14 4.2 Datasets 16 4.2.1 Torrent Datasets 17 4.2.2 Swarm Datasets 17 4.3 Single vs. Bundle 18 4.3.1 Bundling is widespread 18 4.3.2 How files are bundled 20 4.4 Main File Analysis in Bundling 27 4.4.1 Identifying Main Files 28 4.4.2 Constituents of Bundle-k 29 4.5 Publisher Analysis 32 4.5.1 Contribution of Top-20 Publishers 33 4.5.2 Cross-category Publishing of Top-20 Publishers 39 4.6 User Access Pattern Analysis 40 4.6.1 Popularity Analysis 40 4.6.2 Availability Analysis 43 4.6.3 The Number of Files Requested by Users in a Bundle Torrent 44 4.6.4 Swarm Behaviors versus Bundle-k 47 4.7 Discussions 50 V. Content Publishing Practice in BitTorrent 52 5.1 Introduction 52 5.2 The Number of Published Torrents 54 5.3 Publishers Strategies 58 5.3.1 Lifetime of Publishers and their Publishing Rates 59 5.3.2 Content Categories 60 5.3.3 Advertising Strategies of Profit-driven Publishers 63 5.4 Downloaders Behavior 64 5.5 Implications on Publishers Strategies 69 5.5.1 Fake Publishers 69 5.5.2 Profit-driven Publishers 71 VI. Summary & Future Work 73 Bibliography 75 Korean Abstract 80Docto

온라인 게임과 컨텐트 공유 네트워크 분석을 통한 온라인 군집 현상의 이해

학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2015. 2. 권태경.Quantification of collective human behavior and understanding the group characteristics in the Internet is important in user behavior studies since people tend to gather together and form groups due to their inherent nature. On the Internet, people are also often forming a group for a specific purpose such as i) an online group in games (e.g., MMORPGs) to experience various social interactions with other players or accomplish a difficult quest with teammates or ii) a swarm in peer-to-peer network to share a content to utilize a higher download rate with an availability. To this end, we studied the two most well-known major applications in the Internet that people are actively using with different purposesi) MMORPGs and ii) BitTorrent. In this dissertation, we analyze the i) group activities of users in Aion, one of the largest MMORPGs, based on the records of the activities of 94,497 users and ii) crowd phenomena of BitTorrent. First, in a case study of Aion, we focus on (i) how social interactions within a group differ from the ones across groups, (ii) what makes a group rise, sustain, or fall, (iii) how group members join and leave a group, and (iv) what makes a group end. We first find that structural patterns of social interactions within a group are more likely to be close-knit and reciprocative than the ones across groups.We also observe that members in a rising group (i.e., the number of members increases) are more cohesive, and communicate with more evenly within the group than the ones in other groups. Our analysis further reveals that if a group is not cohesive, not actively communicating, or not evenly communicating among members, members of the group tend to leave. Second, we investigate what kinds of crowd phenomena of content exist and why different patterns of crowd phenomena appears and how we can exploit content crowd phenomena considering the content category, publisher, and population of content in BitTorrent. To this end,We conduct comprehensive measurements on content locality in one of the largest BitTorrent portals: The Pirate Bay. In particular, we focus on (i) how content is consumed from spatial and temporal perspectives, (ii) what makes content be consumed with disparity in spatial and temporal domains, and (iii) how we can exploit the content locality. We find that content consumption in real swarms is 4.56 times and 1.46 times skewed in spatial (country) and temporal (time) domains, respectively. We observe that a cultural factor (e.g., language) mainly affects spatial locality of content. Not only the time-sensitivity of content but also the publishing purpose affects temporal locality of content.We reveal that spatial locality of content iii rarely changes on a daily basis (microscopic level), but there is notably spatial spread of content consumption over the years (macroscopic level). Based on the observation, we conduct simulations to show that bundling and caching can exploit the content locality.Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Crowd Phenomena in Massively Multi-player Online Role-Playing Games (MMORPGs) . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Crowd Phenomena in BitTorrent . . . . . . . . . . . . . . . . . . . 3 II. RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1 Crowd Phenomena in MMORPGs . . . . . . . . . . . . . . . . . . 6 2.1.1 Social Interactions in MMORPGs . . . . . . . . . . . . . . 6 2.1.2 Group Activities in MMORPGs . . . . . . . . . . . . . . . 7 2.1.3 Group Activities in Other Online Services . . . . . . . . . . 7 2.2 Crowd Phenomena (Locality) in BitTorrent . . . . . . . . . . . . . 8 2.2.1 Peer Localization . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.2 Crowd Phenomena in BitTorrent . . . . . . . . . . . . . . . 9 2.2.3 Locality in Other Domains . . . . . . . . . . . . . . . . . . 10 III. Group Activities in Online Social Game . . . . . . . . . . . . . . . . 11 3.1 Aion overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.1 Game Features . . . . . . . . . . . . . . . . . . . . . . . . 11 3.1.2 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2 Group Affiliation . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 v 3.2.1 How prevalent are group activities in Aion? . . . . . . . . . 14 3.2.2 Effect of Joining a Group . . . . . . . . . . . . . . . . . . . 16 3.2.3 Social Interactions Within a Group . . . . . . . . . . . . . . 16 3.3 Group Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.3.1 Group Cohesion . . . . . . . . . . . . . . . . . . . . . . . 20 3.3.2 Group Diversity . . . . . . . . . . . . . . . . . . . . . . . . 24 3.3.3 Group Locality . . . . . . . . . . . . . . . . . . . . . . . . 28 3.3.4 Survival Rate . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3.5 Dichotomy in Stable Groups . . . . . . . . . . . . . . . . . 32 3.4 Group Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.4.1 Properties of the Group Network . . . . . . . . . . . . . . . 36 3.4.2 Structural Holes . . . . . . . . . . . . . . . . . . . . . . . 38 3.5 Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.5.1 Why people leave groups? . . . . . . . . . . . . . . . . . . 40 3.5.2 Why a group ends? . . . . . . . . . . . . . . . . . . . . . . 42 IV. Crowd phenomena of BitTorrent in Spatial and Temporal Perspective 46 4.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.1.1 Discovering Swarm Topology . . . . . . . . . . . . . . . . 46 4.1.2 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.1.3 Representativeness . . . . . . . . . . . . . . . . . . . . . . 49 4.2 Spatial Locality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.2.1 Locality Metrics . . . . . . . . . . . . . . . . . . . . . . . 51 4.2.2 Swarm, Community, and Neighbor . . . . . . . . . . . . . 53 vi 4.2.3 Content Categories, Publishers, and Popularity . . . . . . . 55 4.2.4 Spatial Locality Over Time . . . . . . . . . . . . . . . . . . 58 4.3 Temporal Locality . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.3.1 Existence of Temporal Locality . . . . . . . . . . . . . . . 61 4.3.2 Categories, Publishers, and Popularity . . . . . . . . . . . . 63 4.3.3 Temporal Usage Trends . . . . . . . . . . . . . . . . . . . 68 4.4 How to Exploit Locality . . . . . . . . . . . . . . . . . . . . . . . 70 V. Summary & Future Work . . . . . . . . . . . . . . . . . . . . . . . . 74 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76Docto

Deep diving into BitTorrent locality

a

A HOLISTIC REDUNDANCY- AND INCENTIVE-BASED FRAMEWORK TO IMPROVE CONTENT AVAILABILITY IN PEER-TO-PEER NETWORKS

Peer-to-Peer (P2P) technology has emerged as an important alternative to the traditional client-server communication paradigm to build large-scale distributed systems. P2P enables the creation, dissemination and access to information at low cost and without the need of dedicated coordinating entities. However, existing P2P systems fail to provide high-levels of content availability, which limit their applicability and adoption. This dissertation takes a holistic approach to device mechanisms to improve content availability in large-scale P2P systems. Content availability in P2P can be impacted by hardware failures and churn. Hardware failures, in the form of disk or node failures, render information inaccessible. Churn, an inherent property of P2P, is the collective effect of the users’ uncoordinated behavior, which occurs when a large percentage of nodes join and leave frequently. Such a behavior reduces content availability significantly. Mitigating the combined effect of hardware failures and churn on content availability in P2P requires new and innovative solutions that go beyond those applied in existing distributed systems. To addresses this challenge, the thesis proposes two complementary, low cost mechanisms, whereby nodes self-organize to overcome failures and improve content availability. The first mechanism is a low complexity and highly flexible hybrid redundancy scheme, referred to as Proactive Repair (PR). The second mechanism is an incentive-based scheme that promotes cooperation and enforces fair exchange of resources among peers. These mechanisms provide the basis for the development of distributed self-organizing algorithms to automate PR and, through incentives, maximize their effectiveness in realistic P2P environments. Our proposed solution is evaluated using a combination of analytical and experimental methods. The analytical models are developed to determine the availability and repair cost properties of PR. The results indicate that PR’s repair cost outperforms other redundancy schemes. The experimental analysis was carried out using simulation and the development of a testbed. The simulation results confirm that PR improves content availability in P2P. The proposed mechanisms are implemented and tested using a DHT-based P2P application environment. The experimental results indicate that the incentive-based mechanism can promote fair exchange of resources and limits the impact of uncooperative behaviors such as “free-riding”

On Flash Crowd Performance of Peer-Assisted File Distribution

Ph.DDOCTOR OF PHILOSOPH