Cheng, J

Cheng, RCK

Fang, Y

Li, Z

Liu, Q

Lui, JCS

HKU Scholars Hub

Title PASCO: Parallel SimRank Computation at ScaleAuthor(s) Li, Z; Fang, Y; Liu, Q; Cheng, J; Cheng, RCK; Lui, JCSCitation The 2015 ACM Symposium on Cloud Computing (SoCC 2015),Kohala Coast, HI., 27-29 August 2015.Issued Date 2015URL http://hdl.handle.net/10722/214756Rights Creative Commons: Attribution 3.0 Hong Kong LicenseJacobi Method Update xIn Parallel Contribution     1. Enable parallel SimRank computation    2. Test on the largest graph, clue-web(|V|=1B, |E|=43B)         Problem    SimRank Decomposition    P: the transition matrix on graph    D: the diagonal correction matrix to be estimated      1. how to compute D for big graph ?    2. how to query efficiently given D ?   Offline indexing    1. Key observation: self-similarity is 1.0        Indexing linear system          here            2. Generate         by Monte Carlo         simulation, in parallel     3. Solve the linear system via Jacobi         method, in parallel     Graph data grows rapidly     1. Internet of Things     2. World Wide Web   Similarity is fundamental     1. Information retrieval     2. Recommender system     3. Churn prediction   SimRank - two objects are similar if referenced by similar objects            Three fundamental queries    1. Single-pair query – return similarity of two nodes    2. Single-source query – return similarity of every node to a node    3. All-pair query – return similarity between every two nodes    Challenges in SimRank computation    1. High complexity: O(n3) time, O(n2) space    2. Heavy computational dependency (hard to be parallelized)    3. Not allow querying similarities individually SimRank [1] Experiments 1Huawei Noah’s Ark Lab, 2HKU, 3CUHK Zhenguo Li1, Yixiang Fang2, Qin Liu3, Jiefeng Cheng1, Reynold Cheng2, John C.S. Lui3 Walking in the Cloud: Parallel SimRank at Scale ' ( ), ' ( )1,( , )( ', '),( ) ( ) i In i j In ji js i j cs i j i jIn i In j   CloudWalker – Big SimRank, instant response       [1] G. Jeh and J. Widom. Simrank: a measure of structural-ctontext similarity. KDD’02.       [2] D. Fogaras and B. Racz. Scaling link-based similarity search. WWW’05.       [3] T. Maehara, et al. Efficient simrank computation via linearization. CORR’14.  ( 1) ( )1 (1 )k ki ij jj iiix a xa Effectiveness: CloudWalker converges quickly Setup: cluster, datasets, and default parameters - 10 nodes (each with 16 cores, 377GB RAM, 20TB disk) Implementation on Spark Broadcasting is more efficient, but RDD is more scalable  CloudWalker outperforms state of the art MCSP: Monte Carlo simulation for single-pair query MCSS: Monte Carlo simulation for single-source query - constant time complexity: O(TR) MCAP: Monte Carlo simulation for all-pair query - use MCSS repeatedly; time complexity: O(nT2R logd)  Online queries To compute ai, we obtain Ptei using Monte Carlo Simulation   1. Place R random walkers on node i   2. Each walker walks t steps along in-links   3. Count the distribution of walkers Dataset Nodes Edges Size wiki-vote 7.1K 103K 476.8KB wiki-talk 2.4M 5M 45.6MB twitter-2010 42M 1.5B 11.4GB uk-unioni 131M 5.5B 48.3GB clue-web 1B 42.6B 401.1GB Parameter Value Meaning c 0.6 decay factor of SimRank T 10 # of walk steps L 3 # of iterations in Jacobi method R 100 # of walkers in simulating ai R’ 10,000 # of walkers in MCSP and MCSS Dataset D MCSP MCSS wiki-vote 7s 0.004s  0.042s wiki-talk 59s 0.046s 0.179s twitter-2010 975s 0.049s 0.281s uk-union 3323s 0.025s 0.292s 10x larger than the largest graph reported on SimRank  Dataset FMT [2] LIN [3] CloudWalker Prep.         SP.           SS. Prep.          SP.          SS. Prep.        SP.           SS. wiki-vote 43.4s    30.4ms    42.5s 187ms    0.61ms    5.3ms 7s           4ms     42ms wiki-talk N/A          N/A        N/A N/A          N/A         N/A 59s       46ms    180ms twitter-2010 -            -             - 14376s      3.17s      11.9s 975s      49ms    281ms uk-union -            -             - 8291s       9.42s      21.7s 3323s     25ms   291ms clue-web -            -             - -            -             - 110.2h    64.0s     188s s(i,j): similarity of nodes i and j   In(i): in-neighbors of i c: decay factor, 0<c<1 Similarity Propagation  • It captures human perception of similarity • It outperforms other similarity measures, such as co-citation  11 22[ , , , ]nnx D D DInput graph  Node-pair graph  1, 1,2,...,ia x i n 1 1 10( )Tt t ti i ita c P e P e  ia s- constant time complexity: O(T2R logd) Why Spark? • General-purpose in-memory cluster computing • Easy-to-use operations for distributed applications Broadcasting  RDD  Preprocessing, single-pair and single-source queries wiki-vote dataset Dataset D MCSP MCSS wiki-vote 50s 2.7s 2.9s wiki-talk 620s 8.5s 13.9s twitter-2010 8424s 11.8s 22.3s uk-union 6.4h 13.1s 27.2s clue-web 110.2h 64.0s 188.1s Two implementation models • Broadcasting: Graph stored in each machine • RDD (Resilient Distributed Dataset): Graph stored in an RDD 2 22S D cP cDP DPP   P DS cP D To appear in PVLDB’16 

Walking in the Cloud: Parallel SimRank at Scale

http://hub.hku.hk/bitstream/10722/214756/1/Poster.pdf

Walking in the Cloud: Parallel SimRank at Scale

Abstract

Similar works

Full text

Available Versions

HKU Scholars Hub