Advances in cloud computing, 64-bit architectures and huge RAMs enable performing many search related tasks in memory.We argue that term-based partitioned parallel inverted index construction is among such tasks, and provide an efficient parallel framework that achieves this task. We show that by utilizing an efficient bucketing scheme we can eliminate the need for the generation of a global index and reduce the communication overhead without disturbing balancing constraint. We also propose and investigate assignment schemes that can further reduce communication overheads without disturbing balancing constraints. The conducted experiments indicate promising results. © 2012 Springer-Verlag London Limited

Aykanat, C.

Kucukyilmaz, T.

Turk, A.

Tayfun Kucukyilmaz

Ata Turk

Cevdet Aykanat

Crossref

Memory Resident Parallel Inverted Index Construction

Bilkent University Institutional Repository

Memory Resident Parallel Inverted IndexConstructionTayfun Kucukyilmaz, Ata Turk and Cevdet AykanatAbstract Advances in cloud computing, 64-bit architectures and huge RAMsenable performing many search related tasks in memory.We argue that term-basedpartitioned parallel inverted index construction is among such tasks, and providean efficient parallel framework that achieves this task. We show that by utilizing anefficient bucketing scheme we can eliminate the need for the generation of a globalindex and reduce the communication overhead without disturbing balancingconstraint. We also propose and investigate assignment schemes that can furtherreduce communication overheads without disturbing balancing constraints. Theconducted experiments indicate promising results.1 IntroductionInverted index is the de-facto data structure used in state-of-the-art text retrievalsystems. Even though it is quite simple as a data structure, Web-scale generation of aglobal inverted index is very costly [2]. Since the data to be indexed is crawled andstored by distributed or parallel systems (due to performance and scalabilityreasons), parallel index construction techniques are essential. Despite the popularityof document-based partitioned inverted indices, term-based partitioning hasadvantages that can be exploited for better query processing [4].The following studies on index construction [3, 5, 6] extend disk-basedtechniques for parallel systems. In [5, 6], authors propose a parallel disk-basedalgorithm with a centralized approach to generate the global vocabulary. In [5]authors analyze the merging phase of the inverted lists and present three algorithms.In [3], authors start from a document partitioned collection and proposed a software-pipelined inversion architecture.T. Kucukyilmaz  A. Turk  C. Aykanat (&)Computer Engineering Department, Bilkent University, 06800 Ankara, Turkeye-mail: aykanat@cs.bilkent.edu.trE. Gelenbe et al. (eds.), Computer and Information Sciences II,DOI: 10.1007/978-1-4471-2155-8_12,  Springer-Verlag London Limited 201299With the advent of 64-bit architectures, huge memory spaces are available tosingle machines and even very large inverted indices can fit into the totaldistributed memory of a cluster of such systems, enabling memory-based indexconstruction. Given the current advances in network technologies and cloudcomputing and the high availability of low cost memory, in-memory solutions forparallel index construction should be considered seriously.In this work, we propose an efficient, memory-based, parallel index construc-tion framework for generating term-based partitioned inverted indices startingfrom a document-based partitioned collection (possibly due to parallel crawling).In this framework, we propose to mask the communication costs associated withglobal vocabulary construction and communication with a term-to-bucketassignment schema. Furthermore, we investigate several assignment heuristics forimproving both the final storage balance and the communication costs of invertedindex construction. Here, storage balance is important since it relates to queryprocessing loads of processors, whereas the communication cost is important sinceit determines the running time of parallel inversion. Our contributions in this workare prior to optimizations such as compression [7].2 Parallel InversionOur overall parallel inversion scheme has the following phases: local invertedindex construction, term-to-processor assignment, and inverted list exchange andmerge. In this section we describe these three phases in detail.The first task in our framework is generating local inverted indices for alllocal document collections on each processor. As each processor contains anon-overlapping portion of the whole document collection, this operation can beachieved concurrently without communication.After local inverted index construction phase, a term-to-processor assignmentphase has to follow. In order to achieve a term-to-processor assignment, normallya global vocabulary has to be generated. This could be done by sending each termstring, in its word form, to a host processor, where a global vocabulary isconstructed. However in such a scheme, a particular term would be sent to thehost machine by all processors if all processors contain that specific term. Thuswe propose to group terms into a fixed number of buckets prior to the term-to-processor assignment. Using hashing, each word in a local vocabulary isassigned to a bucket. Afterwards, a host processor computes a bucket-to-processorassignment and broadcasts this information to the processors.At the end of local inversion phase, each processor has a local vocabulary and aset of inverted lists for its terms. However, different processors may containdifferent portions of an inverted list for each term. For the final term-basedpartitioned inverted index to be created, the inverted list portions of each termshould be accumulated to a single processor. To this end, each term in the globalvocabulary should be assigned to a particular processor. This term-to-processor100 T. Kucukyilmaz et al.assignment depicts an inverted index partitioning problem. Many different criteriacan be considered when finding a suitable index partitioning, but we focus onbalancing the storage loads of processors and minimizing the communicationoverhead of the inversion process. The storage balance guarantees an evenlydistribution of the final inverted index. As the memory is assumed to be limitedthroughout this work, with an even distribution of the storage loads, larger indicescan fit in the same set of processors. Storage balancing is also expected to inferbalancing on the query processing loads of the processors. Since inversion is ahighly communication-bound process, the minimization of the communicationoverhead ensures that the inverted list exchange phase of the parallel inversionprocess takes less time. In this work, we model the minimization of the commu-nication overhead as the minimization of the total communication volume whilemaintaining balance on communication loads of processors.At the end of the bucket-to-processor assignment phase, all assignments arebroadcast to processors and processors exchange their partial inverted lists in anall-to-all fashion. When sending the local inverted lists to their assigned processor,the vocabulary should also be sent since processors do not necessarily contain allvocabulary terms and do not know which terms will be retrieved from otherprocessors. Although such a communication incur additional costs, since theprocessor-to-host bottleneck due to global vocabulary construction is alreadyavoided, this additional communication cost is easily compensated.The inverted-list exchange between processors is achieved in two steps. In thefirst step, the terms (in word form) and their posting sizes are communicated.At the end of this step, all processors obtain their final local vocabularies and canreserve space for their final local inverted index structures. Then the inverted listsare exchanged. At the end of inverted list exchange, posting lists for each term aremerged and written into their reserved spaces in local inverted indices.3 Bucket-to-Processor Assignment SchemesIn the forthcoming discussions we use the following notations: The vocabulary ofterms is indicated with T : The posting list of each term tj 2 T is distributed amongthe K processors. wkðtjÞ denotes the size of the posting list portion of term tj thatresides in processor pk at the beginning of the inversion, whereas wtotðtjÞ ¼PKk¼1 wkðtjÞ denotes the total posting list size of term tj:We assume that prior to bucket-to-processor assignment, each processor hasbuilt its local inverted index I k and partitioned the vocabulary T ¼ ft1; t2; . . .; tngcontaining n terms, into a predetermined number m of buckets. The number ofbuckets m is selected such that m n and m K: Let B ¼ PðT Þ¼fT 1¼b1; T 2¼b2; . . .; T m¼bmg denote a random term-to-bucket partition, where T idenotes the set of terms that are assigned to bucket bi: In B; wtotðbiÞ denotes thetotal size of the posting lists of terms that belong to bi:Memory Resident Parallel Inverted Index Construction 101In an m-bucket and K-processor system, the bucket-to-processor assignment canbe represented via a K-way partition PðBÞ¼fB1;B2; . . .;Bkg of the bucketsamong the processors. In PðBÞ; wkðbiÞ denotes the total size of the posting listportions of terms that belong to bi and reside in processor pk at the beginning of theinversion. The performance of a bucket-to-processor assignment is measured interms of two metrics. The storage load balance and the communication cost. Thestorage load SðpkÞ of a processor pk induced by the assignment PðBÞ is defined asfollows:SðpkÞ ¼Xbi2BkXtj2biwtotðtjÞ: ð1ÞThe communication cost of a processor pk induced by assignment PðBÞ has twocomponents. Each processor must receive all portions of the buckets assigned toitself from other processors. Thus total receive cost/volume of a processor pk is:RecvðpkÞ ¼Xbi2BkXtj2biðwtotðtjÞ  wkðtjÞÞ: ð2ÞEach processor also sends all postings that are not assigned to it to some otherprocessor. The total send cost of pk is:SendðpkÞ ¼Xbi 62BkXtj2biwkðtjÞ ð3ÞTotal communication cost of a processor is defined as the sum of its send andreceive costs.The MCA scheme is based on the following simple observation [1]. If weassign each bucket bi 2 B to processor pk that has the largest wkðbiÞ value, we willachieve an assignment with globally minimum total communication volume.The BLMCA scheme incorporates a storage balance heuristic to MCA [1]. Thisscheme works in an iterative manner assigning one bucket to a processor at a time.For each bucket, first the processor that will cause the minimum total communi-cation is determined using MCA scheme. If this processor is not the bottleneckprocessor (in terms of storage load) at that iteration, the bucket is assigned to thatprocessor. Otherwise, the bucket is assigned to the minimally loaded processor.In BLMCA, two cost metrics, storage load balance and communication cost arecalculated and at each iteration an assignment decision that optimizes only oneof these metrics is made. The decisions of MCA and BLMCA regarding thecommunication cost minimization only optimizes the total communication costand ignores the maximum communication cost of a processor. In order to mini-mize maximum communication cost, we should consider both the receive cost ofthe assigned processor and the send costs of all other processors.To this end we define the energy E of an assignment PðBÞ based on the storageloads and communication costs of processors. We define two different energyfunctions for a given term-to-processor assignment PðBÞ:102 T. Kucukyilmaz et al.E1ðPðBÞÞ ¼ MaxfMax1 kKfCommðpkÞg;Max1 kKfSðpkÞgg ð4ÞE2ðPðBÞÞ ¼XK1ðCommðpkÞÞ2 þXK1ðSðpkÞÞ2 ð5ÞUtilizing these energy functions, we propose a constructive algorithm that assignsbuckets to processors one-by-one. The energy increase in the system by K possibleassignments of each bucket are considered, and the assignment that incurs theminimum energy increase is performed. That is, for the assignment of a bucket biin the given order, we select the assignment that minimizes EðPðBi1 [ fbigÞÞ EðPðBi1Þ; where Bi1 denotes the set of already assigned buckets. We callE1-based and E2-based assignment schemes as E1A and E2A respectively in ourexperiments.4 ExperimentsIn order to test the performance of the proposed assignment schemes for parallelinversion, we conducted two types of experiments. The first set of experiments aresimulations to report on the storage imbalance and communication volumeperformances of the assignment schemes. The second set of experiments are actualparallel inversion runs provided in order to show how improvements in perfor-mance metrics relate to parallel running times. These experiments are conductedon a PC-cluster with K ¼ 32 nodes, where each node is an Intel Pentium IV3.0 GHz processors with 1 GB RAM connected via an interconnection network of100 Mb/sec fast ethernet.We conducted our experiments on a realistic dataset obtained by crawlingeducational sites across America. The raw size of the dataset is 30 GB andcontains 1,883,037 pages from 18,997 different sites. The biggest site contains10,352 pages while average number of pages per site is 99.1. The vocabulary ofthe dataset consists of 3,325,075 distinct terms. There are 787,221,668 words inthe dataset. The size of the inverted index generated from the dataset is 2.8 GB.Tables 1 and 2 compare the storage load balancing and communicationperformances of the assignment schemes for K ¼ f4; 8; 16; 32; 64; 128g: We alsoimplemented a random assignment (RA) algorithm, which assigns buckets toprocessors randomly, as a baseline assignment scheme. As seen in Table 1, MCAachieves the worst final storage imbalance. This is expected since MCA considersonly minimization of the total communication cost, disregarding storage balanceand as seen in Table 2 MCA achieves lowest average communication cost.BLMCA algorithm on the other hand, achieves best final storage imbalance. Thisis also expected since the primary objective of BLMCA is to balance the processorloads during the assignments instead of minimizing the communication costs. Asseen in Table 2, this storage balancing performance is achieved at the expense ofhigher average communication values per processor. Experiments indicate thatMemory Resident Parallel Inverted Index Construction 103E1A and E2A algorithms both achieve reasonable storage load balance that areeither close or better than the performance of RA scheme. Also as seen in Table 2,for K values higher than 8, E2A achieves the lowest maximum communicationvolumes. Table 2 also indicates that the average and maximum communicationcosts induced by E2A are very close, which means that E2A manages to distributethe communication loads among processors evenly.Table 3 shows the running times of our parallel memory-based index inversionalgorithm under different assignment schemes. In this table, it is assumed that thelocal inverted indices are already created and the time for this operation isneglected. As expected from the results presented in Table 1 and Table 2, MCAinduces the highest inversion time, RA, BLMCA, E1A, and E2A induce similarinversion times and the E2A scheme achieves the lowest inversion times.Table 1 Percent load imbalance valuesInitial FinalK RA MCA BLMCA E1A E2A4 4.4 12.1 38.3 0.0 6.1 5.58 11.7 09.9 60.0 0.1 18.2 14.416 18.2 27.4 66.2 1.7 27.2 20.032 44.1 29.6 83.0 5.4 35.2 31.140 32.2 37.0 77.4 6.2 38.4 31.464 44.7 56.6 92.2 11.5 46.9 33.7128 65.3 94.7 95.6 15.7 64.1 40.4Table 2 Message volume (send + receive) per processor (in terms of 106 postings)RA MCA BLMCA E1A E2AK Avg Max Avg Max Avg Max Avg Max Avg Max4 131.19 145.71 122.09 150.26 127.45 128.62 124.76 125.29 131.24 131.368 76.55 90.58 71.45 119.75 73.40 75.97 72.02 74.53 76.67 76.7916 41.01 49.25 38.32 77.11 39.22 43.44 38.52 42.33 41.60 41.6632 21.19 28.75 19.82 71.13 20.28 26.03 19.94 25.08 20.54 21.6140 17.05 23.96 15.99 44.79 16.32 20.01 16.03 19.22 17.04 17.7164 10.76 17.77 10.09 74.27 10.34 15.42 10.15 14.75 10.86 11.89128 5.42 11.97 5.09 65.59 5.22 10.98 5.12 10.02 6.81 7.95Table 3 Parallel inversion times (in seconds) including assignment and inverted list exchangetimes for different assignment schemesK RA MCA BLMCA E1A E2A4 69.19 81.34 68.63 68.67 68.498 51.42 66.76 46.45 46.59 45.7416 35.89 60.82 33.04 32.90 32.4832 19.31 49.45 18.20 17.91 17.20104 T. Kucukyilmaz et al.5 ConclusionsIn this paper, a memory-based parallel inverted index construction framework wasexamined. An extensive step-by-step experimentation of our model was presentedand further insight were provided using theoretical results and simulations. Also,several problems involving the creation of this framework were identified.References1. Aykanat, C., Cambazoglu, B.B., Findik, F., Kurc, T.: Adaptive decomposition and remappingalgorithms for object-space-parallel direct volume rendering of unstructured grids. J. ParallelDistrib. Comput. 67, 77–99 (2007)2. Cho, J., Garcia-Molina, H.: The evolution of the web and implications for an incrementalcrawler. In: Proceedings of the 26th International Conference on VLDB (2000)3. Melink, S., Raghavan, S., Yang, B., Garcia-Molina, H.: Building a distributed full-text indexfor the web. ACM Trans. Inf. Syst. 19, 217–241 (2001)4. Moffat, A., Webber, W., Zobel, J.: Load balancing for term-distributed parallel retrieval. In:Proceedings of the 29th Annual International ACM SIGIR Conference on Research andDevelopment in Information Retrieval, pp. 348–355 (2006)5. Ribeiro-Neto, B., Moura, E.S., Neubert, M.S., Ziviani, N.: Efficient distributed algorithms tobuild inverted files. In: Proceedings of the 22nd Annual International ACM SIGIR Conferenceon Research and Development in IR, pp. 105–112 (1999)6. Ribeiro-Neto, B.A., Kitajima, J.P., Navarro, G., Sant’Ana, C.R.G., Ziviani, N.: Parallelgeneration of inverted files for distributed text collections. In: Proceedings of the 18thInternational Conference of the Chilean Computer Science Society (1998)7. Zobel, J., Moffat, A., Ramamohanarao, K.: Inverted files versus signature files for text indexing.ACM Trans. Database Syst. 23, 453–490 (1998)Memory Resident Parallel Inverted Index Construction 105

Memory resident parallel inverted index construction

http://repository.bilkent.edu.tr/bitstream/11693/28083/1/Memory%20resident%20parallel%20inverted%20index%20construction.pdf

Memory resident parallel inverted index construction

Abstract

Similar works

Full text

Available Versions

Crossref

Bilkent University Institutional Repository