This note argues about the validity of web-graph data used in the literature

Vigna, Sebastiano

English

arXiv

I discuss the implications of using small data sets for experiments related to the web graph

Dagstuhl Research Online Publication Server

Stanford Matrix Considered HarmfulSebastiano VignaThe idea for this note arose during the “Web Information Retrieval and LinearAlgebra Algorithms” held at Schloss Dagstuhl in February 2007. Many brilliantpeople working on either side (numerical analysis and web search) had a chanceto meet and talk for one week about mathematical and practical aspects of linearmethods for ranking, and in particular (not surprisingly) PageRank and HITS.1There were many scientific aspects that were surprising for both sides and that were(finally, one might say) stated clearly at the workshop. First of all, PageRank is notthe most important factor in Google’s scoring, or in any search engine scoring. It ispart of literally hundreds of features that are somehow combined (e.g., by a standardmachine-learning framework), and its importance has decreased in time (people inthe so-called “search engine optimisation” industry claim that the importance ofPageRank dropped drastically around 2003, but this claim is based just on reverseengineering). Yet papers are published every day starting with claims such as“PageRank is the most important ranking. . . ” and so on. Such statements give adistorted view of reality, but clearly work very well for publishing papers.On the other hand, it is also clear that PageRank (and, more generally, link-basedranking) is useful in other areas, such as deciding which pages to crawl and whichnot.Moreover, when one attacks the real PageRank computation problem suddenlymany sophisticated methods devised by the numerical analysis community arepretty useless because of the very large size of the matrix. This problem actu-ally had the very positive effect of stimulating a lot of new research in numericalanalysis methods that could be applied to large, sparse, irregular stochastic matri-ces of dimension 1010 and beyond. Much of the classical work on such large systemsinvolved matrices with considerable structure from ODEs, PDEs, queuing theory,etc. Actually, many of the more promising approaches for PageRank computationuse some kind of graph structural analysis (usually related to strongly connectedcomponents) that is rather meaningless in the classical problems mentioned above.These considerations bring us to the point of this note. The problem of computingPageRank is interesting from a practical viewpoint only if the size of the matrix islarge and if the type of the matrix is a web graph.What do we mean by “large”? Currently, search engines claim to index a numberof pages in the order of 1010. We cannot expect, as scientists, to replicate exactlywhat an infrastructure with a billion-dollars budget does, but let’s say that we canbe indulgent with ourselves and reduce our problem by three orders of magnitude.1This note contains intentionally no references. After some reckoning, this was the only sensiblechoice, as this note is intended to promote critical discussion instead of providing information forcitation-based importance measures. No claim in this note is new—the facts I report are easilyfound in the literature.1Dagstuhl Seminar Proceedings 07071Web Information Retrieval and Linear Algebra Algorithmshttp://drops.dagstuhl.de/opus/volltexte/2007/1058Defining the type of a web-graph matrix is a much more subtle issue. However,if claiming that a matrix is like a real web graph might be difficult, claiming thatis definitely not like a real web graph is much easier—just look for a significantdivergence in basic statistical facts known for web graphs.The so-called Stanford web matrix is a very small crawl of the Stanford (and maybesurrounding) sites freely available on the web. The matrix is easily available in asparse format that is immediately usable on off-the-shelf commercial computationaltools, so it has become somehow a benchmark, and often the matrix of choicefor testing algorithms or statistical hypotheses about the web. A considerable setof papers in the literature use this matrix to perform experiments and to deriveconclusions from.That matrix has less then 300 000 nodes, and it is thus five orders of magnitudesmaller than real-world web snapshots. There are freely available tools and datasetsthat make it possible to compute PageRank on a laptop for snapshots with 100million nodes in few hours: there is no excuse for using such a poor sample.But size is not the only problem: when we look at absolutely elementary statistics,the Stanford matrix is a complete outlier. As a comparison, I have used a small (onemillion nodes) crawl of the .eu domain, a slightly larger crawl (7.4 million nodes)of Indochina countries, and, finally, two significantly larger crawls of 40 million and100 million nodes of the .uk domain (all these snapshots are publicly available).As a first comparison, let’s look at the density. The most sensible measure we canuse is the average outdegree (proper density is ludicrously small in web graphs). Itis in the range 22-32 for all involved graphs, except for the Stanford Matrix, whichhas average outdegree 8.2.A fundamental issue in web crawling is that nobody can crawl the whole web, so thesuccessors of a significant fraction of pages are not visited. This causes a significantpercentage of the nodes to be dangling (i.e., to have no outlinks). Patching danglingnodes to make the matrix stochastic is a subtle issue, and there are unfortunatelyat least two distinct versions of PageRank around with different solutions. Onecan even forget about the Markov chain and just compute PageRank by solving theassociated linear system as if there were not dangling nodes: the result changes onlyby a scale factor, but the factor depends on how many dangling nodes there are, sothreshold on the norm of the vector used to stop approximations has a completelydifferent meaning in this case.The percentage of dangling nodes is in the range 8-17 in all our examples, exceptfor the Stanford matrix, where it is 0.06 [sic]. The resulting linear system will havea radically different behaviour than any real snapshot.Another relevant class of nodes is that of buckets. A bucket component is a compo-nent that is terminal, but not dangling, in the component DAG. A bucket (node)is a node whose component is a bucket component. When the damping factor αappearing in the definition of PageRank goes to one, all rank concentrates in thebuckets, and the remaining nodes go to rank zero. This phenomenon is very impor-tant, as it implies that when α → 1 PageRank is essentially meaningless (e.g., theentire core component will get rank zero): variants of PageRank suited for α → 1should be computed instead (but you can actually find many papers trying to com-pute PageRank in the α ≈ 1 region). The graphs under examination have about3-12% of buckets, except for the Stanford matrix, with a whopping 41% buckets.2All these measures confirm that statistically the Stanford matrix does not look at allas a web graph. But let’s go further: real web graphs have a kind of self-similaritythat is exploited by all approaches to web-graph compression. In particular, severalnodes have similar successor lists, and a simple way to single out such nodes is topermute the graph so to order lexicographically nodes by their URL. We cannotperform such an experiment because URLs are not available for the Stanford matrix.But we can do better: we can permute the nodes by their Gray codes—each rowis interpreted as a tuple of zeroes and ones, and the nodes are reordered followinga standard recursive Gray code. The resulting matrices exhibit similarity in acoordinate-free way (i.e., without referring to URLs).When we compress a graph using Gray codes and the WebGraph framework stan-dard parameters, we obtain about 2-3 bits per link in all the samples discussed,except for the .eu graph, which requires 4.9 bits per link. The Stanford matrixrequires 13.8 bits per link.There is an interesting phenomenon going on: some typical properties (e.g., highcompressibility) arise in our examples only beyond a certain size (about 10 millionnodes). People invoking the “fractal nature” of the web as an excuse to use smallsamples should thus be very careful (the .eu snapshot, for instance, is not a verygood candidate).Note that I am not suggesting that all web graphs should look the same, or thatwe should set up some standards to define a web graph: there is a healthy diversityof structure in the real world due to culture, wealth, and available tools (content-management systems, for instance, have steadily increased the average outdegreeof the web in the last 5 years). But there are criteria, based on common sense andexperience, that should delimit what we use in our experiments if we want to derivesensible conclusions, and the Stanford matrix largely falls short of such criteria.Post scriptum. David Gleich (whose most useful comments I’m happy to acknowl-edge) made me notice that the computer science community is not providing dataand tools in a language and in a format that the numerical analysis community caneasily exploit, which makes using many realistic, available data sets very difficult.We’re working on that.3

Stanford Matrix Considered Harmful

Abstract

Similar works

Full text

Available Versions

Dagstuhl Research Online Publication Server