This document describes Topian ("Topic-based Model layer for Xapian"), a software layer intended to add support for topical models to Xapian

Chappelier, Jean-Cédric

Eckard, Emmanuel

Infoscience - École polytechnique fédérale de Lausanne

Topian 0.1 Reference ManualEmmanuel Eckard and Jean-Cdric Chappelier17 September 2007AbstractThis document describes Topian (“Topic-based Model layer for Xapian”),a software layer intended to add support for topical models to Xapian.One common technical problem of Information Retrieval engines is to hold andaccess to data. We choose to use Xapian, a performent retrieval library focusingon probabilistic retrieval.Modern Information Retrieval does not limit itself at representing documentsin a vector space via indexing. A transformation of the vector space can takeplace, often involving a reduction in dimensionality. Starting with the LatentSemantic Analysis (LSI) model, a number fof models have been developed in thepast 10 years, including Probabilistic Latent Semantic Analysis (PLSI), NaiveBayesian models, or Latent Dirichlet allocation.We attempt at providing a general software framework to develop such mod-els on the stable, fast and generally efficient basis of Xapian.This document describes Topian (“Topic-based Model layer for Xapian”), asoftware layer intended to add support for topical models to Xapian.1 ConceptionTopian provides two trends to objects: objects directly linked to the docu-ments, and abstract objects dealing with the model. Document representa-tion is implemented in DocsRep objects which encapsulate Xapian objects likeXapian::Database. The DocsRep is what makes the link between the clas-sical document representation, and features of the intended models held inParameters objects. The Parameters are abstract objects which representthe model descriping statistical features of the document topics and word top-ics. This architecture keeps the Parameters distinct and separated from theDocsRep, allowing for selective loading of the data when both direct data andmodel data are not needed for a given process.For instance, the PLSA model is a statistical model based on the parametersP (w|z) (probability that, given a certain topic z, work w occures), P (d|z) (prob-ability that, given a certain topic z, document d occures), and P (z) (probabilitythat a topic z occures). On Topian, this translates into a PLSARep class inher-iting from DocsRep data-wise, and a PLSAParams inheriting from Parameters,model-wise. The classes PLSARep and Parameters have further children to takespecificities of queries versus collection documents into account.1Parameters PLSAParamsPLSADocsParamsPLSAQueriesParamsDocsRepXapian::DatabaseDocsRep PLSARepDocsPLSAQueriesPLSA(parameters where included in docsreps)DocsRep PLSARepDocsPLSAQueriesPLSAParameters PLSAParamsPLSADocsParamsPLSAQueriesParamsXapian::DatabaseXapian::DatabasePLSARepDocsPLSAQueriesPLSACurrent state (clear, but parallel, separation)Multiple Inheritance (seems appealing)Original state (not separation at all)Figure 1: Conception of the Topian layer for PLSA2TREC::DocId foobar, foo, bar, foo2Xapian::DocId 1, 2, 23, 42Topian::DocId 0, 1, 2, 3Figure 2: Example of document identifiers2 Document identifiersDocument identifiers in NLP toolchains can take several types: int, string,etc. In the concrete case of using Xapian and Topian to process TREC data, wefind ourselves with three flavours of identifiers. TREC identifiers are arbitrarystrings; Xapian identifiers, typed Xapian::DocId are non-continuous integersstarting from 1, 0 being reserved for out-of-collection documents; and Topianuses continuous integers starting with 0. We will name these three types ofidentifiers TREC::DocId, Xapian::DocId and Topian::DocId (see figure 2).Xapian documents have a string container whose content can be accessedfrom Xapian::Document::get data() member. The Xapian::DocId of a doc-ument can be retrieved either from the document itself with the Xapian::Document::get docid()member, or by dereferencing an iterator over the Xapian::Documents with the operator*() const.Over the course of the processing, we switch from the TREC identifiers tointernal identifiers, and back again. This allows abstracting the TREC formatif necessary (for instance when using the software on another format), and usingmore efficient algorithms that maps of strings.Two standards containers of the std library could be used to map differenttypes of docids: maps and vectors. Vectors map continuous integer indices to atype. They have an access time in O(1). Maps allow using any type as indice; inparticular, they allow using strings or non-continuous integers as indices. Theiraccess time is in O(log n). Using an iterator, going through a whole container(map or vector) is O(n). Hence it is adventageous to use vectors for punctualaccesses, but performance is not lost if a whole map is processed linearly.We propose to implement the translations between the formats as follow:• Xapian to Topian: the corresponding Topian::DocId is stored in a std::mapcontained in DocsRep. This data must not be contained in the dataattribute of the Xapian::Documents because we need it to contain theTREC::DocId, which is set there at indexing time. Furthermore, data isof the std::string type, and it is unclean to set Topian things in theXapian database.• Topian to Xapian: a std::vector of Xapian::DocId is created.Topian::DocID 0 1 2 3 . . .Xapian::DocID 1 2 23 42 . . .Access being is O(n), this structure can be used for punctual access (byopposition to skimming through the whole collection)• Topian to TREC: a std::vector of TREC::DocId is created. This vectorcan simply be constructed at indexing time, since the Xapian::DocIdindexing scheme is trivial.3Topian::DocID 0 1 2 3 . . .TREC::DocID foobar foo bar foo2 . . .This makes it possible to access the TREC::DocId of any document inO(1).It is straigthfowards that Xapian to TREC is obtained through Topian, alsoin O(1). With the orientation of the data flow, it is not necessary to trans-late TREC::DocId into the internal indices Xapian::DocId and Topian::DocId(which should stay transparent TREC-wise anyway). Hence, we have a wholeset of translators.4

Topian 0.1 Reference Manual

http://infoscience.epfl.ch/record/115459/files/topian.pdf

Topian 0.1 Reference Manual

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne