Text Mining has become a specialized offshoot of Data Mining, Information Retrieval, and Natural Language Processing. One of the major tools of this area is the vector space representation of documents. On the other hand, social network analysis has found its mathematical underpinnings primarily in mathematical graph theory. A graph has a dual representation as an adjacency matrix. So-called two-mode social networks have actors of two different types, frequently individuals and organizations. The adjacency matrix for these two-mode social networks has the same structure as the so-called term-document matrices used in text mining. In the talk we discuss these connections and show how these ideas can be exploited in both fields. In particular, methods for block modeling in social network analysis can be used for document clustering.Army Research Office, Contract W911NF-04-1-0447Army Research Laboratory, Contract W911-NF-07-1-0059National Institute on Alcohol Abuse And Alcoholism, Grant Number F32AA015876Isaac Newton Instituteunpublishednot peer reviewe

Wegman, Edward J.

English

Illinois Digital Environment for Access to Learning and Scholarship Repository

  Document Clustering and Social NetworksEdward J. WegmanGeorge Mason University, College of ScienceClassification Society Annual MeetingWashington University School of MedicineJune 12, 2009  Outline Overview of Text Mining Vector Space Text Models– Latent Semantic Indexing Social Networks– Graph and Matrix Duality– Two Mode Networks– Block Models and Clustering Document Clustering with Mixture Models Conclusions and Acknowledgements  Text Mining Synthesis of …– Information Retrieval Focuses on retrieving documents from a fixed database Bag-of-words methods May be multimedia including text, images, video, audio– Natural Language Processing Usually more challenging questions Vector space models Linguistics: morphology, syntax, semantics, lexicon – Statistical Data Mining Pattern recognition, classification, clustering  Text Mining Tasks Text Classification– Assigning a document to one of several pre-specified  classes Text Clustering– Unsupervised learning – discovering cluster structure Text Summarization– Extracting a summary for a document– Based on syntax and semantics Author Identification/Determination– Based on stylistics, syntax, and semantics Automatic Translation– Based on morphology, syntax, semantics, and lexicon Cross Corpus Discovery– Also known as Literature Based Discovery  Text Preprocessing Denoising– Means removing stopper words … words with little semantic meaning such as the, an, and, of, by, that and so on.– Stopper words may be context dependent, e.g. Theorem and Proof in a mathematics document Stemming– Means removal suffixes, prefixes and infixes to root– An example: wake, waking, awake, woke  wake  Vector Space Model Documents and queries are represented in a high-dimensional vector space in which each dimension in the space corresponds to a word (term) in the corpus (document collection). The entities represented in the figure are q for query and d1, d2, and d3 for the three documents.  The term weights are derived from occurrence counts.  Vector Space Methods The classic structure in vector space text mining methods is a term-document matrix where– Rows correspond to terms, columns correspond to documents, and– Entries may be binary or frequency counts. A simple and obvious generalization is a bigram (multigram)-document matrix where– Rows correspond to bigrams, columns to documents, and again entries are either binary or frequency counts.  Vector Space Methods•  La tent Semantic  Indexing (LSI) is a  tec hnique tha tp rojec ts queries and  doc uments into a  spac e withla tent semantic  d imensions.•  Co-oc c uring terms a re p rojec ted  into the samesemantic  d imensions and  non-c o-oc c uring termsonto d ifferent d imensions.•  In la tent semantic  spac e, a  query and  adoc ument c an have high c osine simila rity even ifthey do not sha re any terms as long as their termsare semantic a lly simila r ac c ord ing  to the c o-oc c urenc e ana lysis.  Latent Semantic Indexing•  LSI is the app lic a tion of Singula r Va lueDec omposition (SVD) to the term-doc ument matrix.•  SVD takes a  matrix  and  represents it as  in aWF Wlower d imensiona l spac e suc h tha t the two-norm isminimized , i.e. .1 1  •  The SVD p rojec ts an -d imensiona l spac e onto a••  T h e-d imensiona l spac e where    Latent Semantic Indexing•  In our app lic a tion to word -doc ument matric es, •is the number of word  types (terms) in the c orpus(doc ument c ollec tion).•  Typ ic a lly  is c hosen between 100 to 150.••  The SVD p rojec tion is c omputed  by dec omposingthe term-doc ument ma trix  into the p roduc t oft 11 three matric est h r e e mtt   †where .– – – – – – – –  Latent Semantic Indexing•  These ma tric es have  c olumns. Thisorthonorma lmeans the c olumn vec tors a re of unit leng th andare orthogona l to eac h other. In partic ula ra r e  o r t h† †(the identity ma trix)•  The d iagona l matrix  c onta ins the • singula r va luesof  in desc end ing order. The  singula r va lueso fofind ic a tes the amount of va ria tion a long the  axis.i i n•  By restric ting  the matric es  and   to the first•  By  •  B y  r e st r c olumns, we ob ta in and         †withw i t h um n sw www www  †  LSI - Some Basic Relations•  ° Ê Ä S @ ² ÉTQo| Ç  s    † † † † † † †† † † † † †† ² ÉTQ o  | Ç t  † † † † † †• • ij²R : Q ojÿ ÿÿ       ] u† † † † † † †† † †† † † † ojÿ ÿ † †•• † † † † † † ojÿ ÿÿ       ] u  † † † † † †  Social Networks Social networks can be represented as graphs– A graph G(V, E), is a set of vertices, V, and edges, E– The social network depicts actors (in classic social networks, these are humans) and their connections or ties– Actors are represented by vertices, ties between actors by edges There is one-to-one correspondence between graphs and so-called adjacency matrices  Example: Author-Coauthor Networks  Graphs versus Matrices  Two-Mode Networks When there are two types of actors– Individuals and Institutions – Alcohol Outlets and Zip Codes– Paleoclimate Proxies and Papers– Authors and Documents– Words and Documents– Bigrams and Documents SNA refers to these as two-mode networks, graph theory as bi-partite graphs– Can convert from two-mode to one-mode   Two-Mode ComputationConsider a bipartite individual by institution social network. Let Am×n be the individual by institution adjacency matrix  with m = the number of individuals and n = the number of institutions. ThenCm×m = Am×nATn×m=Individual-Individual social network adjacency matrix with cii = ∑jaij = the strength of ties to all individuals in i’s social network and cij = the tie strength between individual i and individual j.  Two-Mode ComputationSimilarly, Pn×n = ATn×m Am×n=Institution by Institution social network adjacency matrix with pjj=∑iaij= strength of ties to all institutions in i’s social network with pij the tie strength between institution i and institution j.  Two-Mode Computation Of course, this exactly resembles the computation for LSI. Viewed as a two-mode social network, this computation allows us: – to calculate strength of ties between terms relative to this document database (corpus)– And also to calculate strength of ties between documents relative to this lexicon If we can cluster these terms and these documents, we can discover:– similar sets of documents with respect to this lexicon– sets of words that are used the same way in this corpus  Example of a Two-Mode NetworkOur A matrix  Example of a Two-Mode NetworkOur P matrix  Block Models A partition of a network is a clustering of the vertices in the network so that each vertex is assigned to exactly one class or cluster.  Partitions may specify some property that depends on attributes of the vertices.  Partitions divide the vertices of a network into a number of mutually exclusive subsets. – That is, a partition splits a network into parts.  Partitions are also sometimes called blocks or block models. – These are essentially a way to cluster actors together in groups that behave in a similar way.   Example of a Two-Mode NetworkBlock Model P Matrix - Clustered   Example of a Two-Mode NetworkBlock Model Matrix – Our C Matrix Clustered  Example Data The text data were collected by the Linguistic Data Consortium in 1997 and were originally used in Martinez (2002)– The data consisted of 15,863 news reports collected from Reuters and CNN from July 1, 1994 to June 30, 1995– The full lexicon for the text database included 68,354 distinct words In all 313 stopper words are removed after denoising and stemming, there remain 45,021 words in the lexicon– In the examples that I report here, there are 503 documents only  Example Data A simple 503 document corpus we have worked with has 7,143 denoised and stemmed entries in its lexicon and 91,709 bigrams.– Thus the TDM is 7,143 by 503 and the BDM is 91,709 by 503.– The term vector is 7,143 dimensional and the bigram vector is 91,709 dimensional.– The BPM for each document is 91,709 by 91,709 and, of course, very sparse. A corpus can easily reach 20,000 documents or more.  Term-Document Matrix AnalysisZipf’s Law  Term-Document Matrix Analysissaipeoplonnorthgosaidheretwooutjustcnntimewellknowkoreareportgetrightstatethinknewofficikoreanseeareahousdailiketodaipresidhourlookwhitecit iyearf irsthallbuilds illtalktakehappenunitbecometfirewantmornlastbombbackworkthankpolicjupitthingkimreleasdefensthroughsouthclinictryoff iclivetoldamericangoodcallmake waipilotjoincontinugoverncaseinvestigeventellpointmileagoweekfederquestionplanejohnmilitariairlotbelievhelphelicoptsimpsonhitcrashastronomclintonscenetestsomethsecurrescufarnationoklahomahearkindpossiblkobeilexpectablhopemandeathexplosinformusworldrealliforckillnumberjapanhomeevidissuserbfactasktrainsituatcourspartanythseencourtcomeearthquakshotmadeincidlatestfamilidealcountrisearchfewremainlit t lplacecenterchiefyesterdaisevercloseshowauthoropersalviimpactf indbloodjudgabortpicturinvolvattorneiearthwhethermeetcamenuclearchargsidepersonstartiraqlongmeanspaceproblemsecondneedreturndepartyorkwentnightearliermrwaterbosnian fragmentsuspectlaterstorigroundbobbiapparconcerngivebodileftseemheardshootleaderawaiwashingtonhandsureservicbigfoundsawdamaginjurplanetmissionsafecarminutindicfloodturndnairaqigenerprosecutbosniawordmoveagainstleastresponsappeareffortmonthperhapfeelsortf iveflightconfirmopennatomassachusettquaktelescopcausprobablfbidone123456789101112131415161718192021222324252627 2829303132333435363738394041424344454647484950515253545556575859606162636465666768697071723747576 777879 80 81828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145 146147148149150151152153154155156157158159160161162163164165 166167168169170171172173174175176177178179180181 182183184185186187188189190191192193194195196197981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632645266267268269270271272273274275276277278279280281282283284285286287288289290291292 293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378 3793803813823833843853863873883893903913923933943953963973983940040140240340440540640708409410411412413414415416417418419420421422423424 425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503  Mixture Models for Clustering Mixture models fit a mixture of (normal) distributions We can use the means as centroids of clusters Assign observations to the “closest” centroid Possible improvement in computational complexity

Document Clustering and Social Networks

Abstract

Similar works

Full text

Available Versions

Illinois Digital Environment for Access to Learning and Scholarship Repository

IDEALS @ Illinois