Coates, Heather L.

Johnson, Jennifer

Polley, David E.

IUPUIScholarWorks

Visualizing the Indianapolis Recorder: How Libraries can PromoteDigital ScholarshipTed Polley Jenny JohnsonILF, November 19, 2014I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 0Indianapolis Recorder-Background-One of the top African-American publications in the nation-Weekly publication began in 1899-In 2009 IUPUI University received permission  to digitize collectionI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 1Processing the collection-Contracted with Lyrasis to digitize microfilm-Use CONTENTdm to provide access-OCR extension-5202 issues, 90,461 of pagesI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 2Challenges with Microfilm-Poor Image QualityI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 3Challenges with MicrofilmIssues with OCRI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 4Collection Inquiries-Basic questions from general public about how to use CONTENTdm-basic browsing and searching -Printing specific articles-In recent years there has been an increase in requests from scholars and researchers-Interest in text miningI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 5Potential Scholarly Uses for the RecorderI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 6This resource is potentially valuable to researchers studying:– The History of Indianapolis’ African American community– The evolution of language over time– The changes in content and structure of a newspaper to meet changing community needs and interestsSupporting Digital ScholarshipI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 7Digital humanities include a broad range of scholarly activities drawing from the field of computing and disciplines in the humanities.Projects can include the curation of cultural datasets in digital collections, analysis of large text corpora, social network analysis, and data visualization.As part of its mission, the University Library Center for Digital Scholarship provides support and consultation to IUPUI scholars engaged in, or planning digital humanities projects. Promoting the Recorder to IUPUI ResearchersI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 8To promote this valuable resource to the rest of IUPUI’s campus the Center for Digital Scholarship conducted an exploratory study using basic approaches to text mining and visualization.We focused on characterizing the semantic content of the Recorder, creating topic maps to facilitate exploration of this resource.Selecting a SampleI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 9The full text of the newspaper with metadata can be downloaded in either XML or tab-delimited format.The first (1899) and the last (2005) years in the collection were selected as a sample.The transcripts for every issue for those years were extracted from the dataset, metadata were stripped, and the issues from each year were combined and saved as two text files.  Generating the VisualizationsI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 10The transcripts for each year were visualized using VOSviewer, a free program for analyzing text and generating topic maps.The program calculates term frequency, how often terms co-occur, and statistical distributions of terms to determine their relevancy in the overall corpus.Van Eck & Waltman, 20111899 Top 10 Most Frequently Occurring TermsTerm Occurrences Relevancechurch 802 0.66pastor 745 0.84negro 500 0.62club 433 0.61guest 423 0.93matter 328 0.74meeting 328 0.57smith 304 0.87money 295 0.28johnson 284 0.84I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 111899 Top 10 Most Relevant TermsTerm Occurrences Relevancegold standard 13 2.99silver certificate 20 2.96silver dollar 25 2.88greenback 10 2.78certificate 23 2.73redemption 13 2.64coin 32 2.59metal 11 2.51silver 31 2.49statute 21 2.33I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 121899 Topic Map13I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S1899 Topic Map14I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S2005 Top 10 Most Frequently Occurring TermsTerm Occurrences Relevancechurch 13721 0.76service 13071 0.69home 12964 0.43state 9873 0.29president 9376 0.30pastor 8461 3.16community 8199 0.44team 7854 1.74game 7440 1.60club 7435 0.94I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 152005 Top 10 Most Relevant TermsTerm Occurrences Relevanceregular service 494 5.80pastor Sunday school 185 5.76evening worship 549 5.67prayer meeting 288 5.57bypu 447 5.57p u 459 5.57service Sunday school 243 5.52marion suparior [sic] court 197 5.33morning worship 1615 5.31sundayschool 338 5.24I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 162005 Topic Map17I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S2005 Topic Map18I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I SGoals for OCR ImprovementI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 19Improve information retrievalIncrease the utility of the text corpus as data– Text mining – Cross-linking information about people, places, or organizations from multiple sources– Other types of digital humanities researchWhere does this leave those of us managing digital collections of historical newspapers?Options for ImprovementI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 21Scan from paper copies (instead of microfilm)Manual correctionTrain the OCR engineCreate a custom dictionaryHolley, 2008Arlitsch & Herbert, 2004Decision Making ProcessI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 22Scan from paper copies (instead of microfilm)– Not feasible to get print copies to re-digitizeManual correction– Manual correction is extremely time-consumingTrain the OCR engine– No ability to train the OCR engine within ContentDMCreate a custom dictionary– Possible, if we could over some a few challengesCustom Dictionary (of what?)I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 23Proper nouns – names and place names are harder for OCR engines to process (Tanner et al, 2009)After scanning the front pages of several issues, we decided to create custom dictionaries for Indianapolis and Central Indiana centric:– People– Places– OrganizationsExperiment: Custom DictionaryChallenges– OCR software (ABBYY FineReader) is run on the server– No ability to import custom dictionariesWork-arounds (for this experiment)– Purchase a local installation of ABBYY FineReader 12– Run OCR on the sample manuallyI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 24Experiment DetailsConsiderations– Relatively low time investment (manually running OCR by issue is slow) compared to other approaches– Potential to improve retrieval of proper names most significant to Indianapolis Recorder readers and local historiansSample: – 6 issues randomly selected each year (1960 –1969)– 60 issues (+ 2 special issues) in total– Generated a random sample in Excel (RAND function) across the 10-yr timeframeI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 25ProcessCreating the dictionaries– Manually read front pages of selected issues– Enter people, place, and organization names used into Excel– Converted excel tabs into text files – Copied text files into designated ABBYY dictionary folderPre & Post OCR– Local installation of ABBYY FineReader 12– Pre – OCR was run using default installation and settings– Post – OCR was run with dictionaries in placeI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 26ResultsBoth pre and post OCR files were combined into two text files.These two files were then visualized using VOSviewer to determine if OCR correction makes a difference in resulting visualizations.I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 27Pre OCR Top 10 Most Frequently Occurring TermsTerm Occurrences Relevancecourt 266 0.60ave 196 0.63estate 171 0.72sale 168 0.62funeral service 138 0.85marion county 138 0.69notice 136 0.90defendant 122 0.68complaint 116 0.82clerk 115 0.90I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 28Pre OCR Top 10 Most Relevant TermsTerm Occurrences Relevanceabsence 29 2.13basileus 12 2.13chevrolet 16 2.13controversy 16 2.13eagle 22 2.13james burres 28 2.13middle 16 2.13pta 11 2.13school system 23 2.13ship 17 2.13I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 29Pre OCR Topic MapI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 30Pre OCR Topic MapI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 31Post OCR Top 10 Most Frequently Occurring TermsTerm Occurrences Relevanceestate 197 0.74notice 160 0.92funeral service 145 1.18marion county 143 1.04funeral home 125 1.28report 121 0.75probate court 118 0.95matter 118 0.74defendant 114 0.76clerk 108 0.94I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 32Post OCR Top 10 Most Relevant TermsTerm Occurrences Relevanceabsence 28 1.98chevrolet 17 1.98comment 13 1.98disciple 15 1.98eagle 25 1.98extension 19 1.98highway 13 1.98lafayett 15 1.98lafayett square 13 1.98n central 11 1.98I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 33Pre OCR Topic MapI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 34Pre OCR Topic MapI N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 35AcknowledgementsThanks to Lisa Calvert for her help in creating the dictionaries. Thanks to Anna Proctor and Lucy Williams for their input and thoughtful suggestions during the planning phase of this project.I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 36References1. Allen, R. B., Waldstein, I., & Zhu, W. (2008). Automated Processing of Digitized Historical Newspapers: Identification of Segments and Genres. Lecture Notes in Computer Science, 5362, 379-386.2. Holley, R. (2009). How good can it get?: Analysing and improving OCR accuracy in large scale Historic Newspaper Digitisation Programs. D-lib Magazine, 15(3-4).3. Klijn, E. (2008). The current state of art in newspaper digitisation. A market perspective. D-Lib Magazine, 14(1/2). doi: 10.1045/january2008-klijn4. Strange, C., McNamara, D., Wodak, J., & Wood, I. (2014). Mining for the meanings of a murder: The impact of OCR quality on the use of digitized historical newspapers. Digital Humanities Quarterly, 8(1).5. Van Eck, N.J., & Waltman, L. (2011). Text mining and visualization using VOSviewer. ISSI Newsletter, 7(3), 50-54 .I N D I A N A  U N I V E R S I T Y – P U R D U E  U N I V E R S I T Y  I N D I A N A P O L I S 37

Visualizing the Indianapolis Recorder: How Libraries can Promote Digital Scholarship

https://scholarworks.iupui.edu/bitstream/1805/14776/1/polley_2014_visualizing_the.pdf

Visualizing the Indianapolis Recorder: How Libraries can Promote Digital Scholarship

Abstract

Similar works

Full text

Available Versions

IUPUIScholarWorks