1 research outputs found
Text Extraction and Retrieval from Smartphone Screenshots: Building a Repository for Life in Media
Daily engagement in life experiences is increasingly interwoven with mobile
device use. Screen capture at the scale of seconds is being used in behavioral
studies and to implement "just-in-time" health interventions. The increasing
psychological breadth of digital information will continue to make the actual
screens that people view a preferred if not required source of data about life
experiences. Effective and efficient Information Extraction and Retrieval from
digital screenshots is a crucial prerequisite to successful use of screen data.
In this paper, we present the experimental workflow we exploited to: (i)
pre-process a unique collection of screen captures, (ii) extract unstructured
text embedded in the images, (iii) organize image text and metadata based on a
structured schema, (iv) index the resulting document collection, and (v) allow
for Image Retrieval through a dedicated vertical search engine application. The
adopted procedure integrates different open source libraries for traditional
image processing, Optical Character Recognition (OCR), and Image Retrieval. Our
aim is to assess whether and how state-of-the-art methodologies can be applied
to this novel data set. We show how combining OpenCV-based pre-processing
modules with a Long short-term memory (LSTM) based release of Tesseract OCR,
without ad hoc training, led to a 74% character-level accuracy of the extracted
text. Further, we used the processed repository as baseline for a dedicated
Image Retrieval system, for the immediate use and application for behavioral
and prevention scientists. We discuss issues of Text Information Extraction and
Retrieval that are particular to the screenshot image case and suggest
important future work