Serving images can be a digital object repository\u27s bread and butter, but can touch on logistical and policy complexities. By wrapping the Python based  Loris  image server in a homegrown proxy, and caching with Varnish, we\u27ve been able to improve our image delivery, while keeping stakeholders happy as well

Hukill, Graham

English

Digital Commons@Wayne State University

Wayne State UniversityLibrary Scholarly Publications Wayne State University Libraries7-1-2016Wrapping an Image Server in Proxy and CacheBlanketsGraham HukillWayne State University, ej2929@wayne.eduThis Presentation is brought to you for free and open access by the Wayne State University Libraries at DigitalCommons@WayneState. It has beenaccepted for inclusion in Library Scholarly Publications by an authorized administrator of DigitalCommons@WayneState.Recommended CitationHukill, Graham, "Wrapping an Image Server in Proxy and Cache Blankets" (2016). Library Scholarly Publications. Paper 118.http://digitalcommons.wayne.edu/libsp/118Wrapping an Image Server inProxy and Cache BlanketsCode4Lib Midwest, 2016Graham Hukill, Wayne State University+What is Loris?● IIIF compliant, python based Image Server● Github page"Loris is an implementation of the IIIF Image API 2.0."What is IIIF?● "International Image Interoperability Framework"● Suite of APIs for interacting with IIIF compliant image servers○ Image API○ Presentation API○ Search APIStructure:{scheme}://{server}{/prefix}/{identifier}/{region}/{size}/{rotation}/{quality}.{format}Example:http://www.example.org/image-service/abcd1234/full/full/0/default.jpgReal Image Example:http://digital.library.wayne.edu/loris/fedora:wayne:CFAIEB01c493%7CEB01c493_JP2/full/full/0/default.jpgReal Info Example:http://digital.library.wayne.edu/loris/fedora:wayne:CFAIEB01c493%7CEB01c493_JP2/info.jsonVarnish HTTP CachePython MiddlewareCacheProxyPython MiddlewareProxyProxy: why? ● More control of image delivery pipeline● Improve security and reliability of Loris● Prepare for access challenges / opportunities in the futureUse Python "IIIFImageClient" class(Pulled directly from Emory University's Readux, with some additions)● accept an IIIF image request● parse the components● do things...● reconstitute IIIF URL● make request to Loris● stream back bits to clientLoris ProxyLoris Proxyhttp://digital.library.wayne.edu/loris/fedora:wayne:CFAIEB01c493%7CEB01c493_JP2/full/full/0/default.jpgExample IIIF image request:IIIFImageClient Python ClassLoris Proxy@ /lorisvia middlewareImage RequestrequestAPP_HOST/loris/ID/...requestlocalhost/loris_local/ID/...Proxy for LorisClientServerLoris@ /loris_localvia ApacheIIIFImageClientprocessese.g. Resizing● Collections Stakeholders want to limit resolution of outgoing images to 700 px● Not true for all collections, based on RDF relationship in FedoraDigital Dress Collection Detroit News Collectione.g. Rotation / Mirroring● Some images are scanned horizontally flipped (mirrored), or rotated● Maybe don't want to / can't edit the original?● Add RDF relationships at datastream level that instruct Loris Proxy to adjust<wayne:vmc4617/vmc4617_JP2> <fedora-rels-int:needsRotation> "45" .<wayne:vmc10132/vmc10132_JP2> <fedora-rels-int:needsRotation> "!" .RDF Relationshipse.g. Rotation / Mirroringe.g. Rotation / MirroringVarnish HTTP CacheCacheFilesystem (Directories)LorisImage RequestImage API CacheDefault Loris Cache:FilesystemImages1. Reads images from disk2. Caches Image API callsFileSystem (Directories)LorisImage RequestImage API CacheDefault Loris Cache:HTTPHTTP CacheFedora1. Requests images from Fedora2. Caches to disk in HTTP Cache3. Reads images from disk4. Caches Image API callsIf Python can treat URLs as file-like objects (e.g. StringIO), and Loris is written in Python......why can't Loris treat Fedora URLs as file-like objects and serve them without the additional caching layer?HTTP CacheCache: why?● Improve image delivery speed● Reduce strain on Fedora● Simpler cache cleaning and maintenanceFedora@ /fedoravia ApacheLoris@ /loris_localvia ApacheLoris Proxy@ /lorisvia middlewareVarnish@ :6081Image RequestIIIF Image APIcacheFedora JP2 cachecache and returnrequestAPP_HOST/loris/ID/...requestlocalhost:6081/loris_local/ID/...cache and returnClientServerCache with Varnishrequest /localhost:6081/fedora/...HTTP CacheIIIFImageClientprocesses"I could have sworn we updated that..."● Sometimes resources are updated, moved, etc.● Need to remove specific objects● Created methods in middleware for thatFileSystem (Directories)LorisImage RequestImage API CacheDefault Loris CacheHTTP CacheFedora● Image API cache is turned off, handled by Varnish now● HTTP Cache still exists, as Loris needs to read something from diskVarnishCacheCache Clearing1. Remove cached objectfrom Loris based onIIIF URL identifier2. "Ban" URLs that contain the digital object PID in Varnish3. Voila!Loading Collection tiles (no cache)Loading Collection tiles (with cache)Loading Single Image (no cache)Loading Single Image (with cache)Varnish "hits" and "misses"Some numbersavg. secondsLoris IIIF OFF, Loris HTTP ON, Varnish for IIIF ON, Varnish for Fedora ON 0.053Loris IIIF OFF, Loris HTTP ON, Varnish for IIIF OFF, Varnish for Fedora OFF 0.244Loris IIIF OFF, Loris HTTP ON, Varnish for IIIF OFF, Varnish for Fedora ON 0.256Loris IIIF OFF, Loris HTTP OFF, Varnish for IIIF OFF, Varnish for Fedora OFF 0.353Loris IIIF ON, Loris HTTP ON, Varnish for IIIF OFF, Varnish for Fedora OFF 0.118Fedora@ /fedoravia ApacheLoris@ /loris_localvia ApacheLoris Proxy@ /lorisvia middlewareVarnish@ :6081Image RequestIIIF Image APIcacheFedora JP2 cacherequest /localhost:6081/fedora/...cache and returnrequestAPP_HOST/loris/ID/...requestlocalhost:6081/loris_local/ID/...cache and returnClientServerThanks!Code4Lib Midwest, 2016Graham Hukill, Wayne State UniversityHTTP CacheIIIFImageClientprocesses

Wrapping an Image Server in Proxy and Cache Blankets

https://digitalcommons.wayne.edu/cgi/viewcontent.cgi?article=1122&amp;context=libsp

Wrapping an Image Server in Proxy and Cache Blankets

Abstract

Similar works

Full text

Available Versions

Digital Commons@Wayne State University