As part of its project to develop a new research data management system the University of Lincoln is embracing development practices built around APIs - interfaces to the underlying data and functions of the system which are explicitly designed to make life easy for developers by being machine readable and programmatically accessible

Jackson, Nick

Winn, Joss

English

University of Lincoln Institutional Repository

Eating Your Own Dog FoodJackson, Nick JohnUniversity of Lincolnnijackson@lincoln.ac.ukWinn, Joss LukeUniversity of Lincolnjwinn@lincoln.ac.ukJuly 2012AbstractAs part of its project to develop a new research data management system the University ofLincoln is embracing development practices built around APIs – interfaces to the underlying dataand functions of the system which are explicitly designed to make life easy for developers by beingmachine readable and programmatically accessible.1 The Goals of OrbitalThe Orbital project at the University of Lincoln aims to build a new research data management platform,alongside creating new policies and guidance for the institutional management of data. This platform,also named Orbital, will include a variety of features which distinguish it from a traditional repositoryby encouraging programmatic use and re-use of research data not only after a project has ended, butthroughout the entire life of the project. Once a project has ended, Orbital will encourage deposition of‘complete’ research data in more formal repositories for long-term archiving by allowing the rapid con-densation and collation of research data, automatically supplying metadata where possible, encouraginghigh quality metadata where it is manually supplied, and automating the process of depositing a recordin a repository.2 What is a Repository?The majority of existing repository solutions are built on the premise that items deposited in the reposi-tory will be discrete digital objects1 and that these objects will have associated metadata describing theircontents. In some cases, a deposit will consist of multiple discrete digital objects grouped together witha common set of metadata. In the case of research data, repositories tend to adopt a similar approach —data exists in discrete objects (for example an Excel spreadsheet, a ZIP file of images, a binary blob) andhas attached metadata which describes it. Occasionally these repositories will oﬀer the ability to viewthe contents of certain types of file natively within the application, although this is dependent upon therebeing an available parser or interpreter for the data. While this is trivial to achieve for data stored insimple, open formats such as CSV, it becomes increasingly diﬃcult as data is represented using complexstructures in formats such as XML, and is nearly impossible in many proprietary binary blob formats.Commonly, such repositories also oﬀer a limited choice of ways to deposit and otherwise interact withcontent. In most cases they rely on web-based user interfaces which are accessed through a browser tomanage the complete lifecycle of an item from uploading through curation, management and long-termarchiving. The SWORD2 protocol, while oﬀering a programmatic method of depositing and updatingfiles, does not oﬀer methods of interacting with the data itself, but rather its metadata and the containerfile of the data.In designing Orbital, we are assuming that research data may be too complex or too large for storageand transmission in file containers such as CSV or XML. In surveying researchers at the University ofLincoln, the most pressing requirement was for managed storage of research data during the processof research itself[4]; among the research groups that Orbital works with in the School of Engineering,requirements also centre around the sharing and management of data during the research process. In ourexperience so far, working in the domain of RDM highlights that the approach taken by the repository1Or, in some cases, digital pointers to physical objects.1community for the management of research papers is inadequate for the management of the research dataitself and requires a reconceptualisation of what data actually is. A file-oriented approach is inappropriatefor many use cases, where researchers are collecting and analysing relational data or real-time data frominstrumentation, for example. By designing an RDM tool that respects the form of data (e.g. time-series) rather than imposing a format (e.g. CSV), an API-driven approach to depositing, querying,manipulating and archiving the data, provides interfaces that are native to the domain of research data,rather than research outputs.3 Our Approach3.1 Love Your BackendThe University of Lincoln is moving towards a software architecture in which APIs – interfaces whichallow systems and pieces of software to communicate in an easily machine-readable manner – are definedand built before any interface intended for the end user. Orbital is built as an example of this thinking,that the ‘backend’ of any system should be given attention earlier than the user-facing aspects. This isnot in conflict with a user-centric approach to application development, but rather acknowledges thatuser requirements should be deeply embedded in the design of the application and not simply manifestin a single user interface. In the case of applications which manage raw data, traditional user interfacedesign may be entirely inappropriate and inadequate. Many researchers already have appropriate toolsfor working with data, which could not and need not be replicated by a research data managementtool, such as Orbital. The value of RDM to such researchers is the ability to reliably store, share andselectively publish their data for scrutiny and citation.The result of building a system in this manner is that it is inherently usable and extendable by anyother system with a minimum of eﬀort. In repository systems which lack an API the process of addingdata to the repository may involve multiple steps of exporting data from a database, importing it toan application such as Excel, sorting the data, exporting a spreadsheet, creating a new repository item,uploading the exported data, adding metadata, saving and publishing. Where an API is available thisprocess can be automated and, as part of Orbital’s pilot scheme, users in the School of Engineering aredirectly loading sensor data from a variety of sources.In addition to providing these interfaces to the data, Orbital is exposing the entire feature set of therepository over APIs. This is achieved by ensuring that the development team build APIs which canexpose the required functions to an external interface application rather than including any end-userinterface functionality in Orbital’s ‘core’.Finally, since Orbital is adopting a loosely coupled RESTful2 approach to APIs, no part of Orbital’score functionality has any particular notion of statefulness associated with it. What this means froma practical standpoint is that it is easy to scale the platform horizontally by adding additional serversor storage, since there is no requirement for any process to maintain communication with a particularinstance of a component.3.2 Eat Your Own Dog FoodThe Orbital project consists of two distinct components: Orbital Core, which performs all repository, datamanagement, security and processing functions; and Orbital Manager, which provides a web-accessibleuser interface. The Manager aspect of Orbital communicates with the Core exclusively over the sameAPIs which are publicly available and documented for general use, with no preferential treatment oradditional permissions as shown below in Figure 3.A large part of building APIs lies in the documentation of available functionality both of the un-derlying code and of external interfaces. As Orbital is developed using iterative, agile principles and‘Continuous Integration’3, this documentation is automatically generated, and it is this documentationthat is relied upon by the Orbital developers where they are unfamiliar with a part of the system. Theuse of a single canonical set of documentation for both internal and external development avoids thecreation of undocumented APIs which can only be used with expert system knowledge.2REpresentational State Transfer3https://en.wikipedia.org/wiki/Continuous_integration2Figure 1: The only way to interact with this application is to either be a user, or pretend to be one (forexample via screen-scraping).Figure 2: The most common form of API, consisting of a ‘second view’ on the data and functionality ofan application. This style of API often exposes a limited subset of the application’s functionality.Figure 3: In an API-driven model the API is the only way to interface with the application.33.3 Forget The ContainersThere are already mature research output repository systems which store containers of data (i.e. ‘files’)for the long term, providing archival and discovery services. Orbital is not intended to be a replacementfor such systems. However, there is a distinct lack of general purpose research data management systemswhich store and help manage the data throughout the actual research process. Although Orbital doesallow researchers to upload and store discrete files[5], and we intend to provide a shared, synchronisedfilesystem for researchers[6], a defining feature of the approach of Orbital is that – as far as is possible– data can be removed from container files such as CSV files or Excel spreadsheets and stored in adatabase[2].Access to and control of this data is provided by RESTful APIs, which align with Orbital’s overalldesign. These APIs can be used for the deposition of data either in large blocks or at a very granular level(roughly equivalent to a single row in a spreadsheet), the modification and retrieval of those individualdata points, and querying of the data sets in real time using a variety of database operators, and exportingthe results of those queries as a rich JSON document or a simplified CSV document which can readily beused for further analysis. APIs are also planned to allow for the description of data (including aspectssuch as the type of data) and the addition of metadata on a data set or data point level (for exampleallowing annotation of an anomalous result to be stored with the result itself).Data itself is stored in the MongoDB No-SQL database, which provides a number of advantagesover traditional SQL databases with regard to research data storage. MongoDB is a document storagedatabase which holds representations of data objects as opposed to structured, tabular data. Theseobjects may be heterogeneous with little or nothing in common, a feature which allows individual datapoints to contain diﬀering quantities or properties depending on what is known about them. This meansthat a data point may initially be deposited with relatively little information and subsequently updatedwith additional facts, notes or analysis after the fact and with no impact on the underlying storageengine. As explored in Section 3.4, data may also be subject to automated processing which expandsupon the original data.By removing the data from container files and allowing direct deposition and access, Orbital exposesdata for re-use the moment it has been stored. This dramatically reduces the amount of time spentwaiting for data to be made available to researchers following its collection, which is of particular interestto the project’s pilot users in the Lincoln School of Engineering. Where previously data could takeupwards of a week to be moved from its collection point through various storage systems and thenmanually exported in the appropriate format for further research, Orbital now produces daily datasetsfor the same work.3.4 Make It ExtensibleDue to the document storage model and API accessibility used within Orbital it is relatively simplefor data objects to be updated with additional information by the means of single purpose, automatedapplications. Again in the case of the Lincoln School of Engineering, some data requires processing witha variety of signal filters in order to become usable for further statistical modelling. Previously thiswould have been a batch operation in which the researcher would take the original data, perform theprocessing, and store the output in a separate file. Using Orbital’s data storage engine the results ofthe processing are stored alongside the original data and are similarly queryable or usable in subsequenttasks.By encouraging the breaking down of data into smaller chunks, Orbital also encourages the reductionof the processes involved in research into smaller, more manageable and above all re-usable aspects. Itis envisioned that researchers will be able to build complex automated workflows, including import fromexternal sources, processing of data using pre-built or custom modules, and subsequent export to files orspecialist tools from directly within Orbital.One thing that Orbital isn’t designed to do is replicate the functionality of more traditional andestablished repositories. Although Orbital is designed to be able to hold files in perpetuity followingthe end of a project it will also use protocols such as SWORD2 to allow rapid deposition of a record ofresearch data – including all relevant metadata – with an existing institutional repository.44 Why Data-Driven Development?4.1 We Build Better StuﬀAdopting a data-centric, API driven approach to building Orbital has resulted in the development ofAPIs which are easily understood, easily implementable, behave in a reliable and predictable manner andwhich are robust enough to withstand heavy usage. APIs which lack these criteria are quickly identifiedand refactored, since they are being used by members of the Orbital team to build the Manager platform.Since APIs are the exclusive method of interfacing with the underlying data storage and the functionsof Orbital the APIs must expose a complete set of functionality[3].Reliability of these APIs is also critical to the usability of the entire data storage platform, whichencourages better design of resiliency and error handling; and usability of the API is essential whichencourages better documentation. By forcing the consideration of data types, methods and architectureearly on it is easier to achieve a consistent design and behaviour across the entire Orbital platform, aswell as an underlying structure which encourages the development of reusable, well-defined code.Finally, by combining the aspects of reusable components and consistent APIs, Orbital is able toeasily implement a robust universal security model using the OAuth 2 specification which encompassesall data access regardless of source. Since any frontend can only interface with the data via the controlledAPIs, and since access to the data is governed on a per-user-per-application basis there is a reducedscope for accidental or intentional damage or loss of data, and a lower risk of the accidental disclosureof commercially sensitive data. OAuth 2 also enables the development of third-party tools which caninteract with data in Orbital without exposing any user credentials[1], encouraging the building of toolswhich extend the ecosystem.4.2 Improved Data Visibility from Day OneThe DAF4 survey conducted as part of Orbital’s research shows that although most researchers knowroughly how much data they have, that data is stored primarily on non-managed devices such as internaland external hard-disks, USB memory sticks, and third-party data storage systems[4]. During usermeetings and observations of research by the Orbital team it was found that among these various locationsthere was often confusion over where a particular file was stored, which version any particular file maybe, and if any given file actually contained the data in question.By stripping away the container file and allowing searches to be run across the entirety of a data set– even up to the scale of millions of data points – the entire dataset becomes visible to the researcher atonce for the purposes of finding what they’re looking for.Orbital also has the potential to provide improved information on the quantity of data being storedfor the purposes of planning and reporting, giving academics a truer indication of the amount of resourcesbeing used and giving research administrators a clearer idea of the requirements for future research. Inaddition, since Orbital exposes its complete subset of functionality, it can quickly be integrated with otherUniversity systems such as staﬀ directories or our Awards Management System to assemble disparateinformation regarding research data.4.3 Scale To InfinityA major benefit of using loosely coupled components which communicate using RESTful APIs, alongwith using an inherently scalable database solution such as MongoDB, is that the Orbital platform canscale horizontally to a very large system simply by adding new nodes. An example of an expanded,scaled deployment is given in Figure 4 on the next page.Although in development Orbital is not using a deployment of this size, the application is designedto be inherently scalable as the number of users and size of datasets grows.5 The Problems of This ApproachAPI-driven development, whilst having many benefits, does have downsides which have become apparentduring the development of Orbital. Firstly, development must – in many instances – be ‘doubled up’as developers create both the underlying functionality and API endpoints in Orbital Core, and then an4Data Assets Framework5Figure 4: Example of a scaled Orbital environment, with sharded and replicated database, load balancedCore servers and load balanced Manager frontends.6API interface and user front-end in Orbital Manager. Although this does allow rapid development anddevelopment of better software at a later point initial work can appear slow and unnecessarily complex.Secondly, there is also a significant challenge in ensuring that the transfer of data over APIs is fastand reliable enough for intended applications. The HTTP protocol can introduce an element of overheadwhich can rapidly accumulate in applications which are making a large number of API calls, and insome cases the Orbital project has modified APIs to be more lightweight or to allow for more intelligentgrouping of requests. There has also been a significant amount of work and re-work around error handling— particularly with regard to correctly handling security-based errors, an area in which development isstill ongoing.References[1] Alex Bilbie. What is OAuth? [online]. July 2012. Available from: http://lncn.eu/dfps.[2] Nick Jackson. And now. . . dynamic data! [online]. June 2012. Available from: http://lncn.eu/cjv2.[3] Nick Jackson. Why orbital is all about the api [online]. January 2012. Available from: http://lncn.eu/euw5.[4] Joss Winn. Data assets framework survey summary [online]. April 2012. Available from: http://lncn.eu/rs2.[5] Joss Winn. A minimum viable product: Orbital v0.1 [online]. May 2012. Available from: http://lncn.eu/y26.[6] Joss Winn. Shared, versioned network drives [online]. May 2012. Available from: http://lncn.eu/egj7.7

Eating your own dog food

Abstract

Similar works

Full text

Available Versions

University of Lincoln Institutional Repository