This presentation will tackle the concept of partial compliance from several angles. One angle is on the questions of measurement, metrics and methods. The other angles look at the implied value of (partial) compliance to consumers, providers, and funders

Buchhorn, Markus

The Australian National University

1TDR: What does partial compliance mean?Markus.Buchhorn@anu.edu.au2Why me?Ex-astronomerReally picky about the scientific method, metrics, measurements & metadataMultiple hatsANU, APAC, GrangeNet, and participation in many programsLots of use-cases, in a broad diversity of disciplinesPhysical sciences, social sciences, education and researchScholarly input, as well as scholarly outputsSmall to large scale, short to long termAll of it extremely valuableAPAC/APSR survey of e-research collectionsAround 50 projects analysed in-depthReally keen on the idea of ‘certification’ and ‘recognition’3DisclaimerAsked by APSR/NLA to give this talkSuggested I be ‘contentious’… ;-)Have not lived the experience like some hereNot looking to be editorThough I did spot a few grammatical errorsHave not read the draft repeatedlyKeep finding new anglesHave missed some things, and the updated thinking4What does “what does partial compliance mean”mean?What does “partial compliance” mean?Measurements, metrics and methodsWhat does partial compliance “mean”?i.e. who cares, and why?5What is “trust”?Broad review Philosophy, sociology, dictionaries, …Not by me!Boiled down to:Makes life predictableCreates a communityMakes it easier to collaborate6AcquireCreateObservePurchaseDigitisePre-processCalibrate ValidateVersion 5.1Markus BuchhornRepositionCacheMirrorThumbnailStoreIngestAccessPresentRegister AcceptDownloadVirtualiseFederateStreamPreviewPermit PublishAnnounceOrganiseApproveManage, CurateData MgmtMetadata  MgmtStorage MgmtAccess MgmtArchiveSustainValue DeletePreservePost-processComputeAnalyseMergePresentUseVisualiseCollaborateAuthorDeriveIndividualSourcePolicyInstitutionalDomain StandardsGovtFundersAnonymise ProcessMgmtData process classifiers7What does “partial compliance” mean?It is a measureCan you only be “in” or “out”?Can you be “some number” along the path?On a scale of 0-100, you’re a …Is “compliance” like “pregnancy”?Yes: Getting there is half the work…Yes: You can be or be not…No: You can go backwards, and sidewaysStaying compliant…8Measurements and metricsCan you measure a degree of compliance Per item, per category, overall?Currently: Thought about it, wrote it down, built it, tested itThese are steps on a path, but it’s the quality of the implementation we’re measuringCan we associate some quantitative measurements of progress?Can we compare the impact of individual compliance elements against each other?“this element is twice as important as that one”“they’re all equally important”“this repository is twice as compliant as that one”Probably not…It may depend on who is measuring9Measurements and metricsPolicies: what you’d like to happenCan test for existence, probably can’t measure it – does that help?“I have a policy not to document everything”. It’s valid!Procedures: what you think should happenCan test for existence, probably can’t measure it – does that help?Practises: what actually happensCan measure this, But only at a given point in time Existence of policies, procedures does not mean they are followedWho can guarantee the existence of an institution?10If we have partial compliance…We have some “level of compliance”1-gold star to 5-gold starCan we prioritise compliance requirements?“What do I need for my first gold star?”Can we be more compliant in some areas than others?“really nice policies, shame about the technology”A single number can hide too much11MethodsWho watches the watchers?i.e. who measures the auditors?Different auditors need to provide same answers given same inputs: CalibrationHow much of the audit could we automate?Who keeps an eye on compliance?Most elements involve humansCompliance can be attained and lost, repeatedlyMaintain, review, test, and re-audit; trigger on changes to the audit report package?12Where does it stop, horizontally?Associated repositories for data movementFederated repositoriesData moves for Performance (caching)Protection (mirroring)Policy (de-identification)Outside of my administrative domainBut strongly linked with itHow do I build trust in copies from authoritative sources? Does the local repository inherit some trust? Can a federation be made trustable?13Where does it stop, vertically?Designated Communities, domains Want to trust the dataNeed to trust the processes that created itWhich may be way before the SIP is built1-star lodgement effort into a 5-star repository? Or 5 into 1?Repositories can’t expect to have sufficient domain expertise in-house, for evermorebe able to engage with a domain for evermoreSome domains didn’t exist before, or still exist!deal with every format, software that a domain can use?Unless you treat some of it opaquely?Some of this should not be the repository’s problem14Where does it stop ??Problems with authentication, authorisationExternal identity providers for authentication, External policy providers for authorisation How do we measure trust in them?C3.3 has downstream obligation, but no upstream obligation?Who takes responsibility that policy is correctly expressed, identifiers are correctly provided and these things are correctly implementedDocumentation of accesses, modifications, using identifiers that may not be unique long term re-use of usernames15Don’t we need positives and negatives?B5.2 hasReview inappropriate “access denials”But probably also needReview inappropriate “access approvals”!C3.2 hasRecord accesses that “meet the requirements”But probably also needRecord accesses that “don’t meet the requirements”!16Do we have 3 states of being?Not compliantHow could you be that bad??Fully CompliantHow could you be that good??Partially compliantSufficient, in some/many cases?Users probably care about “just how compliant”And depending on their relationship, different elements matter17What does partial compliance “mean”?i.e. who cares and why?4 key playersConsumersProviders FundersRepository Providers18Consumers careThey want to trust the dataFor each first-time access to a new datasetFor each recurring access to a particular datasetTrust scopethe original data, the process that got it in there, the process that kept it there, the process that got it out of therePredictability, community, collaborationProbably only care about a fraction of the auditable elements, and care about some not-audited elements19Producers careWant the content to reflect what they providedIt’s an additional cost to them to lodge dataWant to leave a legacyCollect once, re-use foreverWant to gain recognition for the effortLodgement of scholarly input data as a form of publicationRequires a repository to be seen like a journalProbably care about most of the elementsMay actually be a stronger relationship20Funders CareNeed to trust the whole scholarly processFrom research funding, through collection, to lodgement, and downstream re-useMay be asked to recognise the effortOr may enforce a requirementRequires measurement of valueRecognition is worth how much?Probably care mostly about how much the users care!21Repository Providers careWhat does it attract for them?Status as trust-able facilityTo providers, consumers, and fundersSupports arguments for ongoing supportHow many repositories have guaranteed futures?22In closingI think this is crucialLots of things will be built on top of thisI think this is hardLots of boundary issuesLots of measurement issuesI think this will all be solvedI think this is all very very good

What Does Partial Compliance Mean?

Abstract

Similar works

Full text

Available Versions

The Australian National University