Ukgovld registry-webinar-v3

UKGovLD Registry

A register. Image by http://www.flickr.com/photos/tekniskamuseet/

UKGovLD Registry Outline

• Background and motivation• Details

– information model– core functionality– advanced topics– status

• Applying it• Discussion and next steps

UKGovLD Registry Motivation

Support coordinated publication of reference information as Linked Data

We want people to use URIs to identify things– codes and code lists– ontologies, thesauri and concepts– units of measure– organizations– spatial objects– licenses– coordinate reference systems– ...

UKGovLD Registry but it’s not trivial ...

Each organization use its own namespace?– need infrastructure and expertise to serve the data– fragmented namespace– where do consumers get the complete authoritative list?

Shared namespace?– how to allocate new identifiers in that shared space?– how to collaboratively maintain identifiers and lists?– how to scale?

Either way:– how to manage the life-cycle of introducing and

deprecating terms?

UKGovLD Registry Linked Data Registry

Infrastructure to enable organizations to create and manage identifiers and authoritative lists– allocation and registration of approved codes,

vocabularies and reference URIs– management of shared namespaces, including

clash-free allocation and delegation– registration of published datasets supporting

discovery and cross-reference

UKGovLD Registry Three functions

• identifier & list management – create, update, reserve, lifecycle, versioning ...

• repository– managed identifiers (information served from registry) as well as

referenced identifiers• namespace management

– delegate a list or part of namespace to others– federate to other repositories

=> very general platform– can register anything you can give a URI and RDF description– use only the aspects you need for a given situation– “registry does X” does not mean it answers every X-shaped need you

might have

UKGovLD Registry Conceptual architecture

router

renderer

requestprocessor

user credentials

roles and bindings

auth

registrycorelogic

Registry RDF store

text index

style and templates

external UI

adminUI

logaudittrail

stor

e AP

I

nginx

proxy

confAPI

UKGovLD Registry Running examples

• Code list of colours– simple, neutral

• UK Local authorities– centrally created identifiers v. local creation?– local control and maintenance?– offloading management





UKGovLD Registry Information model

• hierarchy of Registers• things in registers are Register Items – metadata• thing described by a RegisterItem called an entity

/

/reg1 /def /system

/def/colours /def/weather-types

/def/colours/_red /def/colours/_green

(describes /def/colours/red) (describes /def/colours/green)

Naming convention: metadata item has “_”


complicated by:– item v. entity– versioning

reg:Register entity

reg:register

reg:definition

reg:entity

reg:RegisterItem reg:EntityReferencereg:RegisterItem reg:EntityReference


• default linked data view of Register is simplified• configurable

– alternative membership property or inverse property– so can make a register look like a skos:Collection, skos:ConceptScheme,

owl:Ontology ...– also acts as a LDP container

• but can request full view (?_view=withMetadata)

reg:Register entity

reg:register

reg:definition

reg:entity

reg:RegisterItem reg:EntityReferencereg:RegisterItem reg:EntityReference

induced membership relation default is

rdfs:membercontainer view

full view

UKGovLD Registry Full information model

• Derives from ISO 19135 Geographic information - procedures for item registration

• Also considered OASIS ebRIM– not so suited to flexible RDF model– RegRep API did provide API checklist

• Container model compatible with Linked Data Platform (1st draft!)

• Versioning model based on approached used in Environment Agency linked data

UKGovLD Registry

reg:Register

rdfs:label [1..*]dct:description [1..*]reg:owner [1] (foaf:Agent)reg:manager [1] (foaf:Agent)dct:license [0..*]reg:containedItemClass [0..*]reg:operatingLanguage [0..*]reg:governancePolicy [0..*] (rdfs:Resource)reg:validationQuery [0..*]dct:modified [0..1] (inferred)void:uriLookupEndpoint [0..*]void:uriSpace [0..1]void:exampleResource [0..*]void:openSearchDescription [0..*]

reg:subregister

reg:RegisterItem

rdfs:label [1..*]dct:description [0..*]dct:dateSubmitted [1] (automatic)dct:dateAccepted [0..1]dct:modified [0..1] (inferred)reg:itemClass [1..*]reg:submitter [1] (foaf:Agent)dct:license [0..*]reg:status [1..*]reg:category [0..*] (skos:Concept)reg:notation [0..1]reg:alias [0..*]reg:hasView [0..*]reg:representationOf [0..*]

reg:register

reg:predecessor

version:Version

owl:versionInfo [1]

time:Inteval

version:interval

dct:replacesdct:replacedBy

version:VersionedThing

version:currentVersion

reg:Statusreg:statusNotAccepted

reg:statusSubmittedreg:statusInvalid

reg:statusAcceptedreg:statusValid

reg:statusExperimental reg:statusStable

reg:statusDeprecatedreg:statusSupersededreg:statusRetired

dct:isVersionOf

reg:status

reg:EntityReference

reg:entity [1]reg:sourceGraph [0..1]

reg:definition

Void:Dataset

ldp:Container

ldp:membershipPredicatereg:inverseMembershipPredicate

Full information model

UKGovLD Registry Status lifecycle

UKGovLD Registry Status lifecycle

• controls visibility– not accepted => doesn’t appear (register view, validation)

• guidance to consumers– experimental/stable => no effect on behaviour

• workflow is external– registry provides machinery to

• store status• control who is authorized to change the status• some validation of submissions • list submissions in a given state

– how/whether submissions are scrutinized, how decision is made up to the register manager





UKGovLD Registry Functions: basic access

• read (GET)– entity, item (its metadata), specific versions– register (metadata, list of contents, page-able, prior states)– content negotiation – Turtle, RDF/XML, JSON/LD, HTML

• entity lookup– return what registry knows about any URI

• validate– test a set of URIs to see they are known and Accepted

(within a register sub-tree)• search

– free text search (within a register sub-tree)

UKGovLD Registry Demo

• Browse of colours register• RDF of colours register and a colour

UKGovLD Registry Functions: Registration

• single entity– POST simple entry to register– basic metadata created automatically– UI allows for extensible forms to create entries

• entity with metadata– include explicit RegisterItem to annotate– allows you to register an external URI– can register multiple entries at once this way

• batch registration– register a complete collection – register + contents– e.g. upload a complete skos:ConceptScheme or skos:Collection– can configure bulk types

UKGovLD Registry Status and validation

• some automated checks at registration– minimal properties of entity (label, type)– type matches one of (optional) allowed types for this register– register can specify SPARQL ASK queries for validation

• item status– default is “Submitted”– can set initial status in API or explicit metadata, if you have the authority– can change status through API or UI following the lifecycle– administrators can force a status change outside lifecycle

• approval process– is outside the registry itself– just have property to state the policy and API/UI to do the updates– some operations may have their own complex workflow

UKGovLD Registry Versioning and history

• changes to an item generates new version – metadata record is versioned _red:1 _red:2 ....– separate copy of entity (separate named graph)

• Register is versioned if items added – or their status is changed

• API supports– list versions– version at point in time (including for Register)– get version


• Add a new colour via form• Change its status• Update it• View history• Upload a complete concept scheme• Show registration of referenced entities





UKGovLD Registry Federation, delegation and namespaces

reg:Delegated

reg:delegationTarget

reg:NamespaceForward

reg:forwardingCode [0..1]

reg:DelegatedRegister

reg:enumerationSubject [0..1]reg:enumerationPredicate [0..1]reg:enumerationObject [0..1]

reg:Register

reg:FederatedRegister

reg:forwardingCode [0..1]


Case 1: External entities– identifier published in different namespace– want to include it in authoritative list

Solution:– just register as a referenced entity– already seen this– authoritative because it’s on the list– can record properties of the entity, and maintain

history– no namespace management involved

UKGovLD Registry Referenced entities

/local

/id

/local-authority

Registry External service e.g. opencommunities.org

Hosted by LA directly

UKGovLD Registry

Case 2: Namespace allocation– want someone else to serve part of the registry namespace– might be a single item or a complete register sub tree– e.g. allocating namespace in location.data.gov.uk for serving

INSPIRE spatial object identifiers

Solution:– reg:NamespaceForward– can be a redirect (30X) or proxy (200)– no constraints on whether target acts like a Registry – target ought to serve linked data with URIs in the right

namespace, but not required

Federation, delegation and namespaces

UKGovLD Registry Namespace forward

/local

/id

/local-authority

Registry External web sitecould be anything


Case 3: Federated register– want someone else to run part of the registry

infrastructure but act like one big registry– integrated search, validation etc

Solution:– reg:FederatedRegister– can be a redirect (30X) or proxy (200)– target endpoint must comply with Registry API at

least for search, validation and entity lookup

UKGovLD Registry Federated register

/local

/id

/local-authority

RegistryFederated registry

/local-authority

/id


Case 4: Delegating a register– some one else to serve the list of contents of the

register– but they only have triple store, not full registry

implementation

Solution:– solution eg:DelegatedRegister– specify SPARQL endpoint and triple

pattern to enumerate members

reg:DelegatedRegister

reg:delegationTarget [1]reg:enumerationSubject [0..1]reg:enumerationPredicate [0..1]reg:enumerationObject [0..1]

UKGovLD Registry Delegated register

/local

/id

/local-authority

RegistryExternal SPARQL service


• Proxy namespace• Delegated register (if time)

UKGovLD Registry Security model

• authentication– OpenID (e.g. Google, Google profile)

• authorization– permissions

• Register, Update, StatusUpdate, Force, Grant, GrantAdmin• inherit down the tree• e.g.: Register,Update:/example/local

– can grant to known user or anyone authenticated– bundled into roles

• Maintainer – Update, Grant• Manager – Register, StatusUpdate, Update, Grant• Authorized – Register, Update, StatusUpdate - for experimental areas• Administrator - anything

UKGovLD Registry message queue infrastructure (tbd)

• publish/subscribe on notification of changes– plug in audit trail recording– replicate to duplicate stores for read-only load

balance– hook for workflow

• scheduling analysis of registrations– e.g. analyse registered datasets to map and

visualize identifier usage

UKGovLD Registry Status

• Proof of concept– API implementation build on top of Jena and Lucene– everything except federated search– UI based on simple templating– proxy mode handled by dynamic reconfig of external nginx– open source (license details to be agreed)

• Remaining planned work– notification – registration of datasets– packaging and deployment instructions– optional: some GML rendering support?

• Next steps– pilot applying it

UKGovLD Registry Conceptual architecture

router

renderer

requestprocessor

user credentials

roles and bindings

auth

registrycorelogic

Registry RDF store

text index

style and templates

external UI

adminUI

logaudittrail

stor

e AP

I

nginx

proxy

confAPI





UKGovLD Registry Local authorities example

• illustrated various options– maintain list in register

• some might be managed in registry, LA can edit• some might be served by LA• some might be served by DCLG• single list supporting search, validation

– DCLG run a registry service and federated (or just forward) part of namespace

– DCLG maintain master list in triple store, use delegated register

UKGovLD Registry Dataset registration

• would like to know how identifiers are being used and how they connect data sets

• support registration of RDF data sets– queue separate analysis suite to extract usage information– create metadata– add these back into the registry record – support visual browse

• POC does not include analysis and visualization but provides hooks to third party modules

• e.g. Epimorphics InfoMap visualizer

UKGovLD Registry Dataset analysis

UKGovLD Registry

WMO application of Registry

publishing existing authoritative code-tables as web-accessible

resources

43

UKGovLD Registry WMO data exchange governance

WMO No. 306 Manual on Codes: the ‘crown-jewels’ of WMO … decades of expert effort in developing and maintaining a ‘shared language’ for describing meteorological phenomena.

44

UKGovLD Registry Model-driven data exchange standards

WMO data product specifications are being updated in line with community best practice ; a model driven approach based on the ISO 19100-series of International Standards enables a semantic model to be encoded in a variety of formats – including GML. This is very similar to the approach taken within INSPIRE.

45

GML/XML data products

« co

nfor

ms

to »

SCH

XML Schema 1.0

Schematron .

Validation schema and rules

Validatable data products

*******************

******************

*****************************

« de

rived

from

»

Application Schema(aka ‘conceptual model’)

Technology independent description of content and structure of information to

be exchanged for a given application

FullMoon

UKGovLD Registry Binding data model to WMO code-tables

46

Excerpt from BUFR edition 4 Code- and Flag-tables

IWXXM METAR/SPECI

«CodeList»RunwayDeposits

URI: http://codes.wmo.int/bufr4/codeflag/0-20-086

tagged values:vocabulary = “http://codes.wmo.int/bufr4/codeflag/0-20-086”

extensibility = “none”

«DataType»AerodromeRunwayState

+ runway : Runway [0..1]+ cleared : Boolean [0..1]+ contamination : RunwayContamination [0..1]+ snowClosure : Boolean [0..1]+ depositType : RunwayDeposits [0..1]+ depthOfDeposit : DepthOfDeposit [0..1]+ brakingAction : BrakingAction [0..1]+ frictionCoefficient : FrictionCoefficient [0..1]

UKGovLD Registry Code-table as web-accessible register

47

UKGovLD Registry Referencing terms from data products

48

UKGovLD Registry

NAMESPACE FORWARD:

publishing a simple ontology within the namespace of the registry

49

UKGovLD Registry Pre-determined properties

WMO code-tables (e.g. Common code-table C-6: List of units) have a number of specific properties defined …

50

UKGovLD Registry Published definitions required

To represent the information from the code-tables in RDF we need to reference definitions of those properties …

51

UKGovLD Registry A simple ontology

As the properties support the definition of the code-tables themselves, the definitions are published within the namespace of the registry …

52

UKGovLD Registry A simple ontology (with #frag IDs)

Following de facto practice, the definitions are published using the #fragment identifier pattern …

53

UKGovLD Registry Namespace forward (proxy)

The #fragment identifier pattern means that the registry’s “/ID” ‘containment’ pattern is not applicable – therefore we register the entire ontology as a single DELEGATED resource using a PROXY pattern

54

UKGovLD Registry Namespace forward: configuration

The NAMESPACE FORWARD is configured using the web UI by an authenticated user with appropriate permissions …(but can also be done programmatically via the RESTful API)

55

UKGovLD Registry Externally managed HTTP server

54.246.66.105 is an anonymous HTTP server (hosted on Amazon EC2) with content negotiation configured for HTML, RDF/XML and TTL – all of which are provided as static resources …

56

UKGovLD Registry

ukgovld-registry.dnsalias.net

Apache2 + mod_negotiation

54.246.66.105

NGINX

UKGovLD Registry Tracking changes; version & history

This approach enables one to track changes as new versions of the ontology are published; with each new version published to a new directory one can amend the delegation target to update the registry.

57

UKGovLD Registry Discussion

• use cases for where to apply this?• missing features needed for these?• what would it take to pilot one?

UKGovLD Registry Links

• Design notes and API detailshttps://github.com/der/ukl-registry-poc/wiki

• Proof of concept deploymenthttp://ukgovld-registry.dnsalias.net/ No service guarantees

https://github.com/der/ukl-registry-poc/wiki

http://ukgovld-registry.dnsalias.net/

UKGovLD Registry

UKGovLD Registry Convenient views

• full RegisterItem/Register structure complex• versioning makes that a lot worse//registry

RegisterVersionedThing

//registry:1

RegisterVersion

//registry/_reg

RegisterItemVersionedThing

//registry/_reg:1

RegisterItemVersion

//registry/reg

RegisterVersionedThing

//registry/reg/_foo

RegisterItemVersionedThing

//registry/reg/foo

(entity)//registry/reg:1

RegisterVersion

//registry/reg/_foo:1

RegisterItemVersion

dct:versionOf

dct:versionOf dct:versionOf

dct:versionOf

reg:register

reg:register

reg:definition

reg:definition

//registry/_reg:2

RegisterItemVersion

//registry:2

RegisterVersion

//registry/reg:2

RegisterVersion

//registry/reg/_foo:2

RegisterItemVersion

Documents

Ukgovld registry-webinar-v3