Stream Reasoning Workshop - ods.tu-berlin.de · Stream Reasoning Workshop Technical University Berlin SR2016 ... service

Stream Reasoning WorkshopTechnical University Berlin

SR2016

James Anderson <[email protected]>

<http://dydra.com/>@dydradata @lomoramic

Tara Athan <[email protected]>

<http://www.digitale-vernetzung.org>

http://rdfabout.com/


• What

• Where• Device Production Logs

• Device Production Equipment

Stream Reasosning Workshop

Use Case : Traceability

• What

• Where• Device Production Logs• Device Production Equipment



• Traceability


• Start with a device identifier• Trace through production to origin• Determine :

• Involved equipment : failure analysis

SR 2016

• Supplied customer : claims analysis• IP Interaction


Scenario

Stream Reasosning WorkshopSR 2016


Scenariorootsawacs-hub awacs

datahub/awacs/roots federation

data-hubapplication

deviceID

deviceIDpackage

Indi

vidu

alP

rodu

ct

SP

AR

QL:

trac

eabi

lity-

view

deviceIDpackage

deviceID packingUnitID

custName custName

stripID

stripID

Inpu

tMat

eria

lWor

kord

erworkOrderID

deviceIDworkOrderID

wor

kord

erE

xecu

tion-

Fo

rDev

ice

packingUnitID

AW

AC

S n

avig

atio

n

deviceIDpackingUnitID

logi

stic

s in

fo

data-hub SPARQL processor


SPARQL

SR 2016

construct { # ... }where { { service <https://awacs.local/sparql?auth_token=0123456789> { values ($package $deviceID) { ('SOT1207' '067654389') }

[] a tr:IndividualProduct ; tr:name $deviceID ; tr:package $package ; tr:fromStrip ?stripID ; tr:isPresent ?isPresent . filter (?isPresent = 'True') [] a tr:InputMaterialWorkorder ; tr:name ?stripID ; tr:workorder ?workOrderSHA1 ; tr:mapRole ?stripRole .

[] a tr:WorkorderExecutionForDevice ; tr:name ?workOrderSHA1 ; tr:package $package ; tr:endProduct $deviceID ; tr:mapName ?mapName ; tr:mapRole ?mapRole ; tr:hasDevice ?mapHasDevice .

filter (?mapHasDevice = 'yes') [] a tr:WorkorderAttributes ; tr:name ?workOrderSHA1 ; tr:workorderId ?workorderId ; tr:workstationId ?workstationId ; tr:timestamp ?timestamp . } } bind(concat($package, '.', xsd:string($deviceID)) as ?BATCH_MARKED) bind (iri(concat('http://qa.data.nxp.com/id/trace/individualProduct/', ?BATCH_MARKED)) as ?IndividualProduct) bind (iri(concat('http://qa.data.nxp.com/id/trace/workorder/', ?workOrderSHA1)) as ?stage_uri) bind (if(?stripRole = 'in', tr:MergePack, tr:Assembly) as ?stage)

values (?mapRole ?inputOutputId_pred) { ('in' tr:inputId) ('out' tr:outputId) }

bind (concat(substr(?mapName, 1, 2), lcase(substr(?mapName, 3, 1)), substr(?mapName, 4)) as ?packingUnitID) optional { service <https://www.ws-qa.tibco.nxp.com/DummyTestingURL/RootsSparql/sparql> { # lookup a known puid for SHIP stage [] ded:BATCH_LOT_ID ?packingUnitID ; ded:STAGE_FK 'SHIP' ; ded:CUST_PO_NBR ?CUST_PO_NBR ; ded:CUST_NBR_SHIP_FK ?CUST_NBR_SHIP_FK ; ded:DED_DT ?DT_SHIP ; ded:LOC_CD_FK ?LOC_CD_FK_SHIP ; ded:PROD_12NC_FK ?PROD_12NC_FK . } } bind (iri(concat('http://qa.data.nxp.com/id/plm/salesItem/', ?PROD_12NC_FK)) as ?PROD_12NC_FK_URI) bind (iri(concat('http://qa.data.nxp.com/id/trace/ship/', ?packingUnitID)) as ?ship)


construct?IndividualProduct rdf:type tr:IndividualProduct?IndividualProduct tr:model ?PROD_12NC_FK_URI?IndividualProduct tr:batchMark ?BATCH_MARKED

?IndividualProduct tr:stage ?stage_uri?IndividualProduct tr:stage ?ship

?stage_uri rdf:type ?stage?stage_uri tr:batchLotId ?workorderId

?stage_uri tr:sha1 ?workOrderSHA1?stage_uri tr:location ?workstationId

?stage_uri ?inputOutputId_pred ?mapName?stage_uri tr:date ?timestamp

?ship rdf:type tr:Ship?ship tr:batchLotId ?packingUnitID?ship tr:location ?LOC_CD_FK_SHIP?ship tr:shipTo ?CUST_NBR_SHIP_FK

?ship tr:purchaseOrder ?CUST_PO_NBR?ship tr:date ?DT_SHIP

extend ?ship : ( iri ( concat "http://qa.data.xmpl.com/id/trace/ship/" ?packingUnitID ) )

extend ?PROD_12NC_FK_URI : ( iri ( concat "http://qa.data.xmpl.com/id/plm/salesItem/" ?PROD_12NC_FK ) )

leftjoin

extend ?packingUnitID : ( concat ( substr ?mapName 1 2 ) ( lcase ( substr ?mapName 3 1 ) ) ( substr ?mapName 4 ) ) service <https://www.ws-qa.tibco.xmpl.com/D2RQ/sparql>

join

bind (?mapRole ?inputOutputId_pred) :(("in" tr:inputId)

("out" tr:outputId))

extend ?stage : ( if ( = ?stripRole "in" ) tr:MergePack tr:Assembly )

extend ?stage_uri : ( iri ( concat "http://qa.data.xmpl.com/id/trace/workorder/" ?workOrderSHA1 ) )

extend ?IndividualProduct : ( iri ( concat "http://qa.data.xmpl.com/id/trace/individualProduct/" ?BATCH_MARKED ) )

extend ?BATCH_MARKED : ( concat ?package "." ( string ?deviceID ) )

service <https://agent.remote/sparql?auth_token=0123456789>

<https://agent.remote/sparql?auth_token=0123456789> filter ((?isPresent = 'True') &&(?mapHasDevice = 'yes'))

join

bgpworkOrderSHA1

timestampworkstationIdworkorderId

join

bgpstripID

stripRoledeviceID

mapHasDevicemapRolemapNamepackage

workOrderSHA1

join

bgpdeviceIDisPresentstripIDpackage

bind (?package ?deviceID) :(("SOT1207" "067654389"))

<https://www.ws-qa.tibco.xmpl.com/D2RQ/sparql>

bgpCUST_NBR_SHIP_FK

PROD_12NC_FKLOC_CD_FK_SHIP

DT_SHIPCUST_PO_NBR

packingUnitID

Traceability :Algebra Graph

SR 2016 Stream Reasosning Workshop

CONSTRUCT?IndividualProduct rdf:type tr:IndividualProduct?IndividualProduct tr:model ?PROD_12NC_FK_URI?IndividualProduct tr:batchMark ?BATCH_MARKED







extend ((?PROD_12NC_FK_URI ( iri ( concat "http://qa.data.xmpl.com/id/plm/salesItem/" ?PROD_12NC_FK ) )) (?ship ( iri ( concat "http://qa.data.xmpl.com/id/trace/ship/" ?packingUnitID ) )))

serviceleftjoin <https://www.ws-qa.tibco.xmpl.com/D2RQ/sparql>

<https://www.ws-qa.tibco.xmpl.com/D2RQ/sparql> select (?CUST_NBR_SHIP_FK ?CUST_PO_NBR ?DT_SHIP ?LOC_CD_FK_SHIP ?packingUnitID ?PROD_12NC_FK) extend ((?packingUnitID ( concat ( substr ?mapName 1 2 ) ( lcase ( substr ?mapName 3 1 ) ) ( substr ?mapName 4 ) )))

bgpCUST_NBR_SHIP_FK


DT_SHIPCUST_PO_NBR

packingUnitID

join

bind (?inputOutputId_pred ?mapRole) :((tr:inputId "in")

(tr:outputId "out"))

extend ((?BATCH_MARKED ( concat ?package "." ( string ?deviceID ) )) (?IndividualProduct ( iri ( concat "http://qa.data.xmpl.com/id/trace/individualProduct/" ?BATCH_MARKED ) ))

(?stage_uri ( iri ( concat "http://qa.data.xmpl.com/id/trace/workorder/" ?workOrderSHA1 ) )) (?stage ( if ( = ?stripRole "in" ) tr:MergePack tr:Assembly )))


<https://agent.remote/sparql?auth_token=0123456789> select (?deviceID ?isPresent ?mapHasDevice ?mapName ?mapRole ?package ?stripID ?stripRole ?timestamp ?workorderId ?workOrderSHA1 ?workstationId)

filter ((?isPresent = 'True') &&(?mapHasDevice = 'yes'))

join

bgpworkOrderSHA1


join

bgpstripID

stripRoledeviceID

mapHasDevicemapRolemapNamepackage

workOrderSHA1

join


bind (?package ?deviceID) :(("SOT1207" "067654389"))

SR 2016

Traceability : Hub Plan


project (deviceID isPresent mapHasDevice mapName mapRole package stripID stripRole timestamp workorderId workOrderSHA1 workstationId)

servicejoin ?location_WorkorderAttributes

?location_WorkorderAttributes select (?timestamp ?workorderId ?workOrderSHA1 ?workstationId) join

bgpworkOrderSHA1


bind (?host_WorkorderAttributes ?location_WorkorderAttributes) :(("www.ws-qa.tibco.xmpl.com" "https://www.awacs.com/awacs_cloudsearchwoattr.csv{?cloudsearchsha1}")) filter ((?mapHasDevice = 'yes'))

servicejoin ?location_WorkorderExecutionForDevice

?location_WorkorderExecutionForDevice select (?deviceID ?mapHasDevice ?mapName ?mapRole ?package ?workOrderSHA1)join ((QUOTE, (?host_InputMaterialWorkorder = '*') ||

('*' = ?host_WorkorderExecutionForDevice) || (?host_InputMaterialWorkorder = ?host_WorkorderExecutionForDevice)))

bgpworkOrderSHA1

deviceIDmapHasDevice

mapRolemapNamepackage

bind (?host_WorkorderExecutionForDevice ?location_WorkorderExecutionForDevice) :(("www.ws-qa.tibco.xmpl.com" "https://www.awacs.com/awacs_cloudsearchwoinout.csv{?cloudsearchsha1,cloudsearchpackage,cloudsearch2Did}")) servicejoin ?location_InputMaterialWorkorder

?location_InputMaterialWorkorder select (?stripID ?stripRole ?workOrderSHA1) join

bgpstripID

workOrderSHA1stripRole

bind (?host_InputMaterialWorkorder ?location_InputMaterialWorkorder) :(("www.ws-qa.tibco.xmpl.com" "https://www.awacs.com/awacs_cloudsearchwo.csv{?cloudsearchmapid}")) filter ((?isPresent = 'True'))

servicejoin ?location_IndividualProduct

?location_IndividualProduct select (?deviceID ?isPresent ?package ?stripID) join


bind (?host_IndividualProduct ?location_IndividualProduct) :(("www.ws-qa.tibco.xmpl.com" "https://www.awacs.com/awacs_cloudsearch2Did.csv{?cloudsearchpackage,cloudsearch2Did}"))

bind (?deviceID ?package) :(("067654389" "SOT1207"))

Traceability : Agent Plan


CONSTRUCT?IndividualProduct rdf:type tr:IndividualProduct?IndividualProduct tr:model ?PROD_12NC_FK_URI?IndividualProduct tr:batchMark ?BATCH_MARKED







extend ((?PROD_12NC_FK_URI ( iri ( concat "http://qa.data.xmpl.com/id/plm/salesItem/" ?PROD_12NC_FK ) )) (?ship ( iri ( concat "http://qa.data.xmpl.com/id/trace/ship/" ?packingUnitID ) )))

serviceleftjoin <https://www.ws-qa.tibco.xmpl.com/D2RQ/sparql>

<https://www.ws-qa.tibco.xmpl.com/D2RQ/sparql> select (?CUST_NBR_SHIP_FK ?CUST_PO_NBR ?DT_SHIP ?LOC_CD_FK_SHIP ?packingUnitID ?PROD_12NC_FK) extend ((?packingUnitID ( concat ( substr ?mapName 1 2 ) ( lcase ( substr ?mapName 3 1 ) ) ( substr ?mapName 4 ) )))

bgpCUST_NBR_SHIP_FK


DT_SHIPCUST_PO_NBR

packingUnitID

join

bind (?inputOutputId_pred ?mapRole) :((tr:inputId "in")

(tr:outputId "out"))

extend ((?BATCH_MARKED ( concat ?package "." ( string ?deviceID ) )) (?IndividualProduct ( iri ( concat "http://qa.data.xmpl.com/id/trace/individualProduct/" ?BATCH_MARKED ) ))

(?stage_uri ( iri ( concat "http://qa.data.xmpl.com/id/trace/workorder/" ?workOrderSHA1 ) )) (?stage ( if ( = ?stripRole "in" ) tr:MergePack tr:Assembly )))


<https://agent.remote/sparql?auth_token=0123456789> servicejoin ?location_WorkorderAttributes

?location_WorkorderAttributes select (?timestamp ?workorderId ?workOrderSHA1 ?workstationId) join

bgpworkOrderSHA1


bind (?host_WorkorderAttributes ?location_WorkorderAttributes) :(("www.ws-qa.tibco.xmpl.com" "https://www.awacs.com/awacs_cloudsearchwoattr.csv{?cloudsearchsha1}")) filter ((?mapHasDevice = 'yes'))

servicejoin ?location_WorkorderExecutionForDevice

?location_WorkorderExecutionForDevice select (?deviceID ?mapHasDevice ?mapName ?mapRole ?package ?workOrderSHA1)join ((QUOTE, (?host_InputMaterialWorkorder = '*') ||

('*' = ?host_WorkorderExecutionForDevice) || (?host_InputMaterialWorkorder = ?host_WorkorderExecutionForDevice)))

bgpworkOrderSHA1

deviceIDmapHasDevice

mapRolemapNamepackage

bind (?host_WorkorderExecutionForDevice ?location_WorkorderExecutionForDevice) :(("www.ws-qa.tibco.xmpl.com" "https://www.awacs.com/awacs_cloudsearchwoinout.csv{?cloudsearchsha1,cloudsearchpackage,cloudsearch2Did}")) servicejoin ?location_InputMaterialWorkorder

?location_InputMaterialWorkorder select (?stripID ?stripRole ?workOrderSHA1) join

bgpstripID

workOrderSHA1stripRole

bind (?host_InputMaterialWorkorder ?location_InputMaterialWorkorder) :(("www.ws-qa.tibco.xmpl.com" "https://www.awacs.com/awacs_cloudsearchwo.csv{?cloudsearchmapid}")) filter ((?isPresent = 'True'))

servicejoin ?location_IndividualProduct

?location_IndividualProduct select (?deviceID ?isPresent ?package ?stripID) join


bind (?host_IndividualProduct ?location_IndividualProduct) :(("www.ws-qa.tibco.xmpl.com" "https://www.awacs.com/awacs_cloudsearch2Did.csv{?cloudsearchpackage,cloudsearch2Did}"))

bind (?deviceID ?package) :(("067654389" "SOT1207"))

Traceability : Combined Plan


Use Case : Streaming SPARQL


(select (service ?trafficService (select (bgp (triple ?reading <http://linkedurbandata.org/traffic#speed> ?speed)) (:HAVING (> ?avgSpeed 70) (?passages (count ?reading)) (?avgSpeed (avg ?speed)) (?timestamp (then)))) (extend (join (select (bgp (triple ?incident <http://linkedurbandata.org/traffic#incident> ?trafficCamera) (triple ?incident <http://purl.oclc.org/NET/UNIS/sao/sao#Timestamp> ?timestamp)) (:HAVING (> ?offenses 100) :GROUP-BY (?trafficCamera (??KEY-1 (year ?timestamp)) (??KEY-2 (month ?timestamp))) ?trafficCamera (?offenses (count ?offense)))) (select (filter (bgp (triple ?trafficCamera <http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat) (triple ?trafficCamera <http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long)) (&& (>= ?lat 30623/1275) (&& (<= ?long 39079/1488) (&& (<= ?lat 35191/1176) (>= ?long 18923/800))))) (?trafficCamera))) ?trafficService (iri (concat (str ?trafficCamera) "?revision=R/HEAD/PT10M/PT01M"))) :QUERY-TEXT "SELECT ( COUNT ( ?reading ) AS ?passages ) ( AVG ( ?speed ) AS ?avgSpeed ) ( THEN ( ) AS ?timestamp ) WHERE { ?reading <http://linkedurbandata.org/traffic#speed> ?speed . } HAVING ( ?avgSpeed > 70 ) ") (?trafficCamera ?time ?passages ?avgSpeed))

Use Case : Streaming Algebra


select (?time ?trafficCamera ?passages ?avgSpeed)

filter ( > ?avgSpeed "70"^^<http://www.w3.org/2001/XMLSchema#integer> )

service ?trafficService

select (( ?passages ( count ?reading ) ) ( ?avgSpeed ( avg ?speed ) ) ( ?timestamp ( then ) ))

extend ?trafficService : ( iri ( concat ( str ?trafficCamera ) "?revision=R/HEAD/PT10M/PT01M" ) )

bgp

(?reading <http://linkedurbandata.org/traffic#speed> ?speed)

filter ( > ?offenses "100"^^<http://www.w3.org/2001/XMLSchema#integer> )

join

select (GROUP-BY ( ?trafficCamera ( ??KEY-1 ( year ?timestamp ) ) ( ??KEY-2 ( month ?timestamp ) ) )

?trafficCamera ( ?offenses ( count ?offense ) ))select (?trafficCamera)

bgp

(?incident <http://linkedurbandata.org/traffic#incident> ?trafficCamera) (?incident <http://purl.oclc.org/NET/UNIS/sao/sao#Timestamp> ?timestamp)

filter ( ( >= ?lat "24.01804"^^<http://www.w3.org/2001/XMLSchema#decimal> ) && ( <= ?long "26.26277"^^<http://www.w3.org/2001/XMLSchema#decimal> ) && ( <= ?lat "29.92432"^^<http://www.w3.org/2001/XMLSchema#decimal> ) ( >= ?long "23.65375"^^<http://www.w3.org/2001/XMLSchema#decimal> ) )

bgp

(?trafficCamera lat ?lat) (?trafficCamera long ?long)

Case : Requirements • Construct SPARQL dynamically

SR 2016

• Communication via standard protocols


• Algebra• Dataset designators

Dydra: RSP

[2]

SPARQLEngine

RDFStore

SPARQLQuery

GSPRequest

SPARQLUpdate

Graphor

Solu<onResponse


Dydra: RSP = media type


PostgreSQL

spreadsheetsources

(tablinker)

Java

Per

sist

ent S

tora

geC

loju

re

DYDRA RDF Store

SesameSAIL API

Jenaprovider

API

libpq readview

readtable

writetable

Agri-Esprit : development / research / deployment

text/csv

text/tab-separated-values

text/csv

text/arff

clj-plaza clojure.jena clojure.sesame

learningsources(vector

datasets)

stored procedures

Dydra: Query @Revision

[2]


Dydra : Revision Clauses• Constant

• absolute: REVISION UUID { ... }• relative : REVISION /HEAD(~[0-9]+)?/ { ... }

• Variable• SIP : { ?subject :revision ?revision } ... REVISION ?revision { ... }• Protocol : HTTP://...?REVISION=UUID ... REVISION ?revision { ... }

• temporal : REVISION XPathDateTime { ... }

• Intra-Repository : REVISION ?revision { ?subject a ?class } • Extra-Repository : SERVICE ?revisionEndpoint { ?subject a ?class }


Dydra : Revision Designators• Elementary

• absolute: UUID : 01234567-0123-0123-012345678901• relative : /HEAD(~[0-9]+)?/ : HEAD~1, HEAD~1..HEAD

• Compound• interval : ISOInterval : 20150303T180000Z/PT10M• stream : ISORepeatingInterval : R/20150303T180000Z/PT10M/PT01M

• temporal : XPathDateTime : 20150303T180000Z

SR 2016SR 2016 Stream Reasosning Workshop

Dydra : RSP Generation• What

• How


• Server-Sent Events[1]• Fetch [2]

• [1] : <http://www.w3.org/TR/eventsource/>• [2] : <https://www.chromestatus.com/feature/5804334163951616>

http://www.w3.org/TR/eventsource/

http://www.w3.org/TR/eventsource/

Dydra : RSP Generation : Basis

(http:def-resource-function graph-store-response (resource request response) ;; ... (:post ((resource |/:account/:repository/sparql|) request response (request-type mime:application/sparql) (response-type mime:sparql-results)) ;; given a body with just the query text accept the entire body. (let ((query (http:request-body request))) (graph-store-query resource query request response request-type response-type))) ;;; ... )

• SPARQL protocol


Dydra : RSP Generation : Extended

(http:def-resource-function graph-store-response (resource request response) ;; ... (:post ((resource spocq.si::|/:account/:repository/sparql|) request response (request-type mime:application/sparql) (response-content-type MIME:text/event-stream)) (let ((query (http:request-body request))) (graph-store-stream-query resource query request response request-type response-content-type))) ;;; ... )(defun graph-store-stream-query (resource query request response request-type response-content-type) (declare (ignore request-type)) (with-event-stream (e-stream (http:response-content-stream response)) (loop with stream-content-type = (mime:mime-type-parameter response-content-type :accept) for revision in (repository-revision-ids (resource-repository resource)) do (stream-write-header e-stream "Content-Type" (string (type-of stream-content-type))) (stream-write-header e-stream "Revision-ID" spocq.i::*revision-id*) (stream-write-header e-stream "Memento-Datetime" (http:encode-rfc1123 (repository-write-date revision))) (graph-store-query revision query request response mime:application/sparql-query stream-content-type))


• server-side events

Storage Model: Evolution

2. Initially, "independent copies" @quad term id• efficient HEAD access

• space inefficient• efficient urn:uuid? access

1. Next, "change based" @statement id• efficient HEAD access

• space efficient• inefficient urn:uuid? access

• inefficient mutation at scale

• inefficient mutation at scale

3. Current, "time-slice based" (various)• O|∆| HEAD access

• space efficient• O|∆| urn:uuid? access

• efficient mutation at scaleSR 2016 Stream Reasosning Workshop

Repository Model: 2nd Order• SPARQL 1.1 extensions for a "revisioned store"• Data model

• Graph• Dataset• Repository• Store

SPARQL 1.1 SPARQL 1.1 RSP• Graph• Dataset


Repository Model: 1.1 Semantics Extended• 2nd Order Repository

• Stream model• Named-Graphs comprise "temporal entities"• Default-Graph attributions via "timestamp predicate"

• Entailment extended

• The "temporal predicate" specification• Version succession via PROV:wasRevisionOf


> default 1st Order (metadata) Repository> default graph> named graph *

> named 1st Order Repository sequence

• Basic entailment applied to 1st Order Repository• Window operations yield 1st Order Repository

• Temporal metadata implicitly present for

• "Transaction time predicate" specification : then()

Space

ontology

gistschema.orgSTWefoefo

revisions

1623

7144

@2.69

files (MB)

2.919.194.2

1,384.02.8

quadstotal

47,273156,150771,375

12,807,240260,503

HEAD849

9,144108,967260,503260,503

store (MB)

38.440.6

240.04,270

84.6

bytes/quad

851273311358324


Dydra: Examples • http://dydra.com/schema-org-test/schema-org

• http://dydra.com/schema/efo• http://dydra.com/schema/efo-provenance

• http://dydra.com/schema-org-test/provenance


Thank you

James Anderson <[email protected]>

<http://dydra.com/>@dydradata @lomoramic


Documents

Stream Reasoning Workshop - ods.tu-berlin.de · Stream Reasoning Workshop Technical University Berlin SR2016 ... service