Enabling access to Linked Media with SPARQL-MM

Enabling access to Linked Media with

Thomas Kurz, Kai Schlegel, Prof. Dr. Harald Kosch

3rd International Workshop on Linked Media, WWW 2015, Florence

© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008

The MICO Project and me

• www.mico-project.eu• 36 Months Project• started 10/13• 3.4 Million Euro Project • 7 Partners


The MICO project idea


The MICO project idea

Semantic Web Multimedia

Linked Media

Media Fragment Identifiers

Media Annotation Ontologies

5


Multimedia Retrieval on the Web of Data is finding (fragments of) resources of an structured and unstructured nature (text, image, video,

concepts etc.) that satisfy an information need.

Extension of the standard definition of information retrieval in:

Christopher D. Manning and Prabhakar Raghavan. An Introduction to Information Retrieval. Cambridge University Press, 2009.


Multimedia Query Languages

1. SQL-like approaches (WebSSQL or SQL/MM)

2. Object oriented approaches (MOQL or POQLMM)

3. Focusing XML metadata (MMDOC-QL or XQuery)

4. Visual Query Languages, like (MQuery Visual-MOQL)

5. Query-By-Example approaches, like WS-QBE

6. Metadata agnostic like MPQF

A Survey with ~40 MM Query Languages is coming soon …


Multimedia Query Languages:Requirements and Features

• Spatial Operation

• Temporal Operation

• Freetext Search

• Functions: Aggreg., Sort., Group., Set-Fnct.

• Similarity Search (Top-k)

• Event / Object Concept

• Presentation Part

General Requirements:

Application Independency, Formal Semantics, Extendability, Closed, etc.

Specific Features:

• Fuzzy terms

• Relevance Feedback

• Camera Motion

• Object Motion (Trajectory)

• Query by Example (Image, Video, Sketch, Humming)

• Condition Weighting

• Low-Level-Features (Audio, Video)


Semantic Web Query Languages

• There has been many in the past, but SPARQL

(SPARQL Query Language for RDF) is the winner

of the "natural selection".

• With SPARQL 1.1 many missing features have

become part of the standard.

• SPARQL is pretty good to extend (even without

changing the basic syntax)

BUT: It's missing some important features


What we want to reach

"Find scenes where Barack Obama is left beside the Greenpeace Managing Director during an UN climate conference ordered by length."

Features needed:

• Object Identification

• Temporal Operation

• Spatial Operation

• Freetext Search

• Sorting by Temporal Selector


How could it look like?"Find scenes where Barack Obama is left beside the Greenpeace MD during the UN climate change summit ordered by length."SELECT ?scene WHERE {

?a3 oa:hasBody ?event; oa:hasTarget ?s3. # there are resources about an event

?event schema:Event; schema:summary ?description.# which has a description

FILTER mm:fulltext-search(str (?description), "UN climate conference","en")

# about 'UN climate conference'

?a2 oa:hasBody :Barack_Obama; oa:hasTarget ?s2. # and there are resources about Obama

?a1 oa:hasBody ?p1; oa:hasTarget ?s1. # and there are resources

?p1 :ceo_of dbpedia:Greenpeace. # about the MD of Greenpeace.

FILTER mm:leftBeside(?s2,?s1) # Obama has to be at the left of the MD and

FILTER mm:intersects(?s3,?s2) # has to be appear at the same time like the event.

BIND (mm:boundingBox(?s1,?s2) AS ?scene) # Wrap the results to scenes

ORDER BY DESC(mm:duration(?scene)) # and show the longest first.


Goals• Bring Multimedia Functionalities to SPARQL

(especially spatio-temporal support)

• Give transparent access to common Metadata Models (Web Annotation Selectors, Media Fragment URIs)

• Provide efficient evaluation

• Make it more easy for users


Spatio-Temporal Object Model


Spatio-Temporal Object Model: Example: Rectangle


Supported Standards

• Media Fragment URIs

• Spatial Fragment (currently Rectangle)

• Temporal Fragment (currently Interval)

• Web Annotation Model

• Media Fragment Selectors (http://www.w3.org/TR/media-frags/)

• SVG Basic Shapes (http://www.w3.org/TR/SVG/shapes.html)

http://example.org/video.mp4#t=1,5&xywh=10,20,30,30

http://www.w3.org/TR/media-frags/

http://www.w3.org/TR/SVG/shapes.html

http://example.org/video.mp4#t=1,5&xywh=10,20,30,30


SPARQL-MM FunctionsSpatial Topological Relations

Spatial Predicatesbased in the

Dimensionally Extended nine-Intersection Model

+(DE-9IM)


SPARQL-MM FunctionsSpatial Directional Relations


SPARQL-MM Functions: Temporal Relations


Multimedia Tests and Accessors (not complete yet)xsd:boolean mf:isMediaFragmentURI(Resource r)

xsd:boolean mf:isMediaFragmentSelector(Resource r)

xsd:boolean mf:isSVGSelector(Resource r)

mo:Rectangle mf:getBoundingBox(mo:SpatialEntity e)

xsd:double mf:getArea(mo:SpatialEntity e)

mo:Point mf:getXY(mo:Rectangle r)

xsd:double mf:getWidth(mo:Rectangle r)

xsd:double mf:getWidth(mo:Rectangle r)

xsd:double mf:getDuration(mo:TemporalEntity e)

xsd:double mf:getStart(mo:TemporalEntity e)

xsd:double mf:getEnd(mo:TemporalEntity e)


Fulltext SearchBased in Apache Marmotta Fulltext Search*

• mf:fulltext-search(text, query, [language]): searches “text” for the words occurring in “query”, optionally applying the language-specific processing for the given language; query is a simple text literal (list of words) without any boolean connectors; words are AND connected, i.e. all words have to be found in the text for a successful match.

• mf:fulltext-query(text, query, [language]): searches “text” using the boolean query string passed in “query”, optionally applying language-specific processing for the given language; query is a boolean query string following the syntax used by PostgreSQL.

* http://marmotta.apache.org/kiwi/sparql.html

http://marmotta.apache.org/kiwi/sparql.html


Status of Implementation

• SPARQL-MM 1.0 is out for a while, but:

• it is slow (based on Sesame Function Extensions)

• it is reduced

• it is very prototype (therefore still Snapshot)

https://oss.sonatype.org/content/repositories/snapshots/com/github/tkurz/sparql-mm/

• SPARQL-MM 2.0 will be available soon (planned for 8/2015)

• it is much faster (based on SQL-MM using PostGIS)

• it will be available as Open Source Marmotta Module*

• is has a big feature set

* maybe we should create a LIME Module that include implementations for various Linked Media related things like Web Annotation endpoints etc. ?


Short Demo <repositories> <repository> <id>oss-sonatype</id> <name>oss-sonatype</name> <url>

https://oss.sonatype.org/content/repositories/snapshots/</url>

</repository> </repositories>

<dependency> <groupId>com.github.tkurz</groupId> <artifactId>sparql-mm</artifactId> <version>1.0-SNAPSHOT</version> </dependency>

https://oss.sonatype.org/content/repositories/snapshots/


Future Plans

• SPARQL like for shapesmm:like(mo:Circle(20,20,100), ?f)

• Extend Model to Layersmm:behind(?f1, ?f2)

• SPARQL like for Resourcesmm:like(?u1, ?u2, 0.4)


Thanks for your Attention!

Thomas KurzKnowledge and Media TechnologiesSalzburg Research Forschungsgesellschaft m.b.H.Jakob-Haringer-Straße 5/III | Salzburg, AustriaTel. +43 662 2288-253| Fax +43 662 [email protected]

mailto:[email protected]?subject=

Internet

Enabling access to Linked Media with SPARQL-MM