Upload
raphael-troncy
View
1.211
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Talk given at the "Séminaire Muséologie: Le Web devient audivisuel" at IRI on Tuesday 10/04/2012, Paris, France
Citation preview
Addressing and annotating multimedia fragments
Raphaël Troncy <[email protected]>
Use Case
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 2
Use Case
Aidem received on her Facebook wall a status message containing a Media Fragment URI Use a ‘#’ ! Highlight a
video sequence
Highlight a region to pay attention to
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 3
What are Media Fragments?
t 0 20 35 temporal media fragment
spatial media fragment
track media fragment
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 4
Temporal Media Fragments
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 5
Fragment beginning Fragment end Playback progress
Original resource length
Spatial Media Fragments
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 6
semi-opaque overlay
highlighted fragment
Media Fragments Use Cases
Bookmark / Share parts (fragments) of audio/video content
Annotate media fragments
Mash-ups
Conserve bandwidth
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 7
http://www.w3.org/TR/media-frags-reqs/
Requirements
r01: Temporal fragments: a clipping along the time dimension from a start to an end time that
are within the duration of the media resource
r02: Spatial fragments: a clipping of an image region, only consider rectangular regions
r03: Track fragments: a track as exposed by a container format of the media resource
r04: Named fragments: A temporal media fragment that has been given a name through
some sort of annotation mechanism
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 8
Media URIs
Using URI query part:
Using URI fragment part:
Mixing both:
http://www.example.org/video.ogv?t=60,100
http://www.example.org/video.ogv#t=60,100
http://www.example.org/video.ogv?t=60,100#t=10,15
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 9
The media fragment URI syntax can be used for URI queries
We will focus on URI fragments
URI Fragments vs. URI Queries
#t=20,30 ?t=20,30 secondary resource,
notion of context primary resource,
no notion of context extraction needs to be
expressible in byte ranges no adaptation restrictions
no provisions for communi-cating
fragments to the server
key-value pairs are sent to the server
potentially cacheable not cacheable
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 10
Media Fragments Resolution
For the URI query part: The media file is only processed on server side The UA receives a new video file
For the URI fragment part: Smart UA may strip out the fragment definition and
encode it into custom http headers (Range header) (Media) Servers will handle the request, slice the media
content and serve just the fragment (corresponding byte ranges) … while old ones will serve the whole resource
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 11
Influence of Media Formats
Fragment extraction needs to be expressible in terms of byte ranges
Requirements for the different axes temporal: presence of intra-coded frames
(i.e., random access points) spatial: presence of independently coded spatial regions track: need to be identifiable by a name
Conclusion: temporal and track axes are realistic, spatial fragments can hardly be expressed in terms of byte ranges
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 12
Can I use Media Fragments now?
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 13
<video id='v' src='AudioAPI.webm#t=50,100' onloadedmetadata='update()' onpause='update()' onplay='update()' onseeked='update()' controls></video>
Firefox 9+ (or WebKit)
No bandwidth saving
Can I use Media Fragments now?
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 14
Sysnote over YouTube
No bandwidth saving
Can I use Media Fragments now?
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 15
Ninsuna
Bandwidth saving
Examining HTTP Traffic
HTTP request: retrieving URI parsing key=values pairs from fragment part setting Range header
HTTP response: checking Content-Type
and Content-Range-Mapping headers values attaching custom playback controls to page creating spatial dimension overlay (if specified)
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 16
Example: Requesting a Time Fragment
A web developer specifies a video source with a temporal fragment URI:
key=value pair is analyzed, fragment begin and end time are matched
Media Fragments Extension analyses the fragment part, retrieves beginning and end time and sets proper Range header value:
http://ninsuna.elis.ugent.be/DownloadServlet/mfwg/fragf2f.ogv#t=5,15
t=5,15
Range: t:npt=5-15
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 17
Example: Requesting a Time Fragment
The NinSuna server responds with the 206 Partial Content response and Content-Range-Mapping header showing the mapped time ranges and media fragment in the message payload:
HTTP/1.1 206 Partial Content Content-Type: video/ogg Accept-Ranges: bytes, t, track, id Content-Range: bytes 629578-1690588/4055466 Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33}={bytes 629578-690588/4055466}
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 18
Example: Requesting a Time Fragment
... and it won’t work!
Because the player does not have the media file header
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 19
Example: Requesting a Time Fragment
Player needs to be initialized, thus bare fragment is not playable
To fix this, we add ;include-setup to the Range header value:
Range: t:npt=5-15;include-setup
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 20
Example: Requesting a Time Fragment
The response from the server is slightly different:
HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33;include-setup} ={bytes 0-5998,629578-1690588/4055466} ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End--
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 21
Example: Requesting a Time Fragment
This is why we need to attach a Stream Listener to the HTTP channel:
HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End--
HTTP/1.1 206 Partial Content Content-Type: video/ogg ... {binary data} {binary data}
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 22
Spatial Fragments
Without additional information send to server
Overlay is created by appending four additional DIVs to the webpage
They are styled to create the impression of semi-opaque layer over the video element
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 23
NinSuna: Overview
Fully integrated media adaptation and delivery platform media adaptation and media packaging core is
independent of media formats based on model-driven content adaptation & delivery technique
only high-level adaptation operations such as scene selection and frame rate scaling no transcoding is applied ideal candidate for MF implementation
more information: http://ninsuna.elis.ugent.be
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 24
Support for Media Fragment URI queries both HTTP and RTSP implementation try some URIs at http://ninsuna.elis.ugent.be/MediaFragmentsServer#Test http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_t
railer_2.mp4?track=5;6
Support for Media Fragment URI fragments i.e., support for the MF-specific HTTP headers
Combining Media Fragment URI queries and fragments e.g., http://foo.com/media.mp4?t=10,40#t=5,10
MF-specific Features of NinSuna
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 25
Updating existing infrastructure is not trivial web servers need to be extended with media extractors MF-specific HTTP communication
existing web caches are only effective with HTTP byte ranges specialized media caches need to be developed in the future to
cache the other cases
user agents need to be extended with a Media Fragment URI parser and interpreter MF-specific HTTP communication MF-specific visualization
Impact on the Existing Web Infrastructure
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 26
Goal: make existing media resources, served by generic HTTP Web servers, available as Media Fragments
Biggest problem for content providers willing to support Media Fragments media extractor (dependent on underlying media
formats)
Solution: an approach that works with existing HTTP Web servers (apache, IIS) works with existing Web caches works with not so smart user agents
Media Fragments Proxy
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 27
-get (only) the header info -try to find a fragment-to-byte mapping -construct the redirect response
Media Fragments Proxy
HTTP server
MF Proxy MF User
Agent
http://foo.com/video.ogv#t=11,19
GET /?url=http://foo.com/video.ogv HTTP/1.1 Host: MFProxy.com Accept: video/* Range: t:npt=11-19 Accept-Range-Redirect: bytes
HTTP/1.1 307 Temporary Redirect Location: http://foo.com/video.ogv Accept-Ranges: bytes, t, track Content-Length: 0 Content-Type: video/ogg Content-Range-Mapping: t:npt 10-20/0-50 Range-Redirect: 24000-32000 Vary: Accept-Range-Redirect
GET /video.ogv HTTP/1.1 Host: foo.com Accept: video/* Range: bytes=24000-32000
http://ninsuna.elis.ugent.be/MFProxy?url=<mediaURI>
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 28
Annotating Media Fragments
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 29
yuma.min.js
Annotating Media Fragments
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 30
Benchmarking: Sphinx, HTK, Julius
NER + full text index with the transcription
Interlinking with the Linked Data Cloud to enable semantic search
10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 31
http://www.slideshare.net/troncy
10/04/2012 - - 32 Séminaire Muséologie - Le Web devient audiovisuel