Cyril Concolato

Streaming

SVG Animations

on the Web

Cyril Concolato

HTML5 Dev Conf / The Graphical Web

22-23 Oct 2013

Cyril Concolato

Context and Motivations

Animating using web technologies is getting popular

• Lots of content being produced

« SVG 2 » and « Web Animations 1.0 » being drafted

• Opportunities to improve the technologies

My goal: Enable treating web animations like video

• Why?

• What kind of animations?

• How?

23/10/2013 Streaming SVG animations on the Web 2

Cyril Concolato

Web Animation Use Cases

Animations in User Interfaces

Animations for Data Visualization

Timeline-based Animations

• Graphics animations (cartoons, …)

• Synchronized audio & graphics (karaoke, ads …)

• Synchronized video & graphics (annotations, subtitles)

23/10/2013 Streaming SVG animations on the Web 3

Popcorn.js SVG Wow! Cartoon

Cyril Concolato

Timeline-based Animations Possibilities

Playback control using a clock

• Play(0)/Play(T)

• Pause/Resume

• Speed-up/slow down

Accurate synchronization

• Between animations or

between animation and media

• Dependent clocks and timestamps

23/10/2013 Streaming SVG animations on the Web 4

Similar to video

but not allowed in HTML5 <video>

Cyril Concolato

An SVG video-like JavaScript Player

Principles

Using HTML 5 <video> syntax but for SVG

content

23/10/2013 Streaming SVG animations on the Web

1

2

3

<video controls width="640" height="480">

<source src="myAnimatedFile.svg" type="image/svg+xml" >

</video>

5

Demo 1

Demo 2

1

2

3

4

<video controls width="640" height="480">

<source src="file.mp4" type="video/mp4" >

<track src="annotations.svg" kind="graphics" type="image/svg+xml">

</video>

Cyril Concolato

An SVG video-like JavaScript Player

Status

Simple Implementation

• GUI based on Video.js

• Replacing <video> by <svg> at run-time

• Using XHR to fetch SVG data and insert in the page

• Uses get/setCurrentTime() and un/pauseAnimations()

Limitations

• Bugs!

• Same origin restrictions

• Incremental/progressive loading not working

• Cannot reproduce HTML network-related events (can

play, can play through)

23/10/2013 Streaming SVG animations on the Web 6

Cyril Concolato

An SVG video-like JavaScript player:

Next Step

Network-aware Controlled Playback • Load SVG data just before it’s needed (and let garbage collection

do the rest)

How to achieve controlled playback of SVG ? • Apply streaming concepts

23/10/2013 Streaming SVG animations on the Web 7

Cyril Concolato

Applying streaming concepts to SVG

Streaming “Controlled continuous delivery and consumption of data units”

What’s needed to stream SVG content? • Fragmentation of SVG documents into streaming units

─ Needs guidelines for SVG authoring

• Assign timing to each unit ─ Mapping between byte-offset/time-position of the SVG document

─ Needs a streaming instruction format

• Seek information ─ Needs explicit processing behavior

SVG Streaming draft • http://dev.w3.org/SVG/modules/streaming/

• Feedback welcome

Example: Streaming SVG Cartoons

23/10/2013 Streaming SVG animations on the Web 8

Cyril Concolato

Cartoons on the Web – Today

Largely based on Vector Graphics • Bezier curves filled with solid color or simple gradients and/or stroke

• Sometimes using raster images (e.g. background or texture)

Mostly frame-based content • Sometimes with additional interpolation

• Typically synchronized with a sound track

• Good candidate for streaming

Currently delivered as Flash or video streams • Plugins, coding artefacts or bandwidth problems …

23/10/2013 Streaming SVG animations on the Web 9

Source: www.cartoonnetwork.com

Cyril Concolato

Cartoons on the Web – Next Steps

Could be better integrated in the web platform

• Using of vector graphics primitives for scalability and new primitives for advanced rendering

─ E.g. coons patches, hatches (SVG 2), diffusion curves, …

• Using other web technologies

─ Styling, language translation, …

Attempts to convert Flash to HTML5/CSS/SVG/Canvas

• Many JS-based approaches

─ Google Swiffy

─ Mozilla Shumway

─ Adobe CreateJS

─ PixelPlant

• Some declarative approaches

─ MP4Box (100% SVG)

23/10/2013 Streaming SVG animations on the Web 10

Cyril Concolato

Frame-based SVG Cartoons using MP4Box <svg viewBox="0 0 768 576" ...>

<rect width="100%" height="100%" fill="lightblue"/>

<g id="frame_1" display="none">

<animate attributeName="display" to="inline" begin="0" end="0.16666667"/>

<defs>

<g id="Shape1"><path fill="rgb(111,90,41)" stroke="none" d="M 524.0 -109.0 ..."/>...</g>

<g id="Shape2"><path fill="rgb(111,90,41)" stroke="none" d="M 524.0 -109.0 ..."/>...</g>

<g id="Shape3">...</g>

</defs>

<use xlink:href="#Shape1" transform="translate(0.0 576.0) rotate(0.0) scale(1.0 1.0) "/>

<use xlink:href="#Shape2" transform="translate(0.0 576.0) rotate(0.0) scale(1.0 1.0) "/>

<use xlink:href="#Shape3" transform="translate(0.0 576.0) rotate(0.0) scale(1.0 1.0) "/>

</g>

<g id="frame_2" display="none">

<animate attributeName="display" to="inline" begin="0.16666667" end="0.25"/>

<defs>

<g id="Shape4"><path .../></g>

</defs>

<use xlink:href="#Shape1" transform="translate(0.0 576.0) rotate(0.0) scale(1.0 1.0) "/>

<use xlink:href="#Shape4" transform="translate(0.0 576.0) rotate(0.0) scale(1.0 1.0) "/>

<use xlink:href="#Shape2" transform="translate(0.0 576.0) rotate(0.0) scale(1.0 1.0) "/>

</g>...

</svg>

23/10/2013 Streaming SVG animations on the Web 11

Cyril Concolato

Fragmentation of SVG document

into streaming units

Identify stream header • Static throughout the streaming session

• Required to initialize the parser

Make sure SVG data is properly ordered • Use only backward references

• Document order == time order

Identify units • Assign timestamps

• No overlap in position / possible overlap in time (like WebVTT cues)

Make self-discardable units • <discard>, JavaScript, scoped stylesheets

Mark self-contained units • Random Access Point

23/10/2013 Streaming SVG animations on the Web 12

Cyril Concolato

Streaming Instructions Example

<stream timescale="24" file="cuisine.swf.svg">

<header range-end="204"/>

<unit time="0" range-end="353" rap="1" />

<unit time="2" range-end="38403" rap="0" />

<unit time="3" range-end="39591" rap="0" />

<unit time="5" range-end="41923" rap="0" />

<unit time="6" range-end="45825" rap="0" />

<unit time="7" range-end="51276" rap="0" />

<unit time="10" range-end="52596" rap="1" />

<unit time="12" range-end="55261" rap="0" />

…

</stream>

23/10/2013 Streaming SVG animations on the Web 13

Cyril Concolato

How to generate Streaming Instructions ?

Post-authoring

• Requires byte positions to be final

Hard to determine without author’s instructions

• Where does the frame start/end?

• What’s the timestamp of the unit?

• Where are the random access points?

Possible automatic generation from Flash

Possible with additional elements/attributes

23/10/2013 Streaming SVG animations on the Web 14

Cyril Concolato

Flash-to-SVG streaming

23/10/2013 Streaming SVG animations on the Web 15

SWF

file Converter

SVG

file Packager MP4

file

Extractor

SVG

file

Streaming

Instructions

file

Converter &

Packager

Cyril Concolato

Generating Streaming Instructions

for SVG content converted from Flash

23/10/2013 Streaming SVG animations on the Web 16

MP4Box -add flash.swf:fmt=svg file_with_svg_track.mp4

MP4Box -six 1 file_with_svg_track.mp4

MP4Box -info file_with_svg_track.mp4

Cyril Concolato

SVG Streaming Demo

• Playing an SVG stream

• Viewing Random Access Points and others

• Seeking into an SVG unit not yet downloaded

• Concatenating SVG content (splicing)

• Live and pseudo-live streaming of SVG content

─ Web-based Vector Graphics TV channel

23/10/2013 Streaming SVG animations on the Web 17

Web Server

(byte-range support)

Browser

+

JS Player

SVG

+

Streaming

Instructions

Demo 3

Cyril Concolato

SVG streams and Media Containers

Why? What for? • Packaging audio/video/graphics together

─ To download only one resource

─ To facilitate synchronization of tracks

• Possible use with HTML5 native <video> elements ─ SVG in-band graphical track

• Possible use with HTML 5 Media Source extensions ─ Adaptive streaming of SVG

Formats • ISO/IEC-14496:12 (royalty-free) defines how to store

timed text in MP4

• Could be done for other formats (WebM, Ogg, …)

23/10/2013 Streaming SVG animations on the Web 18

Flash MP4Box

MP4

SVG

MP4

SVG Video

Audio

Cyril Concolato

SVG streams and WebVTT

WebVTT allows metadata tracks and raw cues

23/10/2013 Streaming SVG animations on the Web 19

Flash MP4Box

SVG

WebVTT

WebVTT

SVG

MP4Box –webvtt-raw 1 file_with_svg.mp4

MP4Box –webvtt-raw 1:embedded file_with_svg.mp4

1

2

3

4

<video controls width="640" height="480">

<source src="file.mp4" type="video/mp4" >

<track src="annotations.vtt" kind="metadata">

</video>

Cyril Concolato

SVG embedded in WebVTT

23/10/2013 Streaming SVG animations on the Web 20

WEBVTT Metadata track generated by GPAC MP4Box kind: metadata inBandMetadataTrackDispatchType: image/svg+xml metadata-header: <svg viewBox="0 0 384 288" ...> <rect width="100%" height="100%" fill="lightblue"/> 00:00:00.000 --> 00:00:00.083 isRAP:true <g display="none"> <animate ... /> <defs>...</defs> <use .../> </g> 00:00:00.083 --> 00:00:00.025 isRAP:false <g>...</g>

Cyril Concolato

SVG referenced in WebVTT

23/10/2013 Streaming SVG animations on the Web 21

WEBVTT Metadata track generated by GPAC MP4Box

kind: metadata

inBandMetadataTrackDispatchType: image/svg+xml

baseMediaFile: flash7.svg

text-header-length: 207

00:00:00.000 --> 00:00:00.083 mediaOffset:207 dataLength:311 isRAP:true

00:00:00.083 --> 00:00:00.166 mediaOffset:518 dataLength:320 isRAP:false

00:00:00.166 --> 00:00:00.250 mediaOffset:838 dataLength:207 isRAP:false

...

Cyril Concolato

Summary & Conclusion

Presented • Guidelines for authoring streamable SVG content

• Tools for generating streaming instructions ─ from Flash, will be extended to support SVG natively

• JavaScript-based players for streamable SVG content

Outcome • Provided that your SVG content is well structured and annotated,

you can stream SVG content ─ In different modes: Controlled Progressive Download, Live, On Demand,

Playlists, Adaptive, …

─ Out of band graphics tracks (currently with JS shim)

─ In band tracks in media containers (in browsers in the future)

Future work • Implement adaptive streaming (MP4 based)

• Implement media synchronization (WebVTT)

• Work with CSS-based or JS-based animations

23/10/2013 Streaming SVG animations on the Web 22

Cyril Concolato

Thank you

Questions?

23/10/2013 Streaming SVG animations on the Web 23

Cyril Concolato

Further reading

SVG Related

• HTML 5 Video and SVG Graphics Synchronization

• Carriage of SVG and WebVTT in MP4 (Blog, Text

Track CG)

Streaming related

• Live streaming over HTTP of video and subtitles (Part

1 & Part 2)

• WebVTT streaming

23/10/2013 Streaming SVG animations on the Web 24