Building a Video Encoding Pipeline at The New York Times

Building a Video Encoding Pipeline at

The New York Times

Streaming Media West, 2016

Maxwell Dayvson Da Silva

Sr. Director of Engineering

http://github.com/dayvson

http://twitter.com/dayvson

http://dayvson.me

Flávio Ribeiro

Sr. Software Engineer

http://github.com/flavioribeiro

http://twitter.com/flavioribeiro

http://flavioribeiro.com

First, a little bit of history

The Architecture of the old system

Video CMS(StoryLog)

NYTimes CMS(Scoop)

InternalEncoding

Cluster

Internal Storage

CDN

The flaws

The Flaws1. The whole process relies on a filename

2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMSs are required to publish a

video.

The Flaws1. The whole process relies on a filename

2. Several watch folders (hard to maintain)

3. Unstable transcoding (proprietary software)

4. Hard to scale (cluster in-house)

43380_1_saudi-documentary_wg_720p.mp4

The Flaws1. The whole process relies on a filename

2. Several watch folders (hard to maintain)

3. Unstable transcoding (proprietary software)

4. Hard to scale (cluster in-house)

43380_1_saudi-documentary_wg_720p.mp4Production ID

The Flaws1. The whole process relies on a filename

2. Several watch folders (hard to maintain)

3. Unstable transcoding (proprietary software)

4. Hard to scale (cluster in-house)

43380_1_saudi-documentary_wg_720p.mp4

Version

The Flaws1. The whole process relies on a filename

2. Several watch folders (hard to maintain)

3. Unstable transcoding (proprietary software)

4. Hard to scale (cluster in-house)

43380_1_saudi-documentary_wg_720p.mp4Slug

The Flaws1. The whole process relies on a filename

2. Several watch folders (hard to maintain)

3. Unstable transcoding (proprietary software)

4. Hard to scale (cluster in-house)

43380_1_saudi-documentary_wg_720p.mp4

Resolution

The Flaws

1. The whole process relies on a filename2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video

The Flaws

1. Filenames2. Several watch folders 3. U4. Hard to scale (cluster in-house)

The Flaws

1. The whole process relies on a filename2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video

The Flaws

The Flaws

1. The whole process relies on a filename2. Several watch folders 3. Unstable transcoding4. Hard to scale 5. Two CMS to publish a video

Number of videos published per year.

The Flaws

1. The whole process relies on a filename2. Several watch folders 3. Unstable transcoding 4. Hard to scale5. Two CMSs to publish a video

The challenges

The Challenges1. Fast encoding. News Clips are very time

sensitive2. Scalability3. Reliability4. Company-wide video transcoding solution5. High-quality transcoding

360°, Drone footage, Live Action

The solution

The Video Pipeline

Distribution API

Acquisition API

Transcoding API

Storage

Media Factory Client

Media Factory API CDN

Database

Media Factory Client

Distribution API

Acquisition API

Transcoding API

Storage

Media Factory Client

Media Factory API

Database

CDN

Media Factory Client

1. EcmaScript 6 Library2. Upload Files from Browser to S3

http://github.com/TTLabs/EvaporateJS3. Trigger Jobs on Media Factory API4. Used by other teams

Acquisition API

Distribution API

Acquisition API

Transcoding API

Storage

Media Factory Client

Database

Media Factory API CDN

Acquisition API

1. Back-end integration to upload filesExternal partners

2. Plan to support resumable uploads

DatabaseMedia Factory

Client

Storage

Media Factory API

Distribution API CDN

Acquisition API

Transcoding APIMedia Factory API

Media Factory API

1. All NYTimes business logicsInteract with our internal APIsDefine which presets will be used

based on job type (Standard, 360)http://github.com/nytimes/video-presets

Transcoding API

Distribution API

Acquisition API

Transcoding API

Storage

Media Factory Client

Media Factory API CDN

Database

Transcoding API

1. Integration with multiple encoding cloud services

2. Reliability & Scalability3. Define a common Job specification4. Define a common Preset specification

Transcoding API - Job Example{"provider": "elastictranscoder", "source": "s3://somebucket/folder/my_video.mov", "streamingParams": { "playlistFileName": "hls/playlist.m3u8", "segmentDuration": 5, "protocol": "hls" } "outputs": [ {"preset": "720p_mp4", "fileName": "my_video_720p.mp4"}, {"preset": "1080p_mp4", "fileName": "my_video_1080p.mp4"}, {"preset": "256p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "480p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "720p_hls", "fileName": "hls/my_video_720p.m3u8"}, {"preset": "1080p_hls", "fileName": "hls/my_video_1080p.m3u8"}, {"preset": "2160p_hls", "fileName": "hls/my_video_2160p.m3u8"} ]}

Transcoding API - Preset Example "providers": ["elastictranscoder", "elementalconductor", "encodingcom", "zencoder"], "preset": { "name": "sample_preset", "description": "This is an example preset", "container": "mp4", "profile": "Main", "profileLevel": "3.1", "rateControl": "VBR", "video": { "height": "720", "width": "1080", "codec": "h264", "bitrate": "1000000", "gopSize": "90", "gopMode": "fixed", "interlaceMode": "progressive" }, "audio": { "codec": "aac", "bitrate": "64000" }

Transcoding API

1. Integration with multiple encoding cloud services

2. Reliability & Scalability3. Define a common Job specification4. Define a common Preset specification5. http://nyti.ms/mediafactory

The Distribution API

Database

Media Factory API

Distribution API

Acquisition API

Media Factory Client

Transcoding API

Storage

CDN

Distribution API

1. Fast file transfer leveraging Aspera protocol

2. Plan to support distribution to our partners Youtube, Facebook, other partners

3. Plan to support to multiple CDNs

Just a recap!

Distribution API

Acquisition API

Transcoding API

Storage

Media Factory Client

Media Factory API CDN

Database

Future1. Video Quality

Classify into groupsVMAF, Reingest on the Media Factory API

2. Open SourceDistribution APIAcquisition API

3. Snickers (http://github.com/snickers/snickers)

We’re hiringnyti.ms/technology

@NYTDevs | developers.nytimes.com

Stay updatedopen.blogs.nytimes.com

@NYTDevs | developers.nytimes.com

Connect with us on Slack!http://video-dev.org

@NYTDevs | developers.nytimes.com

Thank you!

{ , } /flavioribeiro

{ , } /dayvson