High Definition Video: Workflow from Cameras to CODECs APAN Fall Meeting Xi’an, China Aug 2007

High Definition Video: Workflow from Cameras to CODECsAPAN Fall Meeting

Xi’an, China Aug 2007Michael Wellings

Director, Engineering

Vice Chair, APAN HD WG

THINK FORWARD. THINK RESEARCHCHANNEL.

Topics of Discussion

• ResearchChannel Background

• Recent and future demonstrations

• Advanced Systems

• iHDTV™ Open Source Project

• HD Video Workflow Discussion

ResearchChannel Background

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006Programs

3200

2100

1700

1270

1000

750

325

610

0

0

500

1000

1500

2000

2500

3000

3500

TotalVOD Programs

ResearchChannel Programs

More than 3,200 hours of programming in theVideo on Demand library

1,100 new programs addedthis year

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006Participants

58

45

35

18

00

0

10

20

30

40

50

60

TotalParticipants

Advanced Network Forum

AJA Video Systems Inc.

Australia's Academic Research Network

(AARNet)

Fujinon

Howard Hughes Medical Institute

IBM Corporation

Intel Corporation

Internet2

Johnson & Johnson

Library of Congress

Microsoft Research

National Academies

National Academy of Engineering

Duke University

George Mason University

Indiana University, School of Informatics

Johns Hopkins University

Massachusetts Institute of Technology

National University of Singapore

New York University

Pennsylvania State University

Rice University

Rutgers, The State University of New Jersey

Stanford University Medical Center

Texas A&M University

Tulane University, A.B. Freeman School of

Business

ResearchChannel :: Participants

Universidad de Puerto Rico

Universidade de São Paulo

University of Alaska - Fairbanks

University of Chicago

University of Hawaii

University of Maryland - College Park

University of Pennsylvania

University of Southern California

University of Virginia

University of Washington

University of Wisconsin-Madison

Virginia Tech

Yale University

National Academy of Sciences

National Institute of Nursing Research

National Institute of Standards and Technology

National Institutes of Health

National Library of Medicine

National Science Foundation

Pacific Northwest Gigapop

Poznań Supercomputing and Networking Center

R1edu.org

SURFnet

Vulcan Northwest, Inc.

Wisconsin Public Television

ResearchChannel : New Participants

American Meteorological Association

Arizona State University

California State University,Monterey Bay

Indiana University

Loma Linda University

Pacific Northwest National Laboratory

University of Kentucky

University of Michigan

Isilon Systems

U.S. Television Distribution

DISH Network satellite system 12 million homes

Cable systems 12.9 million homes

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006TV-total

24.9

21.9

18.9

18.1

12.8

8.6

7.4

0

0

0

0

5

10

15

20

25

Total Satellite and CableViewer Households (Million)

Google Video

Technology Milestones

August 1999First-Ever Streaming of High-Definition Television over the Internet

November 1999Network Sets Record Speed

April 2000First Live HDTV Over Internet Newscast

July 2004First Transmission of Full Bandwidth HD Video Between Computers

September 2005First-Ever Live HD Images from Seafloor to Land Available as IP-Based Feed

November 2005Interactive Multipoint HD Videoconference Demonstrated

November 2006Partner in World-wide testing of ConferenceXP

Technology :: Leading the Way

Continuous Ocean Observing

Consortia Partnerships

Recent and Future Demonstrations

Sept 2005 - iGrid San Diego

• iHDTV - USA118: Global N-Way Uncompressed Interactive Conferencing

• Underwater research using HD from SS Thompson – USA119: 20,000 Terabits Under the Sea

Nov 2005 - SC05 Seattle

• iHDTV /HD Storage, capture and editing using SRB (Storage Resource Broker)

Recent Demonstrations

Nov 2006 SC06 Tampa

• iHD1500 system

Dec 2006 Internet2 Fall Member Meeting Chicago

• iHD1500 system

• HD CoDecs compared

• HD Camera hands-on

Recent Demonstrations

Future Demonstrations

ResearchChannel Annual Meeting Oct07

Cinegrid / GLIF Sept07

Internet 2 Fall Member Meeting Oct07

SC07 Reno Nov07

APAN 08 Hawaii Jan08

Advanced Delivery Methods and Services

ResearchChannel Advanced on-line Services

ResearchChannel-HD

• Currently VC-1 SMPTE 421M

• AVC (h.264, MPEG-4 Part-10) coming soon

• Unicast:• http://media-wm-hd.cac.washington.edu/

ResearchChannel-HDVideo

• Multicast:• http://researchchannel.org/multicast/ResearchChannel-

HDVideo.nsc

• Solicitation for HD content


ResearchChannel satellite feed on the network• 233.0.73.29 MPEG2@ 3.2mbps

Live 1080i HD video of Mt Rainier• 233.0.73.26 MPEG2 MP@HL @ 20mbps

Live 720p HD video of Mt Rainier

http://www.researchchannel.org/tech/desktopclients.asp

HD VOD 720p WMV-HD http://www.researchchannel.org/visions05/hdpresentation.asp

HD VOD 1080i MPEG2 ML@HL

http://www.researchchannel.org/tech/desktophdsamples.asp

KEXP-FM Audio-on-demand; uncompressed audio webcasting

http://www.kexp.org


Developing VOD archive for oceanographic HD video

• Partnership with Isilon – to be demonstrated at SC07 Reno

KEXP-FM on-line services have a world-wide audience that equals Seattle broadcast listener numbers


Visions ’05 - Emmy Award Nominee

First-ever live HD 1080i video/IP streamed around the world• SMPTE 292M multicast and unicast @ 1.5gbps• MPEG 2 MP@HL multicast• WMV-HD multicast and unicast


Seattle Science Foundation Project – testbed for new

Medical technology with HD video archiving –

seattlesciencefoundation.org WWAMI: collaborative medical education via

Advanced video technology

RC HD Lab on-line

Advanced venue on UW campus commissioned

Successful launch of Experimental Internet ResearchChannel-HD

iHDTV Open Source Project

iHDTV™ Software Suite

iHDTV Explained• iHD1500

• iHD270

• HD to the Desktop

WindowsXP-based application suite

Released as Open Source

Account created on SourceForge

iHD_Trusted_Partners_Group formed to direct software development

iHDTV™ Development / Deployment

SourceForge load• SVN Tree transferred

• iHDTV_trusted_partners formed – 47 members

• 4 Core developers @ (UW / U-Mich / U-Wisconsin)

• Executive team to guide development will meet monthly

Deploy globally• N-way implementation this summer with 6

nodes

iHDTV™ iHD270

iHD270 (first introduced aug 99)

• Sony HDCamtm compression

• SDTI data format 270mbps

• AJA Xena I/O

• Requires:• AJA Xena-HD capture cards• Dual Processor P4• Windows XP• Sony HDCam Hardware encoder/decoder• Gige network connection

iHDTV™ iHD270

1920x1080 60i 8bit 4:2:2

• Sony HDCam source material

• 995mbps active lines, 1188mbps total

iHDTV™ iHD1500

iHD1500• Uncompressed SMPTE 292M 4:2:2

• Data rate total approx 1.5 gbps

• Requires:• Uncompressed HD capture cards• PCI Express platforms• Windows XP• 2 x gige or 1x10gige network connection• Jumbo frame transport and routing• Specs:

http://zzz.cac.washington.edu/ihdtv_wiki/index.php/Main_Page

iHDTV™ iHD1500

Latency about 4 frames end to end plus network delay

• 4 frames=133ms or ~ 1/7 sec

iHDTV™ iHD1500

iHD1500 Enhancements:

• New i/o board options

• GUI front end

• Self-installing package

• Uncompressed HD Server:• Single link• Dual link• Quad link

• HD Screen rendering for OptIPortal integration

• NTT i-VISTO interoperability

High Definition Workflow

A Report “in progress…”

HD – a Background

Sample Rates

NTSC 4:2:2• Y, Pb, Pr color matrix

– 4x3.38 (compromise rate from 3.58MHz) = 15.5MHz

– Color difference channels @ 6.75MHz

HD 4:2:2 (really 22:11:11)• Y, Pb, Pr color matrix

– 22x3.38 = 74.25MHz– Color difference channels @ 37.125MHz

10 bit – 1024 gradations

8 bit – 256 gradations

Sample Rates

4:2:2

• Chroma sampled at ½ rate of Luma

4:1:1

• Chroma sampled at ¼ rate of Luma

4:2:0

• Chroma sampled at ½ rate of Luma on every other line of each field

Sample / Bit Rates

Broadcast

• 4:2:0 MPEG-2 TS 19.2mbps

DVD

• 4:2:0 MPEG-2 PS VBR

HD DVD/Blu-Ray

• 4:2:0 VC-1 @ 8mbps

• 4:2:0 AVC @ 8mbps

HD Resolutions

720p 60Hz – SMPTE 274M• 1280x720 24p

• 1280x720 30p

• 1280x720 60p

1080i/p 60Hz – SMPTE 296M• 1920x1080 24p

• 1920x1080 30p

• 1920x1080 60i

• 1920x1080 60p

HD Bitrates – Uncompressed720p 10- bit 4:2:2

720p 60Hz - SMPTE 296M

• 1280x720 30p• 742mbps• 30MHz Y, 15MHz Chroma


Progressive / Interlaced

1080/60i and 1080/30p occupy the same datarate – 1.485gbps

Interlaced pictures produce spacial errors associated with the time difference in motion between fields

This can create a smoothing effect which is easier on the eye during fast motion

High Framerate progressive (60 frame or greater) is best

HD Bitrates – Uncompressed10801/p 10- bit 4:2:2

1080i/p 60Hz – SMPTE 274M


• 1920x1080 60i• 1485mbps• 30MHz Y, 15MHz Chroma


Imaging chips

Solid State Camera front end (tube cameras came first!)

CCD = Charge Coupled Device

CMOS = Complementary metal oxide semiconductor

Imaging chips

CCDs collect charge at each element, and that charge is read out sequentially

Charge to Voltage conversion takes place off-chip

A>D takes place off chip

Digital level = luminance level

These are monochrome sensors!

Imaging chips

CMOS chips collect charge at each element

Charge to Voltage and A>D functions take place on-chip

CMOS imagers exhibit more noise than CCDs, and tend not to be used in higher end cameras

Single chip vs 3 chip

CCDs gather charge according to the amount of light impinging on the element

This charge is converted to a voltage representing light level

This light level does not specifically contain color information

Color reference is obtained through the use of filtering in front of the image sensor

Bayer Filter on-chip

Bayer Filter Diagram

4:2:2 sampling

Green sampled at 2x Red and Blue

Single chip color solution

Color Processing

3 chip

• RGB output from image processing

• Digital sampling and CODECS operate on component video – Y, Pb, Pr (Y’,Cb’,Cr’)

• Y = Luma channel

• Pb= subsampled blue matrix

• Pr= Subsampled red matrix

NAB 2007

HD Cameras

SONY XDCam ex

• 3 chip ½” imager (details unknown)

• MPEG-2 long GOP

• 30-50mbps

• Writes to Express Flash memory cards

• 120 min record time

• Target price $8000

HD Cameras

Grass Valley/Thomson Infinity

• 3 chip CMOS

• 2.4 million pixels each

• Ships with DV25 and JPEG2000 codecs

• MPEG2 optional

• Writes at up to 100mbps to IOMega REVPro drives

• Target price $26K without lens

• Ships in August 07

New HD Cameras at NAB 07

SONY XDCam HD

• Long GOP MPEG-2

• Writes to dual-layer BluRay Pro disk

• Up to 50mbps

• 2/3” 3 chip 2million+ pixel

• ships in 1st QTR 08

• Target price less than $30K without lens

New HD Cameras at NAB 07

Codec comparison Project

SMPTE StEM HD Mini-Movie

• Designed for critical evaluation of HD equipment as well as encoders and decoders

• Delivered as tga files, uncompressed

• Imported into Final Cut Pro

SMPTE Standard Evaluation Material


7 Uncompressed Quicktime Movies on 7 sequences

5 movies exported:

• DVCPro 100

• HDV

• MPEG-4 Basic

• H.264 Part 10 (AVC)

• HD uncompressed

All FCP I/O via AJA Kona 2 board



HD sequence played out to HDCam Tape

• recaptured 8bit 4:2:2 in FCP

HD sequenced played out to NTT HD1000 MPEG-2 encoder

• recaptured from output of NTT HD1000 MPEG-2 decoder as 8bit 4:2:2, and 4:2:0


DVCPro 100 sequence recorded to tape and played back

HDV sequence re-rendered in FCP to uncompressed

HDCAM SR recorded to tape and played back

XDCAM HD recorded to tape and played back


MPEG-4 Basic re-rendered to uncompressed in FCP

MPEG-4 h.264 Part 10 re-rendered to uncompressed in FCP

The edit

Five scenes of interest selected

10 new sequences created for each of the 9 compressed-decompressed files plus the original

HDV and DVCPro were scaled 133 and 150% respectively to match the uncompressed material

Difference mask

Luminance of each compressed file inverted and added to the original

50% luma field indicated no loss

Material lost in codec process exhibited visually due to the subtraction process

The movies

Several HD movies resulted from this project

• 9 with the edited 5 scenes – each an example of a particular codec – showing the difference mask full screen

• 9 with the encoded and decoded video overlayed in the center of the frame of the origional for direct comparison

• 5 complete sequences of codec representation

• The original uncompressed movie

CODEC Comparison

CODECBit Rate (Mbps)

File Size (MB)

Compression Ratio Type Size Color Space Encode Time

HDCam 144 n/a 10.4:1 Hardware 1440X10803:1:1 6:30 min

HDCam SR 440 =1500/B3 Hardware 1920X10804:4:4 6:30 min

DVCProHD 115.24 5100 13:1 Software 1280x1080 4:2:2 17 min

XDCam HD 35 Hardware 6:30 min

MPEG-2 4:2:2@HL 30 n/a 56:1 Hardware 1920X10804:2:2 6:30 min

MPEG-2 MP@HL 4:2:0 18 n/a 93:1 Hardware 1920X10804:2:0 6:30 min

HDV MPEG-2 MP@H14 24.96 1110 60:1 Software 1440X10803:1:1 33 min

MPEG-4 H.264 7.572 344 198:1 Software 1920X10804:2:0 1 hr 5 min

MPEG-4 Basic 5.937 270 253:1 Software 1920X10804:2:0 24 min

VC-1 8 320 188:1 Hardware 1920X10804:2:0 28 min

JPEG2000 46/25* Hardware



equipment

SONY HDW-500 VTR

SONY HDCAM-SR VTR

SONY XDCAM –HD VTR

Panasonic DVCPRO-100

Apple Dual 2.7GHz G5• 4GB RAM• Mac OS-X 10.4.7• Final Cut Pro 5.1.1• AJA Kona 2 I/O

NTT HD 1000 MPEG Encoder

NTT HD 1000 MPEG Decoder

HD Multiburst Waveform Uncompressed Reference

HD Multiburst Waveform DVCPro 100

HD Multiburst Waveform SONY HDCam

HD Multiburst Waveform SONY HDCAM SR

HD Multiburst Waveform HDV

HD Multiburst Waveform H.264 /AVC

HD Multiburst Waveform SONY XDCAM HD 30

HD Multiburst Waveform JPEG 2K 25mbps



Seating Dist

Screens

Small screen up close vs. large screen further away

Dot pitch most important factor

Native HD resolution displays make sense for larger (24 to 50 inch) sizes due to optimum viewing distances

Low-res screens *appear* sharper from a distance – this is due to the apparent dot pitch of the digital display

High Definition Workflow

Apple FCP EDIT-SAN Architecure

Edit System Software

Apple Final Cut Pro / Final Cut Studio 2.0

OSX 10.4

XSAN 1.4

Edit Suite Hardware Systems

4 MacPro Dual Quad Core Xeon platforms

AJA Kona 3 i/o

3 XSERV

30TB XRAID

Shared storage

27.3TB shared media pool

Incremental backup of projects nightly

• IBM Tivoli Storage Manager

• StorageTek Silo archive

• Source tape is backup for media pool currently

Multi-resolution capable

HD Edit workflow

SMPTE 292M

HDW-500A

PESA 292M HD Router

Final Cut Studio edit suites

XSANXRAID Stack

X-SAN Diagram

ResearchChannel HD Migration

SD cameras to be replaced with HD cameras• Likely Sony XDCam HD• MPEG-2 4:2:2 30-50mbps IMX• High speed file dump mode from Pro Blu-Ray dual

layer disc

4:3 safe zone shooting

HD capture and edit as uncompressed HD or IMX

Can output anamorphic, 4:3, 16:9 letterbox for SD

HD project archive

Edit Room Layout

Edit Room Layout

XSERVs in Master Control

XRAID

XSAN

Studio M

Credits

NAB 2007 reports:• Noah Pitzer• Jamie Alls• Jabran Soubeih• Dave Robertson• Jack Hoffman

• Special thanks to:• Staff engineer Jabran Soubeih for CoDec

measurements• Tektronix Guide to Standard and High Definition

Digital Video Measurements– http://www.tek.com/Measurement/App_Notes/

indexes/video_audio/registration/

For more information

www.researchchannel.org

[email protected]

Documents

High Definition Video: Workflow from Cameras to CODECs APAN Fall Meeting Xi’an, China Aug 2007