View
117
Download
0
Category
Preview:
Citation preview
Choosing the Right IP Cores for Low-Latency Video Streaming
Nikos Zervas CEO CAST, Inc. December 2016 www.cast-inc.com
Slide2LowLatencyVideoStreamingIP
Introduction
Low latency is important when there is real-time interaction with the video content Gaming, Video Conferencing, Automotive, Industrial, Medical,
Defense
How low is low latency? For human interaction sub-100ms is very low latency Machine interaction may require even less
Slide3LowLatencyVideoStreamingIP
Introduction
End-to-end or glass-to-glass latency is the amount of time between the instant a frame is captured to the instant the same frame is displayed
Latency is introduced by temporal storage (buffering) Latency is measured in time or buffering (storage) units
Slide4LowLatencyVideoStreamingIP
Latency comes from buffers Mission: Eliminate Excessive Buffering
Buffers are hidden within IP cores, and IP cores may impose external buffers
Focus on what really counts: Pixel buffers do not really count 2000 pixels: 0.03 msec for 1080p30
Video Line buffers count 100 lines: 3.01 msec for 1080p30
Frame buffers count even more 1 frame: 33.33 msec for 30fps
Oneframe 33.33 msOneline 30.86 μsOnepixel 16.08 ns
LatencyFor1080p30Video
Slide5LowLatencyVideoStreamingIP
Choosing the Right IP Core
Throughput is irrelevant: Two cores with same throughput may have totally different latency
Input/output data-ordering may impose external buffers for reordering
Choose “line-based” vs “frame-cased” image processing algorithms
Some cores do not produce any output before they process a full frame, and some do not start processing unless a full frame is available at their input.
TIP
Slide6LowLatencyVideoStreamingIP
Glass-to-Glass Latency
Why do I even need a BRAB and a DSB?
ProcessingStage Buffering Latency(for1080p30)
Capturepost-processing(e.g.bayerfilter,3A,ISP)
Fewlines(e.g.32) <1msec
VideoCompression(JPEG,MPEG-1/2/4orH.264)
8or16linesforconversionfromrasterscanFewthousandofpixelsontheencoderpipeline
0.5msec<0.05msec
Bit-RateAveragingBuffer Anumberofframes(e.g.morethan30)tosub-frame(e.g.1/2frame)
from1secto16msec
NetworkProcessing(e.g.RTP/UDP/IPencapsulation)
FewKbytes <1msec
DecoderStreamBuffer Anumberofframes(e.g.morethan30)tosub-frame(e.g.1/2frame)
from1secto16msec
VideoDecompression(JPEG,MPEG-1/2/4orH.264)
8or16linesforconversionfromrasterscanFewthousandofpixelsonthedecoderpipeline
0.5msec<0.05msec
Displaypre-processing(e.g.Scaling,ChromaResampling)
Fewlines(e.g.32) <1msec
Slide7LowLatencyVideoStreamingIP
Bit-Rate Control and Latency Bit rate is not natively constant nor predictable
Bit rate control averages the bit rate to a constant value
0
5
10
15
20
25
0 100 200 300 400 500 600 700 800 900 1000
Mbp
s&me(msec)
0
5
10
15
20
25
0 100 200 300 400 500 600 700 800 900 1000
Mbp
s
&me(msec)
1stAveragingPeriod 2ndAveragingPeriod 3rdAveragingPeriod
Slide8LowLatencyVideoStreamingIP
BRAB & DSB
!!
!!
Bit-Rate Averaging Buffer (BRAB) on the encoder side “normalizes” encoder bit rate Can be reduced to a zero-latency FIFO, if channel can “absorb”
bit rate picks
Decoder Stream Buffer (DSB) buffers one averaging period worth of stream, before decoding is initiated, to avoid video playback interruption
BRAB
Slide9LowLatencyVideoStreamingIP
BRAB & DSB
!!
!!ü BRAB Latency = DSB Latency = Averaging Period
BRAB
0
5
10
15
20
25
0 100 200 300 400 500 600 700 800 900 1000
Mbp
s
&me(msec)
1stAveragingPeriod 2ndAveragingPeriod 3rdAveragingPeriod
Slide10LowLatencyVideoStreamingIP
Rate Control is What Matters
All rate control algorithms adjust the compression-level based on the remaining bit rate budget in the averaging period (=buffer occupancy)
ü Accurate prediction of future content’s compressed size is infeasible, but some algorithms make better predictions by: Assessing the “complexity” of the remaining content Getting “trained” based on history
ü The frequency of compression-level adjustments varies § More frequent adjustment allows to quickly adopt to content
changes and correct inaccuracies of predictions
Slide11LowLatencyVideoStreamingIP
The Role of Rate Control
Rate Control defines Latency but also Quality
Rate Control is not described in standards
Samecompressionalgorithm,bitrateandaveragingperiod–DifferentEncoders
Slide12LowLatencyVideoStreamingIP
Choosing the Right IP Core
Focus on encoder’s rate control attributes, and evaluate latency and quality (IP core vendor should be able to help you) Test streams do not always stress rate control. Rate control algorithms may overflow or underflow BRAB & DSB, unless they are able to regulate compression levels multiple times within the averaging period
Regardless of rate control attributes, software codecs are not suitable for ultra-low latency
TIP
Slide13LowLatencyVideoStreamingIP
Choosing the Right H.264 IP Core
Choose an H.264 encoder IP core supporting Intra-Refresh
Intra-Refresh allows construction of an intra-coded reference over a number of frames. It eliminates I-frames and the related bit-rate “peaks”
TIP
0
5
10
15
20
25
30
35
0 200 400 600 800 1000
Mbp
s
&me(msec)
0
5
10
15
20
25
30
35
0 200 400 600 800 1000
Mbp
s
&me(msec)
IPPcoding Intra-Refreshcoding
Slide14LowLatencyVideoStreamingIP
JPEG
Choosing the Right JPEG IP Core
A JPEG encoder IP core is not suitable for sub-frame latencies, because the JPEG standard only allows Quantization changes at frame boundaries
Some JPEG encoders can regulate bit-rate also by truncating code-bits from Huffman-coded blocks. The stream remains complaint, and allows intra-frame bit-rate “corrections”
Slide15LowLatencyVideoStreamingIP
Compression Cores from CAST
PLUS:ü HardwareRTP,UDP/IPstacksü CompleteVideo-Over-IPplaTormsü ExperVsetohelpyouarchitectyoursystemandselecttheright
algorithms
CompressionStandard SuggestedMinAveragingPeriod
CompressionLevels
MotionJPEG 1/4Frame ModerateIntra-onlyH.264 1/4Frame HighH.264wIntraRefresh 1/4Frame VeryHighH.264w/oIntraRefresh 3-4Framse VeryHigh
Recommended