Upload
rodrigo-uruchi
View
220
Download
0
Embed Size (px)
Citation preview
7/25/2019 Video Quality Metrics Discussion
1/21
IETF RMCAT WG: Video Quality Metrics
Discussion for Evaluation Criteria
V. . o!aya"ulu#v.srinivasa.so!aya"ulu$intel.co!%
&assnaa Moustafa#'assnaa.!oustafa$intel.co!%
7/25/2019 Video Quality Metrics Discussion
2/21
(ro)le! tate!ent
Video Quality Perception is critical for end*userQoE
To+ic 'as )een discussed in WG, )ut noconclusion yet on includin- t'is in t'ereuire!ents or t'e evaluation !etrics for t'econ-estion control al-orit'!s
/et0or12trans+ort related !etrics are currently+resent in t'e evaluation criteria for R!catcon-estion control al-orit'!s, )ut do not fully
ca+ture t'e i!+act on video QoE.Purpose of this presentation:1. Provide background on video quality metrics that canhelp quantify impact of congestion control on the video QoE2. Stimulate a discussion on adoption of appropriate metricsin evaluation criteria
7/25/2019 Video Quality Metrics Discussion
3/21
#&i-' 3evel% u!!ary of Evaluationcenario Discussions in RMCAT WG
Current #evolvin-% direction see!s to )e *Evaluate overall +erfor!ance, i.e., Video2Audio4/et0or1 Con-estion control al-orit'!
Cou+le of 0ays to include video c'aracteristics:
5. 6se a tra7c !odel t'at ca+tures statistics of t'evideo source4rate control4s'a+in-
8. (ossi)ly use unco!+ressed video seuences 0it' alive encoder #4 +rescri)ed settin-s% and con-estion
control al-o. Wit' eit'er a++roac', 0e s'ould de9ne a !eans
of ca+turin- i!+act on t'e overall video QoE.
7/25/2019 Video Quality Metrics Discussion
4/21
);ective Metrics < =i- (icture Vie0
=ac1-round );ective !etrics develo+ed to !i!ic 'u!an
+erce+tion of video, e.-., >loo1 at t'e receivedvideo, and ;ud-e 'o0 -ood it loo1s?
Distortions )et0een source and videoconsidered a >)lac1 )o@? and did not aect
!etric desi-n
Good for un)iased evaluation of encoders, etc. /on*reference 2 Reduced reference a di7cult
+ro)le! in t'is -eneric case
Full*reference vs. non*reference VQM Full reference: co!+are t'e !easured
video 0it' t'e ori-inal unco!+ressedvideo (/R, M*IM, (EVQ
/on*reference: analy"e t'e video0it'out a co!+arison E.-. (58B8, etc.
IT6*T G.5BB
Intel Con9dential
ourceVideo
Vie0edVideo
Distortions
&.8Encode
&.8Decode
I( #6D(%(ac1ets
(ac1et3oss2itter
(1t 3ossConceal!ent
=uerMana-e!ent
7/25/2019 Video Quality Metrics Discussion
5/21
Video Conferencin- QoE < IT6*TG.5BB
IT6*T G.5BB
!!endation IT6*T G.5BB +ro+oses an al-orit'! t'at esti!ates video+'one uality for +erience #QoE%2uality of service #Qo% +lanners.
7/25/2019 Video Quality Metrics Discussion
6/21
IT6*T G.5BB Video QualityMetric
Video uality is calculated as:
0'ere Icodingre+resents )asic video uality as a function of video )itrate and
fra!e rate
DPplVre+resents de-ree of video uality ro)ustness due to +ac1et loss and
PplV is t'e +ac1et loss rate in
T'ese uantities are calculated usin- a set of 9@ed +ara!eters de+endenton codec ty+e, video for!at, 1ey fra!e interval, and video dis+lay si"e G.5BB +rovides +rovisional values for &.8, VGA for!at, 5 second 1ey fra!e
interval and H.8 inc' dis+lay, coded )it rates )et0een BB 1)+s < 8 M)+s, +ac1et lossrates J and fra!e rates fro! J*8J f+s.
(ara!eter values for !odelin- any ot'er set of conditions 0ould need to )e derivedfro! video uality evaluation
Furt'er en'ance!ents to t'e )asic !odel for dierent codecs,dis+lay for!ats, content de+endency, etc. 'ave )een e@+lored, e.-.: Kos1o0ic", 8BBHL os1o0ic", . et al >En'ance!ents to t'e +inion Model
for Video*Tele+'ony A++lications?, Proc. 5thLatin American NetworkingConference, ++. *H.
7/25/2019 Video Quality Metrics Discussion
7/21
Considerations on use of IT6*T
G.5BB
6se evaluation scenario#s% to run si!ulation of tar-etcon-estion avoidance al-orit'! 0it' a -iven testvideo seuence #trace 9le% Collect trace of +ac1et arrival ti!es, +ac1et losses, at
sender and receiver e-!ent data into s'ort intervals of ti!e #e.-. J secondsN%
For eac' se-!ent i: Calculate video )itrate sent, assu!e 9@ed fra!e rate
Calculate avera-e +ac1et loss rate at receiver
Calculate +er*se-!ent video uality Vq, i
Calculate over t'e entire video seuence:
Mean and variance of t'e set of OVq, iP
&i-'er !ean and lo0er variance )etter overall video uality
7/25/2019 Video Quality Metrics Discussion
8/21
Discussion (ro+ose to add video uality evaluation !etrics for consideration )y R!cat WG.
Desira)le features of t'e !etrics s'ould include: Good correlation 0it' su);ective video uality +erce+tion
Co!)ine dierent +ara!eters to +rovide an inte-rated loo1 at video QoE i!+act
Relatively si!+le to calculate )ased on data fro! net0or1 si!ulations
Ideally, )ased on +u)lis'ed standards
u);ective uality evaluation is 'ard to or-ani"e and e@ecute, es+ecially in a contri)utionevaluation +'ase
Co!!on o);ective evaluation !etrics of video uality
Easier to use in +ro+osal evaluations,
Full*reference /on*reference: latter !ay )e !ore suited for RMCAT evaluation
IT6*T Rec.G.5BB )ased /R VQ !etrics desi-ned for video conferencin- a++lications
Additionally, t'is doesnt reuire co!+ressed )itstrea! ins+ection, etc.: al-orit'! in+uts are 'i-'level, e.-. t'rou-'+ut, +ac1et loss rate, etc.
Soliciting feedback from group on dening consideration for video qualitymetrics as part of evaluation criteria for congestion control algorithms! "urrent phase
#or further consideration: "ould $Q be e%ploited by congestion controlalgorithms!
Potential for incorporating video quality information into &'"P (&)*+", reports!-*ater phase
7/25/2019 Video Quality Metrics Discussion
9/21
/nne%
7/25/2019 Video Quality Metrics Discussion
10/21
=ac1-round R!cat WG is dealin- 0it' con-estion control for Internet data considerin- interactive
+oint*to*+oint real*ti!e !ulti!edia services over RT(
Reuire!ents for con-estion control al-orit'!s are de9ned considerin- 3o0 delay
e!i*Relia)le data delivery
Fairness to ot'er So0s
Ada+tation to net0or1 conditions
Metrics for con-estion control are de9ned to )e Delay, t'rou-'+ut, !ini!i"in- trans!ission rates oscillations, reactivity to transient events and
+ac1et losses and discards
Evaluation Criteria for con-estion control al-orit'!s 'ave )een de9ned considerin- Avoidin- Con-estion Colla+se
ta)ility
Media Tra7c tartu+ =e'avior
Diverse Environ!ents
Varyin- (at' C'aracteristics
Reactin- to Transient Events or Interru+tions
Fairness 0it' i!ilar Cross Tra7c
I!+act on Cross Tra7c
E@tensions to RT(2RTC(
7/25/2019 Video Quality Metrics Discussion
11/21
Video Quality Variation 02 /et0or1Conditions Variation
Sorce! "#$ Cheng et al., %&ea'rement and Anal('i' of Sk(pe Video
)ra*c,+ APCS -/.
PSN"lo'' for di0erent network 1andwidthlimitation'
PSN"lo'' for di0erent packet lo''rate
7/25/2019 Video Quality Metrics Discussion
12/21
I!+act of T'rou-'+ut Variation onVideo Conferencin- A++lications
Sorce! L. De Cicco et al., %Sk(pe Video "e'pon'i2ene'' to 3andwidth
Variation,+ AC& N4SSDAV -/.
7/25/2019 Video Quality Metrics Discussion
13/21
Video Quality Evaluation:
Introduction
Quality of E@+erienceT'e overall e@+erience t'e consu!er 'as 0'en
accessin- and usin- +rovided video services
Quantifyin- Video Quality Mean +inion core #M%
u);ectively done: recruit a -rou+ of +eo+le to 0atc' a set ofvideo cli+s and -ive a nu!eric score to eac' cli+
Auto!atically done: desi-n al-orit'!s to esti!ate a M)ased on c'aracteristics of !edia strea!, net0or1, device,
etc. Video Quality Issues
Video creation, video encodin-2transcodin-, videotrans!ission, video dis+lay
7/25/2019 Video Quality Metrics Discussion
14/21
Video Quality Issues
Video )loc1iness #encodin-%
Video )lurriness #ca+turin-2encodin-%
Video losses #trans!ission%
Video ;er1iness #trans!ission2encodin-2dis+lay%
Video free"in-2re)uerin- #trans!ission% A2V sync +ro)le! #trans!ission2encodin-%
=loc1iness vs. )lurriness
7/25/2019 Video Quality Metrics Discussion
15/21
(/R #(ea1 i-nal*to*/oiseRatio%
Most co!!only used !etric to!easure t'e uality of reconstructionof lossy co!+ression codecs
Ty+ically values )et0een BUJB d=,
'i-'er is )etter
7/25/2019 Video Quality Metrics Discussion
16/21
M*IM
3u!inance l6, (7 Contrast c6,(7 tructure s#@,y%
Ty+ical values )et0een B.U5, 'i-'er is )etter
7/25/2019 Video Quality Metrics Discussion
17/21
(EVQ
ut+ut (EVQ M ran-in- fro! 5 #)ad%UJ #e@cellent%Distortion Indicators: Delay, =ri-'tness, Contrast, (/R, er1iness, =lur,=loc1iness, Fra!e s1i+s and free"es, Te!+oral Activity and s+atial co!+le@ity
7/25/2019 Video Quality Metrics Discussion
18/21
IT6*T G58 /o*Reference );ective
tandards
Consented orA++roved docu!ents
s'o0n in black Tar-eted A++lications:
I(TV services non*ada+tive strea!in-non*interactive
(.58B5.@ < !ulti!ediaQoE(.58B8.@ < video QoE
Tar-eted +rotocols:non*&TT( #RT(, T*on6D(, etc.%
=ittrea!
(ac1et&eaders
DecodedVideo
(.58B5.5#l)rX%(.58B5.8#')rX%(.58B8.5 #l)rX%(.58B8.8 !ode 5 #')rX%
J.bitvqm,J.mm-noref, J.noref,
VQEG-JEG Hybrid Project
In+uts
Intel Con9dential 5
(.58B8.8 !ode 8 #')rX%
X l)r* lo0 )it rate ')r* 'i-' )it rate
7/25/2019 Video Quality Metrics Discussion
19/21
(.58B@.@ Video I!+air!entModel
(.58B8 A++roac' Desi-n a !etric for evaluatin- uality in s+eci9c instance i.e., I(TV services
T'is 'igni8cantl(constrains t'e +ro)le!, and s'ould !a1e t'e solution !uc'!ore feasi)le
Yno0n video encoder encoder out+ut availa)le
Yno0n c'annel i!+air!ent +attern availa)le
Four !ain video distortions accounted for Co!+ression Artifacts
Due to lossy encodin-
licin- Artifacts Due to (ac1et 3oss Conceal!ent #(3C% of lost +ac1ets
Free"in- Artifacts Due to (3C re+lacin- erroneous fra!es 0it' last -ood fra!e #>free"in- 0it'
s1i++in-?%
Re)uerin- Artifacts Due to (3C re+eatin- a fra!e until fra!e rece+tion reco!!ences #>free"in-
0it'out s1i++in- 2 s+innin- 0'eel?%
7/25/2019 Video Quality Metrics Discussion
20/21
(.58B8 #e@. (./=AM%
T0o a++lication areas :
(.58B8.5,Zlo0er resolution !ode?: a!e as (58B5.5:
i.e., QCIF*QVGA*&VGA, !ostly for !o)ile TVand trea!in-
(.58B8.8, Z'i-'er resolution !ode?:
3inear )roadcast TV Video on*de!and: till under tudy
Intel Con9dential 8B
7/25/2019 Video Quality Metrics Discussion
21/21
(.58B8 #e@. (./=AM%
(ac1et &eaders and)itstrea! in+ut only
/ot intended forcodec evaluation
/ot intended forstrea!s 0it'
si-ni9cant rateada+tation
Video (earsoncorrelation of B.H5for (.58B8.5 #H8sa!+les%
$alidated 'est #actors
Reco!!endation
(.58B8.5 (.58B8.8
Audio =R /A /A
Video =R B.BJ < M)+s 6nder study
(ac1et loss 6nder study
Re*)uerin-
VideoResolution
&VGA, QVGA,QCIF
6nder study
Video
encodin-
&.82AVC
)aseline
6nder study
FRs and 1eyfra!e rates
Fra!e rate J*B&"G( len-t's 8*5Bs
6nder study
(rotocol 6D(*)ased 6nder study
(rotocols not
testedX
TC(*)ased 6nder study
X #can )e used, )ut !ay not )e relia)le%
Intel Con9dential 85