Embed Size (px)
Citation preview
Troubleshooting Voice and Troubleshooting Voice and Video Performance over IP Video Performance over IP
using Measurement Tools…using Measurement Tools… Prasad Calyam and Paul Schopis,Prasad Calyam and Paul Schopis,ITEC-Ohio, OARnet and Ohio Supercomputer Center, ITEC-Ohio, OARnet and Ohio Supercomputer Center,
Tools Tutorial,Tools Tutorial, Joint Techs, Salt Lake City, Utah, February 2005Joint Techs, Salt Lake City, Utah, February 2005
Topics of Discussion
Basics of Voice and Video over IP performance measurementMeasurement Tools we have experimented with…“H.323 Beacon” Tool we have developed!
How is this tool different?Tool FeaturesTwo Case-studies
An Useful H.323 Beacon Tool Supplement!!!Ethereal RTP Analysis Module
Conclusion
Basics of Voice and Video over IP performance measurement
Voice and Video Traffic are inherently different from Data Traffic
They use Signaling protocols such as H.323, SIP, …Media (Voice and Video payload) is delivered over IP using RTP packets irrespective of which codec or end-point technology (PC-based Vs Appliance-based)
General ICMP and UDP based tools fail to totally capture the performance bottlenecks faced by actual voice and video traffic at the end-host and in the network
i.e.- ping, traceroute, Iperf, pathrate, … don’t suffice!We need to measure end-user experience of Voice and Video over IP applications also!
Measuring End-user experience…
Two approaches to evaluating end-user experience of audiovisual quality
Subjective Measurements Involve human participants to rate audiovisual quality Can you hear me now? Mean Opinion Score (MOS) Ranking technique (ITU-T P.800) Not just “Good”!
Objective Measurements Automated techniques to rate audiovisual quality
“E-Model” [ITU-T G.107]Perceptual Evaluation of Speech Quality (PESQ) [ITU-T P.862]
Mean Opinion Score (MOS) Concept
Scenario I: A Researcher and an Industry professional want to Videoconference
Internet2 Abilene Network
GigaPOP
OC2
OC192
What are the end-host and network problems?
Case1:Researcher is unable to make a call!
Internet2 Abilene Network
GigaPOP
OC2
OC192
There was a mis-configured firewall blocking necessary ports…
Internet2 Abilene Network
GigaPOP
OC2
OC192
Case2: Industry professional is unable to make a call!
Internet2 Abilene Network
GigaPOP
OC2
OC192
His LAN’s Internet connectivity was non-functional at that time…
Internet2 Abilene Network
GigaPOP
OC2
OC192
Case3: They connected, but of them experienced bad audio & video!
Internet2 Abilene Network
GigaPOP
OC2
OC192
There was congestion at one of the intermediate routers along the path…
Internet2 Abilene Network
GigaPOP
OC2
OC192
There was congestion at one of the intermediate routers along the path…
Internet2 Abilene Network
GigaPOP
OC2
OC192
There was congestion at one of the intermediate routers along the path…
Internet2 Abilene Network
GigaPOP
OC2
OC192
The performance problem can be anywhere in the E2E Path!!!
Internet2 Abilene Network
GigaPOP
OC2
OC192
What are the other common end-to-end performance problems?
Common problems involving endpoint devicesFailure of audio and video hardware, out dated or buggy end-point application softwareFaulty connections and configurations of audio and video interfacesLack of lip-synchronizationMis-configured jitter buffer sizesNon inter-operable end-point application softwareLack of forward error correction and echo cancellation mechanisms in end-point clientsLack of end-user training
Common End-to-End performance problems in VVoIP Systems (2)
Common problems involving network devices Insufficient network capacity for handling multiple high data rate videoconferencing calls
Excessive delay, loss, jitter, out of order packets and re-ordered packets in the network
Duplex mis-match problems
Traffic congestion at peak-usage periods of the network
Mis-configured priorities for real-time audio and video traffic streams in the network
Asymmetric routing with excessive delays on one path
Lack of network engineering resource personnel
Common End-to-End performance problems in VVoIP Systems (3)
Common problems involving application service devices
Misconfigured firewalls that block required ports
Non-H.323 friendly NATs
Misconfigured devices
MCUs
Gatekeeper
Gateway devices
Outdated or buggy application-service device application software
Measurement Tools we have experimented with…
MCU-specific Performance Monitors Per-call monitoring of many network and end-point parameters (delay, jitter, loss, reordering)
Telchemy VQMonBased on E-Model Standard; Integrated into many vendor VoIP troubleshooting products (e.g., Brix Networks, Agilent, etc…)
PESQ Based on PESQ Standard; Used in many products (e.g., Opticom, Acterna, etc…)
Hammer Call AnalyzerPacket Analyzer (Ethereal backend) with many features such as call-flow display, jitter buffer analysis, wave file analysis, etc…
NettachDeveloped originally by PictureTel; Similar to H.323 Beacon where you emulate Voice and Video traffic and measure performance
Many other good tools… (AppareNet, ViDeNet Scout (based on NetIQ Chariot), etc.)
MCU-specific Performance Monitors
Increased Video Sync Loss!
Telchemy VQMon
PESQ
SAMPLE RESULTS: REFERENCE DEGRADED PESQMOS PESQMOS original.wav degraded.wav SQValue=2.930 2.930
Hammer Call Analyzer
Taken from- http://www.empirix.com
Nettach
H.323 Beacon Overview
An application-specific measurement tool To monitor and qualify the performance of H.323 Videoconferencing sessions at the host and in the network (end-to-end)
Useful to an end-user/conference operator/network engineer
Uses OpenH323 and J323Engine libraries
Comparison with other existing tools…
Addresses problems due to H.323 protocol-specific idiosyncrasies
Can be generalized to RTP packets performance over the networkMany in-built tools that generate various kinds of measurement data for pre/during/post Videoconference troubleshooting!
An “easy to install and use” tool that is open source (FREE!)
Commercial tools are VERY expensive!NetIQ Chariot, Spirent Smartbits, Telchemy VQMon,…
Initial call setup failures and haphazard disconnections…
Test Status In-Session, Normal Close, Exception Close
Exception Close Alarms “Possible Firewall/NAT presence obstruction”
“No Internet connectivity”
“Incompatible codec being used”
“Insufficient bandwidth”
“Remote H.323 Beacon Server not online”
Initial call setup failures and haphazard disconnections…
Network Health Status…
Delay, Jitter and Loss data Real-time, offline raw data and test session summary
Network Health Plots…
Watermarks for “Good”, “Acceptable” and “Poor” grade of quality as experienced by end-userDelay: (0-150)ms, (150-300)ms, > 300msJitter: (0-20)ms, (20-50)ms, > 50msLoss: (0-0.5)%, (0.5-1.5)%, >1.5%
Levels obtained from our PAM 2004 paper
Poor
Acceptable
Good
Audio and Video Quality Assessments
Audio and video loopback feature
E-Model-based objective MOS ranking
Slider-based subjective MOS ranking
Customization of tests…
Test results data folder, TCP/UDP/RTP port settings, H.225 and H.245 parameters, preferred codec, watermarks for delay, jitter, loss, …
Use-case I
Effect of a mis-configured firewall on a video stream
Problem report Intermittent frame freezing
Lot of pixilation
No significant audio problems
Sudden disconnections
Use-case I (2)
MCU Software ~15 fps for 384Kbps call and ~0.6% packet loss ~30 fps for 128Kbps call and ~0% packet loss
H.323 Beacon Sluggish call-setup Delayed packet-events Initial jitter variations in poor range
Jitter variations indicated by H.323 Beacon client-to-server test
Increasing the rate-limit and firewall re-configuration solved the problem!
Use-case II
Problem report Poor audio and video reception at the site of user Problem only in the streams from Campus A to Campus B
MCU Software Packet loss (~1%) Packet re-ordering
H.323 Beacon Jitter values consistently in poor range Traceroute and Reverse-Traceroute showed an asymmetric route E-Model results
Campus A to Campus B: 3.78Campus B to Campus A: 4.37
Use-case II (2)
Fig. 2 Time-plot of Delay
Fig. 3 Time-plot of Loss Fig. 4 Time-plot of Mean Opinion Score Ranking
Fig. 1 Time-plot of Equipment Impairment Factor
Problem was solved by contacting the neighboring ISP who was suspected to be experiencing problems!
Scenario I: H.323 Beacon used in a Network Measurement Infrastructure…
Internet2 Abilene Network
GigaPOP
OC2
OC192
3Com
CISCOSYSTEMS
3Com
CISCOSYSTEMS
Core Router
Switch
NMS
CDMA Device
How do you setup the H.323 Beacon in a network?
Scenario II: In a Multi-point setting…
Scenario II: In a Multi-point setting…
Scenario II: In a Multi-point setting…
Ethereal RTP Analysis Module
An useful H.323 Beacon Tool supplement!!!Adds advantages of passive measurement to the H.323 Beacon
In general, to any RTP-based application!
Generates inter-packet jitter, packet-size distribution, packet loss and re-ordering information from input traffic tracesWe have written simple parsing scripts that can process data in a format that facilitates analysis using Minitab or ExcelUse the OPENXTRA version of Ethereal!
http://resource.intel.com/telecom/support/appnotes/9008/9008an.pdf
Ethereal RTP Analysis Module (2)
Steps for analyzing the TracesLoad the packet trace into Ethereal
Trace will contain both forward and reverse direction streams (Check “Source” and “Destination” IP addresses)
Decode streams as RTP (default is UDP) This will mark all related packets as belonging to a specific audio and video codec streams
Analyze individual audio or video streamsImport various information fields as .csv file (“Save as CSV” option)Also has wave file generation relating to an audio stream (“Save Payload” option)
Works only for G.711 Codec streams!Good for PESQ where you want to compare original and degraded wave files to obtain Objective MOS information
Ethereal RTP Analysis Module (3)
General UDP Stream decoded as an H.263
payload stream
Ethereal RTP Analysis Module (4)
Audio Stream Video Stream
Ethereal RTP Analysis Module (5)
Pink-marked packets relate to either lost or re-ordered packets!
Re-ordering; 45413 45415 45414(Observe Sequence #s; could also be 2
consecutive packet losses)
Loss
Ethereal RTP Analysis Module (6)
An Imported CSV File!
Ethereal RTP Analysis Module (6)
Create interesting visualizations to understand various RTP packet characteristics; can do the same for both Voice and Video packets!!!
Questions?
http://www.itecohio.org/beacon
H.323 Beacon Reference:
