| Basics of Voice and Video over IP performance measurement | ||
| Measurement Tools we have experimented with… | ||
| “H.323 Beacon” Tool we have developed! | ||
| How is this tool different? | ||
| Tool Features | ||
| Two Case-studies | ||
| An Useful H.323 Beacon Tool Supplement!!! | ||
| Ethereal RTP Analysis Module | ||
| Conclusion | ||
Basics of Voice and Video over IP performance measurement
| Voice and Video Traffic are inherently different from Data Traffic | ||
| They use Signaling protocols such as H.323, SIP, … | ||
| Media (Voice and Video payload) is delivered over IP using RTP packets irrespective of which codec or end-point technology (PC-based Vs Appliance-based) | ||
| General ICMP and UDP based tools fail to totally capture the performance bottlenecks faced by actual voice and video traffic at the end-host and in the network | ||
| i.e.- ping, traceroute, Iperf, pathrate, … don’t suffice! | ||
| We need to measure end-user experience of Voice and Video over IP applications also! | ||
Measuring End-user experience…
| Two approaches to evaluating end-user experience of audiovisual quality | ||||
| Subjective Measurements | ||||
| Involve human participants to rate audiovisual quality Can you hear me now? | ||||
| Mean Opinion Score (MOS) Ranking technique (ITU-T P.800) Not just “Good”! | ||||
| Objective Measurements | ||||
| Automated techniques to rate audiovisual quality | ||||
| “E-Model” [ITU-T G.107] | ||||
| Perceptual Evaluation of Speech Quality (PESQ) [ITU-T P.862] | ||||
Mean Opinion Score (MOS) Concept
Scenario I: A Researcher and an Industry professional want to Videoconference
Case1:Researcher is unable to make a call!
There was a mis-configured firewall blocking necessary ports…
Case2: Industry professional is unable to make a call!
His LAN’s Internet connectivity was non-functional at that time…
Case3: They connected, but of them experienced bad audio & video!
There was congestion at one of the intermediate routers along the path…
There was congestion at one of the intermediate routers along the path…
There was congestion at one of the intermediate routers along the path…
The performance problem can be anywhere in the E2E Path!!!
What are the other common end-to-end performance problems?
| Common problems involving endpoint devices | ||
| Failure of audio and video hardware, out dated or buggy end-point application software | ||
| Faulty connections and configurations of audio and video interfaces | ||
| Lack of lip-synchronization | ||
| Mis-configured jitter buffer sizes | ||
| Non inter-operable end-point application software | ||
| Lack of forward error correction and echo cancellation mechanisms in end-point clients | ||
| Lack of end-user training | ||
Common End-to-End performance problems in VVoIP Systems (2)
| Common problems involving network devices | ||
| Insufficient network capacity for handling multiple high data rate videoconferencing calls | ||
| Excessive delay, loss, jitter, out of order packets and re-ordered packets in the network | ||
| Duplex mis-match problems | ||
| Traffic congestion at peak-usage periods of the network | ||
| Mis-configured priorities for real-time audio and video traffic streams in the network | ||
| Asymmetric routing with excessive delays on one path | ||
| Lack of network engineering resource personnel | ||
Common End-to-End performance problems in VVoIP Systems (3)
| Common problems involving application service devices | |||
| Misconfigured firewalls that block required ports | |||
| Non-H.323 friendly NATs | |||
| Misconfigured devices | |||
| MCUs | |||
| Gatekeeper | |||
| Gateway devices | |||
| Outdated or buggy application-service device application software | |||
Measurement Tools we have experimented with…
| MCU-specific Performance Monitors | ||
| Per-call monitoring of many network and end-point parameters (delay, jitter, loss, reordering) | ||
| Telchemy VQMon | ||
| Based on E-Model Standard; Integrated into many vendor VoIP troubleshooting products (e.g., Brix Networks, Agilent, etc…) | ||
| PESQ | ||
| Based on PESQ Standard; Used in many products (e.g., Opticom, Acterna, etc…) | ||
| Hammer Call Analyzer | ||
| Packet Analyzer (Ethereal backend) with many features such as call-flow display, jitter buffer analysis, wave file analysis, etc… | ||
| Nettach | ||
| Developed originally by PictureTel; Similar to H.323 Beacon where you emulate Voice and Video traffic and measure performance | ||
| Many other good tools… (AppareNet, ViDeNet Scout (based on NetIQ Chariot), etc.) | ||
MCU-specific Performance Monitors
| Taken from- http://www.empirix.com |
| An application-specific measurement tool | ||
| To monitor and qualify the performance of H.323 Videoconferencing sessions at the host and in the network (end-to-end) | ||
| Useful to an end-user/conference operator/network engineer | ||
| Uses OpenH323 and J323Engine libraries | ||
Comparison with other existing tools…
| Addresses problems due to H.323 protocol-specific idiosyncrasies | |||
| Can be generalized to RTP packets performance over the network | |||
| Many in-built tools that generate various kinds of measurement data for pre/during/post Videoconference troubleshooting! | |||
| An “easy to install and use” tool that is open source (FREE!) | |||
| Commercial tools are VERY expensive! | |||
| NetIQ Chariot, Spirent Smartbits, Telchemy VQMon,… | |||
Initial call setup failures and haphazard disconnections…
| Test Status | |||
| In-Session, Normal Close, Exception Close | |||
| Exception Close Alarms | |||
| “Possible Firewall/NAT presence obstruction” | |||
| “No Internet connectivity” | |||
| “Incompatible codec being used” | |||
| “Insufficient bandwidth” | |||
| “Remote H.323 Beacon Server not online” | |||
Initial call setup failures and haphazard disconnections…
| Delay, Jitter and Loss data | ||
| Real-time, offline raw data and test session summary | ||
| Watermarks for “Good”, “Acceptable” and “Poor” grade of quality as experienced by end-user | ||
| Delay: (0-150)ms, (150-300)ms, > 300ms | ||
| Jitter: (0-20)ms, (20-50)ms, > 50ms | ||
| Loss: (0-0.5)%, (0.5-1.5)%, >1.5% | ||
| Levels obtained from our PAM 2004 paper | ||
Audio and Video Quality Assessments
| Audio and video loopback feature | |
| E-Model-based objective MOS ranking | |
| Slider-based subjective MOS ranking |
| Test results data folder, TCP/UDP/RTP port settings, H.225 and H.245 parameters, preferred codec, watermarks for delay, jitter, loss, … |
| Problem report | ||
| Intermittent frame freezing | ||
| Lot of pixilation | ||
| No significant audio problems | ||
| Sudden disconnections | ||
| MCU Software | ||
| ~15 fps for 384Kbps call and ~0.6% packet loss | ||
| ~30 fps for 128Kbps call and ~0% packet loss | ||
| H.323 Beacon | ||
| Sluggish call-setup | ||
| Delayed packet-events | ||
| Initial jitter variations in poor range | ||
| Problem report | |||
| Poor audio and video reception at the site of user | |||
| Problem only in the streams from Campus A to Campus B | |||
| MCU Software | |||
| Packet loss (~1%) | |||
| Packet re-ordering | |||
| H.323 Beacon | |||
| Jitter values consistently in poor range | |||
| Traceroute and Reverse-Traceroute showed an asymmetric route | |||
| E-Model results | |||
| Campus A to Campus B: 3.78 | |||
| Campus B to Campus A: 4.37 | |||
Scenario I: H.323 Beacon used in a Network Measurement Infrastructure…
Scenario II: In a Multi-point setting…
Scenario II: In a Multi-point setting…
Scenario II: In a Multi-point setting…
| An useful H.323 Beacon Tool supplement!!! | |||
| Adds advantages of passive measurement to the H.323 Beacon | |||
| In general, to any RTP-based application! | |||
| Generates inter-packet jitter, packet-size distribution, packet loss and re-ordering information from input traffic traces | |||
| We have written simple parsing scripts that can process data in a format that facilitates analysis using Minitab or Excel | |||
| Use the OPENXTRA version of Ethereal! | |||
| http://resource.intel.com/telecom/support/appnotes/9008/9008an.pdf | |||
Ethereal RTP Analysis Module (2)
| Steps for analyzing the Traces | |||
| Load the packet trace into Ethereal | |||
| Trace will contain both forward and reverse direction streams (Check “Source” and “Destination” IP addresses) | |||
| Decode streams as RTP (default is UDP) | |||
| This will mark all related packets as belonging to a specific audio and video codec streams | |||
| Analyze individual audio or video streams | |||
| Import various information fields as .csv file (“Save as CSV” option) | |||
| Also has wave file generation relating to an audio stream (“Save Payload” option) | |||
| Works only for G.711 Codec streams! | |||
| Good for PESQ where you want to compare original and degraded wave files to obtain Objective MOS information | |||
Ethereal RTP Analysis Module (3)
Ethereal RTP Analysis Module (4)
Ethereal RTP Analysis Module (5)
Ethereal RTP Analysis Module (6)