perfSONAR Enables Internet2 Cisco TelePresence Service
Internet2 used perfSONAR tools to quickly establish a network for a Cisco TelePresence video conferencing demo at the 2009 Spring Member Meeting. The high demands of a 1080p video connection were met by using methodologies defined by the perfSONAR tools including BWCTL, OWAMP, and Nagios Alarms.
- U.S. Department of Veteran Affairs
Products & Services
Internet2 and the U.S. Department of Veterans Affairs provided a live demonstration at the 2009 Internet2 Spring Member Meeting showcasing how telepresence technologies can advance important telehealth initiatives. The demonstration—representing the first Cisco TelePresence session held over the Internet2 Network—connected a psychiatrist located at the conference hotel with a “wounded warrior” veteran patient at Harvard in Massachusetts for a simulated psychiatric diagnosis session.
Cisco TelePresence incorporates high-quality spatial audio and lifelike, low-latency, 1080p high-definition video in a specially tuned environment to deliver an immersive experience for users. To ensure a consistent, high-quality user experience, networks offering the service must meet stringent performance standards, including:
- 10 milliseconds (ms) jitter or less
- 160 ms delay or less
- Less than 0.05% packet loss
To guarantee that these requirements were met, the demonstration team had to measure delay, jitter and packet loss between connection points—and be prepared to fix any issues that appeared. Unfortunately, multi-domain paths are the most difficult to diagnose, and there were five major network segments included in the demonstration path, with connection points at the hotel in Arlington, VA, the Mid-Atlantic Crossroads (MAX) GigaPoP, Internet2 PoPs in Washington, D.C., and New York, Northern Crossroads (NoX) GigaPoP, and, finally, the Harvard University network. A dependable set of interoperable tools and methodologies was required.
The Internet2 team leveraged perfSONAR-PS to provide the required monitoring options and measurement types. perfSONAR is a multi-institution project that has developed a framework in which a suite of interoperable services and tools support multi-domain network measurements and monitoring activities. Tools in the perfSONAR-PS implementation include BWCTL, OWAMP, perfSONAR- BUOY, PingER, SNMP, NDT, and NPAD. The tools are built on standards being defined by the Open Grid Forum (OGF).
The demonstration team planned to decompose the path into its individual segments by deploying perfSONAR-PS hosts at each connection point and running regular latency tests on the intervening segments. This would allow them to isolate exactly where problems were occurring.
Utilization statistics from routers along the end-to-end path would then be measured, allowing the team to drill down and better understand why problems were occurring.
Using the Internet2 performance tool OWAMP (One-Way Ping, an implementation of the IETF-approved One-Way Active Measurement Protocol) via the perfSONAR-BUOY service, regular testing was conducted along each segment. Data from each location was analyzed by visualization tools created to help engineers zero in on potential network problems by:
- Making it easy to view and understand data
- Providing a variety of views
- Showing the status of a given host, a given path segment, and the entire network
- Alerting personnel to potential problems
To give engineers “the big picture,” the tools provided a grid view of the entire network, showing the latency, jitter and loss between all hosts. To highlight path status, individual graphs showed jitter and loss between hosts, along with interface utilization for the path. Nagios Alarms, an open source tool easily integrated into NOC reporting systems, provided the alert system, notifying network engineers when problems appeared.
To give engineers “the big picture,” perfSONAR- PS tools provided a grid view of the entire network, showing latency, jitter and loss between all hosts.
Nagios Alarms provided the alert system, notifying network engineers when problems appeared.
Jitter readings measured over discrete network segments helped determine the source of the problem. These graphs show jitter as measured from hotel-to-NoX (left) vs. hotel-to-MAX (right).
Using this approach, several potential issues were identified:
- The existence of a highly utilized link in the path
- Too much cross traffic
- Substandard test machine capability/quality
- Conflicts with other software running on the hosts, and
- Network Time Protocol (NTP) Drift
All of these were solved and verified through diagnostics and monitoring. To find the highly utilized link, jitter was measured along the path, first between the hotel and Harvard, then between the hotel and NoX, and so on. Through these measurements, it was determined that the problem link was somewhere between MAX and New York.
Using alternate data sources such as passive SNMP measurements from MAX and Internet2 backbone data provided by perfSONAR, engineers drilled down even further to examine each leg of the path. At this stage, the team discovered several congestion points: the 1Gbps uplink from the hotel to MAX, the 10Gbps MAX core, and 2.5Gbps Internet2 uplink.
Several potential solutions were considered: Identify flows and re-engineer network traffic, re-plumb the demonstration network path or increase capacity.
In the end, the team decided to increase MAX headroom—similarities to the famous TV character unintentional—to 10Gbps. This solved the problem nicely, and when it was time to set up the demonstration in Arlington, it went without a hitch. The team sent this message to Internet2 colleagues: "...we’ve made our first call from the Ballroom to Harvard. The connection was flawless...the CTS node is ready to go."
The demonstration pointed out—in no uncertain terms—that the ability to create and maintain a sustainable and robust end-to-end network path is essential when it comes to implementing the Cisco TelePresence Service. As Internet2 works with its members to roll out the service, deployment of the perfSONAR monitoring system will be facilitated at each endpoint to ensure high-quality network performance from the local campus network to the backbone.
Based on the defined requirements, perfSONAR tools can provide network monitoring personnel with instant updates on network performance changes that adversely affect the performance of the Cisco TelePresence system.
Because of its architecture and standards-based approach, perfSONAR services offer critical support for all brands of telepresence service deployments, to validate network capability requirements as specified by each service provider.
Network performance problems become actionable when performance characteristics can be seen by all interested parties. perfSONAR has proved its worth when it comes to making network performance characteristics visible and actionable.
To find out what’s ahead on the perfSONAR-PS development roadmap, visit psps.perfsonar.net.