Bandwidth Test Controller (BWCTL)

About BWCTL | Downloads | Manual Pages | Cookbook

Mailing List:
Two mailing lists (bwctl-announce and bwctl-users) are hosted at the domaininternet2.edu; both are publicly archived and open to subscriptions.

What's New:

Meetings/Presentations:

Case Studies:

Working Groups:

More Information on Related Projects:

BWCTL

BWCTL Version 1.3

(Bandwidth Control)
"http://www.internet2.edu/performance/bwctl

$Date: 2009-04-21 10:29:40 -0500 (Tues, 21 April 2009) $

BWCTL is a command line client application and a scheduling and policy daemon that wraps the throughput testing tools Iperf, Thrulay, and Nuttcp. These tests can measure maximum TCP bandwidth, with various tuning options available, or, by doing a UDP test, the delay, jitter, and datagram loss of a network.

The bwctl client application works by contacting a bwctld process on the two test endpoint systems. BWCTL will work as a 3-party application. The client can arrange a test between two servers on two different systems. If the local system is intended to be one of the endpoints of the test, bwctl will detect whether a local bwctld is running and will handle the required server functionality if needed. bwctld manages and schedules the resources of the host on which it runs. A more detailed description of the architecture is available in the architecture documentation.

The bwctl client is used to request the type of throughput test wanted. Furthermore, it requests when the test should be executed. bwctld either responds with a tentative reservation or a test denied message. Once bwctl is able to get a matching reservation from both bwctld processes (one for each host involved in the test), it confirms the reservation. Then, the bwctld processes run the test and return the results. The results are returned to the client from both sides of the test. Additionally, the bwctld processes share the results from their respective sides of the test with each other.

BWCTL is used to enable non-specific throughput tests to a host without having to give full user accounts on the given system. Users want the ability to run throughput tests to determine the achievable or available bandwidth between a pair of hosts. It is often useful to test to multiple points along a network path to determine the network characteristics along that path. Typically, users who want to do this path decomposition have to contact the network/system administrators that control the hosts along the path directly. The administrator needs to either run half of the test for the user or provide a user account on the host. Also, network paths of interest usually are controlled by multiple administrators. These hurdles have made this kind of testing difficult in practice.

BWCTL was designed to help with this problem. It allows an administrator to configure a given host as an Iperf, Thrulay or Nuttcp endpoint that can be shared by multiple users without concern that those users will interfere with each other. Specific policy limits can be applied to specific users, and individual tests are scheduled so they will not interfere with each other. BWCTL allows the administrator to classify incoming connections based upon a user name and AES key (generated by a pass phrase) combination or, alternatively, based upon an IP/netmask. Once the connection is classified, the bwctld can determine the exact type and intensities of throughput tests that will be allowed. (More details on the policy controls can be found in the bwctld.limits(8) manual page.)

What's Changed Since Version 1.2a

Allows for different throughput testing tools. Currently supported are Iperf, Thrulay and Nuttcp. Use the -T option on bwctl to specify which throughput testing tool to use. Use bwctl -h to see which are supported.

NOTE: Some of the throughput test options are not available for some of the testers. BWCTL does not support UDP tests, changing the output format or changing the output units with either Nuttcp or Thrulay.

Allows throughput tests to use multiple, parallel streams. This can be specified with the -P option in the bwctl client.
Allows for UDP tests with bandwidth limits of higher than 4.3Gbps. This only works with 1.3 clients and servers.
Allows for comma-separated value output for Iperf. Add "-y c" to the bwctl command line client to use it.
Allows administrators to define scripts to run at the completion of each test, allowing for results to be stored in a database, accounting tracking or any number of other tasks.
Improves robustness of time/scheduling.
Improves error handling of Iperf children processes.
Improves error messages for failure conditions.

What's Changed Since Version 1.1b

Added -f option to bwctld to allow it to run with root permissions.
Added better usage() messages and more information to several user prompts.
Fixed bug that caused all server side errors to be reported incorrectly be the client.
Fixed bwctld bug that disabled srcnode option.
Ported to Solaris.
Ported to OS X.
Removed NTP system call dependency. Added allow_unsync options.
Increased default -L value for bwctl.
Added messages to data output to indicate if Iperf was killed.
Modified all config options to have '_' between words.

Features

Supports version 2.0.x of Iperf.
Supports version 5.3.1 of Nuttcp.

NOTE: BWCTL does not support UDP tests, changing the output format or changing the output units with Nuttcp.

Includes built-in support for Thrulay.

NOTE: BWCTL does not support UDP tests, changing the output format or changing the output units with Thrulay.

Full IPv6 support. No options needed. If the target of a test is specified by a DNS hostname, and that name has both an IPv4 and an IPv6 address defined, the bwctl command line application prefers the IPv6 address.
Data from both sides of the test is returned so that sending rate can be compared with receiving rate.
3-Party communication is supported. The client does not have to be on one of the test endpoint hosts.
A local bwctld is not required. bwctl will detect if there is a local bwctld and use it if it is there; bwctl will mimic the missing functionality if needed.
Port ranges for the bwctld peer connections can be specified to allow bwctld to work in firewall environments.
Port ranges for the throughput tests can be specified to allow passive monitoring to characterize the test streams more easily.
Limits the types of throughput tests that can be run. Sending a set amount of data is not supported because BWCTL needs to know how long the test will take before the test is started.
Supports dynamic TCP window size determination. (Currently, this is limited because the bottleneck capacity is specified as a constant in the config file. In the future, bottleneck capacity could potentially be dynamically determined using techniques such as packet dispersion.)

Requirements

BWCTL includes the Thrulay tester, but it would be best if Iperf 2.0.x is installed on the system.
The bwctld daemon prefers an NTP synchronized system clock to ensure the two endpoints of the test are actually agreeing to the same scheduled time window for test execution. Therefore, BWCTL requires that NTP be running to synchronize the system clock. (See the allow_unsync configuration option described in the bwctld.conf(8) manual page to see how to by-pass this requirement.) NTP should be setup correctly on the system so that bwctld can actually fetch reasonable estimates of the time error from NTP. For the NTP algorithms to work correctly, NTP MUST be configured with no fewer than 4 clocks. (See http://twiki.ntp.org/bin/view/Support/SelectingOffsiteNTPServers for more details.)
gnumake may be required for the build process (see Building the Application).
To get good results, the endpoints will need to be well tuned. See http://www.psc.edu/networking/projects/tcptune/ and http://www-didc.lbl.gov/TCP-tuning/buffers.html for some excellent tips in that regard.

Supported Systems

BWCTL has been tested most extensively on Red Hat Linux 4 and 5 systems, but most recent versions of UNIX should work. The code cleanly compiles on FreeBSD, Linux, and OS X.

BWCTL compiles on Solaris. However, Thrulay does not work on that platform so is not compiled in. If anyone has any interest in updating thrulay to work on Solaris, let us know.

Recommended Hardware

BWCTL does not have any strict hardware requirements. The limits imposed by the specific throughput tests are more taxing to hardware than the scheduling and policy functions added by BWCTL. The hardware requirements are therefore a function of the specific throughput tests that are desired. Internet2 had good luck using the following hardware to perform 1Gbps tests on the Abilene network:

Intel SCB2 motherboard

2 x 1.266 GHz PIII, 512 KB L2 cache, 133 MHz FSB

2 x 512 MB ECC registered RAM (one/slot to enable interleaving)

2 x Seagate 18 GB SCSI (ST318406LC)

SysConnect Gigabit Ethernet SK-9843 SX

Building the Application

The BWCTL software uses the gnu autoconf tools to configure the build process. BWCTL is distributed with a pre-built configure script so the actual autoconf tools should not be needed on the target system. (Although, gnumake may be required.) The configure script can be run with the --help option to determine the full set of configurable options.

A basic build procedure would be:

% gzip -cd bwctl-$VERS.tar.gz | tar xf -
% cd bwctl-$VERS
  # --prefix is only needed if you don't like the default
  #   (/usr/local on most systems)
% ./configure --prefix=/install/root
% make
% make install

Please report any build problems to bwctl-users@internet2.edu.

Configuring bwctld

The basic procedure to configure bwctld is to create a bwctld.conf and, optionally, a bwctld.limits file and a bwctld.keys file. These files need to be installed in the same directory that is specified with the -c option to bwctld. The recommended directory is /install/root/etc. (The etc directory below your install root.) There are examples of these files in the bwctl-$VERS/conf sub directory of the distribution.

bwctld.conf:
Used to configure basic operation of the daemon, such as server listening port, the path for Iperf, and error logging. For a detailed description of the options available, see the bwctld.conf(8) manual page.

bwctld.limits:
Used to configure the policy limits for the daemon. There are two parts to this policy: 1) authentication and 2) authorization. The authentication is done by assigning a class to each new connection. Each class has a set of limits associated with it. The classes are hierarchical, so a connection must pass the limit restrictions of the assigned class as well as all parent classes. For a detailed description of the options available, see the bwctld.limits(8) manual page.

bwctl.keys:
Used to associate identities with AES keys. These identities are then mapped to a class by the bwctld.limits file. For a more detailed description, see the bwctld.keys(8) manual page.

Configuring firewalls

If a firewall is enabled on the machine running bwctld, it will need to be configured to allow incoming clients and tests. Use this set of directions to configure the firewall.

Configuring NTP

To ensure that tests can be run, the machine running bwctld should have its clock synchronized using NTP. Use this set of directions to configure NTP.

Running bwctld

The normal command line to start the daemon is:

 % bwctld -c /install/root/etc

It is possible to run the daemon without a config file directory if enough command line options are specified, but it is easier to use a config file.

To see all the available options:

 % bwctld -h

More details on running the daemon, as well as a complete description of the command line options, is available from the bwctld manual page.

Running bwctl

The basic command line for the client is:

% bwctl [options] [-c catchhost] [-s sendhost]

The -c option indicates that the catchhost should be an Iperf server (think packet catcher). The -s option indicates the sendhost should be an Iperf client (think packet sender). At least one of the -c/-s options must be specified. If only one of these options is specified, the local host is assumed to be the other test endpoint.

To see a list of available options:

% bwctl -h

More details on running the client application, with a complete description of all command line options, is available from the bwctl manual page.

Caveats

Thrulay support is deprecated. If there is enough desire, or if someone is willing to maintain thrulay, the functionality could be undeprecated.

Mailing Lists There are two email lists to support this software:

bwctl-users: This is a general discussion list for users to discuss problems. This list is monitored as bug reports should be sent here.
bwctl-announce: This list is used to announce new versions or significant developments with regard to the software.

Information about these lists, including links to subscribe, is at https://mail.internet2.edu/wws/lists/engineering.

Author

Jeff Boote
boote@internet2.edu
Internet2

Aaron Brown
aaron@internet2.edu
Internet2

Acknowledgments

The following individuals have greatly aided in the development, testing, or debugging of BWCTL:

Shumon Huque (UPenn)
Paul Hyder (NOAA)
Warren Matthews (GaTech)
Sean Peisert (SDSC)
Chris Tracy (MAX)


$Id: index.html 529 2009-01-26 15:29:40Z aaron $