Basic circuit network peering

Peering among circuit-based networks is similar to peering in IP networks. In the IP world, a peer is any network you are connected to and exchange routes with, thus enabling traffic to flow across the connection. Peering between circuit-based networks also requires a physical link between the two networks. This link may be a direct connection between the two networks or it could be a connection across a switch in an exchange point where both networks connect. Two circuit networks that share a link and are able to create circuits across that link can be said to peer.

For example, consider a network where circuits are configured as Ethernet VLANs. In this instance, for a peering to be established on a circuit, all that needs to happen is that there be some known VLAN identifiers that are configured on the link interconnecting the two networks. If a circuit is built to each end of the VLAN and the interfaces are configured to pass traffic on that VLAN, those two networks are said to peer.

Dynamic circuit network peering

The definition above describes the minimal conditions needed for peering to take place. If the respective networks are using IDCs, it is also possible for the IDCs to peer. When they peer, IDCs exchange information about network topology and how to make connections between their networks. In dynamic circuit network peering, this information is passed to peering IDCs out of band. This is one difference between IP peering and circuit peering.

The shared topology information enables circuits to be requested and created across links between networks. In this case, the topology information is similar in its effect to routing information in IP peering.

Transit

Providing transit is an important feature of IP peer networks. Transit occurs when there are at least three networks involved in a connection from one site to another. In the case of three networks, the network in the middle of that circuit is said to provide transit between the two other networks. In effect it is simply a "pseudo wire" connecting the circuit between the other two networks. Again, dynamic circuit transit is not quite the same as IP transit, though it is similar. For transit to occur in the IP case, all that needs to happen is that the network doing the transit must announce all of its routes to both of the connected networks. In the case of circuits, the requesters of the circuit in either connected network must have the ability and authorization to create a dedicated path through the transit network.

In one important sense, both transit cases are the same: they are providing the ability to move a packet from one network through one or more other networks to a third network. The differences are in the mechanism that allows those packets to be transmitted, and in the potential consequences of allowing transit. It is a different thing to allow more best effort traffic into your network than it is to allow users from two communities you peer with but do not serve directly to reserve parts of your infrastructure and thus exclude it from use by your direct customers.

Exchange Point

Peering frequently takes place at an exchange point. An exchange point—for instance, MAN LAN—is a set of equipment operating at layer 2 where multiple networks have links. Within the facility cross connects may be established that will allow the networks to peer. For IP connections, that typically means creating a BGP session between the networks to exchange routing information and enable them to move traffic. In the case of circuits, the exchange point must provide some mechanism to allow traffic to pass from a VLAN on one connector's port to a VLAN on another connector's port. This might be accomplished through a fixed configuration, usually by request, or through an IDC that interacts with connector IDCs and understands the topology of the exchange point. In either case, the result is a peering across the fabric of the exchange point.

Peering across exchange points will be essential to all the interconnection of various circuit-based networks that operate in different administrative domains.

In the circuit case, the Global Lambda Integrated Facility (GLIF) has defined an exchange point called a GLIF Open Lightpath Exchange (GOLE). The basic idea is that any networks connecting to the exchange will be allowed to establish peerings without any GOLE-enforced policy constraints.

Internet2 ION transit between Internet2 international peers across the Internet2 ION backbone will be enabled by default. This policy will be in place in order to assist in the global adoption of dynamic circuit provisioning. Internet2 does this by assisting in the interconnection of international dynamic circuit networks where possible and where needed.

Internet2 will also actively encourage direct connectivity between international peers where the appropriate network infrastructure exists, for instance, at international peering points like MAN LAN. Internet2 ION backbone peering requests will be considered on a case-by-case basis. Each peering request will be documented by a Memorandum of Understanding (MOU) that clearly describes the allowable uses of the Internet2 ION fabric. For each non-US R&E network peer, for example, GÉANT, current Peering Agreements will be modified to include interconnection of Internet2 ION and the service offered by the peer (in addition to the IP peering typically already in place.) As Internet2 ION begins to interconnect and peer with both domestic (like ESnet) and non-US networks (like GÉANT), Internet2 will maintain a list of peers reachable via Internet2 ION. This list will be made available on the Internet2 ION website.

As with the Internet2 IP Network, Internet2 ION peer networks will be encouraged to interconnect behind international exchange points. This interconnectivity is expected to utilize Ethernet as the common bearer service. The Internet2 community and its partners (for example, ESnet) will be asked to develop expectations of such exchange points in consultation with ongoing efforts in the GLIF to define Dynamic GOLEs.

Since the service characteristics of similar circuit services are likely to be slightly different from peer network to peer network, the Internet2 ION working group will produce a list of networks reachable and details about the services Internet2 ION users can expect 'off net.'

By default, Internet2 ION connectivity between US Domestic Peer networks and International Peer Networks will not be enabled. However, where such connectivity is essential, Internet2 will entertain discussions with individual networks about permitting such connectivity. This exception will be made with community review via the Internet2 AOAC.

Consistent with the existing Internet2 IP Network policy, interconnection between Domestic Peer Networks will not be enabled via the Internet2 ION fabric. Domestic circuit networks are encouraged to utilize their own circuit- based fabric to provide interconnectivity with other domestic circuit networks via layer2 exchange points. As with the International-Domestic restriction, exceptions can be made after undergoing community review and approval via the Internet2 Architecture and Operations Advisory Council (AOAC).

Usage

The initial usage policy for the Internet2 ION backbone falls under the existing Internet2 acceptable use policy:

The existing AUP language, extended to the Internet2 ION service prohibits illegal activity, but is largely open in other regards. Unlike the Internet2 IP Network, the Internet2 ION service is available to any network or institution that falls behind an Internet2 Network Connector, regardless of the Internet2 Membership status of that network or site.

Usage Constraints

The Internet2 ION service will only be available across the subset of the community that enables this service within their infrastructures. By default, Internet2 ION is enabled on all connector ports within Internet2. That does not provide any level of user connectivity, for that to happen regional and campus networks must chose to participate. Ideally the service would be enabled at all the connector locations. The aim of the service is to encourage circuit use where there is a requirement for expedited traffic or there is some need to separate flows from the best effort traffic.

The Internet2 ION service initially contained a set of lightweight rules aimed at ensuring the service was not overburdened while the community learned how to use it. Those rules have been removed. Connectors/users may now configure as many short-term or persistent circuits as they choose, the only limitation being the bandwidth available between the endpoints chosen.

• Each Connector may provision circuits on a permanent or persistent basis.
• For either short-term or persistent circuits, the only restriction on the size of the circuit is the available bandwidth at the Connector interface and on the backbone.
• Any circuit may be scheduled for repeated use.
• Persistent circuits are initially configured for 2 years; they may be repeated as well. As the end of the reservation nears, notifications will be sent at intervals of 3 months, 1 week, 1 day and 1 hour before the reservation expires.

Any member of the community or Internet2 staff who believes a pattern of unfair usage is beginning to emerge is expected to bring such observations to the Internet2 ION WG. The Internet2 ION WG will then make a recommendation, which might range from a request to change behavior, to a new set of constraints.

The backbone bandwidth available to the Internet2 ION service will be regularly measured and evaluated for capacity upgrades. Generally speaking the same policies that govern backbone augments on the IP network will apply on the Internet2 ION network.

Accounting

Internet2 will track usage of the Internet2 ION service. The aim will be to be able to provide usage characterization of the Internet2 ION service. . The intent will be to track at least the numbers of virtual circuits that are set up, the duration and characteristics of those circuits as well as overall bandwidth reserved and utilized in aggregate for the service. Usage data will be tracked based on the access control system in units of megabit-hours. Accounting data will be made available to the community, without restriction though rendered anonymous, for all reserved, pending, completed-successfully, and completed-failed circuit requests. If there is a reason this data should not be made publicly available it is up to the user to communicate that to the NOC and insure privacy where it is important.

In addition, information will be available on circuits in use and reserved. The main goal of this is to allow potential users to determine if adequate resources exist to support their requests.

Internet 2 ION Service Fees

Internet2 ION service will be bundled at no cost to Internet2 Connectors allowing them to use up to the limit of their contracted bandwidth for Internet2 ION reservations. Connectors to Internet2 may have one or two connections to the Internet2 backbone. The costs associated with these ports includes access to the Internet2 ION service on those ports without there being any additional fees charged by Internet2.

Other Potential Costs

There may well be other costs that a connector will incur to provide the Internet2 ION service to their members. This will be determined by the technique used to make the connection from the users to the Internet2 ION service. As you will see later there are a variety of means by which this connectivity can be achieved. Some of these will use existing equipment and thus incur no costs, others might involve additional hardware at the regional network and also at the campus. Care should be taken to understand how this connectivity will work in order to fully assess if there are any costs associated with Internet2 ION connectivity.

Internet2 ION API Support

The Internet2 ION Service web tool has been built on top of an interoperable dynamic circuit software suite, hereafter referred to as the Internet2 ION API. Direct calls to the Internet2 ION API will be supported by the Internet2 IDC. Support for tools built by the community to use the API will be provided on an ad-hoc basis via the dynamic circuits working group. The Internet2 ION API service is provided as-is with no further feature development or enhancements beyond critical bug fixes without community discussion.

Internet2 ION Service Management

The Internet2 ION Service will be managed alongside the rest of the Internet2 Network at the Internet2 Network Operations Center (NOC) housed at Indiana University. This will leverage the benefits of the existing Internet2 support infrastructure as well as the necessary network monitoring support to react to Internet2 ION events.

The Internet2 NOC will rely on the consortium of IDC (Inter-domain Controller) developers and perfSONAR-ps developers in the same fashion that they currently rely on network vendors for ongoing support.

The Internet2 NOC will react to requests from Internet2 ION Connectors. It is highly encouraged that Internet2 ION Participants that fall behind an Internet2 ION Connector channel their communications with the Internet2 NOC through their Connector. This ensures that the appropriate entities will be involved in addressing an issue. Internet2 will send outage notifications to both the owner of the account that initiates an Internet2 ION circuit request and the designated operational contact for the directly connected Internet2 ION Connector.

Internet2 expects that all directly connected Internet2 ION Networks will have a 24x7x365 Network Operations center that can respond appropriately to outages and inquiries. The Internet2 NOC will also encourage Internet2 ION Connectors and Peers to share planned outage notifications with as much advance notice as possible. Since Internet2 ION connections can be made between multiple Internet2 ION Participants and Peers, Internet2 will broadcast the outage notification to the NOC email contacts of all Internet2 ION Connectors as well as maintain web-based resources to query on an ad-hoc basis.

Given that many Internet2 ION circuits are expected to exist across infrastructure provided by multiple administrative domains, significant attention needs to be paid to monitoring the various portions of an Internet2 ION circuit. Internet2 will utilize the perfSONAR-ps framework to provide circuit state information to its NOC as well as other NOCs that may be monitoring a circuit on their infrastructure. The Internet2 NOC will monitor Internet2 ION-related events via an appropriate end analysis system (e.g. Cricket or Nagios) built atop the perfSONAR-ps middleware. Internet2 ION participants will be encouraged to similarly monitor their infrastructure so more rapid fault detection and isolation will be possible. A pervasive implementation of perfSONAR monitoring services will create a global view that will give every network operator in the path a view of the end-to-end circuit.

The Internet2 NOC will publish statistics regarding the reliability of each Internet2 ION-dedicated backbone circuit. In addition, it will regularly publish statistics regarding:

• Number of Internet2 ION circuits created on a per-backbone circuit basis and their status (e.g. reserved, in progress, terminated, completed)
• Average circuit creation size on a per-backbone circuit basis
• Average duration of circuit on a per-backbone circuit basis
• Above statistics on a per certificate basis

In the event of an unexpected outage, Internet2 will send targeted outage notifications to the contacts listed for each circuit that was active during the outage, as well as any circuits that are scheduled during the outage.

Fig 1: Internet2 ION static connection

Fig 2: Internet2 ION single campus link

Fig 3: Internet2 ION dual connection

Fig 4: Internet2 ION full dynamic control connection

Fig 5: Internet2 ION Q-in-Q connection

The Internet2 ION service will follow the national-regional-campus model that exists on the Internet2 IP Network today. That is, campus connectivity to the Internet2 ION backbone will be provided via an Internet2 Connector or Regional Optical Network as the access mechanism. It is understood that the Internet2 ION service may not initially be enabled at all Connectors. See Connector Internet2 ION-Enabled Status to check whether your Connector is Internet2 ION-enabled.

Both regional and campus participants need to understand how to connect to the Internet2 ION service. There are many ways this can be accomplished. Included are a series of diagrams that illustrate some of these methods. They are not intended to be proscriptive, but to help explain how Internet2 ION connectivity may be achieved, and to provide some indication of the environment organizations must put in place in order to make this service available.