Internet2 Network of the Future

Steve Corbató

Director, Backbone Network Infrastructure

REUNA

Universidad Austral de Chile

Valdivia

10 de abril 2002

Why Internet2?

The U.S. R&E network – NSFNet – was decommissioned by 1995

Commercial focus shifted to scale the Internet to the general populace

Advanced requirements of U.S. higher education – research, education, and medicine – were not being being prioritized

The U.S. research universities (35 at the start) created Internet2 to insure a collective effort to maintain and to develop advanced Internet capabilities

This presentation

Abilene Network today

Emergence and evolution of optical networking

Next phase of Abilene

Networking hierarchy

Internet2 networking is a fundamentally hierarchical and collaborative activity

International networking

Ad hoc ® Global Terabit Research Network (GTRN)

National backbones

Regional networks

GigaPoPs ® advanced regional networks

Campus networks

Much activity now at the metropolitan and regional scales

Abilene focus

Goals

Enabling innovative applications and advanced services not possible over the commercial Internet

Backbone & regional infrastructure provides a vital substrate for the continuing culture of Internet advancement in the university/corporate research sector

Advanced service efforts

Multicast

IPv6

QoS

Measurement

an open, collaborative approach

Security

Abilene background & milestones

Abilene is a UCAID project in partnership with

Qwest Communications (SONET & DWDM service)

Nortel Networks (SONET kit)

Cisco Systems (routers)

Indiana University (network operations)

ITECs in North Carolina and Ohio (test and evaluation)

Timeline

Apr 1998: Project announced at White House

Jan 1999: Production status for network

Oct 1999: IP version of HDTV (215 Mbps) over Abilene

Apr 2001: First state education network added

Jun 2001: Participation reaches all 50 states & D.C.

Nov 2001: Raw HDTV/IP (1.5 Gbps) over Abilene

Abilene – April, 2002

IP-over-SONET backbone (OC-48c, 2.5 Gbps) 53 direct connections (MREN, NCSA in IL)

4 OC-48c connections

1 Gigabit Ethernet trial

23 will connect via at least OC-12c (622 Mbps) by 1Q02

Number of ATM connections decreasing

211 participants – research universities & labs

All 50 states, District of Columbia, & Puerto Rico

15 regional GigaPoPs support ~70% of participants

Expanded access

46 sponsored participants

21 state education networks (SEGPs)

Slide 8

Slide 9

Abilene international connectivity

Transoceanic R&E bandwidths growing!

GÉANT – 5 Gbps between Europe and New York City

Key international exchange points facilitated by Internet2 membership and the U.S. scientific community

STARTAP & STAR LIGHT – Chicago (GigE)

AMPATH – Miami (OC-3c ® OC-12c)

Pacific Wave – Seattle (GigE)

MAN LAN - New York City – GigE/10GigE EP soon

CA*NET3: Seattle, Chicago, and New York

CUDI: CENIC and Univ. of Texas at El Paso

International transit service

Collaboration with CA*NET3 and STARTAP

Slide 11

Abilene cost recovery model

Raw HDTV/IP testing

Packetized raw High Definition Television (HDTV) - 1.5 Gbps

ISIe, Tektronix, & UW project/DARPA support

Connectivity and testing support

P/NW & MAX Gigapops, Abilene and DARPA Supernet, Level(3)

SC2001 public demo

November, 2001

SEA -> DEN via Level(3)

OC-48c SONET circuit

Implications for support of high performance flows over Abilene

DARPA PIs Meeting: Seattle to Washington DC 1/6/02

Abilene, P/NW & MAX GigaPoPs in Internet2 path

18 hrs of continuous, single-stream raw HD/IP

UDP jumbo frames: 4444 B packet size

Application level measurement

3 billion packets transmitted

0 packets lost, 15 resequencing episodes

e2e network performance

Loss: <8x10 ^-10(90% confidence level)

Reordering: 5x10 ^–9

Transcontinental 1-Gbps TCP (std 1.5 kB MTU) requires loss at the level of 3x10 ^–8 or lower

End-to-End Performance:
‘High bandwidth is not enough’

Bulk TCP flows (> 10 Mbytes transfer)

Current median flow rate over Abilene: 1.9 Mbps

End-to-End Performance Initiative

To enable the researchers, faculty, students and staff who use high performance networks to obtain optimal performance from the current infrastructure on a consistent basis.

True End-to-End Performance requires a system approach

Optical networking technology drivers

Aggressive period of fiber construction on the national & metro scales in U.S.

Many university campuses and regional GigaPoPs with dark fiber

Dense Wave Division Multiplexing (DWDM)

Allows the provisioning of multiple channels (l’s) over distinct wavelengths on the same fiber pair

Fiber pair can carry 160 channels (1.6 Tbps!)

Optical transport is the current focus

Optical switching is still in the realm of experimental networks, but may be nearing practical application

DWDM technology primer

DWDM fundamentally is an analog optical technology

Combines multiple channels (2-160+ in number) over the same fiber pair

Uses slightly displaced wavelengths (l’s) of light

Generally supports 2.5 or 10 Gbps channels

Physical obstacles to long-distance transmission of light

Attenuation

Solved by amplification (OO)

Wavelength dispersion

Requires periodic signal regeneration – an electronic process (OEO)

DWDM system components

Fiber pair

Multiplexing/demultiplexing terminals

OEO equipment at each end of light path

Output: SONET or Ethernet (10G/1G) framing

Amplifiers

All optical (OO)

~100 km spacing

Regeneration

Electrical (OEO) process – costly (~50% of capital)

~500 km spacing (with Long Haul - LH - DWDM)

New technologies can lengthen this distance

Remote huts, operations & maintenance

Telephony’s recent past (from an IP perspective in the U.S.)

IP Networking (and telephony)
in the not so distant future

National optical networking options

1 – Provision incremental wavelengths

Obtain 10-Gbps l’s as with SONET

Exploit smaller incremental cost of additional l’s

1^stl costs ~10x than subsequent l’s

2 – Build dim fiber facility

Partner with a facilities-based provider

Acquire 1-2 fiber pairs on a national scale

Outsource operation of inter-city transmission equipment

Needs lower-cost optical transmission equipment

The classic ‘buy vs. build’ decision in Information Technology

Future of Abilene

Original UCAID/Qwest agreement amended on October 1, 2001

Extension of for another 5 years – until October, 2006

Originally expired March, 2003

Upgrade of Abilene backbone to optical transport capability - l’s (unprotected)

x4 increase in the core backbone bandwidth

OC-48c SONET (2.5 Gbps) to 10-Gbps DWDM

Two leading national initiatives in the U.S.

Next Generation Abilene

Advanced Internet backbone

connects entire campus networks of the research universities

10 Gbps nationally

TeraGrid

Distributed computing (Grid) backplane

connects high performance computing (HPC) machine rooms

Illinois: NCSA, Argonne

California: SDSC, Caltech

4x10 Gbps: Chicago « Los Angeles

Ongoing collaboration between both projects

TeraGrid Architecture – 13.6 TF (Source: C. Catlett, ANL)

Key aspects of next generation Abilene backbone - I

Native IPv6

Motivations

Resolving IPv4 address exhaustion issues

Preservation of the original End-to-End Architecture model

p2p collaboration tools, reverse trend to CO-centrism

International collaboration

Router and host OS capabilities

Run natively - concurrent with IPv4

Replicate multicast deployment strategy

Close collaboration with Internet2 IPv6 Working Group on regional and campus v6 rollout

Addressing architecture

Key aspects of next generation Abilene backbone - II

Network resiliency

Abilene l’s will not be protected like SONET

Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms)

Options:

Currently: MPLS/TE fast reroute

IP-based IGP fast convergence (preferable)

Addition of new measurement capabilities

Enhance active probing (Surveyor)

Latency & jitter, loss, TCP throughput

Add passive measurement taps

Support for computer science research – “Abilene Observatories”

Support of Internet2 End-to-End Performance Initiative

Intermediate performance beacons

Slide 29

Regional optical fanout

Next generation architecture: Regional & state based optical networking projects are critical

Three-level hierarchy: backbone, GigaPoPs/ARNs, campuses

Leading examples

CENIC ONI (California), I-WIRE (Illinois),

SURA Crossroads (Southeastern U.S), Indiana, Ohio

Collaboration with the Quilt

Regional Optical Networking project

U.S. carrier DWDM access is now not nearly as widespread as with SONET circa 1998

30-60 cities for DWDM

~120 cities for SONET

Optical network project differentiation

Slide 32

California & Pacific Northwest
(Source: Greg Scott, CENIC/UCSC)

Conclusions

Abilene future

UCAID’s partnership with Qwest extended through 2006

Backbone to be upgraded to 10-Gbps in three phases

Native v6, enhanced measurement, and increased resiliency are new thrusts

Overall approach to the new technical design and business model is for an incremental, non-disruptive transition

Nicely positioned and collaborative with NSF’s TeraGrid distributed computational backplane effort

National Light Rail

Emerging & expanding collaboration to develop a persistent advanced optical network infrastructure capability to serve the diverse needs of the U.S. higher ed & research communities

Core partners: CENIC & P/NW, Argonne/TeraGrid, UCAID

For more information

Web: www.internet2.edu/abilene

E-mail: abilene@internet2.edu

Slide 36


	Steve Corbató
	Director, Backbone Network Infrastructure

	REUNA
	Universidad Austral de Chile
	Valdivia
	10 de abril 2002


	The U.S. R&E network – NSFNet – was decommissioned by 1995
	Commercial focus shifted to scale the Internet to the general populace
	Advanced requirements of U.S. higher education – research, education, and medicine – were not being being prioritized
	The U.S. research universities (35 at the start) created Internet2 to insure a collective effort to maintain and to develop advanced Internet capabilities


	Abilene Network today
	Emergence and evolution of optical networking
	Next phase of Abilene


Internet2 networking is a fundamentally hierarchical and collaborative activity
	International networking
		Ad hoc ® Global Terabit Research Network (GTRN)
	National backbones
	Regional networks
		GigaPoPs ® advanced regional networks
	Campus networks
Much activity now at the metropolitan and regional scales


Goals
	Enabling innovative applications and advanced services not possible over the commercial Internet
	Backbone & regional infrastructure provides a vital substrate for the continuing culture of Internet advancement in the university/corporate research sector
Advanced service efforts
	Multicast
	IPv6
	QoS
	Measurement
		an open, collaborative approach
	Security


	Abilene is a UCAID project in partnership with
		Qwest Communications (SONET & DWDM service)
		Nortel Networks (SONET kit)
		Cisco Systems (routers)
		Indiana University (network operations)
		ITECs in North Carolina and Ohio (test and evaluation)
	Timeline
		Apr 1998: Project announced at White House
		Jan 1999: Production status for network
		Oct 1999: IP version of HDTV (215 Mbps) over Abilene
		Apr 2001: First state education network added
		Jun 2001: Participation reaches all 50 states & D.C.
		Nov 2001: Raw HDTV/IP (1.5 Gbps) over Abilene


	IP-over-SONET backbone (OC-48c, 2.5 Gbps) 53 direct connections (MREN, NCSA in IL)
		4 OC-48c connections
		1 Gigabit Ethernet trial
		23 will connect via at least OC-12c (622 Mbps) by 1Q02
		Number of ATM connections decreasing
	211 participants – research universities & labs
		All 50 states, District of Columbia, & Puerto Rico
		15 regional GigaPoPs support ~70% of participants
	Expanded access
		46 sponsored participants
		21 state education networks (SEGPs)


	Transoceanic R&E bandwidths growing!
		GÉANT – 5 Gbps between Europe and New York City
	Key international exchange points facilitated by Internet2 membership and the U.S. scientific community
		STARTAP & STAR LIGHT – Chicago (GigE)
		AMPATH – Miami (OC-3c ® OC-12c)
		Pacific Wave – Seattle (GigE)
		MAN LAN - New York City – GigE/10GigE EP soon
		CA*NET3: Seattle, Chicago, and New York
		CUDI: CENIC and Univ. of Texas at El Paso
	International transit service
		Collaboration with CA*NET3 and STARTAP


	Packetized raw High Definition Television (HDTV) - 1.5 Gbps
		ISIe, Tektronix, & UW project/DARPA support
	Connectivity and testing support
		P/NW & MAX Gigapops, Abilene and DARPA Supernet, Level(3)
	SC2001 public demo
		November, 2001
		SEA -> DEN via Level(3)
		OC-48c SONET circuit


DARPA PIs Meeting: Seattle to Washington DC 1/6/02
	Abilene, P/NW & MAX GigaPoPs in Internet2 path
	18 hrs of continuous, single-stream raw HD/IP
	UDP jumbo frames: 4444 B packet size
	Application level measurement
		3 billion packets transmitted
		0 packets lost, 15 resequencing episodes
	e2e network performance
		Loss: <8x10 ^-10(90% confidence level)
		Reordering: 5x10 ^–9
	Transcontinental 1-Gbps TCP (std 1.5 kB MTU) requires loss at the level of 3x10 ^–8 or lower


	Bulk TCP flows (> 10 Mbytes transfer)
		Current median flow rate over Abilene: 1.9 Mbps


	To enable the researchers, faculty, students and staff who use high performance networks to obtain optimal performance from the current infrastructure on a consistent basis.


	Aggressive period of fiber construction on the national & metro scales in U.S.
	Many university campuses and regional GigaPoPs with dark fiber
	Dense Wave Division Multiplexing (DWDM)
		Allows the provisioning of multiple channels (l’s) over distinct wavelengths on the same fiber pair
		Fiber pair can carry 160 channels (1.6 Tbps!)
	Optical transport is the current focus
		Optical switching is still in the realm of experimental networks, but may be nearing practical application


DWDM fundamentally is an analog optical technology
	Combines multiple channels (2-160+ in number) over the same fiber pair
	Uses slightly displaced wavelengths (l’s) of light
	Generally supports 2.5 or 10 Gbps channels
Physical obstacles to long-distance transmission of light
	Attenuation
		Solved by amplification (OO)
	Wavelength dispersion
		Requires periodic signal regeneration – an electronic process (OEO)


	Fiber pair
	Multiplexing/demultiplexing terminals
		OEO equipment at each end of light path
		Output: SONET or Ethernet (10G/1G) framing
	Amplifiers
		All optical (OO)
		~100 km spacing
	Regeneration
		Electrical (OEO) process – costly (~50% of capital)
		~500 km spacing (with Long Haul - LH - DWDM)
		New technologies can lengthen this distance
	Remote huts, operations & maintenance


1 – Provision incremental wavelengths
	Obtain 10-Gbps l’s as with SONET
	Exploit smaller incremental cost of additional l’s
		1^stl costs ~10x than subsequent l’s
2 – Build dim fiber facility
	Partner with a facilities-based provider
	Acquire 1-2 fiber pairs on a national scale
	Outsource operation of inter-city transmission equipment
	Needs lower-cost optical transmission equipment
The classic ‘buy vs. build’ decision in Information Technology


Original UCAID/Qwest agreement amended on October 1, 2001
Extension of for another 5 years – until October, 2006
	Originally expired March, 2003
Upgrade of Abilene backbone to optical transport capability - l’s (unprotected)
	x4 increase in the core backbone bandwidth
		OC-48c SONET (2.5 Gbps) to 10-Gbps DWDM


Next Generation Abilene
	Advanced Internet backbone
		connects entire campus networks of the research universities
	10 Gbps nationally
TeraGrid
	Distributed computing (Grid) backplane
		connects high performance computing (HPC) machine rooms
	Illinois: NCSA, Argonne
	California: SDSC, Caltech
	4x10 Gbps: Chicago « Los Angeles
Ongoing collaboration between both projects


Native IPv6
	Motivations
		Resolving IPv4 address exhaustion issues
		Preservation of the original End-to-End Architecture model
			p2p collaboration tools, reverse trend to CO-centrism
		International collaboration
		Router and host OS capabilities
	Run natively - concurrent with IPv4
	Replicate multicast deployment strategy
	Close collaboration with Internet2 IPv6 Working Group on regional and campus v6 rollout
		Addressing architecture


Network resiliency
	Abilene l’s will not be protected like SONET
	Increasing use of videoconferencing/VoIP impose tighter restoration requirements (<100 ms)
	Options:
		Currently: MPLS/TE fast reroute
		IP-based IGP fast convergence (preferable)
Addition of new measurement capabilities
	Enhance active probing (Surveyor)
		Latency & jitter, loss, TCP throughput
	Add passive measurement taps
	Support for computer science research – “Abilene Observatories”
	Support of Internet2 End-to-End Performance Initiative
		Intermediate performance beacons


Next generation architecture: Regional & state based optical networking projects are critical
	Three-level hierarchy: backbone, GigaPoPs/ARNs, campuses
	Leading examples
		CENIC ONI (California), I-WIRE (Illinois),
		SURA Crossroads (Southeastern U.S), Indiana, Ohio
Collaboration with the Quilt
	Regional Optical Networking project
U.S. carrier DWDM access is now not nearly as widespread as with SONET circa 1998
	30-60 cities for DWDM
	~120 cities for SONET


	Abilene future
		UCAID’s partnership with Qwest extended through 2006
		Backbone to be upgraded to 10-Gbps in three phases
		Native v6, enhanced measurement, and increased resiliency are new thrusts
		Overall approach to the new technical design and business model is for an incremental, non-disruptive transition
		Nicely positioned and collaborative with NSF’s TeraGrid distributed computational backplane effort
	National Light Rail
		Emerging & expanding collaboration to develop a persistent advanced optical network infrastructure capability to serve the diverse needs of the U.S. higher ed & research communities
		Core partners: CENIC & P/NW, Argonne/TeraGrid, UCAID