| TeraPaths: Flow-Based End-to-End QoS Paths through Modern Hybrid WANs |
| Presented by | |
| Dimitrios Katramatos, BNL |
| Outline |
| Background: the TeraPaths project | ||
| Objective | ||
| View of the world (network) | ||
| System architecture | ||
| Establishing flow-based end-to-end QoS paths | ||
| Domain interoperation | ||
| Experience and encountered issues | ||
| Project status and future work | ||
| Objective |
| Provide QoS guarantees at the individual data flow level, all the way to the end hosts, transparently | |||
| Data flows have varying priority/importance | |||
| Video streams | |||
| Critical data | |||
| Long duration transfers | |||
| Default Òbest effortÓ network behavior treats all data flows as equal | |||
| Capacity is not unlimited | |||
| Congestion causes bandwidth and latency variations | |||
| Performance and service disruption problems, unpredictability | |||
| Dynamic flow-based SLAs = schedule network utilization | |||
| Regulate and classify (prioritize) traffic | |||
| Select routing (if possible) | |||
| View of the Network |
| TeraPaths Web Services Architecture |
| Establishing E2E QoS Paths |
| Multiple administrative domains | |||
| Cooperation, trust, but each maintains full control | |||
| Heterogeneous environment | |||
| Domain controller coordination through web services | |||
| Coordination models | |||
| Star | |||
| Requires extensive information for all domains | |||
| Daisy chain | |||
| Requires common flexible protocol across all domains | |||
| Hybrid (end-sites first) | |||
| Independent protocols | |||
| Direct end site negotiation | |||
| Path Setup |
| Path Setup (ii) |
| End site subnets are configured by TeraPaths software instances (TeraPaths Domain Controllers or TDCs) | ||
| TDCs configure end site LANs to prioritize and regulate authorized flows via the DiffServ framework at the network device level | ||
| Source site polices/marks authorized flow packets | ||
| Destination site admits/re-polices/re-marks packets | ||
| End site LANs tx/rx marked packets to/from the WAN | ||
| WAN provides MPLS tunnels or dynamic circuits | ||
| Initiating TDC requests MPLS tunnel or dynamic circuit with matching bandwidth and lifetime, orÉ | ||
| TDC groups flows with common src/dst into MPLS tunnel or dynamic circuit with aggregate bandwidth and lifetime | ||
| WAN preserves packet markings | ||
| Path Setup (iii) |
| WAN domains interoperate | |||
| Each end siteÕs TDC has a single point of contact for WAN services | |||
| TDCs have no knowledge of WAN internals other than what is exposed by the WAN services | |||
| End sites have no direct control over the WAN | |||
| Either tunnel or circuit through WAN | |||
| Cannot mix and match | |||
| Interoperating with WAN Services |
| TeraPaths ÒproxyÓ servers | |||
| Implement interface required by TeraPaths core | |||
| Hide WAN service differences | |||
| Clients to WAN web services (currently OSCARS / DRAGON) | |||
| Close cooperation with ESnet and I2 development teams | |||
| Submit reservations for MPLS tunnels or dynamic circuits | |||
| Handle security requirements | |||
| Handle errors | |||
| MPLS tunnels vs. dynamic circuits | |||
| Utilization requires different approach | |||
| L2 vs. L3 (i) |
| MPLS tunnel starts and ends within WAN domain | ||
| Packets are admitted into the tunnel based on flow ID information (IPsrc, portsrc, IPdst, portdst) | ||
| WAN admission performed at the first router of the tunnel (ingress) | ||
| L2 vs. L3 (ii) |
| Dynamic circuit appears as VLAN connecting end site border routers with single hop | ||
| Cannot use flow ID data directly | ||
| Flow must be directed to the proper VLAN | ||
| WAN admission performed within end site LAN | ||
| Select VLAN with Policy Based Routing (PBR) at both ends | ||
| Route can be selected on a per-flow basis | ||
| Site LAN Setup (DiffServ, PBR) |
| 3rd Party Network Segments |
| Some network segments may not be automatically configurable | |||
| Regional providers | |||
| Campus segments | |||
| Border routers | |||
| Static (once only) configuration required | |||
| Allow DSCP bits to go through | |||
| Only allow specific interfaces | |||
| ACLs and aggregate policers | |||
| Configure VLANs to be used for dynamic circuits | |||
| Trunked VLAN pass-thru | |||
| Virtual border router | |||
| Alternative Site LAN Setup
(DSCP, VLAN pass-thru) |
| VLAN Setup for L2 |
| L2-Specific Issues |
| Limitations with VLANs | ||||
| Tag range (tentatively selected 50 VLANs – 3550 to 3599) | ||||
| Each site may have its own range | ||||
| Tag conflicts | ||||
| Rely on WAN service | ||||
| Eliminate by synchronizing site databases | ||||
| VLAN renaming (if/when possible) | ||||
| Scalability issues | ||||
| Flow grouping | ||||
| Forward flows through same virtual WAN circuit | ||||
| Create circuit with new parameters / switch current flows / cancel old circuit | ||||
| Modify WAN reservations (if/when possible) | ||||
| PBR overhead | ||||
| Virtual border router | ||||
| Sensitive/3rd party network segments | ||||
| VLAN pass-thru | ||||
| Status |
| Currently: basic software ready, infrastructure tested | ||
| API and web interface, simple negotiation | ||
| Multiple service classes per site with statically allocated bandwidth | ||
| Utilization of L3 paths (MPLS tunnels) through ESnet (since 2006) | ||
| Utilization of L2 paths (dynamic circuits) through ESnet and Internet2 (demonstrated at SCÕ07) | ||
| ÒCircle of trustÓ security model, X.509 certificates | ||
| Simple user AAA | ||
| BNL, UMich, BU, SLAC | ||
| Multiple successful pass-thru configurations (BNL, UMich, NoX, Merit, MiLR) | ||
| TeraPaths Testbed during SCÕ07 |
| Weather Map |
| Traffic Regulation (demo) |
| In Progress / Future |
| Testbed | ||
| Expansion to more US ATLAS Tier 2 sites and beyond | ||
| BNL testbed router upgrade to 10Gbps | ||
| Support for different hardware | ||
| Dynamic bandwidth allocation within service classes | ||
| Flow grouping through WAN circuits | ||
| CLI, extended API, configurable negotiation | ||
| Grid-style AAA (GUMS/VOMS) | ||
| Plug-ins: SRM (dCache), others | ||
| Expand collaboration/interoperation | ||
| http://www.terapaths.org | ||