Jumbo Frames on Abilene

The following is an edited version of a personal email note I sent to Andre Broido on 2003-11-05 in response to his questions about jumbo frames on Abilene. (The questions were: How much of Abilene is now transparent for the 9000 byte frames? Which other networks have jumbo frame size enabled, or have plans to enable them? What are the advantages and problems related to increased frame size?) I was encouraged by k claffy to put it up on a web page, so here it goes.

Q: How much of Abilene is now transparent for the 9000 byte frames?

All of it. Last week, about 0.46% of packets on Abilene were larger than 1500B, but last week was unusual: the more usual background percentage of jumbo packets on Abilene is 0.00% (order of magnitude of a million packets per day). The recent spike is undoubtedly related to SC2003 preparation; in general, jumbo frame usage is real-life-event-driven.

Here's a time series plot that displays the approximate percentage of jumbo frames on Abilene as a function of time (each dot represents one week of data from all of Abilene):

(Data from the Internet2 NetFlow Weekly Reports.)

Q: Which other networks have jumbo frame size enabled, or have plans to enable them?

You may wish to see the ``Abilene -- Connector Technology Support'' page maintained by the Abilene NOC.

Roughly half of the connectors are at 4470B MTU. Those whose interfaces don't support it (about half a dozen cases -- mostly the Cisco Trident card), are at less than that. Roughly half are at 9000 or more.

The recommended MTU number is 9000B (to cut off the fractal fuzzy part of 9000+epsilon that gives little or no increase in performance and creates an enormous potential for MTU mismatches). I've been a proponent of this number and might be responsible for it.

The story inside the campus is different, of course. There are a few oasises with jumbo frames, mostly at advanced places (like the supercomputing centers and a few others), but the vast majority of end hosts are, of course, on 1500B.

Q: What are the advantages and problems related to increased frame size?

Advantages:

• Reno TCP congestion control algorithm would work optimally if MTUs were specified in units of seconds, not bytes. A fraction of a millisecond -- say, 500us -- would be reasonable. So, roughly 9kB (or 4470B) for 100Mb/s, 64kB for GigE, and IPv6 with half a megabyte of MTU size for 10GigE. That's not yet happening (the champion of this is Matt Mathis), but any increase pushes us in the right direction.
• Larger MTU means less interrupts. The ATM cell size was the laughing stock when it was designed. In terms of serialization delay, the ATM cell is larger at OC-3 speed than a 1500-byte packet at 10Gb/s speed.

Problems:

• Broken pMTUd (because of firewalls that block frag needed messages). That's being worked on in the pmtud WG in the IETF.
• MTU mismatches. Not enough attention is being devoted to MTU mismatches by the proponents of larger MTUs. If there's a compelling argument against non-1500 Ethernet MTUs, it's this.

Non-problems (mythical):

• Ethernet limitations of MTU size based on CRC
• Ethernet limitations of MTU size based serialization delay
• IP limitations on packet size based on length of length field (16 bits)

For your convenience and so that you get the complete picture, I'll provide rebuttals to these points right away:

• In re Reno TCP: Reno TCP congestion control is broken and needs to be replaced. Between something along the lines of HSTCP and something along the lines of XCP, there must be a solution. (Incidentally, FAST TCP is about halfway between these two points.)
• In re less interrupts: general-purpose computers can still handle packetization overhead at 1Gb/s and by the time we get a >10Gb/s bus for 10GE NICs, will be able to handle packetization at 10Gb/s. Not clear what happens next and which exponent (CPU speed or Ethernet speed) wins, but there's always more and more offloading of functions to NICs.
• In re pMTUd: the new algorithm -- already implemented -- solves the problem. Those who get bitten will be prodded to upgrade.
• In re MTU mismatches: get rid of that switch concept and make everything a router. Modern switches have enough smarts to act like routers anyway. In addition, 10GE has no concept of shared medium and everything is point-to-point. Solves more than MTU mismatch problem, too. Alternatively: add MTU option to DHCP and IPv6 autoconfiguration.

References

• Internet2 NetFlow Weekly Reports
• Abilene -- Connector Technology Support
• Matt Mathis, ``Raising the Internet MTU'' [including pointers to implementations]
• IETF Path MTU Discovery (pmtud) WG charter