| Fun with L3 VPNs |
| aka, cutting VRFs until they bleed all over each other and give me a migraine |
| MAX VRFs, a very very very brief history (because youÕve heard it all before) |
| In the beginning, MAX had no VRFs, and that was OK. | |
| Then Dan added a few, and started proselytizing. | |
| Then, we added some more. And yea, verily, there was more proselytizing. | |
| Now... we have NLR to add to the network. Guess what happens next? |
| "Internet2 routes" |
| Internet2 routes | |
| MAX R&E customers | |
| NGIX R&Es (ie NISN, DREN, NREN, ESNET, USGS) |
| Desired Goals |
| Keep inet.0 sacrosanct | |
| MAX customers need access to one another | |
| NLR gets its own VRF with NGIX R&Es | |
| new BLEND VRF of I2 + NLR with NGIX R&Es |
| Routing ÔproductsÕ enabled |
| Classic I2 R&E paths for I2-members | |
| New NLR-only option for I2-non-members | |
| Pre-mixed blend for those who primarily want R&E redundancy but donÕt care about path | |
| Roll-your-own for those who want granular control over the path on a per-destination basis |
| Crossleaking routes |
| auto export - aka magic | ||
| rib groups | ||
| Limitations |
| One import policy to handle everything | ||
| NLR should get: NGIX R&Es and MAX customers | ||
| BLEND should get: I2 routes, *plus* the above. | ||
| Tried to match and set based on routing instance with Òto instance BLAHÓ as policy, but it did not work. | ||
| By only being able to match and accept a route once into multiple places, I had to leak the same to all. | ||
| Means I2 routes end up in NLR VRF, and vice versa | ||
| Solution |
| How do we make the best of this nastiness? | ||
| Local-pref games | ||
| Community games | ||
| Basically, pref down upon crossleak, and donÕt announce the interloper to customers | ||
| Example: BLEND.inet.0 |
| Initial inet.0 localprefs: | ||
| BLEND duplicated that, except that: | ||
| customers leaked from inet.0 and NLR.inet. as 95 | ||
| I2 and NLR both leaked in as 60 | ||
| Net results / Rationalization |
| Why customers leaked as 95? | ||
| Prefer VRF-native route at 100 to leaked one at 95 | ||
| Why I2/NLR as 60? | ||
| In the other VRFs, the interloper is now less preferred | ||
| Resultant lprefs in inet.0 |
| With leaking in from NLR.inet.0, inet.0 is now: |
| Benefits |
| NLR.inet.0 looks exactly the same as inet.0, just with the players reversed | ||
| Each VRF has the same routes, just slanted differently | ||
| NLR-only routes available to those interested | ||
| I2-only routes available to existing members | ||
| Redundant and equal mix available as well | ||
| Sample leaked route |
| This is a route in inet.0 that originates as a static route in BLEND. Bits have been changed to protect the guilty: |
| An interesting position... |
| IÕve now got three complete ÒR&E routing tablesÓ, each slanted differently: | ||
| I2 primary, with NLR present but preffed down | ||
| NLR primary, with I2 present but preffed down | ||
| NLR and I2 equal, to let BGP do its thing | ||
| So, what can we see? | ||
| Interesting I2 route stats |
| inet.0 has 10222 routes preferring the I2 peer (all routes not heard from customers or NGIX R&Es) | |
| NLR.inet.0 has 2912 routes preferring the I2 peer (unique to I2 since preffed below everything else) | |
| BLEND.inet.0 has 5626 routes preferring I2 |
| Interesting NLR route stats |
| NLR.inet.0 has 7761 routes preferring the NLR peer (all routes not heard from customers or NGIX R&Es) | |
| inet.0 has 451 preferring the NLR peer (unique to NLR since preffed below everything else) | |
| BLEND.inet.0 has 5057 routes preferring NLR |
| Observations on stats |
| BLEND is pretty darn well blended at this point | |
| Initially (six months ago), much of Ôbest pathÕ selection came all the way down to Ôoldest routeÕ in BLEND, so it was slanted a lot more to I2 as compared to the Ônew kidÕ BGP session. | |
| NLR has MANY fewer unique routes, but 2/3 of their total number are preferred when evenly mixed. | |
| Are dual-connected networks doing TE to prefer NLR, or did normal route churn over the last few months even things out? |
| IPv6 and Multicast |
| IPv6 posed no problem at all. Duplicated v4 rib groups and configs for v6. Worked like a charm for unicast. | ||
| I donÕt have multicast working in the VRFs. | ||
| SAs some in and work fine. People in the same VRF can see each other fine. | ||
| Crossleaked doesnÕt. Tree doesnÕt seem to build right to cross the boundary. | ||
| Anyone have experience with L3 VPNs and multicast? | ||
| Multicast workaround |
| Since NLR routes are present in inet.0 (albeit preffed down), multicast enabled members can receive the routes and have theirs visible on NLR with the right communities applied. | |
| Nowhere near as balanced as BLEND, but gets their routes on NLR for now. | |
| Sub-optimal but functional for now. |
| Continued progress |
| Too many compromises, not as ÔcleanÕ a solution as I wanted. | |||
| inet.0 has NLR routes in it, so is not sacrosanct | |||
| too many reindeer games to get lpref working right | |||
| After this was implemented, worked with Juniper to find a better answer. Took quite a while to get anywhere but eventually we did. | |||
| Secret sauce |
| Turns out there is the potential to match on a route multiple times inside the same policy, when importing it to multiple ribs, and its bloody obvious in retrospect: | ||
| match on Òto ribÓ as part of the policy and have different actions based on destination routing table! | ||
| Tried Òto instanceÓ in the early experiments but it did not work, Òto ribÓ never registered as a possibility. | ||
| Saucy example |
| As I said, bloody flipping obvious, this works in the lab: |
| In theory... |
| Which should allow me to do something like this, but IÕve not tested it yet, caveat emptor: |
| Next steps, aka v2.0 |
| Test Òto ribÓ and be sure it does what it should, everywhere it should, giving me the granularity I initially wanted. | |
| Figure out multicast and VRFs (implementing this policy will drop the preffed-down NLR routes from inet.0, so the multicast workaround will not function once things get cleaned up. |
| questions? |