Update on ExpressRoute at Georgia Tech Implemented in Partnership with Internet2
By Adam Smith, IT Service Delivery Lead, Georgia Tech
The implementation of a Microsoft Azure ExpressRoute circuit at Georgia Tech has been an insightful endeavor. Initial setup was completed without much difficulty and the ExpressRoute has been functional in the basic sense, however leveraging the ExpressRoute circuit has posed some other challenges, some of which were expected while others not. These challenges are incorporating external private address space, routing issues, and cloud network architecture.
After peering was setup we were quickly able to connect the Express Route following Microsoft documentation. However, we were unable to connect to private resources despite proper setup. After engaging Microsoft Premier support and Internet2 to validate our routing configuration we dove into more detail and found that return traffic from our private address space in Azure was being stripped out by our firewall’s bogon filter rules. This being a relatively simple fix, we were able to proceed forward. However, the task of enabling an external, private address space being relatively uncommon for us it caught us somewhat by surprise.
Having successfully connected to resources in our Azure ExpressRoute on our private address space, we ran into some asymmetric routing issues that we are currently still working to resolve. One of the things that is unique about large research universities such as Georgia Tech is that we have a lot of public IP address space, and our public address space is assigned out to not just servers and services but to client machines and devices as well. In working with various vendors over the years this is a typical challenge as our network does not reflect that of the typical corporation which leverages significant private address space and very little public address space – using public IPs for only services that truly need it. This has proven to be the case as well when we have been working with our Microsoft support team to address the asymmetric routing issues over our ExpressRoute.
The asymmetric routing issues come down to the fact that we are using private IP address space in the cloud, and connecting to our public IPs on premises through our ExpressRoute. This works seamlessly when using only private IP address space in the Azure cloud. However, if a public IP is assigned to a device or service that is also connected to our ExpressRoute enabled VNet in Azure, and a connection is attempted on the public IP, then the traffic will be routed out through the public internet then back through the ExpressRoute resulting in Asymmetric routing and causing the connection to fail. From what we understand this doesn't usually happen in scenarios where there is private address space on premises and private in the cloud as well. We are currently working with our Microsoft Premier team as well as Azure solutions architects to find the right solution. One may be to route all traffic in our Azure Subscriptions through the ExpressRoute at the risk of adding more traffic and saturating the connection while others are implementing more fine-grained routing policies.
Another challenge that we are working to resolve is network architecture within the cloud itself. Now that we have an ExpressRoute setup we are working to determine the best architecture to use the ExpressRoute effectively. We have been working with Microsoft Solutions Architect to design a model that will allow for granular control between VNets when needed (or not) while leveraging network efficiencies. Access to the networks, management, and delegation thereof is certainly a factor. Once we work out the asymmetric routing issues we may find more issues here.
Finally, with networks setup and routing properly (not necessarily in this order), we will also need to address network security. Should we leverage the built-in Network Security Groups or place a more fully featured Network Virtual Appliance in our Azure Network? At which point or points do we put this and how will it, or the NSGs, tie into our existing firewall management toolset? These are all issues that need to be addressed and ironed out as we work towards extending our network into Azure.