Cisco ACI Floating L3Out

Overview of Floating L3Out

ACI uses L3Out to connect to external L3 domains via routing (dynamic routing protocol or static). There are multiple options and tools to optimize the L3Out for effective L3 communications between ACI and external network services. One of those is Floating L3Out.

In a standard / regular L3Out logical interface profile is used to logically group all logical interfaces required for the route peering / adjacency. In an environment with virtual routers/FWs/LBs, route peering with ACI requires adding all possible logical interfaces to the profile as the cluster not the host is the VM hosting boundary. This solution is not scalable and management, and troubleshooting can easily get complex as the number of hosts in the cluster grows.

Floating L3Out enables engineers to configure L3Out without specifying logical interfaces. Floating L3Out makes configuration, management, and troubleshooting easier. Only specific leaf nodes, called anchor leaf nodes establish routing adjacencies with external routers.

Anchor leaf node – is a leaf node that establishes route peering / L3 adjacencies with the external routers. As of Cisco ACI release 6.0(1), the verified scalability number of anchor leaf nodes is 6 per L3Out.

Non-anchor leaf node – is a leaf node that is on the possible path of the external routers to ACI but acts as a pass-through without establishing route peering / L3 adjacencies. Non-anchor leaf node uses a shared floating IP address (A common IP address for non-anchor leaf nodes). As of ACI release 6.0(1), the verified scalability number of non-anchor leaf nodes is 32 per L3Out.

Floating L3Out works by specifying domains. It works with both VMM (starting release 4.2(1) with VMware vSphere Distributed Switch (VDS) ) and Physical domains (starting release 5.0(1)).

Lab topology and Configuration Steps for Floating L3Out

• Floating L3Out with VMM domain relies on VMM information to track vRouter location, the floating IP address is deployed only when the virtual router is connected to the leaf node.
• Floating L3Out with physical domain, the floating IP address is deployed if the leaf port uses an AEP that has an L3Out domain associated to the floating L3Out.

Configuration

1. Create Floating L3Out

The steps necessary to create the floating L3Out is similar to any standard L3Out. The difference is on the node and interface section of the L3Out creation wizard.

Tenant -> Networking -> L3Outs -> Create L3out

1.1 Create L3Out Identity

Create L3Out will pop up a wizard window to configure the necessary L3Out configurations. On the first window (Identity window) input the mandatory fields :

• Name for the L3out
• VRF on which the L3Out and its routing protocol are deployed
• L3 domain to allow the L3Out to use a set of interfaces and VLANs

Then select the right protocol, OSPF for this lab exercise . Depending on the protocol choice different options pop-up. if static route is the requirement click next without selecting a routing protocol.

Only one routing protocol per L3Out is allowed with one exception. BGP and OSPF can be configured in the same L3Out as an exception in order to be able to use OSPF as the IGP for BGP.

1.2 Node and Interface

The main function of the L3Out Node and Interface Profiles is to specify which switch nodes should be border leaf switches (anchor nodes) and which interfaces should speak a routing protocol.

Under Interface Types

1. Select floating SVI as layer 3 interface type and ‘Physical’ for domain type (since we using physical domain with VMware infrastructure in the lab)
2. Select the physical domain name, and input floating IP address, encap VLAN and MTU

* Both the VLAN pools attached to the physical and L3 domain must include the VLAN we use as encapsulation VLAN for the floating SVI

Under Nodes

• Choose the anchor node with router ID and loopback address
• Input the primary IP address for the route adjacency

* All L3Outs running the same routing protocol in the same VRF on the same node share the same router ID

1.3 Protocols

This section of the wizard is used to define any attributes of the protocol (OSPF in this case) that may be required based on the design and requirements.

1.4 External EPG

1. Input the name and select contract
2. External EPG is required for the L3Out (control plane) to work, even if it is empty. In this step create external EPG to classify traffic coming from external router into the right ext EPG. input the subnet (0.0.0.0/0 for the lab test to catch all) with ’External Subnets for External EPG’ flag selected. ’External Subnets for External EPG’ doesn’t have any impact on routing, it’s only for classification and policy enforcement.

The classification for external EPG is based on a longest prefix match (LPM) and it’s VRF level.

Click finish and the configuration for the floating L3Out is done!

1.5 External router configuration

For this demonstration a simple OSPF configuration is applied on the external (CSR) router and loopback 0 is used to simulate a host behind the external router.

!
interface GigabitEthernet2 
  ip address 172.16.1.1 255.255.255.0
  ip ospf 1 area 0
!
interface Loopback0
  ip address 10.1.3.1 255.255.255.0
  ip ospf network point-to-point
  ip ospf 1 area 0
!
router ospf 1
  router-id 10.1.3.1
!

1.6 Floating L3Out Verification

2. Moving the external router to non-anchor leaf node connected host

When a virtual router move to host connected to a non-anchor leaf node, the non-anchor leaf node deploys L3Out BD, floating IP, ext EPG & policy, notify EP (172.16.1.1 and it’s MAC) to spines. The peering / adjacency remains the same (the dotted red line in the drawing).

2.1. The traffic flow from host connected to the external router (10.1.3.1) to red vm (172.16.10.10)

• Traffic will hit leaf 2 (non-anchor leaf node)
• Since leaf 2 has all the information (L3Out BD with floating IP, ext EPG and policy required), it will enforce policy and route traffic to leaf 4 and then red VM.
• The path is shown in the drawing below with orange lines

2.2. The traffic flow from internal, red VM (172.16.10.10) to host connected to the external router (10.1.3.1)

• Traffic will hit leaf 4 (regular leaf node) and leaf 4 routing table points the next hop as the anchor leaf node VTEP IP

Leaf104# show ip route  vrf SR-MPLS:VRF-1
IP Route Table for VRF "SR-MPLS:VRF-1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.1.1.1/32, ubest/mbest: 1/0
    *via 10.0.96.64%overlay-1, [1/0], 04:18:09, bgp-64500, internal, tag 64500
10.1.3.0/24, ubest/mbest: 1/0
    *via 10.0.96.64%overlay-1, [200/5], 02:40:08, bgp-64500, internal, tag 64500

• Route traffic to leaf 1 (anchor leaf node), if you have multiple anchor leaf nodes it uses ECMP
• Leaf 1 (anchor leaf node) will send to the next hop (172.16.1.1) which is learned from leaf 2 (non-anchor leaf node)
• Leaf 2 (non-anchor leaf node) will send it to the virtual router and then to the host
• The orange line shows the traffic flow

The traffic is bouncing back between anchor and non-anchor leaf nodes. The traffic flow is not optimized.

2.3 Optimizing the traffic flow from internal VMs / endpoints to external endpoints connected through the non-anchor leaf node

To optimize the traffic flow showed in the above drawing, we need:

• Anchor leaf nodes redistribute the external route to ACI fabric by keeping the external router as a next-hop (next-hop propagation)
• Anchor and non-anchor leaf nodes advertise the direct-attached IP to ACI fabric
• Internal leaf nodes will have routing with next-hop as the external router
• Traffic from internal endpoint to external endpoint optimization is supported after 5.0(1) eBGP and 5.2(1) OSPF

3. Configuration steps to optimize the traffic flow

Configuration –

1. update/increase the local max ECMP in BGP address family context policy ( for MP-BGP to install multiple path)
2. Configure match and set rule for next-hop propagation and apply the route map for route control to the L3Out
3. Redistribute direct-attached host route into the fabric

3.1 Local Max ECMP

This step is used for the MP-BGP process to install multiple routes into the routing tables of leaf nodes.

3.2 Match and Set rule for Next-hop Propagation

This step is used to configure the L3Out to advertise external prefixes with out changing the next hop address

Tenants -> Policies -> Protocols -> Match Rules

Add 10.1.30/24 & 172.16.1.0/24 for this lab exercise. All possible route need to be added. Aggregate can be used to simplify / minimize the number of entries.

Tenants -> Policies -> Protocols -> Set Rules

Tenants -> Policies -> Protocols -> Route Maps for Route Control

3.3 Under the L3Out apply the route map for route control and direct-attached host redistribution

Attach the route map for route control created in the above step with match and set rules to the L3Out main policy. Advertise the direct-attached host in both anchor and non-anchor nodes using the L3Out main policy page.

Tenants -> Networking -> L3Outs

Now, the optimization is configured we can go ahead and verify the path

3.4. Optimized traffic flow from internal, red VM (172.16.10.10) to host connected to the external router (10.1.3.1)

• Traffic will hit leaf 4 (regular leaf node) and leaf 4 routing table points the next hop as external routers ip

Leaf104# show ip route vrf SR-MPLS:VRF-1
IP Route Table for VRF "SR-MPLS:VRF-1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.1.1.1/32, ubest/mbest: 1/0
    *via 10.0.96.64%overlay-1, [1/0], 00:43:39, bgp-64500, internal, tag 64500
10.1.3.0/24, ubest/mbest: 1/0
    *via 172.16.1.1%SR-MPLS:VRF-1, [200/5], 00:00:01, bgp-64500, internal, tag 64500

The output from the leaf switch shows that the next-hop for 10.1.3.0/24 (the subnet for the external endpoint) is 172.16.1.1 (the external router IP)

• Leaf 4 will route the traffic to leaf 2 (non-anchor leaf node connected to the external router host) since next-hop is the external router
• Leaf 2 will decapsulate and forward the traffic to the external router then forwarded to external endpoint by the external router

3.5. Verification of Optimized Floating L3Out

Reference

https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all/floating-l3outs/simplify-outside-network-connections-using-floating-l3outs.html

About