One Arm Load Balancer with ACI PBR Destination in an L3out

2 Comments

Belete Ageze 2xCCIE | CCDE

Introduction

In a network design for load balancer, it is required for the ingress and egress traffic to pass through the same load balancer except for direct server return (DSR). To steer the return traffic to the same load balancer as incoming use the following deployment options:

  • Load balancer as a default gateway for servers
  • Source Network Address Translation (SNAT)
  • Policy Based Redirect (PBR)

When inserting a load balancer into a Cisco ACI fabric, it is important to understand the desired traffic flow, the advantage of using the ACI fabric anycast gateway, the benefit of selective traffic redirection and if DSR is required. Insert load balancers into ACI fabric using one or more of the following deployment options:

  • Two-arm (inline) with LB as a gateway – SNAT or PBR is not required because the load balancer is in the traffic path based on routing.
  • Two-arm (inline) with ACI fabric as a gateway – the load balancer is deployed between two different VRFs (VRF stitching). SNAT or PBR is not required because the load balancer is in the traffic path based on routing.
  • Two-arm with ACI fabric as a gateway and SNAT or PBR – SNAT or PBR is required to make return traffic back to the load balancer.
  • One-arm with ACI fabric as a gateway and SANT or PBR – SNAT or PBR is required to make return traffic back to the load balancer.

“arm” refers to interfaces or VLAN interfaces. “one-arm” and “two-arm” are nothing, but the number of interfaces created/used on the load balancer. In one-arm traffic enters and leaves the load balancer on the same interface. In two-arm design incoming traffic and return traffic use different interfaces.

One Arm Load Balancer with Fabric as the Gateway

  • Only one interface / VLAN interface is used (LACP can be used to bundle interface for higher bandwidth).
  • Simple to implement.
  • Virtual IP address should be in the same subnet of physical servers or behind an l3out.
  • For servers default gateway is the ACI fabric.
  • SNAT (Client IP is not retained) or PBR (Client IP is retained) required for symmetric traffic flow. If SNAT or PBR is not used the return traffic will not be redirected to the LB. The traffic will flow back to the source using the routing and bridging of the fabric. The source will discard the traffic since the source IP is different than the destination IP of the original traffic sent by the source.

SNAT vs. PBR

The use of SNAT on the load balancer or PBR on the ACI fabric can make the return traffic back to the load balancer but the use of PBR offers these advantages:

  • With SNAT, the server cannot capture the true client IP unless passed in XFF (X-Forwarded-For) header and the server is able to log.
  • ACI redirects traffic matched a contract without relying on routing or bridging. ACI PBR will selectively redirect only interesting traffics to the load balancer.
  • For PBR, ACI automatically manages VLAN deployment on the ACI fabric and the virtual networks for service node connectivity.
  • ACI automatically connects and disconnects virtual Network Interface Cards (vNICs) for virtual service appliances.
  • ACI provides a more logical view of service insertion between consumer and provider EPGs.
  • ACI can redirect traffic to the service node without the need for the service node to be the default gateway of the servers.

The post discusses a one arm load balancer (LB), specifically F5 BIG IP, design and deployment in an ACI environment with PBR destination in an l3out.

Starting with APIC Release 5.2, L3 PBR destinations can be in an L3Out instead of an L3 bridge domain, which provides more flexible design options.

one-arm load balancer design and deployment with PBR destination on the same subnet as the servers.

Physical Topology

The ACI physical topology used to test the one arm load balancer with ACI PBR destination in an l3out has two spines, one boarder leaf and two leafs for the service and compute nodes. The F5 BIG-IP VE is setup in an active-standby HA cluster.

One Arm LB with ACI PBR Destination in an L3out - physical drawing

Logical Topology

The server bridge domain will serve as a gateway only for the servers. The service nodes peer with the ACI fabric using l3out. The routing protocol used for peering is BGP. The VIP (172.16.100.100) is behind the service l3out. The clients accessing the servers through the VIP can be both outside and within the fabric.  Th traffic coming from external network (through the l3out) and internal client EPG to the VIP – 172.16.100.100 will route to the load balancer. The incoming traffic doesn’t require PBR because the ACI routing and bridging forwards the traffic to the VIP. The load balancer then routes the traffic to the servers using the configured method. The server then returns the traffic back to the load balancer using PBR (unidirectional PBR). The load balancer update the source IP and forward back the return traffic to ACI and ACI back to the client/source.

The PBR uses service graph to insert LB between the client/source EPG (external or internal client) and provider / server EPG. The service graph rendering creates a shadow EPG for the load balancer interface.

One Arm LB with ACI PBR Destination in an L3out - logical drawing

Requirements

  • The L3Out for the PBR destination must be in either the consumer or provider VRF.
  • L3Out with SVI, routed sub-interface, or routed interface is supported. (Infra L3Out, GOLF L3Out, SDA L3Out, or L3Out using floating SVI for PBR destination is not supported.)
  • Single pod, Multi-Pod, and Remote Leaf are supported. Multi-Site is not supported as of APIC Release 5.2.
  • If the consumer/provider EPG is an L3Out EPG, it must not be under the same L3Out for PBR destinations.
  • If the consumer/provider EPG is an L3Out EPG, it must not be under the service leaf nodes, where the L3Out for PBR destinations resides. If the consumer/provider EPG is a regular EPG—not an L3Out EPG—the consumer, provider, and the L3Out for PBR destinations can be under the same leaf. This consideration is applicable to the case where a consumer/provider EPG communicates with an L3Out EPG for a service device via another service device where PBR destination in an L3Out is enabled.
  • If the service device is in two-arm mode and one of the L3Outs for the PBR destinations learns 0.0.0.0/0 or 0::0 route, both arms of the service device must be connected to the same leaf node or the same vPC pair.
  • Mixing of PBR destinations in an L3 bridge domain and PBR destinations in an L3Out within the same function node in the service graph is not supported.
  • IP SLA tracking is mandatory for the PBR destination in an L3Out for better convergence.
  • The L3Out EPG with 0.0.0.0/0 or 0::0 subnet can’t be used for the L3Out EPG for PBR destinations.

Hardware, Platforms, Software and Assumptions for Step-by-Step Configuration

  • ACI – 5.2(1g)
    • Spine – N9K-C9332C
    • Leaf – N9K-C 93180YC-EX
  • F5 BIG-IP VE – v13.0.0(Build 0.0.1645)
  • The basic ACI configuration is outside the scope of this document
  • Tenant, VRF, BD, EPG and l3out are pre-configured on the ACI fabric
  • The F5 BIG-IP VE configuration is outside the scope of this document
  • VLANs, self-IPs, HA, nodes, pools, and virtual servers are pre-configured on the F5 BIG-IP
  • BGP is configured on both ACI fabric and F5. LB BIG-IP in HA pair uses the floating IP as a next hop when advertising subnets using BGP from both active and standby nodes.
  • The F5 LB is in an unmanaged mode service graph with PBR
  • The IP addressing setup is as shown below
  • Client/consumer subnet – 10.10.10.0/24
  • External client/consumer subnet – 192.168.10.0/24
  • Server / provider subnet – 10.10.11.0/24
  • Virtual server / VIP– 172.16.100.100/24 
  • Self-IP subnet –
    • F1 – 172.16.10.3/24
    • F2 – 172.16.10.4/24
    • Floating – 172.16.10.5/24

Step-by-Step Configuration

1. Create L4-L7 Device

You can add one or more L4-L7 devices. This document will create two F5 LB devices as an active/standby high availability cluster.

Tenant > Services > L4-L7 > Devices

One Arm LB with ACI PBR Destination in an L3out

  • Enter the name for the device.
  • Select the service type – ADC for laod balancer.
  • Select the device type – Physical or Virtual.
  • Choose VMM domain or physical domain depending on the device type selection.
  • Click the + to add devices and cluster interfaces.
  • The path / interface configuration of the concrete device is mandatory for PBR destinations in an L3Out and should much the one on the L3Out configuration.

One Arm LB with ACI PBR Destination in an L3out

2. Create Service Graph Template

The Layer 4 – 7 Service Graph is a feature in Cisco ACI to insert Layer 4 – 7 service devices such as a firewall, load balancer, and IPS between the consumer and provider EPGs.

Tenant > Services > L4-L7 > Service Graph Template

One Arm LB with ACI PBR Destination in an L3out

  • Pull the device created on step above to the consumer/provider block.
  • Give a service graph name.
  • Choose between one-arm and two-arm.
  • Select route redirect.
  • If a bidirectional contract is needed (for F5 pools and nodes health monitoring) between the server EPG and the Self-IP, the Direct Connect option should be set as true on the service graph template. 
  • Under the service graph template created go to policy tap then connections and change the Direct Connect option to true for both connectors.

One Arm LB with ACI PBR Destination in an L3out

3. Create L4-L7 Policy Based Redirect (PBR) Policy

Configure the PBR node IP address, IP SLA monitoring and redirect health group. Starting with APIC release 5.2, MAC configuration is not mandatory for L3 PBR if IP-SLA tracking is enabled.

Tenant > Policies > L4-L7 Policy-Based Redirect

One Arm LB with ACI PBR Destination in an L3out

  • Provide a name for the Policy Based Redirect.
  • Create IP SLA Monitoring Policy.
  • Click + sign to create the L3 destination.
  • Provide the IP address for L3 destination (the floating Self-IP).
  • Provide the MAC address of the L3 destination (can auto discovered if IP SLA monitoring policy is defined for release 5.2 and later).
  • Create a Redirect Health Group.

4. Apply the L4-L7 Service Graph Template to EPGs

Apply the L4-L7 service graph template to EPGs using the ‘Apply L4-L7 Service Graph Template’. Manually creating a device selection policy and applying the service graph to the contract can achieve the same.

Tenant > Services > L4-L7 > Service Graph Template > Select the right Service Graph Template > right click > Apply L4-L7 Service Graph Template.

One Arm LB with ACI PBR Destination in an L3out

  • Click ‘Apply L4-L7 Service Graph Template’.
  • Select between EPG and ESG.
  • Choose the consumer / provider EPGs.
  • Select an existing contract or new contract.
  • On next page select the connector as route peering
  • Select the service l3out external EPG
  • This wizard will create the Device Selection Policy.
  • Confirm the consumer and provider connector have the right network associated and the right L4-L7 Policy-Based Redirect.
  • Confirm L3 destination is selected only on the provider side (since this is a unidirectional PBR for F5).

Tenant > Services > L4-L7 > Device Selection Policies > select the policy under consideration

5. Deployed Graph Instance

A deployed graph instance show-up under ‘Tenant > Services > L4-L7 > Deployed Graph Instances’ after the above steps successful completion.

6. Verification

6.1 ‘Show Service Redir Info’

The ‘show service redir info’ output provides info on the destination group, the list of destinations and the health group status.

6.2 Traffic Flow and Contract for One Arm LB with ACI PBR Destination in an L3out– Internal Clients

One Arm LB with ACI PBR Destination in an L3out - internal client traffic flow

The output of ‘show zoning-rule’ below explains the contract and traffic flow of internal client accessing servers behind F5 LB VIP on port 22.

As shown on the traffic flow representation above the first flow is from client to ACI fabric and then to the F5 LB. So ACI needs contract to allow communication between the client EPG (32771) and the L3out EPG (F5-exEPG – 32775). The red annotation on the ‘show zoning-rule’ output highlights the contract for this communication.

The next flow is from the F5 service EPG (10936) to the backend servers (16386). The green annotation on the ‘show zoning-rule’ output highlights the contract for this communication.

Then the server (16386) replies back to the client (32771). The blue annotation on the ‘show zoning-rule’ output highlights the contract for this communication. This is a redirect contract since the server to client communication uses unidirectional PBR.

The final step on the traffic flow is from F5 LB L3out EPG (F5-exEPG – 32775) to the client EPG (32771). The pink annotation on the ‘show zoning-rule’ output highlights the contract for this communication.

The direct connect option on the service graph template configures the contract for the keepalive traffic between the L3Out EPG -32775 (not the service EPG – 10936) and server EPG (16386) . The yellow annotation on the ‘show zoning-rule’ output highlights the contract for this communication.

6.3 Traffic Flow and Contract One Arm LB with ACI PBR Destination in an L3out– External Clients

One Arm LB with ACI PBR Destination in an L3out - external client traffic flow

The output of the ‘show zoning-rule’ below explains the contract and traffic flow of external client accessing servers behind F5 LB VIP on port 22.

As shown on the traffic flow representation above the first flow is from external client to ACI fabric and then to the F5 LB. So ACI needs contract to allow communication between the external client EPG (49159) and the L3out EPG (F5-exEPG – 32775). The red annotation on the ‘show zoning-rule’ output highlights the contract for this communication.

The next flow is from the F5 service EPG (5491) to the backend servers (16386). The green annotation on the ‘show zoning-rule’ output highlights the contract for this communication.

Then the server (16386) replies back to the client (49159). The blue annotation on the ‘show zoning-rule’ output highlights the contract for this communication. This is a redirect contract since the server to client communication uses unidirectional PBR.

The final step on the traffic flow is from F5 LB L3out EPG (F5-exEPG – 32775) to the client EPG (49159). The pink annotation on the ‘show zoning-rule’ output highlights the contract for this communication.

The direct connect option on the service graph template configures the contract for the keepalive traffic between the L3Out EPG -32775 (not the service EPG – 10936) and server EPG (16386). The yellow annotation on the ‘show zoning-rule’ output highlights the contract for this communication.

6.4 Access to the VIP from the Internal Consumer (Client) Computer

The SSH from internal client to the VIP (172.16.100.100) works as expected. The LB method also works as the output shows different backend server (round robin) for each SSH try.

6.5 Access to the VIP from the External Consumer (Client)

The SSH from external client to the VIP (172.16.100.100) works as expected. The LB method also works as the output shows different backend server (round robin) for each SSH try.

Core-router # ssh cisco@172.16.100.100 vrf F5PBR source-ip 192.168.10.10
Outbound-ReKey for 172.16.100.100:22
Inbound-ReKey for 172.16.100.100:22
cisco@172.16.100.100's password: 
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-100-generic x86_64)
Last login: Sat Aug 27 22:08:55 2022 from 192.168.10.10
cisco@Web-2:~$ exit
logout
Connection to 172.16.100.100 closed.

Core-router# ssh cisco@172.16.100.100 vrf F5PBR source-ip 192.168.10.10
Outbound-ReKey for 172.16.100.100:22
Inbound-ReKey for 172.16.100.100:22
cisco@172.16.100.100's password: 
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-100-generic x86_64)
Last login: Sat Aug 27 23:27:44 2022 from 192.168.10.10
cisco@web-3:~$ exit
logout
Connection to 172.16.100.100 closed.

Core-router# ssh cisco@172.16.100.100 vrf F5PBR source-ip 192.168.10.10
Outbound-ReKey for 172.16.100.100:22
Inbound-ReKey for 172.16.100.100:22
cisco@172.16.100.100's password: 
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-100-generic x86_64)
Last login: Sun Aug 28 00:31:30 2022 from 192.168.10.10
cisco@Web-1:~$

2 thoughts on “One Arm Load Balancer with ACI PBR Destination in an L3out”

Leave a Comment

Your email address will not be published. Required fields are marked *