In a network design for load balancer, it is required for the ingress and egress traffic to pass through the same load balancer except for direct server return (DSR). To steer the return traffic to the same load balancer as incoming use the following deployment options:
- Load balancer as a default gateway for servers
- Source Network Address Translation (SNAT)
- Policy Based Redirect (PBR)
When inserting a load balancer into a Cisco ACI fabric, it is important to understand the desired traffic flow, the advantage of using the ACI fabric anycast gateway, the benefit of selective traffic redirection and if DSR is required. Insert Load balancers into ACI fabric using one or more of the following deployment options:
- Two-arm (inline) with LB as a gateway – SNAT or PBR is not required because the load balancer is in the traffic path based on routing.
- Two-arm (inline) with ACI fabric as a gateway – the load balancer is deployed between two different VRFs (VRF stitching). SNAT or PBR is not required because the load balancer is in the traffic path based on routing.
- Two-arm with ACI fabric as a gateway and SNAT or PBR – SNAT or PBR is required to make return traffic back to the load balancer.
- One-arm with ACI fabric as a gateway and SANT or PBR – SNAT or PBR is required to make return traffic back to the load balancer.
“arm” refers to interfaces or VLAN interfaces. “one-arm” and “two-arm” are nothing, but the number of interfaces created/used on the load balancer. In one-arm traffic enters and leaves the load balancer on the same interface. In two-arm design incoming traffic and return traffic use different interfaces.
The post discusses a one-arm load balancer (LB), specifically F5 BIG IP, design and deployment in an ACI environment.
One arm LB with ACI PBR destination in an L3Out
One Arm Load Balancer with Fabric as the Gateway
- Only one interface / VLAN interface is used (LACP can be used to bundle interface for higher bandwidth).
- Simple to implement.
- Virtual IP address should be in the same subnet of physical servers or behind an l3out.
- For servers default gateway is the ACI fabric.
- SNAT (Client IP is not retained) or PBR (Client IP is retained) required for symmetric traffic flow. If SNAT or PBR is not used the return traffic will not be redirected to the LB. The traffic will flow back to the source using the routing and bridging of the fabric. The source will discard the traffic since the source IP is different than the destination IP of the original traffic sent by the source.
SNAT vs. PBR
The use of SNAT on the load balancer or PBR on the ACI fabric can make the return traffic back to the load balancer but the use of PBR offers these advantages:
- With SNAT, the server cannot capture the true client IP unless passed in XFF (X-Forwarded-For) header and the server is able to log.
- ACI redirects traffic matched a contract without relying on routing or bridging. ACI PBR will selectively redirect only interesting traffics to the load balancer.
- For PBR, ACI automatically manages VLAN deployment on the ACI fabric and the virtual networks for service node connectivity.
- ACI automatically connects and disconnects virtual Network Interface Cards (vNICs) for virtual service appliances.
- ACI provides a more logical view of service insertion between consumer and provider EPGs.
- ACI can redirect traffic to the service node without the need for the service node to be the default gateway of the servers.
Physical Topology
The ACI physical topology used to test the one-arm load balancer with PBR has two spines, one boarder leaf and two leafs for the service and compute nodes. The F5 BIG-IP VE is setup in an active-standby HA cluster.
Logical Topology
The server bridge domain will serve as a gateway both for the servers and the load balancers. The clients accessing the servers through the VIP can be both outside of the fabric and within the fabric. Th traffic coming from external network (through the l3out) and internal client EPG to the VIP – 10.10.11.100 will route to the load balancer. The incoming traffic doesn’t require PBR because the ACI routing and bridging forwards the traffic to the VIP. The load balancer then routes the traffic to the servers using the configured method. The server then returns the traffic back to the load balancer using the configured PBR (unidirectional PBR). The load balancer update the source IP and forward back the return traffic to ACI and ACI back to the client/source.
The PBR uses service graph to insert LB between the client/source EPG (external or internal client) and provider / server EPG. The service graph rendering creates a shadow EPG for the load balancer interface.
Requirements
- Detail requirements and design considerations are listed on the Cisco document on the link – https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739971.html
Hardware, Platforms, Software and Assumptions for Step-by-Step Configuration
- ACI – 5.2(1g)
- Spine – N9K-C9332C
- Leaf – N9K-C 93180YC-EX
- F5 BIG-IP VE – v13.0.0(Build 0.0.1645)
- The basic ACI configuration is outside the scope of this document
- Tenant, VRF, BD, EPG and l3out are pre-configured on the ACI fabric
- The F5 BIG-IP VE configuration is outside the scope of this document
- VLANs, self-IPs, HA, nodes, pools, and virtual servers are pre-configured on the F5 BIG-IP
- The F5 LB is in an unmanaged mode service graph with PBR
- The IP addressing setup is as shown below
- Client/consumer subnet – 10.10.10.0/24
- External client/consumer subnet – 192.168.10.0/24
- Server / provider subnet – 10.10.11.0/24
- Virtual server / VIP– 10.10.11.100/24
- Self-IP subnet –
- F1 – 10.10.11.3/24
- F2 – 10.10.11.4/24
- Floating – 10.10.11.5/24
Step-by-Step Configuration for One Arm LB PBR
1. Create L4-L7 Device
You can add one or more L4-L7 devices. This document will create two F5 LB devices as an active/standby high availability cluster.
Tenant > Services > L4-L7 > Devices
- Enter the name for the device.
- Select the service type – ADC for laod balancer.
- Select the device Type – Physical or Virtual.
- Choose VMM domain or physical domain depending on the device type selection.
- Click the + to add devices and Cluster interfaces.
2. Create Service Graph Template
The Layer 4 – 7 Service Graph is a feature in Cisco ACI to insert Layer 4 – 7 service devices such as a firewall, load balancer, and IPS between the consumer and provider EPGs.
Tenant > Services > L4-L7 > Service Graph Template
- Pull the device created on step above to the consumer/provider block.
- Give a service graph name.
- Choose between one-arm and two-arm.
- Select route redirect.
- If a bidirectional contract is needed (for F5 pools and nodes health monitoring) between the server EPG and the Self-IP, the Direct Connect option should be set as true on the service graph template. Under the service graph template created go to policy tap then connections and change the Direct Connect to true for both connectors.
3. Create L4-L7 Policy Based Redirect (PBR) policy
Configure the PBR node IP address, IP SLA monitoring and redirect health group. Starting with APIC Release 5.2, MAC configuration is not mandatory for L3 PBR if IP-SLA tracking is enabled. MAC is not configured since this document is based on 5.2(1g).
Tenant > Policies > L4-L7 Policy-Based Redirect
- Provide a name for the Policy-Based Redirect.
- Create IP SLA Monitoring policy.
- Click + sign to create the L3 destination.
- Provide the IP address for L3 destination (the floating Self-IP).
- Provide the MAC address of the L3 destination (can auto discovered if IP SLA monitoring policy is defined for release 5.2 and later).
- Create a redirect Health Group.
4. Apply the L4-L7 Service Graph Template to EPGs
Apply the L4-L7 service graph template to EPGs using the ‘Apply L4-L7 Service Graph Template’. Manually creating a device selection policy and applying the service graph to the contract can achieve the same.
Tenant > Services > L4-L7 > Service Graph Template > Select the right Service Graph Template > right click > Apply L4-L7 Service Graph Template.
- Click ‘apply L4-L7 Service Graph Template’.
- Select between EPG and ESG.
- Choose the consumer / provider EPGs.
- Select an existing contract or new contract.
- On next page select ‘General’ connector type.
- Select the server BD.
- This wizard will create the Device Selection Policy.
- Confirm the consumer and provider connector have the right network associated, the right L4-L7 Policy-Based Redirect and L3 destination is selected only on the provider side (since this is a unidirectional PBR for F5).
Tenant > Services > L4-L7 > Device Selection Policies > select the policy under consideration
5. Deployed Graph Instance
A deployed graph instance show-up under ‘Tenant > Services > L4-L7 > Deployed Graph Instances’ after the above steps successful completion.
6. Verification
6.1 ‘show service redir info’
The ‘show service redir info’ output provides info on the destination group, the list of destinations and the health group status.
6.2 Traffic Flow and Contract – Internal Clients
The output of ‘show zoning-rule’ below explains the contract and traffic flow of internal client accessing servers behind F5 LB VIP on port 22.
As shown on the traffic flow representation above the first flow is from client to ACI fabric and then to the F5 LB. So ACI needs contract to allow communication between the client EPG (32771) and the service EPG (Shadow EPG – 16390). The red annotation on the ‘show zoning-rule’ output highlights the contract for this communication.
The next flow is from the F5 service EPG (16390) to the backend servers (16386). The green annotation on the ‘show zoning-rule’ output highlights the contract for this communication.
Then the server (16386) replies back to the client (32771). The blue annotation on the ‘show zoning-rule’ output highlights the contract for this communication. This is a redirect contract since the server to client communication uses unidirectional PBR.
The final step on the traffic flow is from F5 LB service EPG (16390) to the client EPG (32771). The pink annotation on the ‘show zoning-rule’ output highlights the contract for this communication.
6.3 Traffic Flow and Contract- External Clients
The output of the ‘show zoning-rule’ below explains the contract and traffic flow of external client accessing servers behind F5 LB VIP on port 22.
As shown on the traffic flow representation above the first flow is from external client to ACI fabric and then to the F5 LB. So ACI needs contract to allow communication between the external client EPG (49159) and the service EPG (Shadow EPG – 16390). The red annotation on the ‘show zoning-rule’ output highlights the contract for this communication.
The next flow is from the F5 service EPG (16390) to the pool servers (16386). The green annotation on the ‘show zoning-rule’ output highlights the contract for this communication.
Then the server (16386) replies back to the client (49159). The blue annotation on the ‘show zoning-rule’ output highlights the contract for this communication. This is a redirect contract since the server to client communication uses unidirectional PBR.
The final step on the traffic flow id from F5 LB service EPG (16390) to the client EPG (49159). The pink annotation on the ‘show zoning-rule’ output highlights the contract for this communication.
6.4 Access to the VIP from the Internal Consumer (Client) Computer
The SSH from internal client to the VIP (10.10.11.100) works as expected. The LB method also works as the output shows different backend server (round robin) for each SSH try.
6.5 Access to the VIP from the External Consumer (Client) Computer
The SSH from external client to the VIP (10.10.11.100) works as expected. The LB method also works as the output shows different backend server (round robin) for each SSH try.
Core-Router# ssh cisco@10.10.11.100 source-ip 192.168.10.10 vrf F5PBR
The authenticity of host '10.10.11.100 (10.10.11.100)' can't be established.
ECDSA key fingerprint is SHA256:NYdS3UdGHpRBX6rNFWq01/R2LkyZgixQKwruSXd9SSE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.10.11.100' (ECDSA) to the list of known hosts.
Outbound-ReKey for 10.10.11.100:22
Inbound-ReKey for 10.10.11.100:22
cisco@10.10.11.100's password:
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-100-generic x86_64)
Last login: Thu Aug 18 15:10:49 2022 from 10.10.10.10
cisco@Web-1:~$ exit
logout
Connection to 10.10.11.100 closed.
Core-Router # ssh cisco@10.10.11.100 source-ip 192.168.10.10 vrf F5PBR
Outbound-ReKey for 10.10.11.100:22
Inbound-ReKey for 10.10.11.100:22
cisco@10.10.11.100's password:
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-100-generic x86_64)
Last login: Thu Aug 18 15:10:15 2022 from 10.10.10.10
cisco@Web-2:~$ exit
logout
Connection to 10.10.11.100 closed.
Core-Router # ssh cisco@10.10.11.100 source-ip 192.168.10.10 vrf F5PBR
Outbound-ReKey for 10.10.11.100:22
Inbound-ReKey for 10.10.11.100:22
cisco@10.10.11.100's password:
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-100-generic x86_64)
Last login: Thu Aug 18 15:11:04 2022 from 10.10.10.10
cisco@web-3:~$ exit
logout
Connection to 10.10.11.100 closed.
Let’s say we don’t use SNAT on F5. So when traffic is sent from F5 to Server, source IP is still 10.10.10.10. Don’t we need a contract between Client (32771) and Server (16386)? Or this traffic matches the rule from Shadow EPG (16390) to Server (16386)?
And BTW, where do I see on the ACI what’s the Shadow EPG value for a particular Device?
Thanks!
Yes, the traffic flow from client to server is in two segments. The first from client to the VIP IP of the F5 and then from F5 to the backend servers. The contract from client EPG to service EPG covers the first segment and the contract from service EPG to server EPG covers the second segment.
The class ID of the service (shadow) EPGs can be found in the ‘Function Node’ under Tenants > [Tenants name] > Services > L4-L7 > Deployed Graph Instances.
Thank you
Sorry if I’m insisting too much, but I’m insisting on the “without SNAT” part. You said “the contract from service EPG to server EPG covers the second segment.” Service EPG contains EP from 10.10.11.0/24, but the source IP address of the traffic is 10.10.10.0/24 (original Clients IP). This contract will permit the flow?
Thanks a lot for the quick response!
The most common EPG classification is by matching the ACI incoming port and VLAN tagging of the traffic. The traffic from the F5 reaches the provider connector with the encap-VLAN assigned to the service EPG. So, even if the source IP of the traffic is the client IP the pcTag used for the traffic will be the service EPG’s.
* With PBR Data-plane learning is disabled on the service BD to avoid the learning of client IP from the F5 data traffic.
Thanks,
Thanks a lot for taking the time to reply.
Very clear and precious content!