VXLAN EVPN vPC Attached External / L4-L7 Configuration – BGP

Leave a Comment

Belete Ageze 2xCCIE | CCDE

Overview

In a VXLAN EVPN fabric, establishing external routed connectivity and integrating Layer 4 to Layer 7 (L4-L7) services are crucial for ensuring network security and optimizing traffic flow. Typically, external routed connections are linked to specific leaf switches known as border leaf switches. These switches handle traffic entering and exiting the VXLAN fabric. On the other hand, L4-L7 services, which include firewalls, load balancers, and other security appliances, are connected to leaf switches referred to as service leaf switches.

The choice of connectivity method depends heavily on your network services’ specific requirements and overall deployment strategy. For instance, connecting these devices through a virtual PortChannel (vPC) topology provides redundancy and load balancing, enhancing both performance and reliability.

This blog delves into the detailed CLI configuration needed to set up VXLAN EVPN external / L4-L7 connectivity with vPC topology. We will cover key configuration steps, best practices, and considerations to ensure a seamless integration of external routed connections and L4-L7 services into your VXLAN EVPN fabric.

Assumptions

Use case and Configuration

BGP Configuration

BGP Routed / L4-L7 External

Topology and Overview

Workloads in the data center require connectivity to external networks to function effectively. This external connectivity is essential for linking internal systems to networks outside the VXLAN EVPN fabric, such as the internet, a Wide Area Network (WAN), or other Layer 3 network segments. In this section, we cover how an internal workloads with the IP range 10.10.110.0/24 can access external resources represented by the IP range 192.168.10.0/24 in the topology used for this demonstration. The configuration for external connectivity is established through a virtual Port-Channel (vPC) to an external router, utilizing eBGP as the routing protocol between the VXLAN EVPN fabric and the external network. The topology is depicted below.

  • In this scenario, peering directly with the Distributed Anycast Gateway (DAG) is not supported. Instead, we need to establish peering with dedicated loopbacks on each VTEP within the VRF we are using for peering.
  • To achieve reachability to these dedicated loopbacks from the external router or L4-L7 devices, static routes to the DAG can be used.
  • The VTEPs will peer with the IP address assigned to the VLAN that connects the external router or L4-L7 devices.
  • The external router or L4-L7 devices will then peer with the dedicated loopbacks on each VTEP.
  • In addition, you must establish per-VRF peering via a vPC peer-link. Alongside this, you can enable the advertisement of prefix routes (EVPN Route-Type 5) using the advertise-pip command.
  • This peering should use an infrastructure VLAN, non-VXLAN enabled VLAN without a VNI. The VLAN should be configured as ‘system nve infra-vlans <vlan id>’
  • For vPC setups with fabric peering, the per-VRF peering is not necessary.

This approach ensures that each VTEP can establish a stable and reliable peering session with the external network or L4-L7 services, maintaining optimal routing and forwarding within the VXLAN EVPN fabric.

Configuration

### BL-1

### vPC Configuration
interface mgmt0
  vrf member management
  ip address 10.14.14.11/24

vpc domain 2
  peer-switch
  peer-keepalive destination 10.14.14.12 source 10.14.14.11
  delay restore 150
  peer-gateway
  auto-recovery reload-delay 360
  ip arp synchronize

interface Ethernet1/3
  description "PO 10 (vpc-peer-link) member BL-1-Ethernet1/3 to BL-2-Ethernet1/3"
  switchport mode trunk
  channel-group 10 mode active

interface port-channel10
  description "vpc-peer-link BL-1-BL-2
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
!
interface Ethernet1/4
  switchport 
  switchport mode trunk
  channel-group 100 mode active

interface Ethernet1/5
  switchport 
  switchport mode trunk
  channel-group 200 mode active

interface port-channel100
  switchport mode trunk
  spanning-tree port type normal
  vpc 100

interface port-channel200
  switchport mode trunk
  spanning-tree port type normal
  vpc 200
!

### VLAN and interface Configuration

vlan 100
  name route-peering
  vn-segment 20100
vlan 1000
!
system nve infra-vlans 1000
!
interface Vlan100
  no shutdown
  vrf member myvrf_1
  no ip redirects
  ip address 172.16.10.1/29 tag 12345
  fabric forwarding mode anycast-gateway
!
interface Vlan1000
  no shutdown
  vrf member myvrf_1
  no ip redirects
  ip address 10.10.210.45/30 tag 12345
!
interface loopback100
  vrf member myvrf_1
  ip address 10.10.100.1/32 tag 12345
!

### BGP Configuration

router bgp 65125
  address-family l2vpn evpn
    advertise-pip
  vrf myvrf_1
    address-family ipv4 unicast
    neighbor 10.10.210.46
      remote-as 65125
      update-source Vlan1000
      address-family ipv4 unicast
        next-hop-self
    neighbor 172.16.10.0/29
      remote-as 65325
      update-source loopback100
      ebgp-multihop 5
      address-family ipv4 unicast
### Ext-1

interface port-channel100
  switchport
  switchport mode trunk
  spanning-tree port type normal

interface Ethernet1/2
  switchport
  switchport mode trunk
  channel-group 100 mode active
  no shutdown

interface Ethernet1/3
  switchport
  switchport mode trunk
  channel-group 100 mode active
  no shutdown
!
vlan 100
!
interface Vlan100
  no shutdown
  ip address 172.16.10.4/29
!
ip route 10.10.100.1/32 172.16.10.1
ip route 10.10.100.2/32 172.16.10.1
!
outer bgp 65325
  address-family ipv4 unicast
    network 192.168.10.0/24
  neighbor 10.10.100.1
    remote-as 65125
    update-source Vlan100
    ebgp-multihop 5
    address-family ipv4 unicast
  neighbor 10.10.100.2
    remote-as 65125
    update-source Vlan100
    ebgp-multihop 5
    address-family ipv4 unicast
!

### BL-2

### vPC Configuration
interface mgmt0
  vrf member management
  ip address 10.14.14.12/24

vpc domain 2
  peer-switch
  peer-keepalive destination 10.14.14.11 source 10.14.14.12
  delay restore 150
  peer-gateway
  auto-recovery reload-delay 360
  ip arp synchronize

interface Ethernet1/3
  description "PO 10 (vpc-peer-link) member BL-2-Ethernet1/3 to BL-1-Ethernet1/3"
  switchport mode trunk
  channel-group 10 mode active

interface port-channel10
  description "vpc-peer-link BL-2-BL-1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
!
interface Ethernet1/4
  switchport 
  switchport mode trunk
  channel-group 100 mode active

interface Ethernet1/5
  switchport 
  switchport mode trunk
  channel-group 200 mode active

interface port-channel100
  switchport mode trunk
  spanning-tree port type normal
  vpc 100

interface port-channel200
  switchport mode trunk
  spanning-tree port type normal
  vpc 200
!

### VLAN and interface Configuration

vlan 100
  name route-peering
  vn-segment 20100
vlan 1000
!
system nve infra-vlans 1000
!
interface Vlan100
  no shutdown
  vrf member myvrf_1
  no ip redirects
  ip address 172.16.10.1/29 tag 12345
  fabric forwarding mode anycast-gateway
!
interface Vlan1000
  no shutdown
  vrf member myvrf_1
  no ip redirects
  ip address 10.10.210.46/30 tag 12345
!
interface loopback100
  vrf member myvrf_1
  ip address 10.10.100.2/32 tag 12345
!

### BGP Configuration

router bgp 65125
  address-family l2vpn evpn
    advertise-pip
  vrf myvrf_1
    address-family ipv4 unicast
    neighbor 10.10.210.46
      remote-as 65125
      update-source Vlan1000
      address-family ipv4 unicast
        next-hop-self
    neighbor 172.16.10.0/29
      remote-as 65325
      update-source loopback100
      ebgp-multihop 5
      address-family ipv4 unicast
### Ext-2

interface port-channel200
  switchport
  switchport mode trunk
  spanning-tree port type normal

interface Ethernet1/2
  switchport
  switchport mode trunk
  channel-group 200 mode active
  no shutdown

interface Ethernet1/3
  switchport
  switchport mode trunk
  channel-group 200 mode active
  no shutdow
!
vlan 100
!
interface Vlan100
  no shutdown
  ip address 172.16.10.5/29
!
ip route 10.10.100.1/32 172.16.10.1
ip route 10.10.100.2/32 172.16.10.1
!
outer bgp 65325
  address-family ipv4 unicast
    network 192.168.10.0/24
  neighbor 10.10.100.1
    remote-as 65125
    update-source Vlan100
    ebgp-multihop 5
    address-family ipv4 unicast
  neighbor 10.10.100.2
    remote-as 65125
    update-source Vlan100
    ebgp-multihop 5
    address-family ipv4 unicast
!

Verification

The show ip route command will be used to verify if the route from one VRF (Blue) is learned by the other VRF (Green) through the router or L4-L7 device, ensuring inter-VRF reachability.

To verify the inter-VRF connectivity, a ping test between hosts in the Blue VRF and the Green VRF is used to confirm reachability.

External route learned on the boarder leaf
Ping from the external host 192.168.10.10 to internal host 10.10.110.10
Ping from internal host 10.10.110.10 to external host 192.168.10.10

BGP L4-L7 between VXLAN EVPN VRFs (Inter-VRF Routing)

Topology and Overview

Routed L4-L7 connectivity is essential for VXLAN EVPN multi-tenancy, ensuring controlled and secure communication between tenants/VRFs and/or external networks. Depending on the requirements and the capabilities of the L4-L7 devices, you can connect to routed L4-L7 in various ways. In this section, we will cover BGP peering between the VXLAN EVPN fabric and an L4-L7 device connected via vPC.

For this demonstration, a Nexus 9K simulates the L4-L7 role. The setup and configuration on the VXLAN EVPN side are identical whether you are using a firewall, load balancer, or any other L4-L7 device. The only variation will be on the L4-L7 devices themselves, as the configuration is vendor and model specific. The topology is depicted below.

  • In this scenario, peering directly with the Distributed Anycast Gateway (DAG) is not supported. Instead, we need to establish peering with dedicated loopbacks on each VTEP within the VRF we are using for peering.
  • To achieve reachability to these dedicated loopbacks from the external router or L4-L7 devices, static routes to the DAG can be used.
  • The VTEPs will peer with the IP address assigned to the VLAN that connects the external router or L4-L7 devices.
  • The external router or L4-L7 devices will then peer with the dedicated loopbacks on each VTEP for each VRF, in this case, Blue and Green VRFs.
  • In addition, you must establish per-VRF peering via a vPC peer-link. Alongside this, you can enable the advertisement of prefix routes (EVPN Route-Type 5) using the advertise-pip command.
  • This peering should use an infrastructure VLAN, non-VXLAN enabled VLAN without a VNI. The VLAN should be configured as ‘system nve infra-vlans <vlan id>’
  • For vPC setups with fabric peering, the per-VRF peering is not necessary.
  • The local-as feature with no-prepend and replace-as options will be used so that the Green VRF peer will be able to receive a route originated from the Blue VRF within the fabric, and vice versa.

Configuration 

### BL-1

### vPC Configuration
interface mgmt0
  vrf member management
  ip address 10.14.14.11/24

vpc domain 2
  peer-switch
  peer-keepalive destination 10.14.14.12 source 10.14.14.11
  delay restore 150
  peer-gateway
  auto-recovery reload-delay 360
  ip arp synchronize

interface Ethernet1/3
  description "PO 10 (vpc-peer-link) member BL-1-Ethernet1/3 to BL-2-Ethernet1/3"
  switchport mode trunk
  channel-group 10 mode active

interface port-channel10
  description "vpc-peer-link BL-1-BL-2
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
!
interface Ethernet1/4
  switchport 
  switchport mode trunk
  channel-group 100 mode active

interface Ethernet1/5
  switchport 
  switchport mode trunk
  channel-group 200 mode active

interface port-channel100
  switchport mode trunk
  spanning-tree port type normal
  vpc 100

interface port-channel200
  switchport mode trunk
  spanning-tree port type normal
  vpc 200
!

### VLAN and interface Configuration

vlan 200
  name route-peering-vrf-Blue
  vn-segment 200200
vlan 400
  name route-peering-vrf-Green
  vn-segment 200400
valn 2200
vlan 2400

system nve infra-vlans 2200,2400
!
interface Vlan200
  no shutdown
  vrf member blue
  no ip redirects
  ip address 172.16.20.1/29 tag 12345
  fabric forwarding mode anycast-gateway

interface Vlan400
  no shutdown
  vrf member green
  no ip redirects
  ip address 172.16.40.1/29 tag 12345
  fabric forwarding mode anycast-gateway
!
interface Vlan2200
  no shutdown
  vrf member blue
  no ip redirects
  ip address 10.10.220.45/30 tag 12345

interface Vlan2400
  no shutdown
  vrf member green
  no ip redirects
  ip address 10.10.240.45/30 tag 12345
!
interface loopback200
  vrf member blue
  ip address 10.10.220.1/32 tag 12345

interface loopback400
  vrf member green
  ip address 10.10.240.1/32 tag 12345
!

### BGP Configuration

router bgp 65125
  router-id 10.10.100.11
  address-family ipv4 unicast
  address-family l2vpn evpn
    advertise-pip
  vrf blue
    neighbor 10.10.220.46
      remote-as 65125
      update-source Vlan2200
      address-family ipv4 unicast
        next-hop-self
    neighbor 172.16.20.0/29
      remote-as 65325
      local-as 65020 no-prepend replace-as
      update-source loopback200
      ebgp-multihop 5
      address-family ipv4 unicast
  vrf green
    neighbor 10.10.240.46
      remote-as 65125
      update-source Vlan2400
      address-family ipv4 unicast
        next-hop-self
    neighbor 172.16.40.0/29
      remote-as 65325
      local-as 65040 no-prepend replace-as
      update-source loopback400
      ebgp-multihop 5
      address-family ipv4 unicast
### Ext-1

interface port-channel100
  switchport
  switchport mode trunk
  spanning-tree port type normal

interface Ethernet1/2
  switchport
  switchport mode trunk
  channel-group 100 mode active
  no shutdown

interface Ethernet1/3
  switchport
  switchport mode trunk
  channel-group 100 mode active
  no shutdown
!
vlan 200
vlan 400
!
interface Vlan200
  no shutdown
  ip address 172.16.20.4/29

interface Vlan400
  no shutdown
  ip address 172.16.40.4/29
!
ip route 10.10.220.1/32 172.16.20.1
ip route 10.10.220.2/32 172.16.20.1
ip route 10.10.240.1/32 172.16.40.1
ip route 10.10.240.2/32 172.16.40.1
!
outer bgp 65325
  address-family ipv4 unicast
  neighbor 10.10.220.1
    remote-as 65020
    update-source Vlan200
    ebgp-multihop 5
    address-family ipv4 unicast
  neighbor 10.10.220.2
    remote-as 65020
    update-source Vlan200
    ebgp-multihop 5
    address-family ipv4 unicast
  neighbor 10.10.240.1
    remote-as 65040
    update-source Vlan400
    ebgp-multihop 5
    address-family ipv4 unicast
  neighbor 10.10.240.2
    remote-as 65040
    update-source Vlan400
    ebgp-multihop 5
    address-family ipv4 unicast
!

### BL-2

### vPC Configuration
interface mgmt0
  vrf member management
  ip address 10.14.14.12/24

vpc domain 2
  peer-switch
  peer-keepalive destination 10.14.14.11 source 10.14.14.12
  delay restore 150
  peer-gateway
  auto-recovery reload-delay 360
  ip arp synchronize

interface Ethernet1/3
  description "PO 10 (vpc-peer-link) member BL-2-Ethernet1/3 to BL-1-Ethernet1/3"
  switchport mode trunk
  channel-group 10 mode active

interface port-channel10
  description "vpc-peer-link BL-2-BL-1
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
!
interface Ethernet1/4
  switchport 
  switchport mode trunk
  channel-group 100 mode active

interface Ethernet1/5
  switchport 
  switchport mode trunk
  channel-group 200 mode active

interface port-channel100
  switchport mode trunk
  spanning-tree port type normal
  vpc 100

interface port-channel200
  switchport mode trunk
  spanning-tree port type normal
  vpc 200
!

### VLAN and interface Configuration

vlan 200
  name route-peering-vrf-Blue
  vn-segment 200200
vlan 400
  name route-peering-vrf-Green
  vn-segment 200400
valn 2200
vlan 2400

system nve infra-vlans 2200,2400
!
interface Vlan200
  no shutdown
  vrf member blue
  no ip redirects
  ip address 172.16.20.1/29 tag 12345
  fabric forwarding mode anycast-gateway

interface Vlan400
  no shutdown
  vrf member green
  no ip redirects
  ip address 172.16.40.1/29 tag 12345
  fabric forwarding mode anycast-gateway
!
interface Vlan2200
  no shutdown
  vrf member blue
  no ip redirects
  ip address 10.10.220.46/30 tag 12345

interface Vlan2400
  no shutdown
  vrf member green
  no ip redirects
  ip address 10.10.240.46/30 tag 12345
!
interface loopback200
  vrf member blue
  ip address 10.10.220.2/32 tag 12345

interface loopback400
  vrf member green
  ip address 10.10.240.2/32 tag 12345
!

### BGP Configuration

router bgp 65125
  router-id 10.10.100.12
  address-family ipv4 unicast
  address-family l2vpn evpn
    advertise-pip
  vrf blue
    neighbor 10.10.220.45
      remote-as 65125
      update-source Vlan2200
      address-family ipv4 unicast
        next-hop-self
    neighbor 172.16.20.0/29
      remote-as 65325
      local-as 65020 no-prepend replace-as
      update-source loopback200
      ebgp-multihop 5
      address-family ipv4 unicast
  vrf green
    neighbor 10.10.240.45
      remote-as 65125
      update-source Vlan2400
      address-family ipv4 unicast
        next-hop-self
    neighbor 172.16.40.0/29
      remote-as 65325
      local-as 65040 no-prepend replace-as
      update-source loopback400
      ebgp-multihop 5
      address-family ipv4 unicast
### Ext-2

interface port-channel200
  switchport
  switchport mode trunk
  spanning-tree port type normal

interface Ethernet1/2
  switchport
  switchport mode trunk
  channel-group 200 mode active
  no shutdown

interface Ethernet1/3
  switchport
  switchport mode trunk
  channel-group 200 mode active
  no shutdown
!
vlan 200
vlan 400
!
interface Vlan200
  no shutdown
  ip address 172.16.20.5/29

interface Vlan400
  no shutdown
  ip address 172.16.40.5/29
!
ip route 10.10.220.1/32 172.16.20.1
ip route 10.10.220.2/32 172.16.20.1
ip route 10.10.240.1/32 172.16.40.1
ip route 10.10.240.2/32 172.16.40.1
!
outer bgp 65325
  address-family ipv4 unicast
  neighbor 10.10.220.1
    remote-as 65020
    update-source Vlan200
    ebgp-multihop 5
    address-family ipv4 unicast
  neighbor 10.10.220.2
    remote-as 65020
    update-source Vlan200
    ebgp-multihop 5
    address-family ipv4 unicast
  neighbor 10.10.240.1
    remote-as 65040
    update-source Vlan400
    ebgp-multihop 5
    address-family ipv4 unicast
  neighbor 10.10.240.2
    remote-as 65040
    update-source Vlan400
    ebgp-multihop 5
    address-family ipv4 unicast
!

Verification

The show ip route command will be used to verify if the route from one VRF (Blue) is learned by the other VRF (Green) through the router or L4-L7 device, ensuring inter-VRF reachability.

To verify the inter-VRF connectivity, a ping test between hosts in the Blue VRF (10.10.20.10) and the Green VRF (10.10.40.10) is used to confirm reachability.

Show Commands Used for Troubleshooting

  • Sh mac address-table address xxxx.xxxx.xxxx
  • Sh system internal l2fm l2dbg macdb address xxxx.xxxx.xxxx vlan 10
  • Sh sys inter l2fm event-hist deb | in xxxx.xxxx.xxxx
  • Sh ip arp vrf xxxxx
  • Sh forwarding vrf VRF03 adjacency
  • Sh l2route evpn mac evi 20 (vlan-id)
  • Sh l2route evpn mac-ip evi 20 (vlan-id)
  • Sh system internal l2rib event-history mac
  • Sh system internal l2rib event-history mac-ip
  • Sh bgp l2vpn evpn vni-id xxxxx route-type 2
  • Sh bgp l2vpn evpn vni-id xxxxx (vni-id)
  • Sh bgp l2vpn evpn xxxx.xxxx.xxxx
  • Sh bgp internal event-history event | in xxxx.xxxx.xxxx
  • Sh nve multsite dci-links
  • Sh nve interface nve 1 detail
  • Sh nve peers
  • Sh ip route 10.10.40.200/32 vrf xxxxx

https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/102x/configuration/vxlan/cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-release-102x/m_configuring_layer_4-layer_7_network_services_integration.html

Leave a Comment

Your email address will not be published. Required fields are marked *