Everyone is looking for new ways to keep up with increasing demands of flexible services consumption, get to market faster, increase agility and productivity, and find savings wherever possible. Easy & secure connectivity is a foundation to successfully deliver a digital transformation demanded by users. VxLAN EVPN Multi-site architecture is one of the widely deployed DC network solutions that can be scaled to thousands of switches across a wide range of geographical regions.
VLANs have been used to provide network segmentation in data center networks. But it’s limitation on addressing the growing need of scale, multi-tenancy and resiliency have made it unfit to the ever-changing demand of the new digital transformation paradigm. VLAN uses 12-bit identifier which limits the segmentation to about 4000 distinct logical networks. The spanning tree loop prevention mechanism also results in an inefficient use of available network links as a way of ensuring a loop free network topology. As modern apps are now a mesh of micro-services with truly distributed codes and data, VLAN based infrastructure limits the ability to build a large, secure, and multi-tenant DC infrastructure.
VxLAN is an overlay technology designed to provide Layer 2 and Layer 3 connectivity extension over a generic IP network. VxLAN, with its 24-bit identifier has the capability to scale the layer 2 segment isolation to about 16 million distinct logical segments. Since the underlay is IP based no spanning tree required and use the links available efficiently. So VxLAN addresses the shortcoming of VLAN based DC fabric seen today. VxLAN flood and learn even with the capability for scaled logical segments don’t practically provide the needed large, secure, and multi-tenant DC infrastructure. So, control plane learning needs to be used to enjoy the scale of VxLAN logical segments. Multi-Protocol BGP with l2vpn evpn address family is used as a control plane to exchange layer 2 and Layer 3 information.
VxLAN and MPBGP creates a powerful technology used to build a large, secure, and resilient multi-tenant web scale fabric that can scale to host hundreds of thousands of systems.
Cisco’s VxLAN EVPN Multi-site fabric uses VxLAN encapsulation and BGP as a control plane for learning endpoints.
VxLAN EVPN Multi-site between two Sites
The VXLAN BGP EVPN fabric can be extended at Layer 2 and Layer 3 with various technologies. However, this document is focused on how this extension can be achieved by using EVPN Multi-site architecture, an integrated interconnectivity approach for VXLAN BGP EVPN fabrics. VXLAN EVPN Multi-site architecture is independent of the transport network between sites.
In this document, VxLAN EVPN Multi-site with two sites (SITE1 and SITE2) and inter-site network (ISN) will be configured for seamlessly extending layer 2 and layer 3 using anycast BGWs. All configurations necessary for full operation will be included.
The Setup uses Nexus 9K switches and NXOS 9.3(6).
Assumptions –
- This document assumes that the reader has a basic familiarity with VXLAN BGP EVPN terminologies.
- The topology uses the same switches for the role of Spine and BGW. It’s a collapsed spine and BGW.
- The lab uses an anycast BGWs.
- The link between the spines at each site added for L2vpn evpn route type 4 (Ethernet segment route) exchange for the BUM designated forwarder (DF) election. Without the link BGW-to-BGW communication is through the only path available, the site-internal VTEPs (leaf nodes). Although this approach doesn’t create any problems from a traffic volume or resiliency perspective, the use of a control-plane exchange between the BGW traversing the leaf node is not natural.
- Nexus 92160 is used as a route server (RS is used to avoid a higher number of full mesh eBGP peering and complexity when we have multiple sites; like route reflectors in iBGP). High availability is recommended in production RS deployment.
- The lab uses the I-E-I model. Focuses on using Interior Gateway Protocol (IGP, in this case OSPF) for underlay and iBGP for overlay in site-internal fabric with eBGP-eBGP at the external site (DCI).
- Tag is used on IP address configuration for identifying the required IPs for redistributing connected routes in BGP.
- Multi-site ID 100 for SITE1 and 200 for SITE2 is used.
- Extend VLAN 20,30, 40 between SITE1 and SITE2.
Table 1. IP Addressing for the Lab Setup
Fig 1. The Logical Representation of the Lab Setup
Expected Result
- Full reachability between hosts on both sites.
- Test using ping between Host-10.10.20.100 (SITE1), Host-10.10.20.200 (SITE2), Host-10.10.30.100(SITE1) and Host-10.10.40.200(SITE2).
Step-by-Step Configuration
The following steps will be used to fully configure an operational VxLAN EVPN Multi-site data center infrastructure.
Step 1 – IP addresses, features, underlay routing (OSPF) configuration
Step 2 – VLAN, VRF, VNI, and site-internal overlay (iBGP) configuration
Step 3 – Site-external overlay, route server, BGW configuration
Step 1 – IP Addresses, Features, Underlay Routing (OSPF) Configuration
!!!!! SITE1 !!!!!
!!!!!Spine-BGW-9336-1!!!!!
#enable features required for VxLAN EVPN
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
# underlay routing and interfaces
router ospf UNDERLAY
router-id 10.10.100.1
log-adjacency-changes detail
interface Ethernet1/31
description Link to leaf1
mtu 9216
ip address 192.168.1.1/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
interface loopback0
description Loopback for Router ID
ip address 10.10.100.1/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback1
description Loopback for VTEP (PIP)
ip address 10.10.100.11/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback254
description Loopback for PIM
ip address 10.254.254.254/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
# Multicast RP configuration
ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24
ip pim anycast-rp 10.254.254.254 10.10.100.1
ip pim anycast-rp 10.254.254.254 10.10.100.2
!!!!!Spine-BGW-9336-2!!!!!
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
router ospf UNDERLAY
router-id 10.10.100.2
log-adjacency-changes detail
interface Ethernet1/31
description Link to leaf1
mtu 9216
ip address 192.168.1.5/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
interface loopback0
description Loopback for Router ID
ip address 10.10.100.2/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback1
description Loopback for VTEP (PIP)
ip address 10.10.100.12/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback254
description Loopback for PIM
ip address 10.254.254.254/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24
ip pim anycast-rp 10.254.254.254 10.10.100.1
ip pim anycast-rp 10.254.254.254 10.10.100.2
!!!!!Leaf-93180-1!!!!!
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
router ospf UNDERLAY
router-id 10.10.100.3
log-adjacency-changes detail
interface Ethernet1/53
description Link to Spine1
mtu 9216
ip address 192.168.1.2/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/54
description Link to Spine1
mtu 9216
ip address 192.168.1.6/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
interface loopback0
description Loopback for Router ID
ip address 10.10.100.3/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback1
description Loopback for VTEP (PIP)
ip address 10.10.100.13/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24
!!!!! SITE2 !!!!!
!!!!!Spine-BGW-93240-1!!!!!
#enable features required for VxLAN EVPN
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
# underlay routing and interfaces
router ospf UNDERLAY
router-id 10.10.200.1
log-adjacency-changes detail
interface Ethernet1/54
description Link to leaf1
mtu 9216
ip address 192.168.2.1/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
interface loopback0
description Loopback for Router ID
ip address 10.10.200.1/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback1
description Loopback for VTEP (PIP)
ip address 10.10.200.11/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback254
description Loopback for PIM
ip address 10.254.254.254/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
# Multicast RP configuration
ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24
ip pim anycast-rp 10.254.254.254 10.10.200.1
ip pim anycast-rp 10.254.254.254 10.10.200.2
!!!!!Spine-BGW-9336-2!!!!!
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
router ospf UNDERLAY
router-id 10.10.200.2
log-adjacency-changes detail
interface Ethernet1/31
description Link to leaf1
mtu 9216
ip address 192.168.2.5/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
interface loopback0
description Loopback for Router ID
ip address 10.10.200.2/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback1
description Loopback for VTEP (PIP)
ip address 10.10.200.12/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback254
description Loopback for PIM
ip address 10.254.254.254/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24
ip pim anycast-rp 10.254.254.254 10.10.200.1
ip pim anycast-rp 10.254.254.254 10.10.200.2
!!!!!Leaf-93180-1!!!!!
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
router ospf UNDERLAY
router-id 10.10.200.3
log-adjacency-changes detail
interface Ethernet1/51
description Link to Spine1
mtu 9216
ip address 192.168.2.2/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/52
description Link to Spine1
mtu 9216
ip address 192.168.2.6/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
interface loopback0
description Loopback for Router ID
ip address 10.10.200.3/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback1
description Loopback for VTEP (PIP)
ip address 10.10.200.13/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
ip pim rp-address 10.254.254.254 group-list 239.239.239.0/24
At this step the underlay at each site is established; reachability between IPs (interface and loopbacks) is achieved.
Step 2 – VLAN, VRF, VNI and Site-internal Overlay (iBGP) Configuration
!!!!! SITE1 !!!!!
!!!!!Spine-BGW-9336-1!!!!!
# BGP L2vpn evpn control plane for site-#internal fabric
router bgp 65501
router-id 10.10.100.1
neighbor 10.10.100.3
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
!!!!!Spine-BGW-9336-2!!!!!
router bgp 65501
router-id 10.10.100.2
neighbor 10.10.100.3
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
!!!!!Leaf-93180-1!!!!!
# VLAN, VRF and VNI(virtual network #identifier)
vlan 1,20,30,40,300
vlan 20
name L2L3HostSegment
vn-segment 20020
vlan 30
name L2L3HostSegmentSite1only
vn-segment 20030
vlan 40
name L2L3HostSegmentSite2only
vn-segment 20040
vlan 300
name PROD-VRF
vn-segment 30300
vrf context PROD
vni 30300
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
fabric forwarding anycast-gateway-mac eeee.eeee.eeee
interface Vlan20
no shutdown
vrf member PROD
ip address 10.10.20.1/24 tag 12345
fabric forwarding mode anycast-gateway
interface Vlan30
no shutdown
vrf member PROD
ip address 10.10.30.1/24 tag 12345
fabric forwarding mode anycast-gateway
interface Vlan300
description PROD-VRF
no shutdown
mtu 9216
vrf member PROD
ip forward
# network virtual interface (nve) and vni to # multicast address mapping for replication
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 20020
mcast-group 239.239.239.20
member vni 20030
mcast-group 239.239.239.30
member vni 30300 associate-vrf
route-map FABRIC-REDIST-SUBNET permit 10
match tag 12345
# BGP L2vpn evpn control plane for site-#internal fabric
router bgp 65501
router-id 10.10.100.3
neighbor 10.10.100.1
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.10.100.2
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf PROD
address-family ipv4 unicast
advertise l2vpn evpn
redistribute direct route-map FABRIC-REDIST-SUBNET
maximum-paths ibgp 2
# RT and RD for each L2 segment
evpn
vni 20020 l2
rd auto
route-target import auto
route-target export auto
vni 20030 l2
rd auto
route-target import auto
route-target export auto
!!!!! SITE2 !!!!!
!!!!!Spine-BGW-93240-1!!!!!
# BGP L2vpn evpn control plane for site-#internal fabric
router bgp 65502
router-id 10.10.200.1
neighbor 10.10.200.3
remote-as 65502
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
!!!!!Spine-BGW-93240-2!!!!!
router bgp 65502
router-id 10.10.200.2
neighbor 10.10.200.3
remote-as 65502
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
!!!!!Leaf-93180-1!!!!!
# VLAN, VRF and VNI(virtual network #identifier)
vlan 1,20,30,40,300
vlan 20
name L2L3HostSegment
vn-segment 20020
vlan 30
name L2L3HostSegmentSite1only
vn-segment 20030
vlan 40
name L2L3HostSegmentSite2only
vn-segment 20040
vlan 300
name PROD-VRF
vn-segment 30300
vrf context PROD
vni 30300
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
fabric forwarding anycast-gateway-mac eeee.eeee.eeee
interface Vlan20
no shutdown
vrf member PROD
ip address 10.10.20.1/24 tag 12345
fabric forwarding mode anycast-gateway
interface Vlan40
no shutdown
vrf member PROD
ip address 10.10.40.1/24 tag 12345
fabric forwarding mode anycast-gateway
interface Vlan300
description PROD-VRF
no shutdown
mtu 9216
vrf member PROD
ip forward
# network virtual interface (nve) and vni to # multicast address mapping for replication
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 20020
mcast-group 239.239.239.20
member vni 20040
mcast-group 239.239.239.40
member vni 30300 associate-vrf
route-map FABRIC-REDIST-SUBNET permit 10
match tag 12345
# BGP L2vpn evpn control plane for site-#internal fabric
router bgp 65502
router-id 10.10.200.3
neighbor 10.10.200.1
remote-as 65502
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.10.200.2
remote-as 65502
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf PROD
address-family ipv4 unicast
advertise l2vpn evpn
redistribute direct route-map FABRIC-REDIST-SUBNET
maximum-paths ibgp 2
# RT and RD for each L2 segment
evpn
vni 20020 l2
rd auto
route-target import auto
route-target export auto
vni 20040 l2
rd auto
route-target import auto
route-target export auto
At this step the underlay and overlay at each site is established; reachability between IPs (interface and loopbacks) and hosts within the same site is achieved.
Step 3 – Site-external Overlay, Route Server, BGW Configuration
!!!!! SITE1 !!!!!
!!!!!Spine-BGW-9336-1!!!!!
# multi-site id and interfaces required for # multisite function
evpn multisite border-gateway 100
delay-restore time 30
interface Ethernet1/35
description Link to ISN
mtu 9216
ip address 192.168.3.1/30 tag 54321
no shutdown
evpn multisite dci-tracking
interface Ethernet1/1
description Link to Spine2
mtu 9216
ip address 192.168.1.9/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
evpn multisite fabric-tracking
interface loopback100
description Loopback for VTEP (VIP)
ip address 10.10.100.100/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
# VLAN, VRF and VNI(virtual network #identifier) needed on the BGW
vlan 1,20,30,40,300
vlan 20
name L2L3HostSegment
vn-segment 20020
vlan 30
name L2L3HostSegmentSite1only
vn-segment 20030
vlan 40
name L2L3HostSegmentSite2only
vn-segment 20040
vlan 300
name PROD-VRF
vn-segment 30300
route-map SITE-REDIST-DIRECT permit 10
match tag 54321
vrf context PROD
vni 30300
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
interface Vlan300
description PROD-VRF
no shutdown
mtu 9216
vrf member PROD
ip forward
# network virtual interface (nve), vni to #multicast address mapping for site-#internal replication, ingress replication #for site-external, VIP(L100) as BGW #interface
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
multisite border-gateway interface loopback100
member vni 20020
multisite ingress-replication
mcast-group 239.239.239.20
member vni 20030
multisite ingress-replication
mcast-group 239.239.239.30
member vni 30300 associate-vrf
#BGP ipv4 unicast address family to the #route server using the DCI interface and #L2vpn evpn address family for site-#internal and site-external (with RS)
router bgp 65501
address-family ipv4 unicast
redistribute direct route-map SITE-REDIST-DIRECT
maximum-paths 4
neighbor 10.10.100.2
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.10.150.100
remote-as 65510
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
rewrite-evpn-rt-asn
neighbor 192.168.3.2
remote-as 65510
update-source Ethernet1/35
address-family ipv4 unicast
# RT and RD for each L2 segment
evpn
vni 20020 l2
rd auto
route-target import auto
route-target export auto
vni 20030 l2
rd auto
route-target import auto
route-target export auto
!!!!!Spine-BGW-9336-2!!!!!
evpn multisite border-gateway 100
delay-restore time 30
interface Ethernet1/35
description Link to ISN
mtu 9216
ip address 192.168.3.5/30 tag 54321
no shutdown
evpn multisite dci-tracking
interface Ethernet1/1
description Link to Spine1
mtu 9216
ip address 192.168.1.10/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
evpn multisite fabric-tracking
interface loopback100
description Loopback for VTEP (VIP)
ip address 10.10.100.100/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
vlan 1,20,30,40,300
vlan 20
name L2L3HostSegment
vn-segment 20020
vlan 30
name L2L3HostSegmentSite1only
vn-segment 20030
vlan 40
name L2L3HostSegmentSite2only
vn-segment 20040
vlan 300
name PROD-VRF
vn-segment 30300
route-map SITE-REDIST-DIRECT permit 10
match tag 54321
vrf context PROD
vni 30300
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
interface Vlan300
description PROD-VRF
no shutdown
mtu 9216
vrf member PROD
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
multisite border-gateway interface loopback100
member vni 20020
multisite ingress-replication
mcast-group 239.239.239.20
member vni 20030
multisite ingress-replication
mcast-group 239.239.239.30
member vni 30300 associate-vrf
router bgp 65501
address-family ipv4 unicast
redistribute direct route-map SITE-REDIST-DIRECT
maximum-paths 4
neighbor 10.10.100.2
remote-as 65501
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.10.150.100
remote-as 65510
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
rewrite-evpn-rt-asn
neighbor 192.168.3.6
remote-as 65510
update-source Ethernet1/35
address-family ipv4 unicast
evpn
vni 20020 l2
rd auto
route-target import auto
route-target export auto
vni 20030 l2
rd auto
route-target import auto
route-target export auto
!!!!! SITE2 !!!!!
!!!!!Spine-BGW-93240-1!!!!!
# multi-site id and interfaces required for # multisite function
evpn multisite border-gateway 200
delay-restore time 30
interface Ethernet1/55
description Link to ISN
mtu 9216
ip address 192.168.4.1/30 tag 54321
no shutdown
evpn multisite dci-tracking
interface Ethernet1/1
description Link to Spine2
mtu 9216
ip address 192.168.2.9/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
evpn multisite fabric-tracking
interface loopback100
description Loopback for VTEP (VIP)
ip address 10.10.200.100/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
# VLAN, VRF and VNI(virtual network #identifier) needed on the BGW
vlan 1,20,30,40,300
vlan 20
name L2L3HostSegment
vn-segment 20020
vlan 30
name L2L3HostSegmentSite1only
vn-segment 20030
vlan 40
name L2L3HostSegmentSite2only
vn-segment 20040
vlan 300
name PROD-VRF
vn-segment 30300
route-map SITE-REDIST-DIRECT permit 10
match tag 54321
vrf context PROD
vni 30300
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
interface Vlan300
description PROD-VRF
no shutdown
mtu 9216
vrf member PROD
ip forward
# network virtual interface (nve), vni to #multicast address mapping for site-#internal replication, ingress replication #for site-external, VIP(L100) as BGW #interface
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
multisite border-gateway interface loopback100
member vni 20020
multisite ingress-replication
mcast-group 239.239.239.20
member vni 20040
multisite ingress-replication
mcast-group 239.239.239.40
member vni 30300 associate-vrf
#BGP ipv4 unicast address family to the #route server using the DCI interface and #L2vpn evpn address family for site-#internal and site-external (with RS)
router bgp 65502
address-family ipv4 unicast
redistribute direct route-map SITE-REDIST-DIRECT
maximum-paths 4
neighbor 10.10.200.2
remote-as 65502
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.10.150.100
remote-as 65510
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
rewrite-evpn-rt-asn
neighbor 192.168.4.2
remote-as 65510
update-source Ethernet1/55
address-family ipv4 unicast
# RT and RD for each L2 segment
evpn
vni 20020 l2
rd auto
route-target import auto
route-target export auto
vni 20040 l2
rd auto
route-target import auto
route-target export auto
!!!!!Spine-BGW-93240-2!!!!!
evpn multisite border-gateway 200
delay-restore time 30
interface Ethernet1/55
description Link to ISN
mtu 9216
ip address 192.168.4.5/30 tag 54321
no shutdown
evpn multisite dci-tracking
interface Ethernet1/1
description Link to Spine1
mtu 9216
ip address 192.168.2.10/30
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
evpn multisite fabric-tracking
interface loopback100
description Loopback for VTEP (VIP)
ip address 10.10.200.100/32 tag 54321
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
vlan 1,20,30,40,300
vlan 20
name L2L3HostSegment
vn-segment 20020
vlan 30
name L2L3HostSegmentSite1only
vn-segment 20030
vlan 40
name L2L3HostSegmentSite2only
vn-segment 20040
vlan 300
name PROD-VRF
vn-segment 30300
route-map SITE-REDIST-DIRECT permit 10
match tag 54321
vrf context PROD
vni 30300
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto
route-target both auto evpn
interface Vlan300
description PROD-VRF
no shutdown
mtu 9216
vrf member PROD
ip forward
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
multisite border-gateway interface loopback100
member vni 20020
multisite ingress-replication
mcast-group 239.239.239.20
member vni 20040
multisite ingress-replication
mcast-group 239.239.239.40
member vni 30300 associate-vrf
router bgp 65502
address-family ipv4 unicast
redistribute direct route-map SITE-REDIST-DIRECT
maximum-paths 4
neighbor 10.10.200.2
remote-as 65502
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.10.150.100
remote-as 65510
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
rewrite-evpn-rt-asn
neighbor 192.168.4.6
remote-as 65510
update-source Ethernet1/55
address-family ipv4 unicast
evpn
vni 20020 l2
rd auto
route-target import auto
route-target export auto
vni 20040 l2
rd auto
route-target import auto
route-target export auto
!!!!! ISN (DCI) !!!!!
!!!!!Route Server!!!!!
#features needed on the route server
nv overlay evpn
feature ospf
feature bgp
#ip address on DCI interfaces facing to each sites BGWs
interface Ethernet1/51
description Link to Spine1-site2
mtu 9216
ip address 192.168.4.2/30
no shutdown
interface Ethernet1/52
description Link to Spine2-site2
mtu 9216
ip address 192.168.4.6/30
no shutdown
interface Ethernet1/53
description Link to Spine1-site1
mtu 9216
ip address 192.168.3.2/30
no shutdown
interface Ethernet1/54
description Link to Spine2-site1
mtu 9216
ip address 192.168.3.6/30
no shutdown
interface loopback0
ip address 10.10.150.100/32
#route map to keep the next hop unchanged when advertising BGP routes from one site to #the other
route-map NEXT-HOP-UNCHANGED permit 10
set ip next-hop unchanged
#BGP config
#unicast address family with the DCI interfaces
#l2vpn evpn address family with loop backs of BGWs.
router bgp 65510
address-family ipv4 unicast
network 10.10.150.100/32
address-family l2vpn evpn
retain route-target all
neighbor 10.10.100.1
remote-as 65501
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
route-map NEXT-HOP-UNCHANGED out
rewrite-evpn-rt-asn
neighbor 10.10.100.2
remote-as 65501
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
route-map NEXT-HOP-UNCHANGED out
rewrite-evpn-rt-asn
neighbor 10.10.200.1
remote-as 65502
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
route-map NEXT-HOP-UNCHANGED out
rewrite-evpn-rt-asn
neighbor 10.10.200.2
remote-as 65502
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
route-map NEXT-HOP-UNCHANGED out
rewrite-evpn-rt-asn
neighbor 192.168.3.1
remote-as 65501
address-family ipv4 unicast
neighbor 192.168.3.5
remote-as 65501
address-family ipv4 unicast
neighbor 192.168.4.1
remote-as 65502
address-family ipv4 unicast
neighbor 192.168.4.5
remote-as 65502
address-family ipv4 unicast
At this step the reachability between hosts on different sites is established. We confirm by doing ping between the hosts.
Ping test between Host-10.10.20.100 (SITE1) & Host-10.10.20.200 (SITE2)
Ping test between Host-10.10.20.100 (SITE1) & Host-10.10.40.200 (SITE2)
Ping test between Host-10.10.30.100 (SITE1) & Host-10.10.40.200 (SITE2)
Ping test between Host-10.10.30.100 (SITE1) & Host-10.10.20.200 (SITE2)
Show output for L2 and L3 extensions
Show commands used to troubleshoot if necessary.
o Sh mac address-table address xxxx.xxxx.xxxx
o Sh system internal l2fm l2dbg macdb address xxxx.xxxx.xxxx vlan 10
o Sh sys inter l2fm event-hist deb | in xxxx.xxxx.xxxx
o Sh ip arp vrf xxxxx
o Sh forwarding vrf VRF03 adjacency
o Sh l2route evpn mac evi 20 (vlan-id)
o Sh l2route evpn mac-ip evi 20 (vlan-id)
o Sh system internal l2rib event-history mac
o Sh system internal l2rib event-history mac-ip
o Sh bgp l2vpn evpn vni-id xxxxx route-type 2
o Sh bgp l2vpn evpn vni-id xxxxx (vni-id)
o Sh bgp l2vpn evpn xxxx.xxxx.xxxx
o Sh bgp internal event-history event | in xxxx.xxxx.xxxx
o Sh nve multsite dci-links
o Sh nve interface nve 1 detail
o Sh nve peers
o Sh ip route 10.10.40.200/32 vrf xxxxx
Hi Belete,
Thanks for the great and informative content on the very important topic.
Let’s say your customer is in Finance or Healthcare sector. Of course regulations might be different for each sector but let’s assumes we need to encrypt the data.
May I ask you what would be the security (mostly I’m mentioning data encryption when I say “security” here 🙂 ) options/solutions that you could consider on these kind of structures? I saw there is MACSec over VXLAN implementation on some vendors. Would you go with similar solution?
Thanks in advance for your thoughts.