ACI Multi-site Intersite L3Out

Leave a Comment

Intersite L3out Overview

Starting from release 4.2(1) ACI allows to configure an intersite L3Out on a multi-site ACI fabric. This feature enables for an endpoint in a site to send traffic to Layer 3 resources accessible via a remote L3Out connection. Before release 4.2(1) endpoints deployed in a given site can communicate with the external network domain only through a local L3Out connection.

Inter-site L3Out also support transit routing across sites (L3Out to L3Out) and L3Out in one site to serve as a backup for other sites.

Traffic will be directly encapsulated to the TEP of the remote BL nodes. External prefixes are exchanged across sites (ISN) via MP-BGP VPNV4/VPNv6 sessions between spines.

In this article we will learn about the following topics:

  • Inter-site L3Out uses cases
  • How to configure inter-site L3Out

Inter-Site L3Out uses case 1

An endpoint in site-1 accessing resources outside of the DC through an L3out in site-2

Figure 1.

  1. An endpoint in site-1 accessing resources outside of the DC through an L3out in site-2
  2. An endpoint in site-1 accessing data center resources in site-2 connected through L3Out like, mainframe, virtualization environment (NSX-T), container environment (OCP), FW, LB…

Inter-Site L3Out uses case 2

Figure 2.

  1. An endpoint in site-1 use local L3Out under normal circumstances.
  2. An L3out in site-2 is used as a backup for endpoints in site-1 during local L3Out failure and vise versa i.e. an endpoint in site-2 will use site-1 L3Out during local L3Out failure.

Inter-Site L3Out uses case 3

Figure 3.

  • Intersite L3Out used as a transit route. Layer 3 users / systems connected to site-1 accessing a layer 3 resources reachable only through site-2 using the remote L3Out and vise versa.
  • Some layer 3 resources are reachable only through one site. Like partner access when partner is connected to only one of the data center sites.
  • Communication between resources like mainframe in site-1 and mainframe in site-2 or container (OCP) in site-1 and container (OCP) in site-2

Lab Setup and Topology Diagram

Figure 4.

  1. Lab setup is on ACI 5.2(7f), ND 2.2(2d)& NDO 4.1.1i
  2. VM and L3Out are in VRF1
  3. VM is in BD1, AP1 and EPG1
  4. L3Out is the provider and EPG1 is the consumer of the contract
  5. VM -> 192.168.0.10 & ext host is -> 192.168.100.10

In this example, we will use the intersite L3Out feature to provide L3 connectivity within our multi-site ACI fabric. An endpoint (VM-192.168.0.10) in site-2 will use site-1’s L3Out to access an external endpoint (192.168.100.10) learnt in site-1.

ACI Multi-site Inter-site L3Out Configuration Steps

  1. Configure External Tunnel Endpoint Pool (ETEP)
  2. Create Tenant and Schema
  3. Create extended template with VRF, Contract and Filter
  4. Create L3Out for SITE1
  5. Create site specific template for SITE1
  6. Create site specific template for SITE2
  7. Apply contract between EPG1 and ext EPG
  8. Verification

1. Configure External Tunnel Endpoint Pool (ETEP)

Intersite L3Out needs a separate external TEP (ETEP) pool for each site that is part of the Multi-Site domain. The ETEP pools need to be routable and configuration is managed by the Nexus Dashboard Orchestrator. The boarder leaf is assigned an additional TEP address from the ETEP pool for establishing leaf-to-leaf VXLAN tunnel between sites.

Infrastructure -> Site Connectivity -> Configure

Figure 5a. Adding External TEP (ETEP) pool

Choose the site and pod -> under External TEP Pools click ‘Add TEP pool‘

Figure 5b. Adding External TEP (ETEP) pool

Enter the External TEP pool ip range -> Click ‘ok’ -> click ‘Deploy’ on the site connectivity window

Figure 5c. Adding external TEP (ETEP) pool

2. Create Tenant and Schema

Figure 6. Create tenant from NDO

Figure 7. Create Schema from NDO

3. Create extended template with VRF, Contract and Filter

Under a schema:

  1. Click ‘Add New Template’
  2. Choose ‘ACI Multi-Cloud’ template type
  3. Name the template and associate with the right tenant
  4. Associate the template with the right site(s)
  5. Create objects

From the schema window click ‘Add New Template’ and choose ‘ ACI Multi-Cloud’ template type

Figure 8a. Create template

Name the template and associate with the right tenant

Figure 8b. Create template and associate with the right tenant

Associate the template with the right site(s)

Figure 8c. Attaching template to site(s)

Create objects needed for the template

Figure 8d. Add objects to template

Figure 8e. Objects created on extended template

4 . Create L3Out for SITE1

Using NDO, create L3Out for SITE1. With NDO release 4.1(1), the entire configuration of L3Outs can be done on NDO. For release before 4.1(1) , create the L3Out object and complete the config in APIC.

Figure 9a. Create L3Out for SITE 1

Configuring the L3Out properties

Figure 9b. Configure L3Out properties

5 . Create Site Specific Template for SITE1

SITE1 only needs external EPG for this lab test

Figure 10. Create and deploy template specific to SITE1

6 . Create Site Specific Template for SITE2

SITE2 needs bridge domain, application profile and EPG

Figure 11. SITE2 specific template

7. Apply contract between EPG1 and ext EPG

Applying contract is the step needed to establish communication between the sites for EPG1 and ext-EPG-OSPF. This step creates shadow objects in each sites

8. Verification

Verification is by testing reachability from VM in EPG1 of SITE2 to host connected to SITE1 using L3Out

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739609.html#IntroducingInterSiteL3Outfunctio

About

Leave a Comment

Your email address will not be published. Required fields are marked *