1. Introduction
ACI (Application Centric Infrastructure) is a Software Defined Data Center (SDDC) solution from Cisco. It debuts in 2014. It is based on a web scale nexus 9K hardware in a spine and leaf architecture. It is managed and controlled by a cluster of Application Policy Infrastructure Controllers (APICs). It is highly scalable that can span multiple geo locations with multi-pod and multi-site architecture. NDO (Nexus Dashboard Orchestrator) running as a service on Nexus Dashboard (ND), a compute platform for hosting and consuming all Data Center services in one place, is used to manage ACI fabrics distributed across multiple geo locations. Each fabric, with one or more PODs, represent a region in cloud availability terminology. The PODs within the fabric represent the availability zone.
ACI can be accessed and managed using UI, API and CLI. The open API makes the fabric highly programmable and suitable to provision the infrastructure as a service. One can use CI/CD pipe lines, and automation tools like Ansible, Terraform, Python, POSTMAN, and others to provision the infrastructure for easier and error free service deployment.
Leveraging and consuming apps from Cisco DC App Center improves the experience of network provisioning, operational management, continuous assurance, compliance checks, and real-time telemetry data analysis.
ACI adopt allow-list based security architecture where explicit definition of traffic flow need to be defined for traffic to flow between end point groups (EPGs), a construct used to group end points with similar security and policy requirements. The security architecture plays a foundational role toward Zero Trust architecture and Micro Segmentation initiatives in data center.
This blog post focuses on ACI contract as a tool to achieve Zero Trust and Micro Segmentation in ACI data center and uses ACI 5.2(x) for any reference to the UI and for outputs used to demonstrate features.
2. ACI Contract
In ACI, by default unicast communication between EPGs (Endpoint Groups) or ESGs (Endpoint Security Groups) are not allowed. Unicast traffic between end points with in EPG or ESG is allowed. BUM traffic such as Broadcast, Unknown unicast and Multicast Protocols, and protocols like ARP, DHCP, OSPF, EIGRP, PIM are always allowed.
EPG is a collection of end points with similar policy requirements residing in the same bridge domain ( a layer 2 boundary). A Bridge Domain (BD) can have one or more EPGs. EPG is equivalent to Security Zones with in BD. Each EPG is mapped to a number called “class-id” or “pcTag” in hardware.
ESG is a collection of end points with similar policy requirements residing in the same VRF ( a layer 3 boundary). A VRF can have one or more ESGs. ESG is equivalent to Security Zones with in VRF. Each ESG is mapped to a number called “class-id” or “pcTag” in hardware.
Contract is a filtering rules based on EPGs or ESGs (not on IP addresses). The action can be permit, deny or redirect to a L4L7 device.
Every EPG and ESG is mapped to class-id in hardware for policy enforcement. Policy enforcement is at ingress if the destination end point is known and flagged with policy applied at source flag. If the destination is not known at the ingress leaf the policy enforcement is done at the egress leaf.
Cisco ACI security architecture is based on allow-list where explicit definition of traffic flow need to be defined. Contract is a foundation for ACI security architecture where communication between EPGs|ESGs is defined. The contract relationship is between ESGs, EPGs (regular or uSig EPGs) or within EPG|ESG for intra-EPG contract.
Contracts is also used to:
- Define route leaking between VRFs in the same tenants
- Define route leaking between VRFs in different tenants
- QOS class assignment
- Policy based redirect to service devices/nodes
- WAN SLA policy (For Cisco SD-WAN integration)
Contract is applied in a provider / consumer relationship. The provider provides the service and the consumer consumes the provided service. EPG 2 is providing web service (port 80 and 443) and EPG 1 is consuming the web service on port 80 and 443. The defined contract is applied to EPG 1 as consumed and EPG 2 as provided.
3. EPG/ESG classification
An endpoint can belong to one EPG. Physical, virtual, and container endpoints can coexist in the same EPG. How to define which EPG an endpoint belongs to is based on the EPG type, as described below:
- EPG is classified based on static binding of leaf interface and VLAN ID or VxLAN ID and/or dynamic binding of the EPG to VMM domain (VMM integration will identify the leaf, port, and encapsulation VLAN)
- L3out EPG is based on the longest prefix match of the IP subnet configured under the extEPG, not based on an L3out the endpoint is learned from.
Example:
The subnet under L3out-1 extEPG is 10.0.0.0/8 and the subnet under L3out-2 extEPG is 10.100.0.0/16. An endpoint from L3out-1 with IP address 10.100.10.10 will be classified as member of extEPG from L3out-2 not ExtEPG from L3out-1 and a contract applied to extEPG from L3out-2 will be applied. This will result in an unintended policy enforcement. Subnets assigned to extEPG should be as specific as possible to match the subnets behind the L3out.
There are some specific class-ids used to represent the L3out extEPG when the extEPG subnet is defined as 0.0.0.0/0. If the traffic is from source L3out to destination EPG the class id for the extEPG is set with that of the VRF . If traffic is from source EPG to destination L3out, the class id for the extEPG is set to 15.
- uSeg-EPG is based on IP, MAC and VM attributes or a combination of IP, MAC, and VM attributes.
- ESG is based on selectors (Policy Tag, EPG, IP subnet). Policy Tags are objects that can be attached to ACI objects as a label. BD subnet, IP Address, MAC Address, and Static endpoints can use policy tag for ESG classification.
4. Contract Structure
4.1 Standard Contract
A contract is a policy construct used to define communication between EPGs|ESGs. Contract contains one or more subject. A subject is a construct contained within a contract and references a filter. A subject contains one or more filters. A filter contains one or more filter entries. A filter entry is a rule specifying fields such as the TCP port and protocol type.
The steps to create and use a contract are:
- Create filter
- Create contract
- Add contracts to provider and consumer EPGs|ESGs
- Tenant > Contracts > Filters
Filter and filter entry – a filter is collection of filter entries. Unspecified in the filter entry means any. if filter entry is configured with the “stateful” flag, ACI programs the filters in the reverse direction to allow packets only if they have the ACK bit set. Stateful option is disabled by default. The stateful option doesn’t work with compression
2. Tenant > Contracts > Standard
The scope of a service contract between two or more EPGs is important especially when the contract is used for inter-VRF and/or inter-tenant contracts.
‘Apply Both Directions’ – sets the contract filter to apply on both ingress and egress traffic directions.
‘Reverse Filter Ports’ – enables the filter to apply on both ingress and egress traffic.
Filter chain contains filters with name, directives, action and priority. Directives can be log or policy compression. for action permit or deny are the options. Priority is only used for deny action.
A subject is a collection of filters. a subject can be associated with service graph.
3. Add contracts to provider and consumer EPGs|ESGs
Fig 16 – Apply contract to EPG|ESG
4.2 Contract inheritance
Contract inheritance Establishes a relationship between EPGs/ESGs, so that one EPG/ESG can inherit the contract relations of other EPGs/ESGs. When new contract relations are added to the higher EPG/ESG (the master EPG/ESG), those with inheritance relation will automatically get the same contract associations. EPG/ESGs must be on the same tenant. Contract inheritance is not applied to vzAny (vzAny can’t refer to a master EPG or be a master EPG). Contract inheritance can simplify the configuration task, but it does not reduce TCAM resource consumption.
The above screenshot shows DeliAP-EPG1 using DeliAb-EPG2 as a contract master and inheriting contract relationship from the master. The EPG inheriting the contract can also have EPG specific contracts in addition to the inherited contracts.
4.3 Contract labels
Labels are a configuration option that makes it possible to select which EPGs|ESGs can consume or provide contracts with other EPGs|ESGs. By using Labels to ”group” those EPGs|ESGs can be potentially simplify contract configuration.
Label configurations can be done per EPG or per contract.
- Per EPG configurations: EPG labels or subject labels
- per contract configurations: Contract labels or subject labels
Below is an example of subject label on per contract configuration. In this example, EPG1 and EGP2 that have same Label “Green” can talk each other on port SSH, and EPG1 and EPG3 that have same Label “Blue” can talk to each other on http/https.
Adding label to EPG1, EGP2 and EPG3. ADD “Green” label to EPG1, for ssh communication and “Blue”, for http/https communication. Add “Blue” label to EGP2 and “Green” label EPG3.
Output of a ‘show zoning-rule scope 2359296’ – contract relationship between EPGs with in VRF with segment ID (scope) 2359296.
- EPG1 (pcTag – 49158) can communicate on port 80/443 (http/https) with EGP2(pcTag – 49165) using Blue subject label.
- EPG1 (pcTag – 49158) can communicate on port 22 (ssh) with EPG3(pcTag – 16394) using Green subject label.
5. Contract Types
Contract can be applied with different level of granularity. The order of the contract or policy enforcement from macro to micro is:
1. VRF level ‘Policy Control Enforcement Preference’ selected as Unenforced: in this mode all communication within the VRF is allowed, no contract is needed.
Fig 22 – VRF policy enforcement
2. Contract between EPGs|ESGs in different tenants: an inter-tenant contract is used to policy control between EPGs|ESGs in different tenants. Inter-tenant configures both policy filtering and route leaking. The contract scope must be set correctly. Global scope is the most common and appropriate scope for inter-tenant contracts.
The configuration of inter-tenant contract depends on whether the contracts are:
- intra-VRF or inter-VRF (discussed next)
- between Common tenant and user tenant
- between user tenants
Contracts are not visible across tenants by default with the exception of common tenant contracts. For a contract to be consumed or provided by EPG|ESG the contract object has to be visible by EPG|ESG.
For inter-tenant contract between common and user tenant, the configuration steps are the same as standard contract if the contract is defined in common tenant. If the contract is defined on the user tenant, the contract need to be exported. The common tenant will see the exported contract as a “contract interface”.
For inter-tenant contract between user tenants, the provider tenant need to export the contract and the consumer tenant will see it as a “contract interface” unless the contract used is defined in common tenant.
3. Contract between EPGs|ESGs in different VRFs: an inter-VRF contract is used for communication between EPGs|ESGs in different VRFs. Inter-VRF contract configures both policy filtering and route leaking and it needs more steps than the regular contracts.
- Set the contract scope to either Tenant or Global
- The scope of the subnet in the consumer BD must be set to ‘Shared between VRFs’
- In the Provider EPG you must add a Subnet more specific than or equal to the Provider BD: The scope of the subnet must be set to ‘ Shared between VRFs’ and select ‘No Default SVI Gateway’ .
Policy Enforcement
- EPG to EPG – at the consumer VRF
- EPG to L3out(as a provider) – at the consumer VRF
- EPG to L3out(as a consumer) – at the ingress leaf
- L3out to L3out – at the ingress leaf
4. vzAny: vzAny represents a collection of EPGs|ESGs|extEPGs that belongs to the same VRF so a contract applied at the ‘EPG|ESG Collection for VRF’ level is applied to all EPGs|ESGs|extEPGs within the VRF.
Fig 23 – vzAny contract
5. Preferred Group: preferred group is used to allow unrestricted communication/access between a subset of EPGs|ESGs in a VRF that are member of the preferred group. Communication between members of the preferred group and non-members uses the regular contract application.
Steps to enable preferred group
- Enable preferred group at the VRF level
Fig 24 – Preferred group
2. Add EPG|ESG to a preferred group member by changing ‘preferred Group Member’ to include.
6. Regular Contracts: Communication between endpoints in EPG|ESG|extEPG requires a standard contract. Communication between endpoints with in EPG|ESG|extEPG is allowed without contract.
Fig 26 – Standard contract
7. uSeg EPG: uSeg EPG groups endpoints based on their attributes. uSeg-EPG is equivalent to a regular EPG in many ways but classification is based on attributes, it doesn’t have encapsulation/port dependancy. The endpoint must be first known to a regular (base) EPG through encapsulation. Communication between uSeg-EPGs or uSeg-EPGs and EPG needs contract even if both are part of the same base EPG. For bare metal servers IP and MAC are the attributes to group endpoints into uSeg EPG. For VMs with VMM domain the attributes listed below can be used.
Fig 27 – Micro seg EPG attributes
7. Intra-EPG isolation and contract: Intra-EPG contract is used to restrict communication with in EPG|ESG|extEPG|uSeg-EPG for traffics only allowed by the contract. The contract defines on which protocol and ports the hosts of a given EPG|ESG|extEPG|uSeg-EPG can communicate
Fig 28 – Intra-EPG contract
Intra-EPG isolation: Enabling intra-EPG isolation enforces that communication between endpoints in an EPG to deny. This is equivalent of PVLANs (private VLANs) in a non ACI network. If layer two is extended to non ACI network like vDS in VMware PVLAN need to be configured. In case of VMM integration, ACI programs PVLAN on the port-group for the EPG. If there is an intermediate switch between the ACI leaf and a vDS, you must configure PVLAN on the intermediate switch.
If communication between EPGs|ESG|extEPG that have intra-EPG isolation enabled is needed you need to also configure Proxy-ARP.
Fig 30 – Intra-EPG isolation
6. Policy Enforcement
Zoning rules get programmed in the TCAM (Ternary Content Addressable Memory) once the contract is associated with a consumer and a provider EPG if the consumer or provider EPG present on the leaf. Zoning rules are per VRF, and each entry defines an action based on the source EPG, the destination EPG, and filter matching.
Fig 32 – Each EPG has a unique ID called a class ID or pcTag and each VRF has a unique ID called a VRF scope or segment ID.
Fig 33 – Class ID and Scope UI
Contract policy is applied on leaf switches, not spine switches. The following table summarize policy enforcement direction based on different criteria.
The VXLAN headers has a flag that indicates whether policy enforcement on the packet has been applied or not. This is done via the “policy applied bit.”
