Explore the Cisco APIC User Interface

You will explore the APIC user interface, the primary management tool for the Cisco ACI
environment.
Step 1
Connect to your StudentPC. Log in as student with password 1234QWer.
Step 2
On your StudentPC, open a browser and go to [Link] The hostname apic will be resolved
to the IP address [Link]. Accept the security warning or create a security exception to
access the user interface. Log in as admin with password 1234QWer. Then close the What's
New in 5.2.1g page.
Answer

After login, close the What's New in 5.2.1g page:

Secure HTTP (HTTPS) is required to access the Cisco APIC user interface using default settings.
Step 3
Examine the top-most portion of the user interface, also referred to as the menu bar (System,
Tenants, Fabric, Virtual Networking, Admin, Operations, Apps, Integrations).
Answer

You will see some alerts, among others about an insufficient number of in-service controllers.
The minimum recommended setup for the Cisco APIC is a cluster consisting of three controllers.
In this lab environment, you will use a cluster with a single Cisco APIC server. You may see the
alerts by clicking the bell symbol in the top-right corner.
Step 4
Select the Tenants menu and choose a predefined tenant, common. A submenu bar will appear
below the menu bar. The Navigation pane displays on the left side of the Cisco APIC user
interface below the submenu bar. This pane provides centralized navigation to all elements of
the submenu category. If you choose a component in the Navigation pane, its objects will
display in the work pane on the right side of the Cisco APIC user interface. The work pane
displays details about the component that is selected in the Navigation pane.
Answer

A tenant is a logical container for application policies that enable an administrator to exercise
domain-based access control. A tenant represents a unit of isolation from a policy perspective,
such as a customer in a service provider setting, an organization or domain in an enterprise
setting, or just a convenient grouping of policies. By default, there are three preconfigured
tenants in Cisco ACI: common, mgmt and infra.
Step 5
Briefly browse the remaining menus:
 • System: The menu for Cisco APIC information, systemwide setting, fabric-wide view for
fault, event and audit log, and so on.
• Fabric: Cisco ACI inventory and configuration point for leaf and spine nodes—includes
but is not limited to port channel and vPC configurations.
• Virtual Networking: Configuration menu for VM Manager integration, such as VMware
vCenter, Microsoft System Center Virtual Machine Manager (SCVMM) or KVM.
• Admin: Menu for controlling the Operation, Administration, and Maintenance (OAM)
aspects, such as authentication, authorization, and accounting (AAA), Syslog, Simple
Network Management Protocol (SNMP), software upgrade, configuration backup,
techsupport.
• Operations: Menu for visibility, troubleshooting, and capacity profiling.
• Apps: Menu for AppCenter applications provided by Cisco or a third-party vendor to run
on Cisco APIC that provide useful features for visibility, troubleshooting, and so on.
• Integrations: Menu for managing integrations with external Cisco Device Managers,
such as Cisco Unified Computing System (UCS) Manager and Viptela vManage network
management system (NMS).
Examine the Discovered Fabric
Your environment has been prepared for you by the lab automation system. You will examine
the fabric and validate its state. You will use the same tools that you would employ when
setting up the fabric from scratch.
Step 6
Go to Fabric > Inventory > Topology and choose the Topology tab in the work pane to view your
environment.
Answer

Your Cisco APIC cluster consists of a single Cisco APIC server, dual-homed to both leaf switches.
Each leaf is connected to a spine switch. You will see the same topology view if you go to Fabric
> Inventory > Pod 1 and select the Topology tab from the work pane.
Note
This course's convention uses > to denote consecutive menu items in a navigation path.
Step 7
Expand the Fabric > Inventory > Fabric Membership and examine the switches listed in the
Registered Nodes tab. If necessary, scroll to the right to see all information.
Answer

Your topology consists of three physical nodes (two leaves and one spine) and no virtual nodes.
The node names and node IDs have been assigned to the switches during registration. The node
IDs and IP addresses shown in the outputs will differ from the ones you will see in your fabric.
Note
Cisco ACI forwarding is based on a VXLAN overlay. Leaf nodes are virtual tunnel endpoints
(VTEPs), which, in Cisco ACI terminology, are known as physical tunnel endpoints (PTEPs). The
TEP addresses are displayed in the IP column. The TEP address pool [Link]/16 has been
configured on the Cisco APIC using the initial setup dialog. The Cisco APIC assigns the TEP
addresses to the fabric switches through DHCP, so the infrastructure IP addresses in your fabric
will be different from the figure.
Step 8
Go back to the fabric topology page Fabric > Inventory > Topology > Topology and double-click
each device to verify its connections to other fabric elements.
Answer
The figure below shows the output after selecting leaf-a. Close the sub-page by clicking the X
symbol.

The LLDP is responsible for discovering directly adjacent neighbors. Once a switch node is
discovered via LLDP and registration (node ID and name assignment) is performed, APIC assigns
a TEP IP through DHCP and Cisco APIC communicates with the switch over the TEP IP using a
messaging system called Intra-Fabric Messaging (IFM).
Note
You may see also other connections.
Step 9
Go to Fabric > Inventory > Pod 1, select leaf-a, and examine the information shown in the
General tab.
Answer

The information available in this page includes model type, serial number, management IP
addresses (missing at this point), and other chassis information.
Step 10
Click the Interface tab and hover the cursor over some interfaces to investigate their properties.
Answer

Note
Your output may be different from the above example.
Step 11
Expand the menu of a switch in the navigation pane, and skim through the available inventory
information, such as the various interface types.
Answer

Step 12
Use PuTTY to connect through secure shell (SSH) to the Cisco APIC. Enter apic as the hostname.
Log in as admin with password 1234QWer. Run the show firmware upgrade status command to
verify the software versions on the fabric switches.
Answer
apic1# show firmware upgrade status
 Pod Node Current-Firmware Target-Firmware Status Upgrade-Progress(%) Download-
Status Download-Progress(%)
---------- ---------- -------------------- -------------
1 1 apic-5.2(1g) success 100 - -
1 101 n9000-15.2(1g) n9000-15.2(1g) success 100 downloaded 100
1 102 n9000-15.2(1g) n9000-15.2(1g) success 100 downloaded 100
1 201 n9000-15.2(1g) n9000-15.2(1g) success 100 downloaded 100
Although your output may differ from the example above, depending on the upgrade history,
the switch software versions must be identical and consistent with the Cisco APIC software
release. You can use other options of the show firmware command to examine other firmware-
related information. To obtain context-sensitive help and command auto-completion, press the
Tab key twice or type "?" like on Cisco Nexus Operating System (NX-OS) standalone devices.
Note
Most of the information discovered through the CLI can also be obtained in the Cisco APIC user
interface. For example, to collect the firmware information, go to Admin > Firmware.
The default shell on the Cisco APIC is called Cisco NX-OS style CLI and resembles the Cisco NX-
OS configuration. Before its implementation, ACI could be configured only through REST API or
Cisco APIC user interface (which uses REST API in background). Although APIC user interface is
still the most popular configuration method, users could opt to Cisco NX-OS style CLI. However,
users cannot mix Cisco NX-OS style CLI and REST API (or user interface). “show” commands and
other useful non-configuration commands in the Cisco NX-OS style CLI can still be used even
when the configuration is performed through REST API (or user interface). When executing
Linux commands, it is recommended to type “bash” to change your shell. Typing “exit” lets you
go back to Cisco NX-OS style CLI shell.
Step 13
On Cisco APIC, run the acidiag -h command to view the available Cisco ACI diagnostics options.
Answer
apic1# acidiag -h
usage: acidiag [-h] [-v]

{avread,fnvread,fnvreadex,fnvreadall,rvread,rvreadle,crashsuspecttracker,bootother,bootcurr,
dbgtoken,version,platform,refreshav,touch,dmelogdecode,journal,logs,oob,scheduler,cleanup,
cluster,hwcheck,validateimage,validatenginxconf,preservelogs,verifyapic,bond0test,linkflap,run
,installer,start,stop,restart,reboot,drrmode,vapicjoin,gluster,dmestack,dmecore}
...

positional arguments:

{avread,fnvread,fnvreadex,fnvreadall,rvread,rvreadle,crashsuspecttracker,bootother,bootcurr,
dbgtoken,version,platform,refreshav,touch,dmelogdecode,journal,logs,oob,scheduler,cleanup,
cluster,hwcheck,validateimage,validatenginxconf,preservelogs,verifyapic,bond0test,linkflap,run
,installer,start,stop,restart,reboot,drrmode,vapicjoin,gluster,dmestack,dmecore}
sub-command help
avread read appliance vector
 fnvread read fabric node vector
fnvreadex read fabric node vector (extended mode)
fnvreadall read fabric node vector readall
rvread read replica vector
rvreadle read replica leader summary
crashsuspecttracker
read crash suspect tracker state
bootother on next boot, boot other Linux Partition, and display
updated /etc/[Link]
bootcurr on next boot, boot current Linux Partition, and
display updated /etc/[Link]
dbgtoken show debug token
version show ISO version
platform show platform
refreshav refresh AV for standalone APIC cluster to initiate
fabric recovery
touch touch special files
dmelogdecode DME log decode
journal Contents of journal logs
logs show log history
oob oob options
scheduler scheduler
cleanup fs cleanup utility
cluster cluster command options
hwcheck Quick check of APIC Hardware
validateimage validate image
validatenginxconf validate nginx conf
preservelogs stash away logs in preparation for hard reboot
verifyapic run apic installation verify command
bond0test ==SUPPRESS==
linkflap flap a link
run run specific commands and capture output
installer installer
start start a service
stop stop a service
restart restart a service
reboot reboot
drrmode drrmode options
vapicjoin join existing vapic cluster
gluster gluster admin and health tool

optional arguments:
-h, --help show this help message and exit
-v, --verbose verbose
Step 14
View the fabric node vector using the acidiag fnvread command. View the TEP IP addresses
assigned to the switches over DHCP through the Infra VLAN.
Answer
apic1# acidiag fnvread
ID Pod ID Name Serial Number IP Address Role State LastUpdMsgId
------------------------------------------------------
101 1 leaf-a FDO23161CZ0 [Link]/32 leaf active 0
102 1 leaf-b FDO23161MNG [Link]/32 leaf active 0
201 1 spine FDO231113UJ [Link]/32 spine active 0
The IP addresses in your fabric will be different from the example below. The state of each
node should be active.
Step 15
From Cisco APIC, connect to leaf-a IP address using the ssh command. You can use either the
switch name (leaf-a) or its IP address. Cisco APIC will automatically resolve the switch name to
its TEP IP address. The password is 1234QWer.
Answer
apic1# ssh leaf-a ----or--- ssh [Link]

Password: 1234QWer
Cisco Nexus Operating System (NX-OS) Software
TAC support: [Link]
Copyright (c) 2002-2021, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
[Link] and
[Link]
leaf-a#
Note
The SSH connections to the fabric switches will go through the Infra VLAN.
Step 16
Open three more PuTTY sessions to Cisco APIC. In one session connect through Secure Shell
(SSH) to leaf-b, in the second to the spine, use the third connection for Cisco APIC itself,
allowing an easy state verification.
Step 17
On each switch, examine the LLDP neighbors using the show lldp neighbors command.
Answer
leaf-a# show lldp neighbors
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
 (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
[Link] Eth1/1 120 BR Gi1/0/3
apic1 Eth1/2 120 eth2-1
spine Eth1/49 120 BR Eth1/1
Total entries displayed: 3
The leaves are connected to Cisco APIC over Eth1/2, and to the spine over Eth1/49. They will
also see an external switch connected to Eth1/1 and may or may not see a server connected to
Eth1/3.
Note
There may be additional devices connected to the leafs in the output of the show lldp
neighbors command. These devices can safely be ignored.
Although most of the standard show commands are available on Cisco ACI leaf or spine
switches (Cisco Nexus 9000 in ACI mode), the show command cannot be abbreviated to sh in
Cisco Nexus 9000 ACI mode, as a result of the Bash implementation. For example, sh lldp
neighbor would return an error. For context-sensitive help, press ESC twice. For command auto-
completion, press the Tab key.
leaf-b# show lldp neighbors
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
[Link] Eth1/1 120 BR Gi1/0/4
apic1 Eth1/2 120 eth2-2
spine Eth1/49 120 BR Eth1/2
Total entries displayed: 3
Note
LLDP discovers the dual-homing of Cisco APIC to both leaves, but not which connection is active
and which backup. You will later examine the bonding on Cisco APIC.
spine# show lldp neighbors
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
leaf-a Eth1/1 120 BR Eth1/49
leaf-b Eth1/2 120 BR Eth1/49
Total entries displayed: 2
Step 18
On Cisco APIC, verify the LLDP neighbors on the fabric-facing interfaces eth2-1 and eth2-2 using
the acidiag run lldptool command. Examine the information in the Appliance Vector TLV.
Answer
apic1# acidiag run lldptool in eth2-1
Chassis ID TLV
MAC: [Link]
Port ID TLV
Local: Eth1/2
Time to Live TLV
120
Port Description TLV
topology/pod-1/paths-101/pathep-[eth1/2]
System Name TLV
leaf-a
System Description TLV
topology/pod-1/node-101
System Capabilities TLV
System capabilities: Bridge, Router
Enabled capabilities: Bridge, Router
Management Address TLV
IPv4: [Link]
Ifindex: 83886080
Cisco 4-wire Power-via-MDI TLV
4-Pair PoE supported
Spare pair Detection/Classification not required
PD Spare pair Desired State: Disabled
PSE Spare pair Operational State: Disabled
Cisco Port Role TLV
4
Cisco Port Mode TLV
0
Cisco Port State TLV
1
Cisco Model TLV
N9K-C93180YC-FX
Cisco Serial Number TLV
FDO23161CZ0
Cisco Firmware Version TLV
n9000-15.2(1g)
Cisco Node Role TLV
1
Cisco Infra VLAN TLV
3967
Cisco Name TLV
leaf-a
Cisco Fabric Name TLV
Fabric
Cisco Node IP TLV
IPv4:[Link]
Cisco Node ID TLV
 101
Cisco POD ID TLV
1
Cisco Appliance Vector TLV
Id: 1
IPv4: [Link]
UUID: 9df7d5a0-ca14-11eb-beda-e526c7a0aa53
LLDP-MED Capabilities TLV
Device Type: netcon
Capabilities: LLDP-MED, Network Policy, Extended Power via MDI-PSE
LLDP-MED Network Policy TLV
01400000
End of LLDPDU TLV
apic1# acidiag run lldptool in eth2-2
Chassis ID TLV
MAC: [Link]
Port ID TLV
Local: Eth1/2
Time to Live TLV
120
Port Description TLV
topology/pod-1/paths-102/pathep-[eth1/2]
System Name TLV
leaf-b
System Description TLV
topology/pod-1/node-102
System Capabilities TLV
System capabilities: Bridge, Router
Enabled capabilities: Bridge, Router
Management Address TLV
IPv4: [Link]
Ifindex: 83886080
Cisco 4-wire Power-via-MDI TLV
4-Pair PoE supported
Spare pair Detection/Classification not required
PD Spare pair Desired State: Disabled
PSE Spare pair Operational State: Disabled
Cisco Port Role TLV
4
Cisco Port Mode TLV
0
Cisco Port State TLV
1
Cisco Model TLV
 N9K-C93180YC-FX
Cisco Serial Number TLV
FDO23161MNG
Cisco Firmware Version TLV
n9000-15.2(1g)
Cisco Node Role TLV
1
Cisco Infra VLAN TLV
3967
Cisco Name TLV
leaf-b
Cisco Fabric Name TLV
Fabric
Cisco Node IP TLV
IPv4:[Link]
Cisco Node ID TLV
102
Cisco POD ID TLV
1
Cisco Appliance Vector TLV
Id: 1
IPv4: [Link]
UUID: 9df7d5a0-ca14-11eb-beda-e526c7a0aa53
LLDP-MED Capabilities TLV
Device Type: netcon
Capabilities: LLDP-MED, Network Policy, Extended Power via MDI-PSE
LLDP-MED Network Policy TLV
01400000
End of LLDPDU TLV
The LLDP organizes information in Type-Length-Values (TLVs). In addition to basic data, such as
device name, IP address, signal-to-noise, device model and adjacent interface, the TLVs are
used to exchange further details, such as the Infra VLAN ID or AV. The AV includes the IP
address and UUID of Cisco APIC, to which the switches are registered.
Note
In the next step, you will cross-check this information against Cisco APIC.
Step 19
On Cisco APIC, view the appliance vector using the acidiag avread command. Cross-check the
chassis ID with the Cisco APIC UUID obtained from the leafs in the Cisco appliance vector TLV.
Answer
apic1# acidiag avread
Local appliance ID=1 ADDRESS=[Link] TEP ADDRESS=[Link]/16 ROUTABLE IP
ADDRESS=[Link] CHASSIS_ID=9df7d5a0-ca14-11eb-beda-e526c7a0aa53
Cluster of 1 lm(t):1(zeroTime) appliances (out of targeted 1 lm(t):1(2021-06-
11T[Link].787+00:00)) with FABRIC_DOMAIN name=Fabric set to version=5.2(1g)
lm(t):1(2021-06-11T[Link].215+00:00); discoveryMode=PERMISSIVE lm(t):0(1970-01-
01T[Link].001+00:00); drrMode=OFF lm(t):0(1970-01-01T[Link].001+00:00);
kafkaMode=OFF lm(t):0(1970-01-01T[Link].001+00:00)
appliance id=1 address=[Link] lm(t):1(2021-06-10T[Link].051+00:00) tep
address=[Link]/16 lm(t):1(2021-06-10T[Link].051+00:00) routable address=[Link]
lm(t):1(zeroTime) oob address=[Link]/24 lm(t):1(2021-06-10T[Link].131+00:00)
version=5.2(1g) lm(t):1(2021-06-10T[Link].188+00:00) chassisId=9df7d5a0-ca14-11eb-beda-
e526c7a0aa53 lm(t):1(2021-06-10T[Link].188+00:00) capabilities=0X7EEFFFFFFFFF--0X2020--
0X1 lm(t):1(2021-06-11T[Link].539+00:00) rK=(stable,present,0X206173722D687373)
lm(t):1(2021-06-10T[Link].134+00:00) aK=(stable,present,0X206173722D687373)
lm(t):1(2021-06-10T[Link].134+00:00) oobrK=(stable,present,0X206173722D687373)
lm(t):1(2021-06-10T[Link].134+00:00) oobaK=(stable,present,0X206173722D687373)
lm(t):1(2021-06-10T[Link].134+00:00) cntrlSbst=(APPROVED, FCH2128V0F0) lm(t):1(2021-
06-10T[Link].188+00:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0
lm(t):0(zeroTime)) podId=1 lm(t):1(2021-06-10T[Link].051+00:00) commissioned=YES
lm(t):1(zeroTime) registered=YES lm(t):1(2021-06-10T[Link].051+00:00) standby=NO
lm(t):1(2021-06-10T[Link].051+00:00) DRR=NO lm(t):0(zeroTime) apicX=NO lm(t):1(2021-06-
10T[Link].051+00:00) virtual=NO lm(t):1(2021-06-10T[Link].051+00:00) active=YES(2021-
06-10T[Link].051+00:00) health=(applnc:255 lm(t):1(2021-06-10T[Link].737+00:00) svc's)
---------------------------------------------
clusterTime=<diff=-7610 common=2021-06-11T[Link].430+00:00 local=2021-06-
11T[Link].040+00:00 pF=<displForm=0 offsSt=0 offsVlu=0 lm(t):1(2021-06-
11T[Link].180+00:00)>>
---------------------------------------------
Cisco APIC UUID is shown as chassis ID in the AV. The following flow describes how AV is
populated and communicated between each Cisco APIC and switch node:
1. Initially, each Cisco APIC generates an AV with only its own information (APIC ID, IP,
UUID, and so on).
2. When the first leaf is discovered by apic1, the leaf gets apic1’s AV from apic1.
3. A spine and a second leaf will get apic1’s AV from apic1 once they are discovered.
4. If a second Cisco APIC is connected to one of the leaf nodes, the leaf gets the AV for the
second APIC and it will report it to other Cisco APICs (in this case, apic1).
5. apic1 now knows that there is apic2 and starts communication directly with apic2 using
the TEP addresses and the infra network provided by leaf and spine switches.
6. apic1 and 2 will share the updated AV information to all registered switches. If any
conflict exists, such as chassis ID mismatch in any of the above steps, the APIC cannot
join the cluster.
Step 20
On the APIC, view the interfaces using the ifconfig command.
Answer
apic1# ifconfig
bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500
<...output omitted...>
bond1: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500
<...output omitted...>

bond0.3967: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1496

inet [Link] netmask [Link] broadcast [Link]
<...output omitted...>

eth1-1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500

<...output omitted...>

eth1-2: flags=6147<UP,BROADCAST,SLAVE,MULTICAST> mtu 1500

<...output omitted...>

eth2-1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500

<...output omitted...>

eth2-2: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500

<...output omitted...>

ifb0: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500

<...output omitted...>

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 1500

inet [Link] netmask [Link]
<...output omitted...>

lxcbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

<...output omitted...>

oobmgmt: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet [Link] netmask [Link] broadcast [Link]
<...output omitted...>

tep0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

<...output omitted...>

tep1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

<...output omitted...>

tep2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

<...output omitted...>

tep3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

<...output omitted...>
tep4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
<...output omitted...>

tep5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

<...output omitted...>

tep6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

<...output omitted...>

tep7: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

<...output omitted...>

teplo-1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

<...output omitted...>
The most relevant interfaces are explained below:
• bond0: A logical bond that bundles the physical interfaces attached to the fabric (eth2-1
and eth2-2).
• bond1: A logical bond that provides OOB connectivity.
• bond0.3967: Subinterface of the bond0 interface that carries Infra traffic, such as
packets encapsulated with Infra VLAN (3967) 802.1Q header. The IP address of this
subinterface is [Link]/32. It belongs to the TEP address pool ([Link]/16) that was
configured in the setup utility.
• oobmgmt: Logical interface for OOB management ([Link]/24) configured during
the initial setup
Step 21
On Cisco APIC, examine the bond0 configuration, defined in the file /proc/net/bonding/bond0.
Identify the active link.
Answer
The bonding mode is set to fault-tolerance (active-backup). In the example below, eth2-2,
facing leaf-b, is active. Although you are not monitoring the actual discovery procedure, leaf-b
must have been discovered first.
apic1# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None
Currently Active Slave: eth2-2
MII Status: up
MII Polling Interval (ms): 60
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2-1

MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: [Link]
Slave queue ID: 0

Slave Interface: eth2-2

MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: [Link]
Slave queue ID: 0
Step 22
On leaf-a, view the interfaces using the show interface brief command.
Answer
You should find that:
• The Cisco APIC facing interface (Eth1/2) is a trunking switch port.
• The spine-facing interface (Eth1/49) is a routed port and has a subinterface. The
subinterface provides a logical connection to the spine. Later you will see that the leaf
physical tunnel endpoint (PTEP) address is applied as "unnumbered" to this interface to
enable IP connectivity within the fabric.
• Several Loopback and Tunnel interfaces are up. Loopback 0 represents, as you will see,
the TEP.
leaf-a# show interface brief
------------------------------------------------------
Port VRF Status IP Address Speed MTU
------------------------------------------------------
mgmt0 -- up 1000 9000

-----------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
-----------------------------------------------------
Eth1/1 0 eth trunk up out-of-service 1000(D) --
Eth1/2 0 eth trunk up none 10G(D) --
Eth1/3 0 eth trunk up out-of-service 10G(D) --
...
Eth1/49 -- eth routed up none 40G(D) --
Eth1/49.7 2 eth routed up none 40G(D) --
...
Interface Status Description
----------------------------------------------------
Lo0 up --
Lo1023 up --

----------------------------------------------------
Interface Secondary VLAN(Type) Status Reason
-----------------------------------------------------
Vlan8 -- up --

----------------------------------------------------
Interface Status IP Address Encap type MTU
----------------------------------------------------
Tunnel1 up -- ivxlan 9000
Tunnel2 up -- ivxlan 9000
Tunnel3 up -- ivxlan 9000
Tunnel4 up -- ivxlan 9000
Tunnel5 up -- ivxlan 9000
Tunnel6 up -- ivxlan 9000
Step 23
On leaf-a, list the VRF instances using the show vrf command.
Answer
leaf-a# show vrf
VRF-Name VRF-ID State Reason
black-hole 3 Up --
overlay-1 4 Up --
Note
Cisco ACI uses a dedicated VRF as an infrastructure to carry VXLAN traffic. The transport
infrastructure for VXLAN traffic is known as overlay-1, which exists as part of the tenant “infra.”
Leaf nodes are known as PTEPs (physical tunnel endpoints). Cisco ACI maintains a mapping
database containing information about where (that is, on which TEP) an endpoint MAC and IP
address reside. The mapping database is maintained by the Council of Oracles Protocol (COOP)
on the spine switches.
Step 24
On leaf-a, view the IP interfaces using the show ip interface brief vrf overlay-1 command.
Answer
leaf-a# show ip interface brief vrf overlay-1
IP Interface Status for VRF "overlay-1"(4)
eth1/49 unassigned protocol-up/link-up/admin-up
eth1/49.7unnumbered protocol-up/link-up/admin-up
(lo0)
eth1/50 unassigned protocol-down/link-down/admin-up
eth1/51 unassigned protocol-down/link-down/admin-up
eth1/52 unassigned protocol-down/link-down/admin-up
eth1/53 unassigned protocol-down/link-down/admin-up
eth1/54 unassigned protocol-down/link-down/admin-up
vlan7 [Link]/27 protocol-up/link-up/admin-up
lo0 [Link]/32 protocol-up/link-up/admin-up
lo1023 [Link]/32 protocol-up/link-up/admin-up
You should find that:
• The TEP IP address obtained via DHCP from the APIC is assigned to Loopback 0. It is
known as a physical tunnel endpoint (PTEP). The PTEP address is also applied as
unnumbered to a subinterface of the spine-facing link. In this example, the PTEP
address, assigned to Loopback 0, is [Link].
• There is another TEP IP address, known as the Fabric TEP (FTEP), used to encapsulate
traffic in VXLAN to a vSwitch TEP, if present. Cisco ACI defines a unique FTEP address
that is identical on all leaf nodes to allow mobility of downstream TEP devices. In this
example, the FTEP address is [Link].
• All these IP addresses belong to the overlay-1 VRF.
Note
The overlay-1 VRF contains /32 routes to each PTEP, vPC TEP, APIC TEP, and spine-proxy TEP.
Each TEP address exists as a loopback in the overlay-1 VRF.
Step 25
On leaf-b, view the IP interfaces using the show ip interface brief vrf overlay-1 command.
Answer
leaf-b# show ip interface brief vrf overlay-1
IP Interface Status for VRF "overlay-1"(4)
Interface Address Interface Status
eth1/49 unassigned protocol-up/link-up/admin-up
eth1/49.7 unnumbered protocol-up/link-up/admin-up
(lo0)
eth1/50 unassigned protocol-down/link-down/admin-up
eth1/51 unassigned protocol-down/link-down/admin-up
eth1/52 unassigned protocol-down/link-down/admin-up
eth1/53 unassigned protocol-down/link-down/admin-up
eth1/54 unassigned protocol-down/link-down/admin-up
vlan8 [Link]/27 protocol-up/link-up/admin-up
lo0 [Link]/32 protocol-up/link-up/admin-up
lo1023 [Link]/32 protocol-up/link-up/admin-up
The PTEP address obtained through DHCP from Cisco APIC is assigned to Loopback 0 and is also
applied as unnumbered to a subinterface on the spine-facing link. The PTEP address in this
example is [Link]. The Fabric TEP (FTEP) is [Link].
Step 26
On leaf-a, view the available variants of the show vlan command.
Answer
leaf-a# show vlan ESC ESC
<CR> Carriage return
all-ports Show all ports on VLAN
brief All VLAN status in brief
encap-id VLAN status by VLAN wire encap id
 extended VLAN extended info like encaps
fcoe FCOE Configuration
id VLAN status by VLAN id
internal Show internal information of vlan-mgr
reserved Internal reserved VLANs
summary VLAN summary information
vnid-id VLAN status by VXLAN wire vnid id
leaf-a# show vlan extended
VLAN Name Encap Ports
---- -------------------------------- ---------------
8 infra:default vxlan-16777209, Eth1/2
vlan-3967
The VXLAN to VLAN mapping is automatically provisioned by Cisco ACI. Cisco ACI uses multiple
types of VLANs and uses a translation process to map one VLAN type to another. The VXLAN ID
is consistent across all leaves. The Encap VLAN shows the encapsulation on the edge of the
network. The Platform Independent (PI) VLAN is seen when executing show commands on the
switches. The PI VLAN maps to Encap VLANs and VXLANs. VLAN IDs do not need to be
consistent across all leaves.
The Infra VLAN (3967) is enabled on the APIC-facing port (Eth1/2). The Infra VLAN is mapped to
a Platform-Independent (PI) VLAN and the Infra VXLAN. In this example, the Infra VLAN uses the
Encap VLAN 3967 and is mapped to the PI VLAN 8 and the Infra VXLAN 16777209. The Infra
VLAN has been set to 3967 using the Setup Utility during Cisco APIC bringup.
Note
The details on each VLAN type are explained later. The key point in Fabric Discovery phase is to
make sure that the Infra VLAN is trunked on a port connected to Cisco APIC.
Step 27
On the spine, view the interfaces using the show interface brief command.
Answer
spine# show interface brief
-----------------------------------------------------
Port VRF Status IP Address Speed MTU
----------------------------------------------------
mgmt0 -- up 1000 9000
---------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
---------------------------------------------------
Eth1/1 -- eth routed up none 40G(D) --
Eth1/1.36 2 eth routed up none 40G(D) --
Eth1/2 -- eth routed up none 40G(D) --
Eth1/2.35 2 eth routed up none 40G(D) --
<... output omitted ...>
----------------------------------------------------
Interface Status Description
---------------------------------------------------
Lo0 up --
Lo1 up --
Lo2 up --
Lo3 up --
Lo4 up --
Lo5 up --
Lo6 up --
Lo7 up --
Lo8 up --
Lo9 up --
--------------------------------------------------
Interface Status IP Address Encap type MTU
--------------------------------------------------
Tunnel1 up -- ivxlan 9000
Tunnel2 up -- ivxlan 9000
Tunnel3 up -- ivxlan 9000
Tunnel4 up -- ivxlan 9000
The spine acts solely as a routed node and does not have switched interfaces, which exist at the
edge of the fabric. The commands show vlan (extended) and show system internal epm vlan all
will return empty outputs on the spine switch.
Step 28
On the spine, view the IP interfaces using the show ip interface brief vrf command.
Answer
spine# show ip interface brief vrf overlay-1
IP Interface Status for VRF "overlay-1"(4)
Interface Address Interface Status
eth1/1 unassigned protocol-up/link-up/admin-up
eth1/1.36 unnumbered protocol-up/link-up/admin-up
(lo0)
eth1/2 unassigned protocol-up/link-up/admin-up
eth1/2.35 unnumbered protocol-up/link-up/admin-up
(lo0)
<... output omitted ...>
lo0 [Link]/32 protocol-up/link-up/admin-up
lo1 [Link]/32 protocol-up/link-up/admin-up
lo2 [Link]/32 protocol-up/link-up/admin-up
lo3 [Link]/32 protocol-up/link-up/admin-up
lo4 [Link]/32 protocol-up/link-up/admin-up
lo5 [Link]/32 protocol-up/link-up/admin-up
lo6 [Link]/32 protocol-up/link-up/admin-up
lo7 [Link]/32 protocol-up/link-up/admin-up
lo8 [Link]/32 protocol-up/link-up/admin-up
lo9 [Link]/32 protocol-up/link-up/admin-up
Note that:
• The PTEP IP address obtained through DHCP from Cisco APIC (in this example
[Link]) is assigned to Loopback 0.
• This IP address is reused on leaf-facing subinterfaces using the unnumbered method.
• There are several other IP addresses, all in the overlay-1 VRF.
Note
Each TEP address exists as a loopback on the overlay-1 VRF. In addition to their individual PTEP
addresses, spines can be addressed by several other TEP addresses, including the proxy TEP.
You can view this address by running the show isis dteps vrf overlay-1 command on a leaf
switch.
Step 29
On a leaf, identify the spine-proxy TEP addresses using the show isis dteps vrf overlay-1
command. They are designated by their type (proxy-acast).
Answer
leaf-a# show isis dteps vrf overlay-1

IS-IS Dynamic Tunnel End Point (DTEP) database:

DTEP-Address Role Encapsulation Type
[Link] LEAF N/A PHYSICAL
[Link] SPINE N/A PHYSICAL,PROXY-ACAST-MAC
[Link] SPINE N/A PHYSICAL,PROXY-ACAST-V4
[Link] SPINE N/A PHYSICAL
[Link] SPINE N/A PHYSICAL,PROXY-ACAST-V6
Note
The spine-proxy TEP address is an anycast IP address that exists across all spines, used for
forwarding lookups into the mapping database (Council of Oracle Protocol [COOP]). There is a
separate spine-proxy TEP address for each address family (IPv4, IPv6 and MAC).
Step 30
On the spine, view the routing table of the overlay-1 VRF. Examine the routes to Cisco APIC,
leaf-a PTEP, and leaf-b PTEP.
Answer
spine# show ip route vrf overlay-1
<... legend omitted ...>
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/2.35, [115/11], [Link], isis-isis_infra, isis-l1-ext
[Link]/16, ubest/mbest: 1/0
*via , null0, [1/0], [Link], static
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/2.35, [115/11], [Link], isis-isis_infra, isis-l1-ext
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo7, [0/0], [Link], local, local
*via [Link], lo7, [0/0], [Link], direct
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo8, [0/0], [Link], local, local
 *via [Link], lo8, [0/0], [Link], direct
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo9, [0/0], [Link], local, local
*via [Link], lo9, [0/0], [Link], direct
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/1.36, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/2.35, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo0, [0/0], [Link], local, local
*via [Link], lo0, [0/0], [Link], direct
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo4, [0/0], [Link], local, local
*via [Link], lo4, [0/0], [Link], direct
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo5, [0/0], [Link], local, local
*via [Link], lo5, [0/0], [Link], direct
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo6, [0/0], [Link], local, local
*via [Link], lo6, [0/0], [Link], direct
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo2, [0/0], [Link], local, local
*via [Link], lo2, [0/0], [Link], direct
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo3, [0/0], [Link], local, local
*via [Link], lo3, [0/0], [Link], direct
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo1, [0/0], [Link], local, local
*via [Link], lo1, [0/0], [Link], direct
The routes to Cisco APIC, leaf-a PTEP, and leaf-b PTEP, appear as Intermediate System-to-
Intermediate System (IS-IS) level 1 routes, directed via the subinterfaces of the respective leaf-
facing links. The route to Cisco APIC ([Link]/32) points to the leaf with the active link in the
bond interface.
Note
In this example, leaf-b, reachable over eth1/2, has the active link in the bond interface.
[Link] is the PTEP address of leaf-b. [Link] is the PTEP address of leaf-b.
Step 31
On the leaf with the backup bond link (in this example leaf-a), view the routing table and
identify the routes to Cisco APIC, to the spine PTEP and the other leaf PTEP.
Answer
leaf-a# show ip route vrf overlay-1
<... legend omitted ...>
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/12], [Link], isis-isis_infra, isis-l1-ext
[Link]/27, ubest/mbest: 1/0, attached, direct
*via [Link], vlan8, [0/0], [Link], direct
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/12], [Link], isis-isis_infra, isis-l1-ext
[Link]/32, ubest/mbest: 1/0, attached
*via [Link], vlan8, [0/0], [Link], local, local
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo1023, [0/0], [Link], local, local
*via [Link], lo1023, [0/0], [Link], direct
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 2/0, attached, direct
*via [Link], lo0, [0/0], [Link], local, local
*via [Link], lo0, [0/0], [Link], direct
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/3], [Link], isis-isis_infra, isis-l1-int10.0.32.66/32,
ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
[Link]/32, ubest/mbest: 1/0
*via [Link], eth1/49.7, [115/2], [Link], isis-isis_infra, isis-l1-int
All three IS-IS level 1 routes should point over a subinterface of the spine-facing interface
Eth1/49. All other IS-IS routes are for other spine TEP IP addresses, including the proxy TEP.
Note
In this example, [Link] is the spine PTEP, [Link] is the PTEP address of leaf-a.
Step 32
On the leafs, display the endpoints using the show endpoint command.
Answer
leaf-a# show endpoint
Legend:
S - static s - arp L - local O - peer-attached
 V - vpc-attached a - local-aged p - peer-aged M - span
B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy
E - shared-service m - svc-mgr
+-----------------------------
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+-----------------------------
overlay-1 [Link] L lo0
leaf-b# show endpoint
<... legend omitted ...>
+-----------------------------
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+------------------------------
overlay-1 [Link] L lo0
8/overlay-1 vxlan-16777209 3890.a540.76ea L eth1/2
The leaf with the active bond link (leaf-b in this example), will see the MAC address of the APIC.
Step 33
On any switches, examine additional information of your choice. This step completes the
activity.