Cumulus Linux 3.

5
User Guide

Table of Contents
 Cumulus Linux User Guide

Table of Contents
Introducing Cumulus Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
What's New in Cumulus Linux 3.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Open Source Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Hardware Compatibility List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Quick Start Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Configuring Breakout Ports with Splitter Cables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Testing Cable Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Configuring Switch Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Configuring a Loopback Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Installation Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Managing Cumulus Linux Disk Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Installing a New Cumulus Linux Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Upgrading Cumulus Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
x86 vs ARM Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Reprovisioning the System (Restart Installer) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Uninstalling All Images and Removing the Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Booting into Rescue Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Inspecting Image File Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Installing a New Cumulus Linux Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Understanding these Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Installing via a DHCP/Web Server Method with DHCP Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Installing via a DHCP/Web Server Method without DHCP Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Installing via a Web Server with no DHCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Installing via FTP or TFTP without a Web Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Installing via a Local File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Installing via USB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Installing a New Image when Cumulus Linux Is already Installed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Upgrading Cumulus Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Upgrades: Comparing the Network Device Worldview vs. the Linux Host Worldview . . . . . . . . . . . 49
Upgrading Cumulus Linux Devices: Strategies and Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Upgrading Cumulus Linux: Choosing between a Binary Install vs. Package Upgrade . . . . . . . . . . . 56
Rolling Back a Cumulus Linux Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

[Link] 2
 Cumulus Linux User Guide

Third Party Package Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Installation and Upgrade Workflow in Cumulus Linux 3.0 and Later . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Using Snapshots during Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Caveats When Migrating Configuration Files Between Cumulus Linux 2.5.z and 3.0 and Later . . . .
59
Using the Config File Migration Script to Identify and Move Files to Cumulus Linux 3.0 and Later
60
Using Automation Tools to Back Up 2.5.z Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Using Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Installing the Snapshot Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Taking and Managing Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Rolling Back to Earlier Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Configuring Automatic Time-based Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Adding and Updating Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Updating the Package Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Listing Available Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Adding a Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Listing Installed Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Upgrading to Newer Versions of Installed Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Adding Packages from Another Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Zero Touch Provisioning - ZTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Zero Touch Provisioning Using a Local File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Zero Touch Provisioning Using USB (ZTP-USB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Zero Touch Provisioning over DHCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Writing ZTP Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
Testing and Debugging ZTP Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Manually Using the ztp Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

System Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Network Command Line Utility - NCLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
What's New and Different in NCLU in Version 3.5? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Installing NCLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Configuring User Accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Restarting the netd Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Backing up the Configuration to a Single File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Advanced Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Setting Date and Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

[Link] 3
 Cumulus Linux User Guide

Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Setting the Time Zone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Setting the Date and Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Setting Time Using NTP and NCLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Specifying the NTP Source Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
NTP Default Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Authentication, Authorization and Accounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
SSH for Remote Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
User Accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Using sudo to Delegate Privileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
LDAP Authentication and Authorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
TACACS Plus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
RADIUS AAA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Netfilter - ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
Understanding Traffic Rules In Cumulus Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Installing and Managing ACL Rules with NCLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Installing and Managing ACL Rules with cl-acltool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Installing Packet Filtering (ACL) Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Specifying which Policy Files to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
Hardware Limitations on Number of Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Supported Rule Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Common Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Example Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Default Cumulus Linux ACL Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Filtering Learned MAC Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Managing Application Daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Using systemd and the systemctl Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Identifying Active Listener Ports for IPv4 and IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Identifying Daemons Currently Active or Stopped . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Identifying Essential Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
Configuring switchd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
The switchd File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Configuring switchd Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Restarting switchd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Power over Ethernet - PoE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
How It Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Configuring PoE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Troubleshooting PoE and PoE+ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Configuring a Global Proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
[Link] 4
 Cumulus Linux User Guide

Configuring a Global Proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
HTTP API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

Interface Configuration and Management . . . . . . . . . . . . . . . . . . . . . . . . . 208

Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Basic Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Using NCLU to Set the Admin State of an Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
ifupdown2 Interface Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Bringing All auto Interfaces Up or Down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Configuring a Loopback Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
ifupdown Behavior with Child Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
ifupdown2 Interface Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
ifup Handling of Upper (Parent) Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
Configuring IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Specifying IP Address Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
Purging Existing IP Addresses on an Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Specifying User Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Sourcing Interface File Snippets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
Using Globs for Port Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
Using Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Commenting out Mako Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Adding Descriptions to Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Layer 1 and Switch Port Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Interface Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
Interface Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
Configuring Breakout Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Logical Switch Port Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Using ethtool to Configure Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Verification and Troubleshooting Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Buffer and Queue Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
Example Configuration File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
Configuring Traffic Marking through ACL Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
Configuring Priority Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Configuring Link Pause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Configuring Cut-through Mode and Store and Forward Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
Configuring Explicit Congestion Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

[Link] 5
 Cumulus Linux User Guide

Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

Configuring Hardware-enabled DDOS Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Configuring DHCP Relays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Configuring IPv4 DHCP Relays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Configuring IPv6 DHCP Relays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
Configuring Multiple DHCP Relays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
Configuring a DHCP Relay with VRR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
Configuring the DHCP Relay Service Manually (Advanced) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Troubleshooting the DHCP Relays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Configuring DHCP Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
Configuring DHCP Server on Cumulus Linux Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
Troubleshooting the Log from a DHCP Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
802.1X Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Supported Features and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Installing the 802.1X Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Configuring 802.1X Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Verifying Connections from Linux Supplicants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Configuring Dynamic VLAN Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
Configuring the RADIUS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

Layer 1 and 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

Spanning Tree and Rapid Spanning Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Supported Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Viewing Bridge and STP Status/Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
Customizing Spanning Tree Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
Link Layer Discovery Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
Configuring LLDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
Example lldpcli Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
Enabling the SNMP Subagent in LLDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Prescriptive Topology Manager - PTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
Supported Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
Configuring PTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
Basic Topology Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
ptmd Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

[Link] 6
 Cumulus Linux User Guide

Configuration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

Bidirectional Forwarding Detection (BFD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
Checking Link State with FRRouting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
Using ptmd Service Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
Using ptmctl Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
Bonding - Link Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
Hash Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
Creating a Bond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
Example Configuration: Bonding 4 Slaves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Ethernet Bridging - VLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
Creating a VLAN-aware Bridge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
Creating a Traditional Mode Bridge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
Configuring Bridge MAC Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
Configuring an SVI (Switch VLAN Interface) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
VLAN-aware Bridge Mode for Large-scale Layer 2 Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
Traditional Mode Bridges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
VLAN Tagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
Multi-Chassis Link Aggregation - MLAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
MLAG Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
LACP and Dual-Connectedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
Configuring MLAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354
Example MLAG Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
Checking the MLAG Configuration Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
Configuring MLAG with a Traditional Mode Bridge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
Peer Link Interfaces and the protodown State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
Monitoring Dual-Connected Peers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Configuring Layer 3 Routed Uplinks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
IGMP Snooping with MLAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374
Monitoring the Status of the clagd Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
MLAG Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
STP Interoperability with MLAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
Troubleshooting MLAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
LACP Bypass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
Understanding the LACP Bypass All-active Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382

[Link] 7
 Cumulus Linux User Guide

Configuring LACP Bypass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382

Virtual Router Redundancy - VRR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Configuring a VRR-enabled Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
Example VRR Configuration with MLAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
ifplugd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
IGMP and MLD Snooping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394
Configuring IGMP/MLD Querier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394
Disable IGMP and MLD Snooping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395
Debugging IGMP/MLD Snooping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398

Network Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398

Caveats/Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
Cut-through Mode and Store & Forward Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
MTU Size for Virtual Network Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
Useful Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
Static VXLAN Tunnels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
Example Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
Configuring Static VXLAN Tunnels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
Verifying the Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
Static MAC Bindings with VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406
Example VXLAN Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406
Configuring the Static MAC Bindings VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406
Troubleshooting VXLANs in Cumulus Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
Lightweight Network Virtualization - LNV Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
Understanding LNV Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
Sample LNV Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
Configuring the VLAN to VXLAN Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
Verifying the VLAN to VXLAN Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
Enabling and Managing Service Node and Registration Daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
Configuring the Registration Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
Configuring the Service Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
Verification and Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
Advanced LNV Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
LNV VXLAN Active-Active Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
LNV Full Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
Ethernet Virtual Private Network - EVPN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
[Link] 8
 Cumulus Linux User Guide

Ethernet Virtual Private Network - EVPN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465

Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466
Basic EVPN Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
ARP and ND Suppression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
EVPN and VXLAN Active-Active Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
Inter-subnet Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
Prefix-based Routing — EVPN Type-5 Routes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
Static (Sticky) MAC Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
EVPN Operational Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
Troubleshooting EVPN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
Caveats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
Example Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
VXLAN Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
VXLAN Routing Data Plane and the Broadcom Tomahawk and Trident II+ Platforms . . . . . . . . . 512
Configuring VXLAN Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
VXLAN Routing with Active-Active VTEPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
VXLAN with VRFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
Viewing VXLAN Routing Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
Troubleshooting VXLAN Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529
VXLAN Hyperloop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
Hyperloop Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
VXLAN Hyperloop Troubleshooting Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534
Integrating Hardware VTEPs with Midokura MidoNet and OpenStack . . . . . . . . . . . . . . . . . . 537
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
Troubleshooting MidoNet and Cumulus VTEPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
Integrating with VMware NSX-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556
Bootstrapping the NSX-V Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556
Configuring the Transport Zone and Segment ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562
Configuring the Logical Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564
Verifying the VXLAN Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567
Integrating with VMware NSX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568
Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569
Bootstrapping the NSX Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569
Configuring the Transport Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
Configuring the Logical Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
Verifying the VXLAN Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
Troubleshooting VXLANs in NSX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580
VXLAN Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581
Hybrid Cloud Connectivity with QinQ and VXLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582
[Link] 9
 Cumulus Linux User Guide

Hybrid Cloud Connectivity with QinQ and VXLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582

Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582
Removing the Early Access QinQ Metapackage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
Configuring Single Tag Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
Configuring Double Tag Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589

Layer 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 590
Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591
Managing Static Routes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591
Configuring a Gateway or Default Route . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
Supported Route Table Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
Introduction to Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
Defining Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
Configuring Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
Protocol Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 600
Network Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 600
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 600
Clos Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
Over-Subscribed and Non-Blocking Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
Containing the Failure Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
FRRouting Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603
About zebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603
Upgrading from Quagga to FRRouting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603
Configuring FRRouting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
Configuring FRRouting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
Interface IP Addresses and VRFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
Using the FRRouting vtysh Modal CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
Reloading the FRRouting Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616
Comparing NCLU and vtysh Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616
Open Shortest Path First - OSPF - Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 619
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 619
Scalability and Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 620
Configuring OSPFv2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
Scaling Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623

[Link] 10
 Cumulus Linux User Guide

Unnumbered Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629

Applying a Route Map for Route Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
ECMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
Topology Changes and OSPF Reconvergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631
Debugging OSPF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632
Open Shortest Path First v3 - OSPFv3 - Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
Configuring OSPFv3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633
Unnumbered Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
Debugging OSPF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
Border Gateway Protocol - BGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634
Autonomous System Number (ASN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
eBGP and iBGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
Route Reflectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636
ECMP with BGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637
Configuring BGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 638
Using BGP Unnumbered Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 639
BGP add-path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644
Fast Convergence Design Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647
Using Peer Groups to Simplify Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
Configuring BGP Dynamic Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
Configuring BGP Peering Relationships across Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 650
Configuring MD5-enabled BGP Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652
Configuring BGP TTL Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654
Configuring Graceful BGP Shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656
Configuration Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657
Troubleshooting BGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658
Enabling Read-only Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664
Applying a Route Map for Route Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664
Protocol Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667
Bidirectional Forwarding Detection - BFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 668
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 668
Using BFD Multihop Routed Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 668
BFD Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669
Configuring BFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669
BFD in BGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669
BFD in OSPF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 671
OSPF Show Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 671
Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673
Echo Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673
Troubleshooting BFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674
Equal Cost Multipath Load Sharing - Hardware ECMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675
[Link] 11
 Cumulus Linux User Guide

Equal Cost Multipath Load Sharing - Hardware ECMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675

Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675
Understanding Equal Cost Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675
Understanding ECMP Hashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676
Resilient Hashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 679
Redistribute Neighbor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 683
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684
Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684
Target Use Cases and Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684
How It Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685
Configuration Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685
Known Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689
Virtual Routing and Forwarding - VRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 692
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694
Configuring VRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694
FRRouting Operation in a VRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 698
Example Commands to Show VRF Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 701
Using BGP Unnumbered Interfaces with VRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709
Using DHCP with VRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 711
Using ping or traceroute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716
Management VRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716
Enabling Management VRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 717
Running Services within the Management VRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 718
OSPF and BGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 722
Using SSH within a Management VRF Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723
Viewing the Routing Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723
Using the mgmt Interface Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724
Management VRF and DNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725
Incompatibility with cl-ns-mgmt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725
Protocol Independent Multicast - PIM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 726
PIM Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727
PIM Sparse Mode (PIM-SM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 730
Configuring PIM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736
Source Specific Multicast Mode (SSM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 741
IP Multicast Boundaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742
Multicast Source Discovery Protocol (MSDP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 743
Verifying PIM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744
PIM in a VRF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 748
BFD for PIM Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 751
Troubleshooting PIM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 751
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 755

[Link] 12
 Cumulus Linux User Guide

Monitoring and Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 755

Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 756
Using the Serial Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 756
Configuring the Serial Console on ARM Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 756
Configuring the Serial Console on x86 Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 757
Getting General System Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 758
Diagnostics Using cl-support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 758
Sending Log Files to a syslog Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 759
Using NCLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 759
Logging Technical Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 759
Local Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 760
Enabling Remote syslog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 761
Writing to syslog with Management VRF Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 761
Rate-limiting syslog Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 762
Harmless syslog Error: Failed to reset [Link] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 763
Syslog Troubleshooting Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 763
Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 765
Single User Mode - Boot Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766
Resource Diagnostics Using cl-resource-query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 767
Monitoring System Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 768
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 769
Monitoring Hardware Using decode-syseeprom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 769
Monitoring Hardware Using sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 770
Monitoring Switch Hardware Using SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 772
Monitoring System Units Using smond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 772
Keeping the Switch Alive Using the Hardware Watchdog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 773
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 773
Network Switch Port LED and Status LED Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 774
Monitoring Virtual Device Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 777
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 777
Sample VXLAN Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 777
Sample VLAN Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 778
Configuring the Counters in switchd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 779
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 780
Buffer Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 781
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 781
Understanding Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 781
Configuring Buffer Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 782
Restarting the asic-monitor Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 784
Understanding Triggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 785
Understanding Monitoring Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 785
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 785
Understanding the cl-support Output File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 786
Troubleshooting Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 786

[Link] 13
 Cumulus Linux User Guide

Troubleshooting the etc Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 789

Troubleshooting Network Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 803
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 803
Enabling Logging for Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 803
Using ifquery to Validate and Debug Interface Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 804
Debugging Mako Template Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 805
ifdown Cannot Find an Interface that Exists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 806
Removing All References to a Child Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 806
MTU Set on a Logical Interface Fails with Error: "Numerical result out of range" . . . . . . . . . . . . . . . 807
Interpreting iproute2 batch Command Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807
Understanding the "RTNETLINK answers: Invalid argument" Error when Adding a Port to a Bridge
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807
MLAG Peerlink Interface Drops Many Packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807
Monitoring Interfaces and Transceivers Using ethtool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 808
Network Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 812
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 812
Checking Reachability Using ping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 813
Printing Route Trace Using traceroute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 813
Manipulating the System ARP Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 814
Generating Traffic Using mz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 814
Creating Counter ACL Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 815
Configuring SPAN and ERSPAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816
Monitoring Control Plane Traffic with tcpdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 824
Caveats and Errata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825
Using NCLU to Troubleshoot Your Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825
Monitoring System Statistics and Network Traffic with sFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 828
SNMP Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 831
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 831
Introduction to SNMP (Simple Network Management Protocol) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 832
Configuring Ports for SNMP to Listen for Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 832
Quick Start Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 832
Starting the SNMP Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835
Configuring SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835
snmpwalk a Switch from Another Linux Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 848
SNMP Traps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 850
Supported MIBs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 860
About Pass Persist Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 863
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 863
Using Nutanix Prism as a Monitoring Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 864
Monitoring Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 875
Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876
System Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 878

[Link] 14
 Cumulus Linux User Guide

Process Restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 880

Layer 1 Protocols and Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 880
Layer 2 Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 887
Layer 3 Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 889
Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 892
Protocols and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 894
Device Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 894

Network Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 895

Data Center Host to ToR Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 896
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 896
Layer 2 - Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 896
Layer 3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 900
Network Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 908
Cumulus Networks Services Demos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 909
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 909
Reference Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 910
Docker on Cumulus Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 912
Setting up Docker on Cumulus Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 912
Performance Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 914
OpenStack Neutron ML2 and Cumulus Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 914
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 915
Configuring the REST API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 915
Installing and Configuring the Cumulus Networks Modular Layer 2 Mechanism Driver . . . . . . . . 916
Demo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 916
Anycast Design Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 917
Anycast Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 917
Anycast with TCP and UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 918
Resilient Hashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 919
Applications for Anycast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 921
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 922
RDMA over Converged Ethernet - RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 923
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 923
Enabling RDMA over Converged Ethernet with PFC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 923
Enabling RDMA over Converged Ethernet with ECN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 924
Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 926
©2018 Cumulus Networks. All rights reserved

CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are trademarks and service marks of
Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior written consent of
Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus
Torvalds, owner of the mark on a worldwide basis. All other marks are used under fair use or license from their respective owners.

Introducing
[Link]
Cumulus Linux 15
Cumulus Linux 3.5 User Guide

Introducing Cumulus Linux

Cumulus Linux is the networking industry's first full-featured Linux operating system. The Debian Jessie-
based, networking-focused distribution runs on hardware produced by a broad partner ecosystem,
ensuring unmatched customer choice regarding silicon, optics, cables, and systems.
This user guide provides in-depth documentation covering installing Cumulus Linux, system configuration
and management, network solutions, and monitoring and troubleshooting recommendations. In addition,
the quick start guide provides an end-to-end setup process to get you started with Cumulus Linux.
This documentation is current as of January 17, 2017 for version 3.5.1. Please visit the Cumulus Networks
Web site for the most up to date documentation.
Read the release notes for new features and known issues in this release.

What's New in Cumulus Linux 3.5

Cumulus Linux 3.5 contains a number of new platforms, features and improvements:
New platforms include:
Accton OMP-800 chassis/Cumulus Express CX-10256-S (100G)
Delta 9032-v1 (100G Tomahawk) and AG7648 (10G Trident II)
Broadcom Maverick-based 10G switches, including Dell S4128F-ON
Edgecore AS5812 AC with 3Y PSU
Facebook Wedge-100S now generally available
Mellanox Spectrum A1 chipsets in the 2100, 2410 and 2700 models; Mellanox 2740 (100G)
and 2740B (40G)
Quanta LY7 (10G)
10GBASE-LR BiDi optics
Symmetric VXLAN routing (see page 510)
VLAN-aware bridge support for ovs-vtepd, for VXLAN solutions using controllers
OSPF is now VRF-aware (see page 692)
Voice VLAN
PIM (see page 725) now supports overlapping IP addresses and IP multicast boundaries
Bridge layer 2 protocol tunnels
The SNMP Cumulus-Counters-MIB (see page 860) file includes a new table pfcClCountersTable
for link pause and priority flow control counters
See what's new and different with NCLU in this release
It also contains these early access features and platforms:
For further information regarding bug fixes and known issues present in this release, refer to the product
release notes.

16 23 January 2018
 Cumulus Networks

Open Source Contributions

Cumulus Networks has forked various software projects, like CFEngine, Netdev and some Puppet Labs
packages in order to implement various Cumulus Linux features. The forked code resides in the Cumulus
Networks GitHub repository.
Cumulus Networks developed and released as open source some new applications as well.
The list of open source projects is on the open source software page.

Hardware Compatibility List

You can find the most up to date hardware compatibility list (HCL) here. Use the HCL to confirm that your
switch model is supported by Cumulus Networks. The HCL is updated regularly, listing products by port
configuration, manufacturer, and SKU part number.

Quick Start Guide

[Link] 17
Cumulus Linux 3.5 User Guide

Quick Start Guide

This quick start guide provides an end-to-end setup process for installing and running Cumulus Linux, as
well as a collection of example commands for getting started once installation is complete.

Prerequisites
Prior intermediate Linux knowledge is assumed for this guide. You should be familiar with basic
text editing, Unix file permissions, and process monitoring. A variety of text editors are pre-
installed, including vi and nano.
You must have access to a Linux or UNIX shell. If you are running Windows, you should use a
Linux environment like Cygwin as your command line tool for interacting with Cumulus Linux.

If you're a networking engineer but are unfamiliar with Linux concepts, refer to this
reference guide for examples of the Cumulus Linux CLI and configuration options, and
their equivalent Cisco Nexus 3000 NX-OS commands and settings for comparison. You
can also watch a series of short videos introducing you to Linux in general and some
Cumulus Linux-specific concepts in particular.

Contents
This chapter covers ...
Installation (see page 19)
Upgrade to the Latest Version (see page 20)
Getting Started (see page 20)
Login Credentials (see page 20)
Serial Console Management (see page 20)
Wired Ethernet Management (see page 20)
Configuring the Hostname and Timezone (see page 21)
Verifying the System Time (see page 22)
Installing the License (see page 22)
Configuring Breakout Ports with Splitter Cables (see page 23)
Testing Cable Connectivity (see page 23)
Configuring Switch Ports (see page 24)
Layer 2 Port Configuration (see page 24)
Layer 3 Port Configuration (see page 26)
Configuring a Loopback Interface (see page 27)

18 23 January 2018
 Cumulus Networks

Installation
To install Cumulus Linux, you use ONIE (Open Network Install Environment), an extension to the traditional
U-Boot software that allows for automatic discovery of a network installer image. This facilitates the
ecosystem model of procuring switches, with a user's own choice of operating system loaded, such as
Cumulus Linux.

If Cumulus Linux 3.0.0 or later is already installed on your switch, and you need to upgrade the
software only, you can skip to Upgrading Cumulus Linux (see page 20) below.

The easiest way to install Cumulus Linux with ONIE is via local HTTP discovery:

1. If your host (like a laptop or server) is IPv6-enabled, make sure it is running a web server.
If the host is IPv4-enabled, make sure it is running DHCP as well as a web server.
2. Download the Cumulus Linux installation file to the root directory of the web server. Rename this file
onie-installer.
3. Connect your host via Ethernet cable to the management Ethernet port of the switch.
4. Power on the switch. The switch downloads the ONIE image installer and boots it. You can watch the
progress of the install in your terminal. After the installation finishes, the Cumulus Linux login prompt
appears in the terminal window.

These steps describe a flexible unattended installation method. You should not need a console
cable. A fresh install via ONIE using a local web server should generally complete in less than 10
minutes.
You have more options for installing Cumulus Linux with ONIE. Read Installing a New Cumulus
Linux Image (see page 34) to install Cumulus Linux using ONIE in the following ways:
DHCP/web server with and without DHCP options
Web server without DHCP
FTP or TFTP without a web server
Local file
USB

ONIE supports many other discovery mechanisms using USB (copy the installer to the root of the drive),
DHCPv6 and DHCPv4, and image copy methods including HTTP, FTP, and TFTP. For more information on
these discovery methods, refer to the ONIE documentation.
After installing Cumulus Linux, you are ready to:
Log in to Cumulus Linux on the switch.
Install the Cumulus Linux license.
Configure Cumulus Linux. This quick start guide provides instructions on configuring switch ports
and a loopback interface.

[Link] 19
Cumulus Linux 3.5 User Guide

Upgrade to the Latest Version

If you are running a Cumulus Linux version earlier than 3.0.0, you must perform a complete install, as
described above (see page ). If you already have Cumulus Linux 3.0.0 or later installed on your switch,
read Upgrading Cumulus Linux (see page 29) for considerations before start the process.

Getting Started
When bringing up Cumulus Linux for the first time, the management port makes a DHCPv4 request. To
determine the IP address of the switch, you can cross reference the MAC address of the switch with your
DHCP server. The MAC address should be located on the side of the switch or on the box in which the unit
was shipped.

Login Credentials
The default installation includes one system account, root, with full system privileges, and one user account,
cumulus, with sudo privileges. The root account password is set to null by default (which prohibits login),
while the cumulus account is configured with this default password:

CumulusLinux!

In this quick start guide, you will use the cumulus account to configure Cumulus Linux.

For best security, you should change the default password (using the passwd command) before
you configure Cumulus Linux on the switch.

All accounts except root are permitted remote SSH login; sudo may be used to grant a non-root account
root-level access. Commands which change the system configuration require this elevated level of access.
For more information about sudo, read Using sudo to Delegate Privileges (see page 105).

Serial Console Management

Users are encouraged to perform management and configuration over the network, either in band or out
of band (see page 51). Use of the serial console is fully supported; however, many customers prefer the
convenience of network-based management.
Typically, switches ship from the manufacturer with a mating DB9 serial cable. Switches with ONIE are
always set to a 115200 baud rate.

Wired Ethernet Management

Switches supported in Cumulus Linux always contain at least one dedicated Ethernet management port,
which is named eth0. This interface is geared specifically for out-of-band management use. The
management interface uses DHCPv4 for addressing by default. You can set a static IP address with the
Network Command Line Utility (NCLU).

20 23 January 2018
 Cumulus Networks

Example IP Configuration
Set the static IP address with the interface address and interface gateway NCLU
commands:

cumulus@switch:~$ net add interface eth0 ip address [Link]

/24
cumulus@switch:~$ net add interface eth0 ip gateway [Link]
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands produce the following snippet in the /etc/network/interfaces file:

auto eth0
iface eth0
address [Link]/24
gateway [Link]

Configuring the Hostname and Timezone

To change the hostname, run net add hostname, which modifies both the /etc/hostname and /etc
/hosts files with the desired hostname.

cumulus@switch:~$ net add hostname <hostname>

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The command prompt in the terminal doesn't reflect the new hostname until you either log out of
the switch or start a new shell.

When you use this NCLU command to set the hostname, DHCP does not override the hostname
when you reboot the switch. However, if you disable the hostname setting with NCLU, DHCP does
override the hostname the next time you reboot the switch.

To update the timezone, use the NTP interactive mode:

[Link] 21
Cumulus Linux 3.5 User Guide

1. Run the following command in a terminal:

sudo dpkg-reconfigure tzdata

2. Follow the on screen menu options to select the correct location.

Programs that are already running (including log files), and users currently logged in, will not see
timezone changes made with interactive mode.

Verifying the System Time

Before you install the license, verify that the switch's date and time are correct. You must correct the time
(see page 95) if it is wrong, as the wrong date may have other impacts on the switch like an inability to
synchronize with Puppet or return errors like this one after you restart switchd:

Warning: Unit file of [Link] changed on disk, 'systemctl daemon-reload' recommended.

Installing the License

Cumulus Linux is licensed on a per-instance basis. Each network system is fully operational, enabling any
capability to be utilized on the switch with the exception of forwarding on switch panel ports. Only eth0 and
console ports are activated on an unlicensed instance of Cumulus Linux. Enabling front panel ports
requires a license.
You should have received a license key from Cumulus Networks or an authorized reseller. Here is a sample
license key:

user@[Link]|thequickbrownfoxjumpsoverthelazydog312

There are three ways to install the license onto the switch:
Copy it from a local server. Create a text file with the license and copy it to a server accessible from
the switch. On the switch, use the following command to transfer the file directly on the switch, then
install the license file:

cumulus@switch:~$ scp user@my_server:/home/user/my_license_file.

txt .
cumulus@switch:~$ sudo cl-license -i my_license_file.txt

Copy the file to an HTTP server (not HTTPS), then reference the URL when you run cl-license:

cumulus@switch:~$ sudo cl-license -i <URL>

22 23 January 2018
 Cumulus Networks

Copy and paste the license key into the cl-license command:

cumulus@switch:~$ sudo cl-license -i

<paste license key>
^+d

You don't have to reboot the switch to activate the switch ports. Once you install the license,
restart the switchd service. All front panel ports become active and show up as swp1, swp2, and
so forth.

cumulus@switch:~$ sudo systemctl restart [Link]

If a license is not installed on a Cumulus Linux switch, the switchd service does not start. Once
you install the license, start switchd as described above.

Configuring Breakout Ports with Splitter Cables

If you are using 4x10G DAC or AOC cables, or want to break out 100G or 40G switch ports, configure the
breakout ports. For more details, see Layer 1 and Switch Port Attributes (see page 241).

Testing Cable Connectivity

By default, all data plane ports (every Ethernet port except the management interface, eth0) are disabled.
To test cable connectivity, administratively enable a port:

cumulus@switch:~$ net add interface swp1

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

To administratively enable all physical ports, run the following command, where swp1-52 represents a
switch with switch ports numbered from swp1 to swp52:

cumulus@switch:~$ net add interface swp1-52

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

To view link status, use net show interface all. The following examples show the output of ports in
"admin down", "down" and "up" modes:

[Link] 23
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ net show interface all

Name Speed MTU Mode
Summary
----- ------------------------ ------- ----- -------------
--------------------------------------
UP lo N/A 65536 Loopback IP:
[Link]/32, [Link]/8, ::1/128
UP eth0 1G 1500 Mgmt IP:
[Link]/24(DHCP)
UP swp1 (hypervisor_port_1) 1G 1500 Access/L2
Untagged: br0
UP swp2 1G 1500 NotConfigured
ADMDN swp45 0M 1500 NotConfigured
ADMDN swp46 0M 1500 NotConfigured
ADMDN swp47 0M 1500 NotConfigured
ADMDN swp48 0M 1500 NotConfigured
ADMDN swp49 0M 1500 NotConfigured
ADMDN swp50 0M 1500 NotConfigured
UP swp51 1G 1500 BondMember
Master: bond0(DN)
UP blue N/A 65536 NotConfigured
DN bond0 N/A 1500 Bond Bond
Members: swp51(UN)
UP br0 N/A 1500 Bridge/L3 IP:
[Link]/24

Untagged Members: swp1

802.1
q Tag: Untagged
STP:
RootSwitch(32768)
UP red N/A 65536 NotConfigured
ADMDN rename13 0M 1500 NotConfigured
ADMDN vagrant 0M 1500 NotConfigured

Configuring Switch Ports

Layer 2 Port Configuration

Cumulus Linux does not put all ports into a bridge by default. To create a bridge and configure one or more
front panel ports as members of the bridge, use the following examples as guides.

24 23 January 2018
 Cumulus Networks

Examples

Example One
In the following configuration example, the front panel port swp1 is placed into a bridge called
bridge. The NCLU commands are:

cumulus@switch:~$ net add bridge bridge ports swp1

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The commands above produce the following /etc/network/interfaces snippet:

auto bridge
iface bridge
bridge-ports swp1
bridge-vlan-aware yes

Example Two
A range of ports can be added in one command. For example, add swp1 through swp10, swp12,
and swp14 through swp20 to bridge:

cumulus@switch:~$ net add bridge bridge ports swp1-10,12,14-20

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

This creates the following /etc/network/interfaces snippet:

auto bridge
iface bridge
bridge-ports swp1 swp2 swp3 swp4 swp5 swp6 swp7 swp8
swp9 swp10 swp12 swp14 swp15 swp16 swp17 swp18 swp19 swp20
bridge-vlan-aware yes

To view the changes in the kernel, use the brctl command:

[Link] 25
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ brctl show

bridge name bridge id STP enabled interfaces
bridge 8000.443839000004 yes swp1
swp2

A script is available to generate a configuration that places all physical ports in a single bridge.

Layer 3 Port Configuration

The NCLU can also be used to configure a front panel port or bridge interface as a layer 3 port.
In the following configuration example, the front panel port swp1 is configured as a layer 3 access port:

cumulus@switch:~$ net add interface swp1 ip address [Link]/30

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

This creates the following /etc/network/interfaces snippet:

auto swp1
iface swp1
address [Link]/30

To add an IP address to a bridge interface, it must be put into a VLAN interface:

cumulus@switch:~$ net add vlan 100 ip address [Link]/24

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

This creates the following /etc/network/interfaces snippet:

auto bridge
iface bridge
bridge-vids 100
bridge-vlan-aware yes

auto vlan100
iface vlan100
address [Link]/24
vlan-id 100
vlan-raw-device bridge

26 23 January 2018
 Cumulus Networks

To view the changes in the kernel use the ip addr show command:

cumulus@switch:~$ ip addr show

...

4. swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast

master bridge state UP group default qlen 1000
link/ether [Link] brd [Link]

...

14: bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue

state UP group default
link/ether [Link] brd [Link]
inet6 fe80::4638:39ff:fe00:4/64 scope link
valid_lft forever preferred_lft forever
...

Configuring a Loopback Interface

Cumulus Linux has a loopback preconfigured in /etc/network/interfaces. When the switch boots up,
it has a loopback interface, called lo, which is up and assigned an IP address of [Link].

The loopback interface lo must always be specified in /etc/network/interfaces and must

always be up.

To see the status of the loopback interface (lo), use the net show interface lo command:

cumulus@switch:~$ net show interface lo

Name MAC Speed MTU Mode
-- ------ ----------------- ------- ----- --------
UP lo [Link] N/A 65536 Loopback

IP Details
------------------------- --------------------
IP: [Link]/8, ::1/128
IP Neighbor(ARP) Entries: 0

Note that the loopback is up and is assigned an IP address of [Link].

To add an IP address to a loopback interface, configure the lo interface with NCLU:

cumulus@switch:~$ net add loopback lo ip address [Link]/32

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Multiple loopback addresses can be configured by adding additional address lines:

[Link] 27
Cumulus Linux 3.5 User Guide

Multiple loopback addresses can be configured by adding additional address lines:

cumulus@switch:~$ net add loopback lo ip address [Link]/24

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

This creates the following snippet in /etc/network/interfaces:

auto lo
iface lo inet loopback
address [Link]/32
address [Link]/24

Installation
28 Management 23 January 2018
 Cumulus Networks

Installation Management
A Cumulus Linux switch can have only one image of the operating system installed. This section discusses
installing new and updating existing Cumulus Linux disk images, and configuring those images with
additional applications (via packages) if desired.
Zero touch provisioning is a way to quickly deploy and configure new switches in a large-scale environment.

Managing Cumulus Linux Disk Images

The Cumulus Linux operating system resides on a switch as a disk image. This section discusses how to
manage the image, including installation and upgrading.

Contents
This chapter covers ...
Installing a New Cumulus Linux Image (see page 29)
Upgrading Cumulus Linux (see page 29)
x86 vs ARM Switches (see page 29)
Reprovisioning the System (Restart Installer) (see page 30)
Uninstalling All Images and Removing the Configuration (see page 30)
Booting into Rescue Mode (see page 31)
Inspecting Image File Contents (see page 32)
Related Information (see page 33)

Installing a New Cumulus Linux Image

For details, read the chapter, Installing a New Cumulus Linux Image (see page 34).

Upgrading Cumulus Linux

There are two ways you can upgrade Cumulus Linux:
Upgrading only the changed packages, using apt-get update and apt-get upgrade. This is
the preferred method.
Perform a binary (full image) install of the new version, using ONIE. This is used when moving
between major versions or if you want to install a clean image.
The entire upgrade process is described in Upgrading Cumulus Linux (see page 48).

x86 vs ARM Switches

You can easily determine whether your switch is on an x86 or ARM platform by using the uname -m
command.
For example, on an x86 platform, uname -m outputs x86_64:

[Link] 29
Cumulus Linux 3.5 User Guide

cumulus@x86switch$ uname -m
x86_64

While on an ARM platform, uname -m outputs armv7l:

cumulus@ARMswitch$ uname -m
armv7l

You can also visit the HCL (hardware compatibility list) to look at your hardware to determine the processor
type.

Reprovisioning the System (Restart Installer)

You can reprovision the system, wiping out the contents of the whole switch.
To initiate the provisioning and installation process, use onie-select -i:

cumulus@switch:~$ sudo onie-select -i

WARNING:
WARNING: Operating System install requested.
WARNING: This will wipe out all system data.
WARNING:
Are you sure (y/N)? y
Enabling install at next reboot...done.
Reboot required to take effect.

A reboot is required for the reinstall to begin.

If you change your mind, you can cancel a pending reinstall operation by using onie-select -c
:

cumulus@switch:~$ sudo onie-select -c

Cancelling pending install at next reboot...done.

Uninstalling All Images and Removing the Configuration

To remove all installed images and configurations, returning the switch to its factory defaults, use onie-
select -k:

30 23 January 2018
 Cumulus Networks

cumulus@switch:~$ sudo onie-select -k

WARNING:
WARNING: Operating System uninstall requested.
WARNING: This will wipe out all system data.
WARNING:
Are you sure (y/N)? y
Enabling uninstall at next reboot...done.
Reboot required to take effect.

A reboot is required for the uninstall to begin.

If you change your mind you can cancel a pending uninstall operation by using onie-select -c
:

cumulus@switch:~$ sudo onie-select -c

Cancelling pending uninstall at next reboot...done.

Booting into Rescue Mode

If your system becomes broken is some way, you may be able to correct things by booting into ONIE rescue
mode. In rescue mode, the file systems are unmounted and you can use various Cumulus Linux utilities to
try and fix the problem.
To reboot the system into the ONIE rescue mode, use onie-select -r:

cumulus@switch:~$ sudo onie-select -r

WARNING:
WARNING: Rescue boot requested.
WARNING:
Are you sure (y/N)? y
Enabling rescue at next reboot...done.
Reboot required to take effect.

A reboot is required to boot into rescue mode.

[Link] 31
Cumulus Linux 3.5 User Guide

If you change your mind you can cancel a pending rescue boot operation by using onie-
select -c:

cumulus@switch:~$ sudo onie-select -c

Cancelling pending rescue at next reboot...done.

Inspecting Image File Contents

The Cumulus Linux installation disk image file is executable. From a running switch, you can display the
contents of the Cumulus Linux image file by passing the info option to the image file. For example, if the
image file is called onie-installer and is located in /var/lib/cumulus/installer, you can get
information about the disk image by running:

cumulus@switch:~$ sudo /var/lib/cumulus/installer/onie-installer info

Verifying image checksum ... OK.
Preparing image archive ... OK.
Control File Contents
=====================
Description: Cumulus Linux
OS-Release: 2.1.0-0556262-201406101128-NB
Architecture: amd64
Date: Tue, 10 Jun 2014 [Link] -0700
Installer-Version: 1.2
Platforms: im_n29xx_t40n mlx_sx1400_i73612 dell_s6000_s1220
Homepage: [Link]

Data Archive Contents

=====================
128 2014-06-10 [Link] [Link]
44 2014-06-10 [Link] [Link].sha1
104276331 2014-06-10 [Link] [Link]
44 2014-06-10 [Link] [Link].sha1
5391348 2014-06-10 [Link] [Link]
44 2014-06-10 [Link] [Link].sha1
cumulus@switch:~$

You can also extract the contents of the image file by passing the extract option to the image file:

32 23 January 2018
 Cumulus Networks

cumulus@switch:~$ sudo /var/lib/cumulus/installer/onie-installer

extract PATH
Verifying image checksum ... OK.
Preparing image archive ... OK.
[Link]
[Link].sha1
[Link]
[Link].sha1
[Link]
[Link].sha1
Success: Image files extracted OK.
cumulus@switch:~$ sudo ls -l
total 107120
-rw-r--r-- 1 1063 3000 128 Jun 10 18:44 [Link]
-rw-r--r-- 1 1063 3000 44 Jun 10 18:44 [Link].sha1
-rw-r--r-- 1 1063 3000 104276331 Jun 10 18:44 [Link]
-rw-r--r-- 1 1063 3000 44 Jun 10 18:44 [Link].
sha1
-rw-r--r-- 1 1063 3000 5391348 Jun 10 18:44 [Link]
-rw-r--r-- 1 1063 3000 44 Jun 10 18:44 [Link].
sha1

Finally, you can verify the contents of the image file by passing the verify option to the image file:

cumulus@switch:~$ sudo /var/lib/cumulus/installer/onie-installer

verify
Verifying image checksum ... OK.
Preparing image archive ... OK.
[Link]
[Link].sha1
[Link]
[Link].sha1
[Link]
[Link].sha1
Success: Image files extracted OK.
cumulus@switch:~$ sudo ls -l
total 107120
-rw-r--r-- 1 1063 3000 128 Jun 10 18:44 [Link]
-rw-r--r-- 1 1063 3000 44 Jun 10 18:44 [Link].sha1
-rw-r--r-- 1 1063 3000 104276331 Jun 10 18:44 [Link]
-rw-r--r-- 1 1063 3000 44 Jun 10 18:44 [Link].
sha1
-rw-r--r-- 1 1063 3000 5391348 Jun 10 18:44 [Link]
-rw-r--r-- 1 1063 3000 44 Jun 10 18:44 [Link].
sha1

Related Information
[Link] 33
Cumulus Linux 3.5 User Guide

Related Information
Open Network Install Environment (ONIE) Home Page

Installing a New Cumulus Linux Image

Before you install Cumulus Linux, the switch can be in two different states:
The switch has no image on it (so the switch is only running ONIE) or you desire or require a clean
installation. In this case, you can install Cumulus Linux in one of the following ways, using:
DHCP/a web server with DHCP options (see page 35)
DHCP/a web server without DHCP options (see page 36)
A web server with no DHCP (see page 36)
FTP or TFTP without a web server (see page 37)
Local file installation (see page 38)
USB (see page 38)
The switch already has Cumulus Linux installed on it, so you only need to upgrade it (see page 48).

ONIE is an open source project, equivalent to PXE on servers, that enables the installation of
network operating systems (NOS) on bare metal switches.

Understanding these Examples

The sections in this chapter are ordered from the most repeatable to the least repeatable methods. For
instance, DHCP can scale to hundreds of switch installs with zero manual input, compared to something
like USB installs. Installing via USB is fine for a single switch here and there but is not scalable.
You can name your Cumulus Linux installer binary using any of the ONIE naming schemes
mentioned here.
In the examples below, [PLATFORM] can be any supported Cumulus Linux platform, such as x86_64,
or arm.

Contents
This chapter covers ...
Understanding these Examples (see page 34)
Installing via a DHCP/Web Server Method with DHCP Options (see page 35)
Installing via a DHCP/Web Server Method without DHCP Options (see page 36)
Installing via a Web Server with no DHCP (see page 36)
Installing from Cumulus Linux (see page 36)
Installing from ONIE (see page 36)
Installing via FTP or TFTP without a Web Server (see page 37)
Installing from Cumulus Linux (see page 37)
Installing from ONIE (see page 37)

34 23 January 2018
 Cumulus Networks

Installing via a Local File (see page 38)

Installing from Cumulus Linux (see page 38)
Installing from ONIE (see page 38)
Installing via USB (see page 38)
Preparing for USB Installation (see page 38)
Instructions for x86 Platforms (see page 41)
Instructions for ARM Platforms (see page 44)
Installing a New Image when Cumulus Linux Is already Installed (see page 47)
Entering ONIE Mode from Cumulus Linux (see page 47)

Installing via a DHCP/Web Server Method with DHCP Options

Installing Cumulus Linux in this manner is as simple as setting up a DHCP/web server on your laptop and
connecting the eth0 management port of the switch to your laptop.
Once you connect the cable, the installation proceeds as follows:

1. The bare metal switch boots up and asks for an address (DHCP request).
2. The DHCP server acknowledges and responds with DHCP option 114 and the location of the
installation image.
3. ONIE downloads the Cumulus Linux binary, installs and reboots.
4. Success! You are now running Cumulus Linux.

The most common method is for you to send DHCP option 114 with the entire URL to the web
server (this could be the same system). However, there are many other ways to use DHCP even if
you don't have full control over DHCP. See the ONIE user guide for help.

Here's an example DHCP configuration with an ISC DHCP server:

subnet [Link] netmask [Link] {

range [Link] [Link];
option default-url = "[Link]
}

Here's an example DHCP configuration with dnsmasq (static address assignment):

[Link] 35
Cumulus Linux 3.5 User Guide

dhcp-host=sw4,[Link],[Link],set:sw4
dhcp-option=tag:sw4,114,"[Link]
[PLATFORM]"

If you don't have a web server, you can use this free Apache example.

Installing via a DHCP/Web Server Method without DHCP Options

If you have a laptop on the same network and the switch can pull DHCP from the corporate network, but
you cannot modify DHCP options (maybe it's controlled by another team), do the following:

1. Place the Cumulus Linux binary in a directory on the web server.

2. Run the installer manually, since DHCP options can't be modified, either from ONIE or the Cumulus
Linux command prompt.
From ONIE, run the onie-nos-install command:

ONIE:/ #onie-nos-install [Link]

install-[PLATFORM].bin

From Cumulus Linux, run the onie-install command:

cumulus@switch:~$ sudo onie-install -a -i [Link]

/path/to/cumulus-install-[PLATFORM].bin && sudo reboot

Installing via a Web Server with no DHCP

If your laptop is on the same network as the switch eth0 interface but no DHCP server is available, you can
still install directly from Cumulus Linux or using ONIE.

Installing from Cumulus Linux

From Cumulus Linux, run the onie-install command:

cumulus@switch:~$ sudo onie-install -a -i [Link]

/cumulus-install-[PLATFORM].bin && sudo reboot

Installing from ONIE

Do the following steps to run the install using ONIE. Note that ONIE is in discovery mode:

36 23 January 2018
 Cumulus Networks

1. To disable discovery mode, run:

onie# onie-discovery-stop

or, on older ONIE versions if that command isn't supported:

onie# /etc/init.d/[Link] stop

2. Assign a static address to eth0 via ONIE (using ip addr add):

ONIE:/ #ip addr add [Link]/24 dev eth0

3. Place the Cumulus Linux installer image in a directory on your web server.
4. Run the installer manually, since there are no DHCP options:

ONIE:/ #onie-nos-install [Link]

install-[PLATFORM].bin

Installing via FTP or TFTP without a Web Server

If your laptop is on the same network as the switch eth0 interface but no DHCP server is available, you can
still install directly from Cumulus Linux or using ONIE.

Installing from Cumulus Linux

If you are not utilizing DHCP options, run one of the following commands (tftp for TFTP or ftp for FTP),
from the Cumulus Linux command prompt:

cumulus@switch:~$ sudo onie-install -a -i [Link]

/cumulus-install-[PLATFORM].bin && sudo reboot

cumulus@switch:~$ sudo onie-install -a -i t[Link]

/cumulus-install-[PLATFORM].bin && sudo reboot

Installing from ONIE

To install without DHCP options using ONIE, do the following:

1. Set up DHCP or static addressing for eth0, as in the examples above.

2. If you are utilizing static addressing, disable ONIE discovery mode.
3. Place the Cumulus Linux installer image into a TFTP or FTP directory.

[Link] 37
Cumulus Linux 3.5 User Guide

4. If you are not utilizing DHCP options, run one of the following commands (tftp for TFTP or ftp for
FTP):

ONIE# onie-nos-install [Link]

[PLATFORM].bin

ONIE# onie-nos-install t[Link]

[PLATFORM].bin

Installing via a Local File

You can still install referencing a local file, directly from Cumulus Linux or using ONIE.

Installing from Cumulus Linux

From Cumulus Linux, run the onie-install command:

cumulus@switch:~$ sudo onie-install -a -i /path/to/local/file/cumulus-

install-[PLATFORM].bin && sudo reboot

Installing from ONIE

1. Set up DHCP or static addressing for eth0, as in the examples above.
2. If you are utilizing static addressing, disable ONIE discovery mode.
3. Use scp to copy the Cumulus Linux binary to the switch.
Note: Windows users can use WinScp.
4. Run the installer manually from ONIE:

ONIE:/ #onie-nos-install /path/to/local/file/cumulus-install-

[PLATFORM].bin

Installing via USB

Following the steps below produces a clean installation of Cumulus Linux. This wipes out all pre-existing
configuration files that may be present on the switch. Instructions are offered for x86 and ARM platforms,
and also cover the installation of a license after the software installation.

Make sure to back up (see page 48) any important configuration files that you may need to
restore the configuration of your switch after the installation finishes.

Preparing for USB Installation

38 23 January 2018
 Cumulus Networks

Preparing for USB Installation

1. Download the appropriate Cumulus Linux image for your x86 or ARM platform from the Cumulus
Networks Downloads page.
2. Prepare your flash drive by formatting in one of the supported formats: FAT32, vFAT or EXT2.
Optional: Preparing a USB Drive inside Cumulus Linux

It is possible that you could severely damage your system with the following utilities, so
please use caution when performing the actions below!

[Link] 39
Cumulus Linux 3.5 User Guide

a. Insert your flash drive into the USB port on the switch running Cumulus Linux and log in to
the switch.
b. Determine and note at which device your flash drive can be found by using output from
cat /proc/partitions and sudo fdisk -l [device]. For example, sudo fdisk -
l /dev/sdb.

These instructions assume your USB drive is the /dev/sdb device, which is
typical if the USB stick was inserted after the machine was already booted.
However, if the USB stick was plugged in during the boot process, it is possible the
device could be /dev/sda. Make sure to modify the commands below to use the
proper device for your USB drive!

c. Create a new partition table on the device:

sudo parted /dev/sdb mklabel msdos

The parted utility should already be installed. However, if it is not, install it with:
sudo -E apt-get install parted

d. Create a new partition on the device:

sudo parted /dev/sdb -a optimal mkpart primary 0% 100%

e. Format the partition to your filesystem of choice using ONE of the examples below:

sudo mkfs.ext2 /dev/sdb1

sudo [Link] -F 32 /dev/sdb1
sudo [Link] /dev/sdb1

To use [Link] or [Link], you need to install the dosfstools package

from the Debian software repositories (step 3 here shows you how to add
repositories from Debian), as they are not included by default.

f. To continue installing Cumulus Linux, mount the USB drive in order to move files to it.

sudo mkdir /mnt/usb

sudo mount /dev/sdb1 /mnt/usb

40 23 January 2018
 Cumulus Networks

3. Copy the image and license files over to the flash drive and rename the image file to:
onie-installer-x86_64, if installing on an x86 platform
onie-installer-arm, if installing on an ARM platform

You can also use any of the ONIE naming schemes mentioned here.

When using a Mac or Windows computer to rename the installation file the file extension
may still be present. Make sure to remove the file extension otherwise ONIE will not be
able to detect the file!

4. Insert the USB stick into the switch, then continue with the appropriate instructions below for your
x86 or ARM platform.

Instructions for x86 Platforms

Click to expand x86 instructions...

1. Prepare the switch for installation:

If the switch is offline, connect to the console and power on the switch.
If the switch is already online in Cumulus Linux, connect to the console and reboot the switch
into the ONIE environment with the sudo onie-select -i command, followed by sudo
reboot. Then skip to step 4 below.
If the switch is already online in ONIE, use the reboot command.

SSH sessions to the switch get dropped after this step. To complete the remaining
instructions, connect to the console of the switch. Cumulus Linux switches display their
boot process to the console, so you need to monitor the console specifically to complete
the next step.

[Link] 41
Cumulus Linux 3.5 User Guide

2. Monitor the console and select the ONIE option from the first GRUB screen shown below.

3. Cumulus Linux on x86 uses GRUB chainloading to present a second GRUB menu specific to the
ONIE partition. No action is necessary in this menu to select the default option ONIE: Install OS.

42 23 January 2018
 Cumulus Networks

4. At this point, the USB drive should be automatically recognized and mounted. The image file should
be located and automatic installation of Cumulus Linux should begin. Here is some sample output:

ONIE: OS Install Mode ...

Version : quanta_common_rangeley-2014.05.05-6919d98-201410171013
Build Date: 2014-10-17T10:13+0800
Info: Mounting kernel filesystems... done.
Info: Mounting LABEL=ONIE-BOOT on /mnt/onie-boot ...
initializing eth0...
scsi [Link] Direct-Access SanDisk Cruzer Facet 1.26 PQ: 0
ANSI: 6
sd [Link] [sdb] 31266816 512-byte logical blocks: (16.0 GB/14.
9 GiB)
sd [Link] [sdb] Write Protect is off
sd [Link] [sdb] Write cache: disabled, read cache: enabled,
doesn't support DPO or FUA
sd [Link] [sdb] Attached SCSI disk
<...snip...>
ONIE: Executing installer: [Link]
Verifying image checksum ... OK.
Preparing image archive ... OK.
Dumping image info...
Control File Contents
=====================
Description: Cumulus Linux
OS-Release: 3.0.0-3b46bef-201509041633-build
Architecture: amd64
Date: Fri, 27 May 2016 [Link] -0700
Installer-Version: 1.2
Platforms: accton_as5712_54x accton_as6712_32x
mlx_sx1400_i73612 dell_s6000_s1220 dell_s4000_c2338
dell_s3000_c2338 cel_redstone_xp cel_smallstone_xp cel_pebble
quanta_panther quanta_ly8_rangeley quanta_ly6_rangeley
quanta_ly9_rangeley
Homepage: [Link]

5. After installation completes, the switch automatically reboots into the newly installed instance of
Cumulus Linux.
6. Determine and note at which device your flash drive can be found by using output from cat /proc
/partitions and sudo fdisk -l [device]. For example, sudo fdisk -l /dev/sdb.

These instructions assume your USB drive is the /dev/sdb device, which is typical if the
USB stick was inserted after the machine was already booted. However, if the USB stick
was plugged in during the boot process, it is possible the device could be /dev/sda. Make
sure to modify the commands below to use the proper device for your USB drive!

[Link] 43
Cumulus Linux 3.5 User Guide

7. Create a mount point to mount the USB drive to:

sudo mkdir /mnt/mountpoint

8. Mount the USB drive to the newly created mount point:

sudo mount /dev/sdb1 /mnt/mountpoint

9. Install your license file with the cl-license command:

sudo cl-license -i /mnt/mountpoint/[Link]

10. Check that your license is installed with the cl-license command.
11. Reboot the switch to utilize the new license.

sudo reboot

Instructions for ARM Platforms

Click to expand ARM instructions...

1. Prepare the switch for installation:

If the switch is offline, connect to the console and power on the switch.
If the switch is already online in Cumulus Linux, connect to the console and reboot the switch
into the ONIE environment with the sudo onie-select -i command, followed by sudo
reboot. Then skip to step 4 below.
If the switch is already online in ONIE, use the reboot command.

SSH sessions to the switch get dropped after this step. To complete the remaining
instructions, connect to the console of the switch. Cumulus Linux switches display their
boot process to the console, so you need to monitor the console specifically to complete
the next step.

44 23 January 2018
 Cumulus Networks

2. Interrupt the normal boot process before the countdown (shown below) completes. Press any key to
stop the autobooting.

U-Boot 2013.01-00016-gddbf4a9-dirty (Feb 14 2014 - [Link])

Accton: [Link]
CPU0: P2020, Version: 2.1, (0x80e20021)
Core: E500, Version: 5.1, (0x80211051)
Clock Configuration:
CPU0:1200 MHz, CPU1:1200 MHz,
CCB:600 MHz,
DDR:400 MHz (800 MT/s data rate) (Asynchronous), LBC:37.500 MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled
<...snip…>
USB: USB2513 hub OK
Hit any key to stop autoboot: 0

3. A command prompt appears, so you can run commands. Execute the following command:

run onie_bootcmd

[Link] 45
Cumulus Linux 3.5 User Guide

4. At this point the USB drive should be automatically recognized and mounted. The image file should
be located and automatic installation of Cumulus Linux should begin. Here is some sample output:

Loading Open Network Install Environment …

Platform: arm-as4610_54p-r0
Version : [Link]
WARNING: adjusting available memory to 30000000
## Booting kernel from Legacy Image at ec040000 …
Image Name: as6701_32x.[Link]
Image Type: ARM Linux Multi-File Image (gzip compressed)
Data Size: 4456555 Bytes = 4.3 MiB
Load Address: 00000000
Entry Point: 00000000
Contents:
Image 0: 3738543 Bytes = 3.6 MiB
Image 1: 706440 Bytes = 689.9 KiB
Image 2: 11555 Bytes = 11.3 KiB
Verifying Checksum ... OK
## Loading init Ramdisk from multi component Legacy Image at
ec040000 …
## Flattened Device Tree from multi component Image at EC040000
Booting using the fdt at 0xec47d388
Uncompressing Multi-File Image ... OK
Loading Ramdisk to 2ff53000, end 2ffff788 ... OK
Loading Device Tree to 03ffa000, end 03fffd22 ... OK
<...snip...>
ONIE: Starting ONIE Service Discovery
ONIE: Executing installer: [Link]
Verifying image checksum ... OK.
Preparing image archive ... OK.
Dumping image info…
Control File Contents
=====================
Description: Cumulus Linux
OS-Release: 3.0.0-3b46bef-201509041633-build
Architecture: arm
Date: Fri, 27 May 2016 [Link] -0700
Installer-Version: 1.2
Platforms: accton_as4600_54t, accton_as6701_32x, accton_5652,
accton_as5610_52x, dni_6448, dni_7448, dni_c7448n, cel_kennisis,
cel_redstone, cel_smallstone, cumulus_p2020, quanta_lb9,
quanta_ly2, quanta_ly2r, quanta_ly6_p2020
Homepage: [Link]

5. After installation completes, the switch automatically reboots into the newly installed instance of
Cumulus Linux.

46 23 January 2018
 Cumulus Networks

6. Determine and note at which device your flash drive can be found by using output from cat /proc
/partitions and sudo fdisk -l [device]. For example, sudo fdisk -l /dev/sdb.

These instructions assume your USB drive is the /dev/sdb device, which is typical if the
USB stick was inserted after the machine was already booted. However, if the USB stick
was plugged in during the boot process, it is possible the device could be /dev/sda. Make
sure to modify the commands below to use the proper device for your USB drive!

7. Create a mount point to mount the USB drive to:

sudo mkdir /mnt/mountpoint

8. Mount the USB drive to the newly created mount point:

sudo mount /dev/sdb1 /mnt/mountpoint

9. Install your license file with the cl-license command:

sudo cl-license -i /mnt/mountpoint/[Link]

10. Check that your license is installed with the cl-license command.
11. Reboot the switch to utilize the new license.

sudo reboot

Installing a New Image when Cumulus Linux Is already Installed

At times it may be necessary to put the switch into ONIE in order to do an install. This may be required
when moving between major releases or re-installing from an early version of 3.y.z. For more information,
see Upgrading Cumulus Linux (see page 58).

Entering ONIE Mode from Cumulus Linux

If Cumulus Linux is already installed on the switch, you can enter ONIE mode in one of two ways, using:
ONIE Recovery Mode to manually install an image from the ONIE prompt:

cumulus@switch:~$ sudo onie-select -r

cumulus@switch:~$ sudo reboot

[Link] 47
Cumulus Linux 3.5 User Guide

ONIE Install Mode to attempt to automatically discover the image from a DHCP server:

cumulus@switch:~$ sudo onie-select -i

cumulus@switch:~$ sudo reboot

Upgrading Cumulus Linux

Cumulus Networks software melds the Linux host world with the networking devices world. Each world
comes with its own paradigm on how to upgrade software. Before we discuss the various ways to upgrade
Cumulus Linux switches, let's review the general considerations and strategies used to upgrade network
devices and Linux hosts.

Contents
This chapter covers ...
Upgrades: Comparing the Network Device Worldview vs. the Linux Host Worldview (see page 49)
Manual vs. Automated Configuration (see page 49)
Pre-deployment Testing of Production Environments (see page 49)
Locations of Configuration Data vs. Executables (see page 49)
Upgrade Procedure (see page 50)
Rollback Procedure (see page 50)
Third Party Packages (see page 50)
Upgrading Cumulus Linux Devices: Strategies and Processes (see page 50)
Automated Configuration Is Preferred over Manual Configuration (see page 50)
Out-of-Band Management Is Worth the Investment (see page 51)
Pre-Deployment Testing of New Releases Is Advised and Enabled (see page 51)
Understanding the Locations of Configuration Data is Required for Successful Upgrades,
Migration, and Backup (see page 51)
Upgrading Switches in an MLAG Pair (see page 55)
Upgrading Cumulus Linux: Choosing between a Binary Install vs. Package Upgrade (see page 56)
Upgrading Using Package Installs (apt-get update && apt-get upgrade) (see page 56)
Upgrading via Binary Install (ONIE) (see page 58)
Rolling Back a Cumulus Linux Installation (see page 58)
Third Party Package Considerations (see page 59)
Installation and Upgrade Workflow in Cumulus Linux 3.0 and Later (see page 59)
Using Snapshots during Upgrades (see page 59)
Caveats When Migrating Configuration Files Between Cumulus Linux 2.5.z and 3.0 and Later (see
page 59)
Using the Config File Migration Script to Identify and Move Files to Cumulus Linux 3.0 and Later (see
page 60)

48 23 January 2018
 Cumulus Networks

Using Automation Tools to Back Up 2.5.z Configurations (see page 61)

Upgrades: Comparing the Network Device Worldview vs. the Linux Host
Worldview

Manual vs. Automated Configuration

Historically, network devices were configured in place, and most network devices required customized
configurations, which led predominantly to configuring the hardware manually. A lack of standardization
between vendors, device types, and device roles hampered the development of APIs and automation tools.
However, in the case of very large data centers, configurations became uniform and repeatable, and
therefore scriptable. Some larger enterprises had to develop their own custom scripts to roll out data
center network configurations. Virtually no industry-standard provisioning tools existed.
In contrast to data center network devices, Linux hosts in the data center number in the thousands and
tend to have similar configurations. This increased scale led Linux sysadmins long ago to move to common
tools to automate installation and configuration, since manually installing and configuring hosts did not
work at the scale of a data center. Nearly all tasks are done via commonly available provisioning and
orchestration tools.

Pre-deployment Testing of Production Environments

Historically, the cost of network device testing has been hampered by the cost of a single device. Setting up
an appropriately sized lab topology can be very expensive. As a result, it is difficult to do comprehensive
topology-based testing of a release before deploying it. Thus, many network admins cannot or will not do
comprehensive system testing of a new release before deploying it.
Alternatively, the cost of a Linux host is cheap (or nearly free when using virtualization), so rigorous testing of
a release before deploying it is not encumbered by budgeting concerns. Most sysadmins extensively test
new releases in the complete application environment.

Locations of Configuration Data vs. Executables

Network devices generally separate configuration data from the executable code. On bootup, the executable
code looks into a different file system and retrieves the configuration file or files, parses the text and uses
that data to configure the software options for each software subsystem. The model is very centralized,
with the executables generally being packaged together, and the configuration data following a set of rules
that can be read by a centralized parser. Each vendor controls the configuration format for the entire box,
since each vendor generally supports only their own software. This made sense since the platform was
designed as an application-specific appliance.
Since a Linux host is a general purpose platform, with applications running on top of it, the locations of the
files are much more distributed. Applications install and read their configuration data from text files usually
stored in the /etc directory tree. Executables are generally stored in one of several bin directories, but the
bin and etc directories are often on the same physical device. Since each module (application or executable)
was often developed by a different organization and shared with the world, each module was responsible
for its own configuration data format. Most applications are community supported, and while there are
some generally accepted guiding principles on how their configuration data is formatted, no central
authority exists to control or ensure compliance.

[Link] 49
Cumulus Linux 3.5 User Guide

Upgrade Procedure
Both network admins and sysadmins generally plan upgrades only to gain new functionality or to get bug
fixes when the workarounds become too onerous. The goal is to reduce the number of upgrades as much
as possible.
The network device upgrade paradigm is to leave the configuration data in place, and replace the executable
files either all at once from a single binary image or in large chunks (subsystems). A full release upgrade
comes with risk due to unexpected behavior changes in subsystems where the admin did not anticipate or
need changes.
The Linux host upgrade paradigm is to independently upgrade a small list of packages while leaving most of
the OS untouched. Changing a small list of packages reduces the risk of unintended consequences. Usually
upgrades are a "forward only" paradigm, where the sysadmins generally plan to move to the latest code
within the same major release when needed. Every few years, when a new kernel train is released, a major
upgrade is planned. A major upgrade involves wiping and replacing the entire OS and migrating
configuration data.

Rollback Procedure
Even the most well planned and tested upgrades can result in unforeseen problems, and sometimes the
best solution to new problems is to roll back to the previous state.
Since network devices clearly separate data and executables, generally the process is to overwrite the new
release executable with the previously running executable. If the configuration was changed by the newer
release, then you either have to manually back out or repair the changes, or restore from an already
backed up configuration.
The Linux host scenario can be more complicated. There are three main approaches:
Back out individual packages: If the problematic package is identified, the sysadmin can downgrade
the affected package directly. In rare cases the configuration files may have to be restored from
backup, or edited to back out any changes that were automatically made by the upgrade package.
Flatten and rebuild: If the OS becomes unusable, you can use orchestration tools to reinstall the
previous OS release from scratch and then automatically rebuild the configuration.
Backup and restore: Another common strategy is to restore to a previous state via a backup
captured before the upgrade.

Third Party Packages

Third party packages are rare in the network device world. Because the network OS is usually proprietary,
third party packages are usually packaged by the network device vendor and upgrades of those packages is
handled by the network device upgrade system.
Third party packages in Linux host world often use the same package system as the distribution into which it
is to be installed (for example, Debian uses apt-get). Or the package may be compiled and installed by
the sysadmin. Configuration and executable files generally follow the same filesystem hierarchy standards
as other applications.

Upgrading Cumulus Linux Devices: Strategies and Processes

Because Cumulus Linux is both Linux and a network device, it has characteristics of both paradigms. The
following describes the Cumulus Linux paradigm with respect to upgrade planning and execution.

Automated Configuration Is Preferred over Manual Configuration

50 23 January 2018
 Cumulus Networks

Automated Configuration Is Preferred over Manual Configuration

Because Cumulus Linux is Linux, Cumulus Networks recommends that even with small networks or test
labs, network admins should make the jump to deploy, provision, configure, and upgrade switches using
automation from the very beginning. The small up front investment of time spent learning an orchestration
tool, even to provision a small number of Cumulus Linux devices, will pay back dividends for a long time.
The biggest gain is realized during the upgrade process, where the network admin can quickly upgrade
dozens of devices in a repeatable manner.
Switches, like servers, should be treated like cattle, not pets.

Out-of-Band Management Is Worth the Investment

Because network devices are reachable via the IP addresses on the front panel ports, many network
admins of small-to-medium sized networks use in-band networks to manage their switches. In this design,
management traffic like SSH, SNMP, and console server connections use the same networks that regular
network traffic traverses — there is no separation between the management plane and the data plane.
Larger data centers create a separate out-of-band network with a completely separate subnet and
reachability path to attach to the management ports — that is accessible via eth0 and the serial console.
This is a situation where smaller companies should learn from the big companies. A separate management
network isn't free, but it is relatively cheap. With an inexpensive Cumulus RMP management switch, an
inexpensive console server, and a separate cable path, up to 48 devices can be completely controlled via
the out-of-band network in the case of a network emergency.
There are many scenarios where in-band networking can fail and leave the network admin waiting for
someone to drive to the data center or remote site to connect directly to the console of a misconfigured or
failing device. The cost of one outage would usually more than pay for the investment in a separate
network. For even more security, attach remote-controllable power distribution units (PDUs) in each rack to
the management network, so you can have complete control to remote power cycle every device in that
rack.

However, if an out-of-band network is not available for you to upgrade, you can use the dtach tool
instead to upgrade in band.

Pre-Deployment Testing of New Releases Is Advised and Enabled

White box switches and virtualization (Cumulus VX) bring the cost of networking devices down, so the ability
for network admins to test their own procedures, configurations, applications, and network topology in an
appropriately-sized lab topology becomes extremely affordable.

Understanding the Locations of Configuration Data is Required for

Successful Upgrades, Migration, and Backup
As with other Linux distributions, the /etc directory is the primary location for all configuration data in
Cumulus Linux. The following list is a likely set of files that should be backed up and migrated to a new
release, but any file that has been changed would need to be examined. Cumulus Networks recommends
you consider making the following files and directories part of a backup strategy.

[Link] 51
Cumulus Linux 3.5 User Guide

Network Configuration Files

File Explanation Cumulus Linux Documentation Debian

Name Documentation
and
Location

/etc Network configuration Layer 1 and Switch Port Attributes (see page N/A
/network/ files, most notably 225)
/etc/network
/interfaces and
/etc/network
/interfaces.d/

/etc/resolv. DNS resolution Not unique to Cumulus Linux: [Link] [Link]

conf /NetworkConfiguration#The_resolv. /doc/manuals
conf_configuration_file /debian-reference
/[Link]

/etc/frr/ Routing application FRRouting Overview (see page 602) N/A

(responsible for BGP
and OSPF)

/etc Configuration file for Quick Start [Link]

/hostname the hostname of the Guide#ConfiguringtheHostnameandTimeZone /HowTo
switch (see page ) /ChangeHostname

/etc Netfilter configuration Netfilter - ACLs (see page 137) N/A

/cumulus
/acl/*

/etc Breakout cable Layer 1 and Switch Port N/A; please read
/cumulus configuration file Attributes#ConfiguringBreakoutPorts (see the guide on
/ports. page ) breakout cables
conf

/etc Switchd configuration Configuring switchd (see page 191) N/A; please read
/cumulus the guide on
/switchd. switchd
conf configuration

52 23 January 2018
 Cumulus Networks

Additional Commonly Used Files

File Explanation Cumulus Linux Debian

Name Documentation Documentation
and
Location

/etc Message of the day Not unique to [Link]

/motd Cumulus Linux /motd#Wheezy

/etc User account information Not unique to [Link]

/passwd Cumulus Linux /doc/manuals
/debian-
reference/ch04.
[Link]

/etc Secure user account information Not unique to [Link]

/shadow Cumulus Linux /doc/manuals
/debian-
reference/ch04.
[Link]

/etc Defines user groups on the switch Not unique to [Link]

/group Cumulus Linux /doc/manuals
/debian-
reference/ch04.
[Link]

/etc/lldpd. Link Layer Discover Protocol (LLDP) daemon Link Layer [Link].
conf configuration Discovery org/wheezy/lldpd
Protocol (see
page 298)

/etc/lldpd. Configuration directory for lldpd Link Layer [Link].

d/ Discovery org/wheezy/lldpd
Protocol (see
page 298)

/etc Name Service Switch (NSS) configuration file TACACS Plus (see N/A
/nsswitch. page 120)
conf

/etc/ssh/ SSH configuration files SSH for Remote [Link]

Access (see page /SSH
101)

Best practice is to place these changes in /etc  

/sudoers.d/ instead of /etc/sudoers itself, as
changes in the former directory are not lost on

[Link] 53
Cumulus Linux 3.5 User Guide

File Explanation Cumulus Linux Debian

Name Documentation Documentation
and
Location

upgrade. Customers upgrading from a release prior

/etc Using sudo to
to 3.2 (such as 3.1.2) to a 3.2 or later release should
/sudoers Delegate
be aware the sudoers file changed in Cumulus
and /etc Privileges (see
/sudoers. Linux 3.2. page 105)
d

If you are using the root user account, consider including /root/.
If you have custom user accounts, consider including /home/<username>/.

Files That Should Never Be Migrated Between Versions or Boxes

File Name and Explanation

Location

/etc/adjtime System clock adjustment data. NTP manages this automatically. It is incorrect when
the switch hardware is replaced. Do not copy.

/etc/bcm.d/ Per-platform hardware configuration directory, created on first boot. Do not copy.

/etc/mlx/ Per-platform hardware configuration directory, created on first boot. Do not copy.

/etc/[Link] Partition table. It should not be modified manually. Do not copy.

/etc/[Link]. A previous partition table; it should not be modified manually. Do not copy.
old

/etc/cumulus Platform hardware-specific files. Do not copy.

/init

/etc/default Created and managed by ifupdown2. Do not copy.

/clagd

/etc/default Grub init table; it should not be modified manually.

/grub

/etc/default Platform hardware-specific file. Created during first boot. Do not copy.
/hwclock

/etc/init Platform initialization files. Do not copy.

/etc/init.d/ Platform initialization files. Do not copy.

54 23 January 2018
 Cumulus Networks

File Name and Explanation

Location

/etc/fstab Static info on filesystem. Do not copy.

/etc/image- System version data. Do not copy.

release

/etc/os-release System version data. Do not copy.

/etc/lsb-release System version data. Do not copy.

/etc/lvm/archive Filesystem files. Do not copy.

/etc/lvm/backup Filesystem files. Do not copy.

/etc/modules Created during first boot. Do not copy.

/etc/modules- Created during first boot. Do not copy.

load.d/

/etc/sensors.d Platform-specific sensor data. Created during first boot. Do not copy.

/root/.ansible Ansible tmp files. Do not copy.

/home Ansible tmp files. Do not copy.

/cumulus/.
ansible

Upgrading Switches in an MLAG Pair

If you have a pair of Cumulus Linux switches as part of an MLAG (multi-chassis link aggregation) pair (see
page 350), you should only upgrade each switch when it is in the secondary role.

If you are upgrading from Cumulus Linux 2.y.z to Cumulus Linux 3.y.z, during the time when one
switch in the pair is on Cumulus Linux 3.y.z and the other switch in the pair is on Cumulus Linux 2.
y.z, a complete outage occurs on these switches and their associated network segments.

The upgrade path is as follows:

1. Upgrade Cumulus Linux on the switch already in the secondary role. This is the switch with the
higher clagd-priority value.
2. Set the switch in the secondary role into the primary role by setting its clagd-priority to a value
lower than the clagd-priority setting on the switch in the primary role.

[Link] 55
 2.

Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo clagctl priority VALUE

3. Upgrade the switch that just took on the secondary role.

4. Put that switch into the primary role again, if you so choose.

cumulus@switch:~$ sudo clagctl priority VALUE

For more information about setting the priority, see Understanding Switch Roles (see page 359).

Upgrading Cumulus Linux: Choosing between a Binary Install vs. Package

Upgrade
Network admins have two ways to upgrade Cumulus Linux:
Upgrading only the changed packages, using apt-get update and apt-get upgrade. This is
the preferred method.
Performing a binary (full image) install of the new version, using ONIE. This is used when moving
between major versions or if you want to install a clean image.
There are advantages and disadvantages to using these methods, which are outlined below.

Upgrading Using Package Installs (apt-get update && apt-get upgrade)

Pros:
Configuration data stays in place while the packages are upgraded. In the event that the new
version changes a configuration file, and you've also changed the configuration file, a prompt
appears during the upgrade process asking which version you want to use or whether you want to
evaluate the differences.
Third-party apps stay in place.
Cons:
This method works only if you are upgrading to a later minor release (like 3.1.x to 3.2.y), or to a later
maintenance release from an earlier version of that minor release (for example, 2.5.2 to 2.5.5 or
3.0.0 to 3.0.1).
Rollback is quite difficult and tedious.
You can't choose the exact release version that you want to run.
When you upgrade, you upgrade all packages to the latest available version.
The upgrade process takes a while to complete, and various switch functions may be intermittently
available during the upgrade.
Some upgrade operations will terminate SSH sessions on the in-band (front panel) ports, leaving the
user unable to monitor the upgrade process. As a workaround, use the dtach tool.
Just like the binary install method, you may have to reboot after the upgrade, lengthening the
downtime.
After you upgrade, user names and group names created by packages may be different on different
switches, depending the configuration and package installation history.

Network Disruptions When Updating/Upgrading

56 23 January 2018
 Cumulus Networks

The apt-get upgrade and apt-get install commands cause disruptions to network
services:
The apt-get upgrade command may result in services being restarted or stopped as
part of the upgrade process.
The apt-get install command may disrupt core services by changing core service
dependency packages.
In some cases, installing new packages with apt-get install may also upgrade additional
existing packages due to dependencies. To review potential issues before installing, run apt-
get install --dry-run to see the additional packages that will be installed and/or upgraded.

If services are stopped, a reboot may be required before those services can be started again.

To upgrade the switch by updating the packages:

1. Back up the configurations off the switch.

2. Fetch the latest update meta-data from the repository.

cumulus@switch$ sudo -E apt-get update

3. Upgrade all the packages to the latest distribution.

cumulus@switch$ sudo -E apt-get upgrade

4. Reboot the switch if the upgrade messages indicate that a system restart is required.

cumulus@switch$ sudo -E apt-get upgrade

... upgrade messages here ...

*** System restart required ***

cumulus@switch$ sudo reboot

5. Verify correct operation with the old configurations on new version.

After you successfully upgrade Cumulus Linux, you may notice some some results that you may
or may not have expected:
apt-get upgrade always updates the operating system to the most current version, so
if you are currently running Cumulus Linux 3.0.1 and run apt-get upgrade on that
switch, the packages get upgraded to the latest versions contained in the latest 3.y.z
release.

[Link] 57
Cumulus Linux 3.5 User Guide

When you run cat /etc/image-release, the output still shows the version of
Cumulus Linux from the last binary install. So if you installed Cumulus Linux 3.1.0 as a full
image install and then upgraded to 3.2.0 using apt-get upgrade, the output from /etc
/image-release still shows: IMAGE_RELEASE=3.0.0. To see the current version of all
the Cumulus Linux packages running on the switch, use dpkg --list or dpkg -l.

Package Upgrade Notes

If you are using some forms of network virtualization (see page 398), including VMware NSX-V (see
page 555) or Midokura MidoNet (see page 537), you may have updated the /usr/share
/openvswitch/scripts/ovs-ctl-vtep file. This file is not marked as a configuration file, so if
the file contents change in a newer version of Cumulus Linux, they will overwrite any changes you
made to the file. Cumulus Networks recommends you back up this file before upgrading.

Upgrading via Binary Install (ONIE)

Pros:
You choose the exact version that you want to upgrade to.
This is the only method for upgrading to a new major (X.0) version. For example, when you are
upgrading from 2.5.5 to 3.0.
Cons:
Configuration data must be moved to the new OS via ZTP while the OS is first booted, or soon
afterwards via out-of-band management.
Moving the configuration file can go wrong in various ways:
Identifying all the locations of config data is not always an easy task. See section above on
Understanding the Locations of Configuration Data (see page 51).
Config file changes in the new version may cause merge conflicts that go undetected.
If config files aren't restored correctly, the user may be unable to attach to the switch from in-band
management. Hence, out-of-band connectivity (eth0 or console) is recommended.
The installer takes a while to complete.
Third-party apps must be reinstalled and reconfigured afterwards.
To upgrade the switch by running a binary install:

1. Back up the configurations off the switch.

2. Install the binary image, following the instructions at Installing a New Cumulus Linux Image (see page
34).
3. Restore the configuration files to the new version — ideally via automation.
4. Verify correct operation with the old configurations on the new version.
5. Reinstall third party apps and associated configurations.

Rolling Back a Cumulus Linux Installation

Rolling back to an earlier release after upgrading the packages on the switch follows the same procedure as
described for the Linux host OS rollback above. There are three main strategies, and all require detailed
planning and execution:
Back out individual packages: If the problematic package is identified, the network admin can
58 23 January 2018
 Cumulus Networks

Back out individual packages: If the problematic package is identified, the network admin can
downgrade the affected package directly. In rare cases the configuration files may have to be
restored from backup, or edited to back out any changes that were automatically made by the
upgrade package.
Flatten and rebuild: If the OS becomes unusable, you can use orchestration tools to reinstall the
previous OS release from scratch and then automatically rebuild the configuration.
Backup and restore: Another common strategy is to restore to a previous state via a backup
captured before the upgrade.
Which method you employ is specific to your deployment strategy, so providing detailed steps for each
scenario is outside the scope of this document.

Third Party Package Considerations

Note that if you install any third party apps on a Cumulus Linux switch, any configuration data will likely be
installed into the /etc directory, but it is not guaranteed. It is the responsibility of the network admin to
understand the behavior and config file information of any third party packages installed on a Cumulus
Linux switch.
After you upgrade the OS using a full binary install, you will need to reinstall any third party packages or any
Cumulus Linux add-on packages, such as vxsnd or vxrd.

Installation and Upgrade Workflow in Cumulus Linux 3.0 and Later

Beginning with version 3.0, Cumulus Linux completely embraces the Linux and Debian upgrade workflow.
In this paradigm, a base image is installed using an installer, then any upgrades within that release train
(major version, like 3.y.z) are done using apt-get update && apt-get upgrade. Any packages that
have been changed since the base install get upgraded in place from the repository.
The huge advantage of this approach is that all switch configuration files remain untouched, or in rare
cases merged (using the Debian merge function) during the package upgrade.
However, when upgrading a switch from a previous release train — for example, Cumulus Linux 2.5 — a
mechanism is required to migrate the configuration files over to the new installation. This is the perfect
opportunity to use automation and orchestration tools to backup the configuration files, examine them to
verify correctness with the new version, and then to redeploy the configuration files on the new installation.

Using Snapshots during Upgrades

Snapshots (see page 61) can aid you when upgrading the switch operating system. Cumulus Linux takes
two snapshots automatically during the upgrade, one right before you run apt-get upgrade, and one
right after. This way, if something goes wrong with the upgrade, or you need to revert to the earlier version,
you can roll back to the snapshot.

Caveats When Migrating Configuration Files Between Cumulus Linux 2.5.z

and 3.0 and Later
Generally, the configuration files in Cumulus Linux 2.5.z should be able to migrate to version 3.0 or later
without any problems, but there are some known issues listed below and there may be additional issues
with a customer's particular setup.
Known caveats when migrating files from version 2.x to 3.0 or later:
Some configuration files should never be migrated between versions or while replacing hardware.
The Files that Should Never be Migrated (see page 54) table above contains a list of files that
should never be migrated.

[Link] 59
Cumulus Linux 3.5 User Guide

/etc/passwd and /etc/shadow should not be migrated to the new version directly. The example
below and the ansible script included with Config File Migration Script explicitly excludes these two
files from the backup archive. The default password for the cumulus user must be changed, and any
locally created users should be added to the new installation after the upgrade completes.
/etc/apt/[Link] must be completely updated with a new 3.0 or later repository and
repository structure. Repositories from Cumulus Linux 2.5 must be removed. If there are any
custom repositories on the switch, they need to be migrated into the new [Link] file or
the sources.d/ directory.

Using the Config File Migration Script to Identify and Move Files to
Cumulus Linux 3.0 and Later
You can use the Config File Migration Script with the --backup option to create a backup archive of
configuration files in version 2.5, copy them off the box, then install them on the new version switch. Note
that you need to follow the previous section about caveats when migrating configuration files.

You cannot use the Config File Migration Script to upgrade from Cumulus Linux 3.0.0 to a later
version. Use apt-get instead, as documented in the release notes.

The following example excludes /etc/apt, /etc/passwd and /etc/shadow from the backup archive.

1. Back up the version 2.5.z files.

Optional: Use the Ansible playbook included with the Config File Migration script to automate the
backup of all your Cumulus Linux 2.5 switches. See the section below on Using Automation Tools to
Backup Configurations (see page 61) for more details.

# Make a temp dir

loc=$(mktemp -d)
# Create a backup archive to the temp dir
sudo ./config_file_changes --backup --backup_dir $loc --exclude
/etc/apt,/etc/passwd,/etc/shadow
# Copy the archive and log file to an external server
sudo scp -r $loc/* user@my_external_server:.

2. Install Cumulus Linux 3.0 or later onto the switch using ONIE (see page 34).
3. Reinstall the files from the config file archive to the newly installed switch.

# On the switch, copy the config file archive back from the
server:
scp user@my_external_server:PATH/SWITCHNAME-config-archive-
DATE_TIME.[Link] .
# Untar the archive to the root of the box
sudo tar -C / -xvf SWITCHNAME-config-archive-DATE_TIME.[Link]

60 23 January 2018
 Cumulus Networks

Be aware that version 2.5.z configurations are not guaranteed to work in Cumulus Linux
3.0 or later. You should test the restoration and proper operation of the Cumulus Linux 2.5.
z configuration in Cumulus Linux 3.0 or later on a non-production switch or in a Cumulus
VX image, since every deployment is unique.

Using Automation Tools to Back Up 2.5.z Configurations

Adopting the use of orchestration tools like Ansible, Chef or Puppet for configuration management greatly
increases the speed and accuracy of the next major upgrade; they also enable the quick swap of failed
switch hardware. Included with the Config Migration Script is an Ansible playbook that can be used to
create a backup archive of Cumulus Linux 2.5.z switch configuration files and to retrieve them to a central
server — automating step 1 of the previous section for all deployed Cumulus Linux 2.5.z switches. This is a
quick start on the road to setting up automated configuration and control for your deployment. For more
details on integrating automation into your Cumulus Linux deployment, see the Automation Solutions
section on [Link].

Using Snapshots
Cumulus Linux supports the ability to take snapshots of the complete file system as well as the ability to roll
back to a previous snapshot. Snapshots are performed automatically right before and after you upgrade
Cumulus Linux and right before and after you commit a switch configuration using NCLU (see page 85). In
addition, you can take a snapshot at any time. You can roll back the entire file system to a specific snapshot
or just retrieve specific files.
The primary snapshot components are:
btrfs — an underlying file system in Cumulus Linux, which supports snapshots.
snapper — a userspace utility to create and manage snapshots on demand as well as taking
snapshots automatically before and after running apt-get upgrade|install|remove|dist-
upgrade. You can use snapper to roll back to earlier snapshots, view existing snapshots, or delete
one or more snapshots.
NCLU (see page 85) — takes snapshots automatically before and after committing network
configurations. You can use NCLU to roll back to earlier snapshots, view existing snapshots, or
delete one or more snapshots.

Contents
This chapter covers ...
Installing the Snapshot Package (see page 62)
Taking and Managing Snapshots (see page 62)
Viewing Available Snapshots (see page 62)
Viewing Differences between Snapshots (see page 63)
Deleting Snapshots (see page 64)
Rolling Back to Earlier Snapshots (see page 65)
Configuring Automatic Time-based Snapshots (see page 66)
Caveats and Errata (see page 67)
root Partition Mounted Multiple Times (see page 67)

[Link] 61
Cumulus Linux 3.5 User Guide

Installing the Snapshot Package

If you're upgrading from a version of Cumulus Linux earlier than version 3.2, you need to install the
cumulus-snapshot package before you can use snapshots.

cumulus@switch:~$ sudo -E apt-get update

cumulus@switch:~$ sudo -E apt-get install cumulus-snapshot
cumulus@switch:~$ sudo -E apt-get upgrade

Taking and Managing Snapshots

As described above, snapshots are taken automatically:
Before and after you update your switch configuration by running net commit, via NCLU.
Before and after you update Cumulus Linux by running apt-get
upgrade|install|remove|dist-upgrade, via snapper.
You can also take snapshots as needed using the snapper utility. Run:

cumulus@switch:~$ sudo snapper create -d SNAPSHOT_NAME

For more information about using snapper, run snapper --help or man snapper(8).

Viewing Available Snapshots

You can use both NCLU and snapper to view available snapshots on the switch.

cumulus@switch:~$ net show commit history

# Date Description
--- -------------------------------
--------------------------------------
20 Thu 01 Dec 2016 [Link] AM UTC nclu pre 'net commit' (user
cumulus)
21 Thu 01 Dec 2016 [Link] AM UTC nclu post 'net commit' (user
cumulus)
22 Thu 01 Dec 2016 [Link] AM UTC nclu pre '20 rollback' (user
cumulus)
23 Thu 01 Dec 2016 [Link] AM UTC nclu post '20 rollback' (user
cumulus)
24 Thu 01 Dec 2016 [Link] AM UTC nclu pre '22 rollback' (user
cumulus)
31 Fri 02 Dec 2016 [Link] AM UTC nclu pre 'ACL' (user cumulus)
32 Fri 02 Dec 2016 [Link] AM UTC nclu post 'ACL' (user cumulus)

However, net show commit history only displays snapshots taken when you update your switch
configuration. It does not list any snapshots taken directly with snapper. To see all the snapshots on the
switch, run:

62 23 January 2018
 Cumulus Networks

cumulus@switch:~$ sudo snapper list

Type | # | Pre # | Date | User |
Cleanup | Description | Userdata
-------+----+-------+---------------------------------+------
+---------+----------------------------------------+--------------
single | 0 | | | root
| | current |
single | 1 | | Sat 24 Sep 2016 [Link] AM UTC | root
| | first root filesystem |
pre | 20 | | Thu 01 Dec 2016 [Link] AM UTC | root |
number | nclu pre 'net commit' (user cumulus) |
post | 21 | 20 | Thu 01 Dec 2016 [Link] AM UTC | root |
number | nclu post 'net commit' (user cumulus) |
pre | 22 | | Thu 01 Dec 2016 [Link] AM UTC | root |
number | nclu pre '20 rollback' (user cumulus) |
post | 23 | 22 | Thu 01 Dec 2016 [Link] AM UTC | root |
number | nclu post '20 rollback' (user cumulus) |
single | 26 | | Thu 01 Dec 2016 [Link] PM UTC | root
| | test_snapshot |
pre | 29 | | Thu 01 Dec 2016 [Link] PM UTC | root |
number | pre-apt | important=yes
post | 30 | 29 | Thu 01 Dec 2016 [Link] PM UTC | root |
number | post-apt | important=yes
pre | 31 | | Fri 02 Dec 2016 [Link] AM UTC | root |
number | nclu pre 'ACL' (user cumulus) |
post | 32 | 31 | Fri 02 Dec 2016 [Link] AM UTC | root |
number | nclu post 'ACL' (user cumulus) |

Viewing Differences between Snapshots

To see a line by line comparison of changes between two snapshots, run:

cumulus@switch:~$ sudo snapper diff 20..21

--- /.snapshots/20/snapshot/etc/cumulus/acl/policy.d/50_nclu_acl.
rules 2016-11-30 [Link].675092103 +0000
+++ /.snapshots/21/snapshot/etc/cumulus/acl/policy.d/50_nclu_acl.
rules 2016-12-01 [Link].029171289 +0000
@@ -1,7 +0,0 @@
-[iptables]
-# control-plane: acl ipv4 EXAMPLE1 inbound
--A INPUT --in-interface swp+ -j ACCEPT -p tcp -s [Link]/32 -d
[Link]/32 --dport 110
-
-# swp1: acl ipv4 EXAMPLE1 inbound
--A FORWARD --in-interface swp1 --out-interface swp2 -j ACCEPT -p tcp
-s [Link]/32 -d [Link]/32 --dport 110
-

[Link] 63
Cumulus Linux 3.5 User Guide

--- /.snapshots/20/snapshot/var/lib/cumulus/nclu/nclu_acl.conf
2016-11-30 [Link].030079000 +0000
+++ /.snapshots/21/snapshot/var/lib/cumulus/nclu/nclu_acl.conf
2016-12-01 [Link].096136000 +0000
@@ -1,8 +1,3 @@
-acl ipv4 EXAMPLE1 priority 10 accept tcp [Link]/32 [Link]/32
pop3 outbound-interface swp2

-control-plane
- acl ipv4 EXAMPLE1 inbound

-iface swp1
- acl ipv4 EXAMPLE1 inbound

You can view the diff for a single file by specifying the name in the command:

cumulus@switch:~$ sudo snapper diff 20..21 /var/lib/cumulus/nclu

/nclu_acl.conf
--- /.snapshots/20/snapshot/var/lib/cumulus/nclu/nclu_acl.conf
2016-11-30 [Link].030079000 +0000
+++ /.snapshots/21/snapshot/var/lib/cumulus/nclu/nclu_acl.conf
2016-12-01 [Link].096136000 +0000
@@ -1,8 +1,3 @@
-acl ipv4 EXAMPLE1 priority 10 accept tcp [Link]/32 [Link]/32
pop3 outbound-interface swp2

-control-plane
- acl ipv4 EXAMPLE1 inbound

-iface swp1
- acl ipv4 EXAMPLE1 inbound

For a higher level view, displaying the names of changed/added/deleted files only, run:

cumulus@switch:~$ sudo snapper status 20..21

c..... /etc/cumulus/acl/policy.d/50_nclu_acl.rules
c..... /var/lib/cumulus/nclu/nclu_acl.conf

Deleting Snapshots
You can remove one or more snapshots using both NCLU and snapper.

Take care when deleting a snapshot, as you cannot restore it once it's been deleted.

To remove a single snapshot or a range of them created with NCLU, run:

64 23 January 2018
 Cumulus Networks

cumulus@switch:~$ net commit delete SNAPSHOT|SNAPSHOT1-SNAPSHOT2

To remove a single snapshot or a range of snapshots using snapper, run:

cumulus@switch:~$ sudo snapper delete SNAPSHOT|SNAPSHOT1-SNAPSHOT2

Snapshot 0 is the running configuration. You can't roll back to it or delete it. However, you can
take a snapshot of it.
Snapshot 1 is the root file system.

The snapper utility preserves a number of snapshots, and automatically deletes older snapshots once the
limit is reached. It does this in two ways.
By default, snapper preserves 10 snapshots that are labeled important. A snapshot is labeled important if
it was created when you run apt-get. To change this number, run:

cumulus@switch:~$ sudo snapper set-config NUMBER_LIMIT_IMPORTANT=<NUM>

You should always make NUMBER_LIMIT_IMPORTANT an even number since two snapshots are
always taken before and after an upgrade. This does not apply to NUMBER_LIMIT, described
next.

snapper also deletes unlabeled snapshots. The default number of snapshots snapper preserves is 5. To
change this number, run:

cumulus@switch:~$ sudo snapper set-config NUMBER_LIMIT=<NUM>

Also, you can prevent snapshots from being taken automatically before and running apt-get
upgrade|install|remove|dist-upgrade. Edit /etc/cumulus/[Link] and set:

APT_SNAPSHOT_ENABLE=no

Rolling Back to Earlier Snapshots

If you need to restore Cumulus Linux to an earlier state, you can roll back to an older snapshot.
For a snapshot created with NCLU, you can revert to a specific snapshot listed in the output from net
show commit history, or you can revert to the previous snapshot by specifying last when you run:

cumulus@switch:~$ net rollback SNAPSHOT_NUMBER|last

[Link] 65
Cumulus Linux 3.5 User Guide

For any snapshot on the switch, you can use snapper to roll back to a specific snapshot. When running
snapper rollback, you must reboot the switch for the rollback to complete:

cumulus@switch:~$ sudo snapper rollback SNAPSHOT_NUMBER

cumulus@switch:~$ sudo reboot

You can also revert to an earlier version of a specific file instead of rolling back the whole file system:

cumulus@switch:~$ sudo snapper undochange 31..32 /etc/cumulus/acl

/policy.d/50_nclu_acl.rules

You can also copy the file directly from the snapshot directory:

cumulus@switch:~$ cp /.snapshots/32/snapshot/etc/cumulus/acl
/policy.d/50_nclu_acl.rules /etc/cumulus/acl/policy.d/

Configuring Automatic Time-based Snapshots

You can configure Cumulus Linux to take hourly snapshots. You need to enable TIMELINE_CREATE in the
snapper configuration:

cumulus@switch:~$ sudo snapper set-config TIMELINE_CREATE=yes

cumulus@switch:~$ sudo snapper get-
config
Key | Value
-----------------------+------
ALLOW_GROUPS |
ALLOW_USERS |
BACKGROUND_COMPARISON | yes
EMPTY_PRE_POST_CLEANUP | yes
EMPTY_PRE_POST_MIN_AGE | 1800
FSTYPE | btrfs
NUMBER_CLEANUP | yes
NUMBER_LIMIT | 5
NUMBER_LIMIT_IMPORTANT | 10
NUMBER_MIN_AGE | 1800
QGROUP |
SPACE_LIMIT | 0.5
SUBVOLUME | /
SYNC_ACL | no
TIMELINE_CLEANUP | yes
TIMELINE_CREATE | yes

66 23 January 2018
 Cumulus Networks

TIMELINE_LIMIT_DAILY | 5
TIMELINE_LIMIT_HOURLY | 5
TIMELINE_LIMIT_MONTHLY | 5
TIMELINE_LIMIT_YEARLY | 5
TIMELINE_MIN_AGE | 1800

Caveats and Errata

root Partition Mounted Multiple Times

You may notice that the root partition gets mounted multiple times. This is due to the way the btrfs file
system handles subvolumes, mounting the root partition once for each subvolume. btrfs keeps one
subvolume for each snapshot taken, which stores the snapshot data. While all snapshots are subvolumes,
not all subvolumes are snapshots.
Cumulus Linux excludes a number of directories when it takes a snapshot of the root file system (and from
any rollbacks):

Directory Reason

/home Excluded to avoid user data loss on rollbacks.

/var/log, /var/support Log file and Cumulus support location. Excluded from snapshots to allow post-
rollback analysis.

/tmp, /var/tmp No need to rollback temporary files.

/opt, /var/opt Third-party software usually are installed in /opt. Exclude /opt to avoid re-
installing these applications after rollbacks.

/srv Contains data for HTTP and FTP servers. Excluded this directory to avoid
server data loss on rollbacks.

/usr/local This directory is used when installing locally built software. Exclude this
directory to avoid re-installing these software after rollbacks.

/var/spool Exclude this directory to avoid loss of mail after a rollback.

/var/lib/libvirt/images This is the default directory for libvirt VM images. Exclude from the snapshot.
Additionally disable Copy-On-Write (COW) for this subvolume as COW and VM
image I/O access patterns do not play nice.

/boot/grub/i386-pc, The GRUB kernel modules must stay in sync with the GRUB kernel installed in
/boot/grub/x86_64-efi, the master boot record or UEFI system partition.
/boot/grub/arm-uboot

[Link] 67
Cumulus Linux 3.5 User Guide

Adding and Updating Packages

You use the Advanced Packaging Tool (apt) to manage additional applications (in the form of packages) and
to install the latest updates.
Before running any apt-get commands or after changing the /etc/apt/[Link] file, you need
to run apt-get update.

Network Disruptions When Updating/Upgrading

The apt-get upgrade and apt-get install cause disruptions to network services:
The apt-get upgrade command may result in services being restarted or stopped as
part of the upgrade process.
The apt-get install command may disrupt core services by changing core service
dependency packages.
In some cases, installing new packages with apt-get install may also upgrades additional
existing packages due to dependencies. To review potential issues before installing, run apt-
get install --dry-run to see the additional packages that will be installed and/or upgraded.

If services are stopped, a reboot may be required before those services can be started again.

Contents
This chapter covers ...
Updating the Package Cache (see page 68)
Listing Available Packages (see page 70)
Adding a Package (see page 71)
Listing Installed Packages (see page 72)
Upgrading to Newer Versions of Installed Packages (see page 72)
Upgrading a Single Package (see page 72)
Upgrading All Packages (see page 72)
Adding Packages from Another Repository (see page 73)
Related Information (see page 74)

Updating the Package Cache

To work properly, APT relies on a local cache of the available packages. You must populate the cache
initially, and then periodically update it with apt-get update:

cumulus@switch:~$ sudo -E apt-get update

68 23 January 2018
 Cumulus Networks

Get:1 [Link] CumulusLinux-3 InRelease

[7,624 B]
Get:2 [Link] CumulusLinux-3-security-
updates InRelease [7,555 B]
Get:3 [Link] CumulusLinux-3-updates
InRelease [7,660 B]
Get:4 [Link] CumulusLinux-3/cumulus Sources
[20 B]
Get:5 [Link] CumulusLinux-3/upstream
Sources [20 B]
Get:6 [Link] CumulusLinux-3/cumulus amd64
Packages [38.4 kB]
Get:7 [Link] CumulusLinux-3/upstream amd64
Packages [445 kB]
Get:8 [Link] CumulusLinux-3-security-updates
/cumulus Sources [20 B]
Get:9 [Link] CumulusLinux-3-security-updates
/upstream Sources [11.8 kB]
Get:10 [Link] CumulusLinux-3-security-
updates/cumulus amd64 Packages [20 B]
Get:11 [Link] CumulusLinux-3-security-
updates/upstream amd64 Packages [8,941 B]
Get:12 [Link] CumulusLinux-3-updates
/cumulus Sources [20 B]
Get:13 [Link] CumulusLinux-3-updates
/upstream Sources [776 B]
Get:14 [Link] CumulusLinux-3-updates
/cumulus amd64 Packages [38.4 kB]
Get:15 [Link] CumulusLinux-3-updates
/upstream amd64 Packages [444 kB]
Ign [Link] CumulusLinux-3/cumulus
Translation-en_US
Ign [Link] CumulusLinux-3/cumulus
Translation-en
Ign [Link] CumulusLinux-3/upstream
Translation-en_US
Ign [Link] CumulusLinux-3/upstream
Translation-en
Ign [Link] CumulusLinux-3-security-updates
/cumulus Translation-en_US
Ign [Link] CumulusLinux-3-security-updates
/cumulus Translation-en
Ign [Link] CumulusLinux-3-security-updates
/upstream Translation-en_US
Ign [Link] CumulusLinux-3-security-updates
/upstream Translation-en
Ign [Link] CumulusLinux-3-updates/cumulus
Translation-en_US
Ign [Link] CumulusLinux-3-updates/cumulus
Translation-en
Ign [Link] CumulusLinux-3-updates/upstream
Translation-en_US

[Link] 69
Cumulus Linux 3.5 User Guide

Ign [Link] CumulusLinux-3-updates/upstream

Translation-en
Fetched 1,011 kB in 1s (797 kB/s)
Reading package lists... Done

Cumulus Networks recommends you use the -E option with sudo whenever you run any apt-
get command. This option preserves your environment variables — such as HTTP proxies —
before you install new packages or upgrade your distribution.

Listing Available Packages

Once the cache is populated, use apt-cache to search the cache to find the packages you are interested
in or to get information about an available package. Here are examples of the search and show sub-
commands:

cumulus@switch:~$ apt-cache search tcp

socat - multipurpose relay for bidirectional data transfer
fakeroot - tool for simulating superuser privileges
tcpdump - command-line network traffic analyzer
openssh-server - secure shell (SSH) server, for secure access from
remote machines
openssh-sftp-server - secure shell (SSH) sftp server module, for SFTP
access from remote machines
python-dpkt - Python packet creation / parsing module
libfakeroot - tool for simulating superuser privileges - shared
libraries
openssh-client - secure shell (SSH) client, for secure access to
remote machines
rsyslog - reliable system and kernel logging daemon
libwrap0 - Wietse Venema's TCP wrappers library
netbase - Basic TCP/IP networking system

cumulus@switch:~$ apt-cache show tcpdump

Package: tcpdump
Status: install ok installed
Priority: optional
Section: net
Installed-Size: 1092
Maintainer: Romain Francoise <rfrancoise@[Link]>
Architecture: amd64
Multi-Arch: foreign
Version: 4.6.2-5+deb8u1
Depends: libc6 (>= 2.14), libpcap0.8 (>= 1.5.1), libssl1.0.0 (>=
1.0.0)
Description: command-line network traffic analyzer

70 23 January 2018
 Cumulus Networks

This program allows you to dump the traffic on a network. tcpdump

is able to examine IPv4, ICMPv4, IPv6, ICMPv6, UDP, TCP, SNMP, AFS
BGP, RIP, PIM, DVMRP, IGMP, SMB, OSPF, NFS and many other packet
types.
.
It can be used to print out the headers of packets on a network
interface, filter packets that match a certain expression. You can
use this tool to track down network problems, to detect attacks
or to monitor network activities.
Description-md5: f01841bfda357d116d7ff7b7a47e8782
Homepage: [Link]
cumulus@switch:~$

The search commands look for the search terms not only in the package name but in other parts
of the package information. Consequently, it matches on more packages than you would expect.

Adding a Package
In order to add a new package, first ensure the package is not already installed in the system:

cumulus@switch:~$ dpkg -l | grep {name of package}

If the package is installed already, ensure it’s the version you need. If it’s an older version, then update the
package from the Cumulus Linux repository:

cumulus@switch:~$ sudo -E apt-get upgrade

If the package is not already on the system, add it by running apt-get install. This retrieves the
package from the Cumulus Linux repository and installs it on your system together with any other packages
that this package might depend on.
For example, the following adds the package tcpreplay to the system:

cumulus@switch:~$ sudo -E apt-get install tcpreplay

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
tcpreplay
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 436 kB of archives.
After this operation, 1008 kB of additional disk space will be used.
Get:1 [Link] CumulusLinux-1.5/main
tcpreplay amd64 4.6.2-5+deb8u1 [436 kB]
Fetched 436 kB in 0s (1501 kB/s)
Selecting previously unselected package tcpreplay.

[Link] 71
Cumulus Linux 3.5 User Guide

(Reading database ... 15930 files and directories currently

installed.)
Unpacking tcpreplay (from .../tcpreplay_4.6.2-5+deb8u1_amd64.deb) ...
Processing triggers for man-db ...
Setting up tcpreplay (4.6.2-5+deb8u1) ...
cumulus@switch:~$

Listing Installed Packages

The APT cache contains information about all the packages available on the repository. To see which
packages are actually installed on your system, use dpkg. The following example lists all the packages on
the system that have "tcp" in their package names:

cumulus@switch:~$ dpkg -l \*tcp\*

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait
/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version
Architecture Description
+++-=============================-===================-
===================-
===============================================================
un tcpd <none>
<none> (no description available)
ii tcpdump 4.6.2-5+deb8u1
amd64 command-line network traffic analyzer
cumulus@switch:~$

Upgrading to Newer Versions of Installed Packages

Upgrading a Single Package

A single package can be upgraded by simply installing that package again with apt-get install. You
should perform an update first so that the APT cache is populated with the latest information about the
packages.
To see if a package needs to be upgraded, use apt-cache show <pkgname> to show the latest version
number of the package. Use dpkg -l <pkgname> to show the version number of the installed package.

Upgrading All Packages

You can update all packages on the system by running apt-get update, then apt-get upgrade. This
upgrades all installed versions with their latest versions but will not install any new packages.

72 23 January 2018
 Cumulus Networks

Adding Packages from Another Repository

As shipped, Cumulus Linux searches the Cumulus Linux repository for available packages. You can add
additional repositories to search by adding them to the list of sources that apt-get consults. See man
[Link] for more information.

For several packages, Cumulus Networks has added features or made bug fixes and these
packages must not be replaced with versions from other repositories. Cumulus Linux has been
configured to ensure that the packages from the Cumulus Linux repository are always preferred
over packages from other repositories.

If you want to install packages that are not in the Cumulus Linux repository, the procedure is the same as
above with one additional step.

Packages not part of the Cumulus Linux Repository have generally not been tested, and may not
be supported by Cumulus Linux support.

Installing packages outside of the Cumulus Linux repository requires the use of apt-get, but, depending
on the package, easy-install and other commands can also be used.
To install a new package, please complete the following steps:

1. First, ensure package is not already installed in the system. Use the dpkg command:

cumulus@switch:~$ dpkg -l | grep {name of package}

2. If the package is installed already, ensure it's the version you need. If it's an older version, then
update the package from the Cumulus Linux repository:

cumulus@switch:~$ sudo -E apt-get update

cumulus@switch:~$ sudo -E apt-get install {name of package}
cumulus@switch:~$ sudo -E apt-get upgrade

3. If the package is not on the system, then most likely the package source location is also not in the
/etc/apt/[Link] file. If the source for the new package is not in [Link], please
edit and add the appropriate source to the file. For example, add the following if you wanted a
package from the Debian repository that is not in the Cumulus Linux repository:

deb [Link] jessie main

deb [Link] jessie/updates main

Otherwise, the repository may be listed in /etc/apt/[Link] but is commented out, as can
be the case with the early-access repository:

[Link] 73
Cumulus Linux 3.5 User Guide

#deb [Link] CumulusLinux-3-early-

access cumulus

To uncomment the repository, remove the # at the start of the line, then save the file:

deb [Link] CumulusLinux-3-early-

access cumulus

4. Run apt-get update then install the package and upgrade:

Network Disruptions When Updating/Upgrading

The apt-get upgrade and apt-get install will cause disruptions to network
services:
The apt-get upgrade command may result in services being restarted or
stopped as part of the upgrade process.
The apt-get install command may disrupt core services by changing core
service dependency packages.
In some cases, installing new packages with apt-get install may also upgrades
additional existing packages due to dependencies. To review potential issues before
installing, run apt-get install --dry-run to see the additional packages that will be
installed and/or upgraded.

cumulus@switch:~$ sudo -E apt-get update

cumulus@switch:~$ sudo -E apt-get install {name of package}
cumulus@switch:~$ sudo -E apt-get upgrade

Related Information
Debian GNU/Linux FAQ, Ch 8 Package management tools
man pages for apt-get, dpkg, [Link], apt_preferences

Zero Touch Provisioning - ZTP

Zero touch provisioning (ZTP) enables network devices to be quickly deployed in large-scale environments.
On first boot, Cumulus Linux will invoke ZTP, which executes the provisioning automation used to deploy
the device for its intended role in the network.
The provisioning framework allows for a one-time, user-provided script to be executed. You can develop
this script using a variety of automation tools and scripting languages, providing ample flexibility for you to
design the provisioning scheme to meet your needs. You can also use it to add the switch to a
configuration management (CM) platform such as Puppet, Chef, CFEngine or possibly a custom, proprietary
tool.
While developing and testing the provisioning logic, you can use the ztp command in Cumulus Linux to
manually invoke your provisioning script on a device.

74 23 January 2018
 Cumulus Networks

ZTP in Cumulus Linux can occur automatically in one of the following ways, in this order:
Via a local file
Using a USB drive inserted into the switch (ZTP-USB)
Via DHCP
Each method is discussed in greater detail below.

Contents
Click to expand...
Zero Touch Provisioning Using a Local File (see page 75)
Zero Touch Provisioning Using USB (ZTP-USB) (see page 76)
Zero Touch Provisioning over DHCP (see page 76)
Triggering ZTP over DHCP (see page 77)
Configuring The DCHP Server (see page 77)
Detailed Look at HTTP Headers (see page 77)
Writing ZTP Scripts (see page 78)
Example ZTP Script (see page 78)
Testing and Debugging ZTP Scripts (see page 79)
Manually Using the ztp Command (see page 83)
Notes (see page 84)

Zero Touch Provisioning Using a Local File

ZTP only looks once for a ZTP script on the local file system when the switch boots. ZTP searches for an
install script that matches an ONIE-style waterfall in /var/lib/cumulus/ztp, looking for the most
specific name first, and ending at the most generic:
'cumulus-ztp-' + architecture + '-' + vendor + '_' + model + '-r' +
revision
'cumulus-ztp-' + architecture + '-' + vendor + '_' + model
'cumulus-ztp-' + vendor + '_' + model
'cumulus-ztp-' + architecture
'cumulus-ztp'
For example:

cumulus-ztp-amd64-cel_pebble-rUNKNOWN
cumulus-ztp-amd64-cel_pebble
cumulus-ztp-cel_pebble
cumulus-ztp-amd64
cumulus-ztp

You can also trigger the ZTP process manually by running the ztp --run <URL> command, where the
URL is the path to the ZTP script.

[Link] 75
Cumulus Linux 3.5 User Guide

Zero Touch Provisioning Using USB (ZTP-USB)

This feature has been tested only with "thumb" drives, not an actual external large USB hard
drive.

If the ztp process did not discover a local script, it tries once to locate an inserted but unmounted USB
drive. If it discovers one, it begins the ZTP process.
Cumulus Linux supports the use of a FAT32, FAT16, or VFAT-formatted USB drive as an installation source
for ZTP scripts. You must plug in the USB stick before you power up the switch.
At minimum, the script should:
Install the Cumulus Linux operating system and license.
Copy over a basic configuration to the switch.
Restart the switch or the relevant serves to get switchd up and running with that configuration.
Follow these steps to perform zero touch provisioning using USB:

1. Copy the Cumulus Linux license and installation image to the USB stick.
2. The ztp process searches the root filesystem of the newly mounted device for filenames matching
an ONIE-style waterfall (see the patterns and examples above), looking for the most specific name
first, and ending at the most generic.
3. The script's contents are parsed to ensure it contains the CUMULUS-AUTOPROVISIONING flag (see
example scripts (see page 78)).

The USB device is mounted to a temporary directory under /tmp (for example, /tmp
/tmpigGgjf/). In order to reference files on the USB, use the environment variable
ZTP_USB_MOUNTPOINT to refer to the USB root partition.

Zero Touch Provisioning over DHCP

If the ztp process did not discover a local/ONIE script or applicable USB drive, it checks DHCP every 10
seconds for up to 5 minutes for the presence of a ZTP URL specified in /var/run/[Link]. The URL
can be any of HTTP, HTTPS, FTP or TFTP.
For ZTP using DHCP, provisioning initially takes place over the management network and is initiated via a
DHCP hook. A DHCP option is used to specify a configuration script. This script is then requested from the
Web server and executed locally on the switch.
The zero touch provisioning process over DHCP follows these steps:

1. The first time you boot Cumulus Linux, eth0 is configured for DHCP and makes a DHCP request.
2. The DHCP server offers a lease to the switch.
3. If option 239 is present in the response, the zero touch provisioning process itself will start.
4. The zero touch provisioning process requests the contents of the script from the URL, sending
additional HTTP headers (see page 77) containing details about the switch.

5.
76 23 January 2018
 Cumulus Networks

5. The script's contents are parsed to ensure it contains the CUMULUS-AUTOPROVISIONING flag (see
example scripts (see page 78)).

6. If provisioning is necessary, then the script executes locally on the switch with root privileges.
7. The return code of the script gets examined. If it is 0, then the provisioning state is marked as
complete in the autoprovisioning configuration file.

Triggering ZTP over DHCP

If provisioning has not already occurred, it is possible to trigger the zero touch provisioning process over
DHCP when eth0 is set to use DHCP and one of the following events occur:
Booting the switch
Plugging a cable into or unplugging it from the eth0 port
Disconnecting then reconnecting the switch's power cord
You can also run the ztp --run <URL> command, where the URL is the path to the ZTP script.

Configuring The DCHP Server

During the DHCP process over eth0, Cumulus Linux will request DHCP option 239. This option is used to
specify the custom provisioning script.
For example, the /etc/dhcp/[Link] file for an ISC DHCP server would look like:

option cumulus-provision-url code 239 = text;

subnet [Link] netmask [Link] {

range [Link] [Link];
option cumulus-provision-url "[Link]
}

Additionally, the hostname of the switch can be specified via the host-name option:

subnet [Link] netmask [Link] {

range [Link] [Link];
option cumulus-provision-url "[Link]
host dc1-tor-sw1 { hardware ethernet [Link]; fixed-
address [Link]; option host-name "dc1-tor-sw1"; }
}

Detailed Look at HTTP Headers

The following HTTP headers are sent in the request to the webserver to retrieve the provisioning script:

Header Value Example

------ ----- -------

[Link] 77
Cumulus Linux 3.5 User Guide

User-Agent CumulusLinux-
AutoProvision/0.4
CUMULUS-ARCH CPU architecture x86_64
CUMULUS-BUILD 3.0.0-5c6829a-
201309251712-final
CUMULUS-LICENSE-INSTALLED Either 0 or 1 1
CUMULUS-MANUFACTURER odm
CUMULUS-PRODUCTNAME switch_model
CUMULUS-SERIAL XYZ123004
CUMULUS-VERSION 3.0.0
CUMULUS-PROV-COUNT 0
CUMULUS-PROV-MAX 32

Writing ZTP Scripts

Remember to include the following line in any of the supported scripts which are expected to be
run via the autoprovisioning framework.

# CUMULUS-AUTOPROVISIONING

This line is required somewhere in the script file in order for execution to occur.

The script must contain the CUMULUS-AUTOPROVISIONING flag. This can be in a comment or remark and
does not needed to be echoed or written to stdout.
The script can be written in any language currently supported by Cumulus Linux, such as:
Perl
Python
Ruby
Shell
The script must return an exit code of 0 upon success, as this triggers the autoprovisioning process to be
marked as complete in the autoprovisioning configuration file.

Example ZTP Script

The following script install Cumulus Linux and its license from USB and applies a configuration:

#!/bin/bash
function error() {
echo -e "\e[0;33mERROR: The Zero Touch Provisioning script failed
while running the command $BASH_COMMAND at line $BASH_LINENO.\e[0m"
>&2
exit 1
}

78 23 January 2018
 Cumulus Networks

# Log all output from this script

exec >> /var/log/autoprovision 2>&1
date "+%FT%T ztp starting script $0"

trap error ERR

#Add Debian Repositories

echo "deb [Link] jessie main" >> /etc/apt
/[Link]
echo "deb [Link] jessie/updates main" >> /etc/apt
/[Link]

#Update Package Cache

apt-get update -y

#Install netshow diagnostics commands

apt-get install -y netshow htop nmap

#Load interface config from usb

cp ${ZTP_USB_MOUNTPOINT}/interfaces /etc/network/interfaces

#Load port config from usb

# (if breakout cables are used for certain interfaces)
cp ${ZTP_USB_MOUNTPOINT}/[Link] /etc/cumulus/[Link]

#Install a License from usb and restart switchd

/usr/cumulus/bin/cl-license -i ${ZTP_USB_MOUNTPOINT}/[Link] &&
systemctl restart [Link]

#Reload interfaces to apply loaded config

ifreload -a

#Output state of interfaces

netshow interface

# CUMULUS-AUTOPROVISIONING
exit 0

Several ZTP example scripts are available in the Cumulus GitHub repository.

Testing and Debugging ZTP Scripts

There are a few commands you can use to test and debug your ZTP scripts.
You can use verbose mode to debug your script and see where your script failed. Include the -v option
when you run ztp:

cumulus@switch:~$ sudo ztp -v -r [Link]

Attempting to provision via ZTP Manual from [Link]

[Link] 79
Cumulus Linux 3.5 User Guide

Broadcast message from root@dell-s6000-01 (ttyS0) (Tue May 10 [Link]

17 2016):

ZTP: Attempting to provision via ZTP Manual from [Link]

/[Link]
ZTP Manual: URL response code 200
ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
ZTP Manual: Executing [Link]
error: ZTP Manual: Payload returned code 1
error: Script returned failure

You can also run ztp -s to get more information about the current state of ZTP.

cumulus@switch:~$ ztp -s
ZTP INFO:

State enabled
Version 1.0
Result Script Failure
Date Tue May 10 [Link] 2016 UTC
Method ZTP DHCP
URL [Link]

If ZTP ran when the switch booted and not manually, you can run the systemctl -l status ztp.
service then journalctl -l -u [Link] to see if any failures occur:

cumulus@switch:~$ sudo systemctl -l status [Link]

[Link] - Cumulus Linux ZTP
Loaded: loaded (/lib/systemd/system/[Link]; enabled)
Active: failed (Result: exit-code) since Wed 2016-05-11 [Link]
UTC; 1min 47s ago
Docs: man:ztp(8)
Process: 400 ExecStart=/usr/sbin/ztp -b (code=exited, status=1
/FAILURE)
Main PID: 400 (code=exited, status=1/FAILURE)

May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP USB: Device not found
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP: Looking
for ZTP Script provided by DHCP
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: Attempting to
provision via ZTP DHCP from [Link]
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP: URL
response code 200
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP: Found
Marker CUMULUS-AUTOPROVISIONING
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP:
Executing [Link]
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP: Payload
returned code 1

80 23 January 2018
 Cumulus Networks

May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: Script returned

failure
May 11 [Link] dell-s6000-01 systemd[1]: [Link]: main process
exited, code=exited, status=1/FAILURE
May 11 [Link] dell-s6000-01 systemd[1]: Unit [Link] entered
failed state.
cumulus@switch:~$
cumulus@switch:~$ sudo journalctl -l -u [Link] --no-pager
-- Logs begin at Wed 2016-05-11 [Link] UTC, end at Wed 2016-05-11
[Link] UTC. --
May 11 [Link] cumulus ztp[400]: ztp [400]: /var/lib/cumulus/ztp:
Sate Directory does not exist. Creating it...
May 11 [Link] cumulus ztp[400]: ztp [400]: /var/run/[Link]: Lock
File does not exist. Creating it...
May 11 [Link] cumulus ztp[400]: ztp [400]: /var/lib/cumulus/ztp
/ztp_state.log: State File does not exist. Creating it...
May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP LOCAL: Looking for
ZTP local Script
May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall
search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6000_s1220-
rUNKNOWN
May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall
search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6000_s1220
May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall
search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell
May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall
search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64
May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall
search for /var/lib/cumulus/ztp/cumulus-ztp
May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP USB: Looking for
unmounted USB devices
May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP USB: Parsing
partitions
May 11 [Link] cumulus ztp[400]: ztp [400]: ZTP USB: Device not found
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP: Looking
for ZTP Script provided by DHCP
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: Attempting to
provision via ZTP DHCP from [Link]
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP: URL
response code 200
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP: Found
Marker CUMULUS-AUTOPROVISIONING
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP:
Executing [Link]
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: ZTP DHCP: Payload
returned code 1
May 11 [Link] dell-s6000-01 ztp[400]: ztp [400]: Script returned
failure
May 11 [Link] dell-s6000-01 systemd[1]: [Link]: main process
exited, code=exited, status=1/FAILURE
May 11 [Link] dell-s6000-01 systemd[1]: Unit [Link] entered
failed state.

[Link] 81
Cumulus Linux 3.5 User Guide

Instead of running journalctl, you can see the log history by running:

cumulus@switch:~$ cat /var/log/syslog | grep ztp

2016-05-11T[Link].132583+00:00 cumulus ztp [400]: /var/lib/cumulus
/ztp: State Directory does not exist. Creating it...
2016-05-11T[Link].134081+00:00 cumulus ztp [400]: /var/run/ztp.
lock: Lock File does not exist. Creating it...
2016-05-11T[Link].135360+00:00 cumulus ztp [400]: /var/lib/cumulus
/ztp/ztp_state.log: State File does not exist. Creating it...
2016-05-11T[Link].185598+00:00 cumulus ztp [400]: ZTP LOCAL:
Looking for ZTP local Script
2016-05-11T[Link].485084+00:00 cumulus ztp [400]: ZTP LOCAL:
Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-
dell_s6000_s1220-rUNKNOWN
2016-05-11T[Link].486394+00:00 cumulus ztp [400]: ZTP LOCAL:
Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-
dell_s6000_s1220
2016-05-11T[Link].488385+00:00 cumulus ztp [400]: ZTP LOCAL:
Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell
2016-05-11T[Link].489665+00:00 cumulus ztp [400]: ZTP LOCAL:
Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64
2016-05-11T[Link].490854+00:00 cumulus ztp [400]: ZTP LOCAL:
Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp
2016-05-11T[Link].492296+00:00 cumulus ztp [400]: ZTP USB: Looking
for unmounted USB devices
2016-05-11T[Link].493525+00:00 cumulus ztp [400]: ZTP USB: Parsing
partitions
2016-05-11T[Link].636422+00:00 cumulus ztp [400]: ZTP USB: Device
not found
2016-05-11T[Link].372857+00:00 cumulus ztp [1805]: Found ZTP DHCP
Request
2016-05-11T[Link].696562+00:00 cumulus ztp [400]: ZTP DHCP: Looking
for ZTP Script provided by DHCP
2016-05-11T[Link].698598+00:00 cumulus ztp [400]: Attempting to
provision via ZTP DHCP from [Link]
2016-05-11T[Link].816275+00:00 cumulus ztp [400]: ZTP DHCP: URL
response code 200
2016-05-11T[Link].817446+00:00 cumulus ztp [400]: ZTP DHCP: Found
Marker CUMULUS-AUTOPROVISIONING
2016-05-11T[Link].818402+00:00 cumulus ztp [400]: ZTP DHCP:
Executing [Link]
2016-05-11T[Link].834240+00:00 cumulus ztp [400]: ZTP DHCP: Payload
returned code 1
2016-05-11T[Link].835488+00:00 cumulus ztp [400]: Script returned
failure
2016-05-11T[Link].876334+00:00 cumulus systemd[1]: [Link]:
main process exited, code=exited, status=1/FAILURE
2016-05-11T[Link].879410+00:00 cumulus systemd[1]: Unit [Link]
entered failed state.

82 23 January 2018
 Cumulus Networks

If you see that the issue is a script failure, you can modify the script and then run ztp manually using ztp -
v -r <URL/path to that script>, as above.

cumulus@switch:~$ sudo ztp -v -r [Link]

Attempting to provision via ZTP Manual from [Link]

Broadcast message from root@dell-s6000-01 (ttyS0) (Tue May 10 [Link]

17 2016):

ZTP: Attempting to provision via ZTP Manual from [Link]

/[Link]
ZTP Manual: URL response code 200
ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
ZTP Manual: Executing [Link]
error: ZTP Manual: Payload returned code 1
error: Script returned failure
cumulus@switch:~$ sudo ztp -s
State enabled
Version 1.0
Result Script Failure
Date Tue May 10 [Link] 2016 UTC
Method ZTP Manual
URL [Link]

Manually Using the ztp Command

To enable zero touch provisioning, use the -e option:

cumulus@switch:~$ sudo ztp -e

Enabling ztp means that ztp will try to occur the next time the switch boots. However, if ZTP
already occurred on a previous boot up or if a manual configuration has been found, ZTP will just
exit without trying to look for any script.
ZTP checks for these manual configurations during bootup:
Password changes
Users and groups changes
Packages changes
Interfaces changes
The presence of an installed license
When the switch is booted for the very first time, ZTP records the state of some important files
that are most likely going to be modified after that the switch is configured. If ZTP is still enabled
after a reboot, ZTP will compare the recorded state to the current state of these files. If they do
not match, ZTP considers that the switch has already been provisioned and exits. These files are
only erased after a reset.

[Link] 83
Cumulus Linux 3.5 User Guide

To reset ztp to its original state, use the -R option. This removes the ztp directory and ztp runs the next
time the switch reboots.

cumulus@switch:~$ sudo ztp -R

To disable zero touch provisioning, use the -d option:

cumulus@switch:~$ sudo ztp -d

To force provisioning to occur and ignore the status listed in the configuration file use the -r option:

cumulus@switch:~$ sudo ztp -r [Link]

To see the current ztp state, use the -s option:

cumulus@switch:~$ sudo ztp -s

ZTP INFO:
State disabled
Version 1.0
Result success
Date Thu May 5 [Link] 2016 UTC
Method Switch manually configured
URL None

Notes
During the development of a provisioning script, the switch may need to be rebooted.
You can use the Cumulus Linux onie-select -i command to cause the switch to reprovision
itself and install a network operating system again using ONIE.

System
84 Configuration 23 January 2018
 Cumulus Networks

System Configuration

Network Command Line Utility - NCLU

The Network Command Line Utility, or NCLU, is a command line interface for Cumulus Networks products
that simplifies the networking configuration process for all users.
NCLU resides in the Linux user space, as seen below. It provides consistent access to networking
commands directly via bash, thereby making configuration and troubleshooting simple and easy — no
need to edit files or enter modes and sub-modes. In addition, NCLU does more than traditional command
line interfaces by:
Embedding help, examples and automatic command checking with suggestions in case you’ve
entered a typo
Running directly from and integrating with bash, while being interoperable with the regular way of
accessing underlying configuration files and automation
Automatically configuring dependent features so you don’t have to

The NCLU wrapper utility is called net. net is capable of configuring L2 and L3 features of the networking
stack, installing ACLs and VXLANs, rolling back and deleting snapshots, as well as providing monitoring and
troubleshooting functionality for these features. /etc/network/interfaces and /etc/frr/[Link]
can both be configured with net, in addition to running show and clear commands related to ifupdown2
and FRRouting.

[Link] 85
Cumulus Linux 3.5 User Guide

Contents
This chapter covers ...
What's New and Different in NCLU in Version 3.5? (see page 86)
Installing NCLU (see page 86)
Getting Started (see page 86)
Tab Completion, Verification and Inline Help (see page 87)
Adding ? (Question Mark) Ability to NCLU (see page 89)
Built-In Examples (see page 90)
Configuring User Accounts (see page 91)
Editing the [Link] File (see page 93)
Restarting the netd Service (see page 93)
Backing up the Configuration to a Single File (see page 93)
Advanced Configuration (see page 94)

What's New and Different in NCLU in Version 3.5?

A number of commands have been added, updated, or removed from NCLU in the new release. Read more
about what's changed in this knowledge base article.

Installing NCLU
If you upgraded Cumulus Linux from a version earlier than 3.2 instead of performing a full binary install, you
need to install the nclu package on your switch:

cumulus@switch:~$ sudo -E apt-get update

cumulus@switch:~$ sudo -E apt-get install nclu
cumulus@switch:~$ sudo -E apt-get upgrade

The nclu package installs a new bash completion script, and displays the following message
when it is manually installed:

Setting up nclu (1.0-cl3u3) ...

To enable the newly installed bash completion for nclu in this
shell, execute...
source /etc/bash_completion

Getting Started
NCLU uses the following workflow for staging and committing changes to Cumulus Linux:

86 23 January 2018
 Cumulus Networks

1. Use the net add and net del commands to stage/remove configuration changes.
2. Use the net pending command to review staged changes.
3. Use net commit and net abort to commit/delete staged changes.

net commit applies the changes to the relevant configuration files, such as /etc/network
/interfaces, then runs necessary follow on commands to enable the configuration, such as
ifreload -a.

Once you have a running configuration, you can review and update it using:
net show: A series of commands for viewing various parts of the network configuration, such as
net show configuration, net show commit history and net show bgp to view the
complete network configuration, a history of commits using NCLU and BGP status, respectively.
net clear: A way to clear net show counters, BGP and OSPF neighbor content, and more.
net rollback: Provides a mechanism to revert back to an earlier configuration.
net commit confirm: Requires the user to press Enter in order to commit changes via NCLU. If
you run net commit confirm but do not press Enter within 10 seconds, the commit is
automatically reverted and nothing changes.
net commit permanent: Retains the snapshot (see page 61) taken when committing the
change. Otherwise, the snapshots created from NCLU commands are cleaned up periodically via a
snapper cron job.
net commit delete: Deletes one or more snapshots created when committing changes with
NCLU.
net del all: Deletes all configurations and stops the IEEE 802.1X service.

This command does not remove management VRF (see page 716) configurations, as
NCLU does not interact with eth0 interfaces and management VRF at all.

Tab Completion, Verification and Inline Help

NCLU provides a number of features to assist users. In addition to tab completion and partial keyword
commands identification, verification checks are set in place to ensure correct syntax is used. The examples
below show the output for incorrect commands:

cumulus@switch:~$ net add bgp router-id [Link]/32

ERROR: Command not found

Did you mean one of the following?

net add bgp router-id <ipv4>

This command is looking for an IP address, not an IP/prefixlen

cumulus@switch:~$ net add bgp router-id [Link]

cumulus@switch:~$ net add int swp10 mtu <TAB>

[Link] 87
Cumulus Linux 3.5 User Guide

<552-9216> :
cumulus@switch:~$ net add int swp10 mtu 9300
ERROR: Command not found

Did you mean one of the following?

net add interface <interface> mtu <552-9216>

NCLU has a comprehensive help system built in to assist usage. In addition to the net man page, you can
use ? and help to display available commands:

cumulus@switch:~$ net help

Usage:
# net <COMMAND> [<ARGS>] [help]
#
# net is a command line utility for networking on Cumulus Linux
switches.
#
# COMMANDS are listed below and have context specific arguments
which can
# be explored by typing "<TAB>" or "help" anytime while using net.
#
# Use 'man net' for a more comprehensive overview.

net abort
net commit [verbose] [confirm] [description <wildcard>]
net commit delete (<number>|<number-range>)
net help [verbose]
net pending
net rollback (<number>|last)
net show commit (history|<number>|<number-range>|last)
net show rollback (<number>|last)
net show configuration
[commands|files|acl|bgp|ospf|ospf6|interface <interface>]

Options:

# Help commands
help : context sensitive information; see section below
example : detailed examples of common workflows

# Configuration commands
add : add/modify configuration
del : remove configuration

# Commit buffer commands

88 23 January 2018
 Cumulus Networks

abort : abandon changes in the commit buffer

commit : apply the commit buffer to the system
pending : show changes staged in the commit buffer
rollback : revert to a previous configuration state

# Status commands
show : show command output
clear : clear counters, BGP neighbors, etc

cumulus@switch:~$ net help bestpath

The following commands contain keyword(s) 'bestpath'

net (add|del) bgp bestpath as-path multipath-relax [as-set|no-as-

set]
net (add|del) bgp bestpath compare-routerid
net (add|del) bgp bestpath med missing-as-worst
net (add|del) bgp vrf <text> bestpath as-path multipath-relax [as-
set|no-as-set]
net (add|del) bgp vrf <text> bestpath compare-routerid
net (add|del) bgp vrf <text> bestpath med missing-as-worst
net add bgp debug bestpath <ip/prefixlen>
net del bgp debug bestpath [<ip/prefixlen>]
net show bgp (<ipv4>|<ipv4/prefixlen>) [bestpath|multipath] [json]
net show bgp (<ipv6>|<ipv6/prefixlen>) [bestpath|multipath] [json]
net show bgp vrf <text> (<ipv4>|<ipv4/prefixlen>)
[bestpath|multipath] [json]

Multiple interfaces can be configured at once:

cumulus@switch:~$ net add int swp7-9,12,15-17,22 mtu 9216

Adding ? (Question Mark) Ability to NCLU

While tab completion is enabled by default, you can also configure NCLU to use the ? (question mark
character) to look at available commands. To enable this feature for the cumulus user, open the following
file:

cumulus@leaf01:~$ sudo nano ~/.inputrc

Uncomment the very last line in the .inputrc file so that the file changes from this:

# Uncomment to use ? as an alternative to

# ?: complete

[Link] 89
Cumulus Linux 3.5 User Guide

to this:

# Uncomment to use ? as an alternative to

?: complete

Save the file and reconnect to the switch. The ? (question mark) ability will work on all subsequent sessions
on the switch.

cumulus@leaf01:~$ net
abort : abandon changes in the commit buffer
add : add/modify configuration
clear : clear counters, BGP neighbors, etc
commit : apply the commit buffer to the system
del : remove configuration
example : detailed examples of common workflows
help : Show this screen and exit
pending : show changes staged in the commit buffer
rollback : revert to a previous configuration state
show : show command output

When the question mark is typed, NCLU will autocomplete and show all available options, but the
question mark won't actually appear on the terminal. This is normal, expected behavior.

Built-In Examples
The NCLU has a number of built in examples to guide users through basic configuration setup:

cumulus@switch:~$ net example

acl : access-list
bgp : Border Gateway Protocol
bond : Bond, port-channel, etc
bridge : A layer2 bridge
clag : Multi-Chassis Link Aggregation
dot1x : Configure, Enable, Delete or Show IEEE 802.1X
EAPOL
link-settings : Physical link parameters
lnv : Lightweight Network Virtualization
management-vrf : Management VRF
mlag : Multi-Chassis Link Aggregation
ospf : Open Shortest Path First (OSPFv2)
vlan-interfaces : IP interfaces for VLANs

cumulus@switch:~$ net example bridge

90 23 January 2018
 Cumulus Networks

Scenario
========
We are configuring switch1 and would like to configure the following
- configure switch1 as an L2 switch for host-11 and host-12
- enable vlans 10-20
- place host-11 in vlan 10
- place host-12 in vlan 20
- create an SVI interface for vlan 10
- create an SVI interface for vlan 20
- assign IP [Link]/24 to the SVI for vlan 10
- assign IP [Link]/24 to the SVI for vlan 20
- configure swp3 as a trunk for vlans 10, 11, 12 and 20
swp3
*switch1 --------- switch2
/\
swp1 / \ swp2
/ \
/ \
host-11 host-12

switch1 net commands

====================
- enable vlans 10-20
switch1# net add vlan 10-20
- place host-11 in vlan 10
- place host-12 in vlan 20
switch1# net add int swp1 bridge access 10
switch1# net add int swp2 bridge access 20
- create an SVI interface for vlan 10
- create an SVI interface for vlan 20
- assign IP [Link]/24 to the SVI for vlan 10
- assign IP [Link]/24 to the SVI for vlan 20
switch1# net add vlan 10 ip address [Link]/24
switch1# net add vlan 20 ip address [Link]/24
- configure swp3 as a trunk for vlans 10, 11, 12 and 20
switch1# net add int swp3 bridge trunk vlans 10-12,20
# Review and commit changes
switch1# net pending
switch1# net commit

Verification
============
switch1# net show interface
switch1# net show bridge macs

Configuring User Accounts

You can configure user accounts in Cumulus Linux with read-only or edit permissions for NCLU:

[Link] 91
Cumulus Linux 3.5 User Guide

You create user accounts with read-only permissions for NCLU by adding them to the netshow
group. A user in the netshow group can run NCLU net show commands, such as net show
interface or net show config, and certain general Linux commands, such as ls, cd or man,
but cannot run net add, net del or net commit commands.

You create user accounts with edit permissions for NCLU by adding them to the netedit group. A
user in the netedit group can run NCLU configuration commands, such net add, net del or
net commit in addition to NCLU net show commands.
The examples below demonstrate how to add a new user account or modify an existing user account called
myuser.
To add a new user account with NCLU show permissions:

cumulus@switch:~$ sudo adduser --ingroup netshow myuser

Adding user `myuser' ...
Adding new user `myuser' (1001) with group `netshow' …

To add NCLU show permissions to a user account that already exists:

cumulus@switch:~$ sudo addgroup myuser netshow

Adding user `myuser' to group `netshow' ...
Adding user myuser to group netshow
Done

To add a new user account with NCLU edit permissions:

cumulus@switch:~$ sudo adduser --ingroup netedit myuser

Adding user `myuser' ...
Adding new user `myuser' (1001) with group `netedit' …

To add NCLU edit permissions to a user account that already exists:

cumulus@switch:~$ sudo addgroup myuser netedit

Adding user `myuser' to group `netedit' ...
Adding user myuser to group netedit
Done

You can use the adduser command for local user accounts only. You can use the addgroup
command for both local and remote user accounts. For a remote user account, you must use the
mapping username, such as tacacs3 or radius_user, not the TACACS (see page 120) or
RADIUS (see page 132) account name.

If the user tries to run commands that are not allowed, the following error displays:

92 23 January 2018
 Cumulus Networks

myuser@switch:~$ net add hostname host01

ERROR: User username does not have permission to make networking
changes.

Editing the [Link] File

Instead of using the NCLU commands described above, you can manually configure users and groups to be
able to run NCLU commands.
Edit the /etc/[Link] file to add users to the users_with_edit and users_with_show lines in the file, then
save the file.
For example, if you want the user netoperator to be able to run both edit and show commands, add the
user to the users_with_edit and groups_with_edit lines in the /etc/[Link] file:

cumulus@switch:~$ sudo nano /etc/[Link]

# Control which users/groups are allowed to run 'add', 'del',

# 'clear', 'net abort', 'net commit' and restart services
# to apply those changes
users_with_edit = root, cumulus, netoperator
groups_with_edit = root, cumulus

# Control which users/groups are allowed to run 'show' commands

users_with_show = root, cumulus, netoperator
groups_with_show = root, cumulus

To configure a new user group to use NCLU, add that group to the groups_with_edit and
groups_with_show lines in the file.

Use caution giving groups edit permissions. For example, you don't want to give edit permissions
to the tacacs group (see page 124).

Restarting the netd Service

Whenever you modify [Link], you must restart the netd service for the changes to take effect:

cumulus@switch:~$ sudo systemctl restart [Link]

Backing up the Configuration to a Single File

You can easily back up your NCLU configuration to a file by outputting the results of net show
configuration commands to a file, then retrieving the contents of the file using the source command.
You can then view the configuration at any time or copy it to other switches and use the source command
to apply that configuration to those switches.
For example, to copy out the configuration of a leaf switch called leaf01, you would run something like the
[Link] 93
Cumulus Linux 3.5 User Guide

For example, to copy out the configuration of a leaf switch called leaf01, you would run something like the
following:

cumulus@leaf01:~$ net show configuration commands >> [Link]

With the commands all stored in a single file, you can now copy this file to another ToR switch in your
network called leaf01 and apply the configuration by running:

cumulus@leaf01:~$ source [Link]

Advanced Configuration
NCLU needs no initial configuration; it's ready to go in Cumulus Linux. However, if you need to modify its
configuration, you must manually update the /etc/[Link] file. This file can be configured to allow
different permission levels for users to edit configurations and run show commands. It also contains a
blacklist that hides less frequently used terms from the tabbed autocomplete.

Configuration Default Setting Description

Variable

show_linux_command False When true,

displays the
Linux
command
running in
the
background.

enable_ifupdown2 True Enables net

wrapping of
ifupdown2
commands.

enable_frr True Enables net

wrapping of
FRRouting
commands.

users_with_edit root, cumulus Sets the

Linux users
with root
edit
privileges.

groups_with_edit root, cumulus

94 23 January 2018
 Cumulus Networks

Configuration Default Setting Description

Variable

Sets the
Linux
groups with
root edit
privileges.

users_with_show root, cumulus Controls

which users
are allowed
to run show
commands.

groups_with_show root, cumulus Controls

which
groups are
allowed to
run show
commands.

ifupdown_blacklist address-purge, bond-ad-actor-sys-prio, bond-ad-actor-system, Hides

bond-mode, bond-num-grat-arp, bond-num-unsol-na, bond-use- corner case
carrier, bond-xmit-hash-policy, bridge-bridgeprio, bridge-fd, command
bridge-hashel, bridge-hashmax, bridge-hello, bridge-maxage, options
bridge-maxwait, bridge-mclmc, bridge-mclmi, bridge-mcmi, from tab
bridge-mcqi, bridge-mcqpi, bridge-mcqri, bridge-mcrouter, complete, to
bridge-mcsqc, bridge-mcsqi, bridge-pathcosts, bridge-port-pvids, simplify and
bridge-port-vids, bridge-portprios, bridge-stp, bridge-waitport, streamline
broadcast, hwaddress, link-type, mstpctl-ageing, mstpctl-fdelay, output.
mstpctl-forcevers, mstpctl-hello, mstpctl-maxage, mstpctl-
maxhops, mstpctl-portp2p, mstpctl-portpathcost, mstpctl-
portrestrrole, mstpctl-portrestrtcn, mstpctl-treeportcost, mstpctl-
treeportprio, mstpctl-txholdcount, netmask, preferred-lifetime,
scope, vxlan-ageing, vxlan-learning, up, down, bridge-ageing,
bridge-gcint, bridge-mcqifaddr, bridge-mcqv4src

Net Tab Complete Output

net provides an environment variable for setting where the net output is directed. To only use
stdout, set the NCLU_TAB_STDOUT environment variable to true. The value is not case sensitive.

Setting Date and Time

Setting the time zone, date and time requires root privileges; use sudo.

Contents
[Link] 95
Cumulus Linux 3.5 User Guide

Contents
This chapter covers ...
Setting the Time Zone (see page 96)
Alternative: Use the Guided Wizard to Find and Apply a Time Zone (see page 96)
Setting the Date and Time (see page 97)
Setting Time Using NTP and NCLU (see page 98)
Specifying the NTP Source Interface (see page 99)
NTP Default Configuration (see page 99)
Related Information (see page 101)

Setting the Time Zone

To see the current time zone, list the contents of /etc/timezone:

cumulus@switch:~$ cat /etc/timezone

US/Eastern

Edit the file to add your desired time zone. A list of valid time zones can be found at the following link.
Use the following command to apply the new time zone immediately.

cumulus@switch:~$ sudo dpkg-reconfigure --frontend noninteractive

tzdata

Alternative: Use the Guided Wizard to Find and Apply a Time Zone
To set the time zone, run dpkg-reconfigure tzdata as root:

cumulus@switch:~$ sudo dpkg-reconfigure tzdata

Then navigate the menus to enable the time zone you want. The following example selects the US/Pacific
time zone:

cumulus@switch:~$ sudo dpkg-reconfigure tzdata

Configuring tzdata
------------------

Please select the geographic area in which you live. Subsequent

configuration
questions will narrow this down by presenting a list of cities,
representing
the time zones in which they are located.

96 23 January 2018
 Cumulus Networks

1. Africa 4. Australia 7. Atlantic 10. Pacific 13. Etc

2. America 5. Arctic 8. Europe 11. SystemV
3. Antarctica 6. Asia 9. Indian 12. US
Geographic area: 12

Please select the city or region corresponding to your time zone.

1. Alaska 4. Central 7. Indiana-Starke 10. Pacific

2. Aleutian 5. Eastern 8. Michigan 11. Pacific-New
3. Arizona 6. Hawaii 9. Mountain 12. Samoa
Time zone: 10

Current default time zone: 'US/Pacific'

Local time is now: Mon Jun 17 [Link] PDT 2013.
Universal Time is now: Mon Jun 17 [Link] UTC 2013.

For more info see the Debian System Administrator’s Manual – Time.

Setting the Date and Time

The switch contains a battery backed hardware clock that maintains the time while the switch is powered
off and in between reboots. When the switch is running, the Cumulus Linux operating system maintains its
own software clock.
During boot up, the time from the hardware clock is copied into the operating system’s software clock. The
software clock is then used for all timekeeping responsibilities. During system shutdown the software clock
is copied back to the battery backed hardware clock.
You can set the date and time on the software clock using the date command. First, determine your
current time zone:

cumulus@switch$ date +%Z

If you need to reconfigure the current time zone, refer to the instructions above.

Then, to set the system clock according to the time zone configured:

cumulus@switch$ sudo date -s "Tue Jan 12 [Link] 2016"

See man date(1) for if you need more information.

You can write the current value of the system (software) clock to the hardware clock using the hwclock
command:

cumulus@switch$ sudo hwclock -w

See man hwclock(8) if you need more information.

[Link] 97
Cumulus Linux 3.5 User Guide

See man hwclock(8) if you need more information.

You can find a good overview of the software and hardware clocks in the Debian System Administrator's
Manual – Time, specifically the section Setting and showing hardware clock.

Setting Time Using NTP and NCLU

The ntpd daemon running on the switch implements the NTP protocol. It synchronizes the system time
with time servers listed in /etc/[Link]. It is started at boot by default. See man ntpd(8) for ntpd
details, and you can check this site for an explanation of the output.
By default, /etc/[Link] contains some default time servers. You can specify the NTP server or servers
you want to use with NCLU (see page 85); include the iburst option to increase the sync speed.

cumulus@switch:~$ net add time ntp server [Link].

org iburst
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands add the NTP server to the list of servers in /etc/[Link]:

# [Link] maps to about 1000 low-stratum NTP servers. Your

server will
# pick a different set every time it starts up. Please consider
joining the
# pool: <[Link]
server [Link] iburst
server [Link] iburst
server [Link] iburst
server [Link] iburst
server [Link] iburst

To set the initial date and time via NTP before starting the ntpd daemon, use ntpd -q. This is same as
ntpdate, which is to be retired and no longer available. See man [Link](5) for details on configuring
ntpd using [Link].

ntpd -q can hang if the time servers are not reachable.

To verify that ntpd is running on the system:

cumulus@switch:~$ ps -ef | grep ntp

ntp 4074 1 0 Jun20 ? [Link] /usr/sbin/ntpd -p /var
/run/[Link] -g -u 101:102

To check the NTP peer status:

98 23 January 2018
 Cumulus Networks

cumulus@switch:~$ net show time ntp servers

remote refid st t when poll reach delay
offset jitter
======================================================================
========
+[Link] [Link] 3 u 140 1024 377 55.659
0.339 1.464
+[Link] [Link] 2 u 259 1024 377 41.587
1.011 1.677
*[Link] [Link] 2 u 210 1024 377 4.008
1.277 1.628
+[Link] [Link] 2 u 743 1024 377 39.319
-0.316 1.384

To remove one or more NTP servers:

cumulus@switch:~$ net del time ntp server [Link].

org iburst
cumulus@switch:~$ net del time ntp server [Link].
org iburst
cumulus@switch:~$ net del time ntp server [Link].
org iburst
cumulus@switch:~$ net del time ntp server [Link].
org iburst
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Specifying the NTP Source Interface

You can change the source interface that NTP uses if you want to use an interface other than eth0, the
default.

cumulus@switch:~$ net add time ntp source swp10

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration snippet in the [Link] file:

...

# Specify interfaces
interface listen swp10

...

NTP Default Configuration

[Link] 99
Cumulus Linux 3.5 User Guide

NTP Default Configuration

The default NTP configuration comprises the following servers, which are listed in the /etc/[Link]
file:
server [Link] iburst
server [Link] iburst
server [Link] iburst
server [Link] iburst
If you need to restore the default NTP configuration, its contents are listed below.
Default [Link] file ...

# /etc/[Link], configuration for ntpd; see [Link](5) for help

driftfile /var/lib/ntp/[Link]
# Enable this if you want statistics to be logged.
#statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
# You do need to talk to an NTP server or two (or three).
#server [Link]
# [Link] maps to about 1000 low-stratum NTP servers. Your
server will
# pick a different set every time it starts up. Please consider
joining the
# pool: <[Link]
server [Link] iburst
server [Link] iburst
server [Link] iburst
server [Link] iburst
# Access control configuration; see /usr/share/doc/ntp-doc/html
/[Link] for
# details. The web page <[Link]
/AccessRestrictions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a
configuration
# that might be intended to block requests from certain clients could
also end
# up blocking replies from your own upstream servers.
# By default, exchange time with everybody, but don't allow
configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
# Local users may interrogate the ntp server more closely.
restrict [Link]
restrict ::1

100 23 January 2018

 Cumulus Networks

# Clients from this (example!) subnet have unlimited access, but only
if
# cryptographically authenticated.
#restrict [Link] mask [Link] notrust
# If you want to provide time to your local subnet, change the next
line.
# (Again, the address is an example only.)
#broadcast [Link]
# If you want to listen to time broadcasts on your local subnet, de-
comment the
# next lines. Please do this only if you trust everybody on the
network!
#disable auth
#broadcastclient
# Specify interfaces, don't listen on switch ports
interface listen eth0

Related Information
Debian System Administrator’s Manual – Time
[Link]
[Link]/wiki/Network_Time_Protocol
[Link]/NTP

Authentication, Authorization and Accounting

SSH for Remote Access

Authentication keys can be generated for securely accessing a Cumulus Linux switch with the ssh-keygen
component of the Secure Shell (SSH) protocol. Cumulus Linux uses the OpenSSH package to provide this
functionality. The section below covers how to generate a SSH key pair.

Contents
This chapter covers ...
Generate an SSH Key Pair (see page 101)
Related Information (see page 103)

Generate an SSH Key Pair

1. Run the ssh-keygen command, and follow the prompts, to generate the key pair:

Configure a Passwordless System

To configure a completely passwordless system, do not enter a passphrase when
prompted in the following step.

[Link] 101
Cumulus Linux 3.5 User Guide

cumulus@leaf01:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/cumulus/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/cumulus/.ssh/id_rsa.
Your public key has been saved in /home/cumulus/.ssh/id_rsa.pub.
The key fingerprint is:
[Link] cumulus@leaf04
The key's randomart image is:
+---[RSA 2048]----+
| +.o o |
| o * o . o |
| o + o O o |
| + . = O |
| . S o . |
| + . |
| . E |
| |
| |
+-----------------+

2. Run the ssh-copy-id command, and follow the prompts, to copy the generated public key to the
desired location:

cumulus@leaf01:~$ ssh-copy-id -i /home/cumulus/.ssh/id_rsa.pub

cumulus@leaf02
The authenticity of host 'leaf02 ([Link])' can't be
established.
ECDSA key fingerprint is [Link]
[Link].
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key
(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed --
if you are prompted now it is to install the new keys
cumulus@leaf01's password:

Number of key(s) added: 1

ssh-copy-id will not work if the username on the remote switch is different to the local
switch. To work around this issue, use the scp command instead:

cumulus@leaf01:~$ scp .ssh/id_rsa.pub cumulus@leaf02:.ssh

/authorized_keys

102 23 January 2018

 Cumulus Networks

Enter passphrase for key '/home/cumulus/.ssh/id_rsa':

id_rsa.pub

3. Connect to the remote switch to confirm the authentication keys are in place:

cumulus@leaf04:~$ ssh cumulus@leaf02

Welcome to Cumulus VX (TM)

Cumulus VX (TM) is a community supported virtual appliance

designed for
experiencing, testing and prototyping Cumulus Networks' latest
technology.
For any questions or technical support, visit our community site
at:
[Link]

The registered trademark Linux (R) is used pursuant to a

sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a
world-wide
basis.
Last login: Thu Sep 29 [Link] 2016

Related Information
Debian Documentation - Password-less logins with OpenSSH
Wikipedia - Secure Shell (SSH)

User Accounts
By default, Cumulus Linux has two user accounts: cumulus and root.
The cumulus account:
Default password is CumulusLinux!
Is a user account in the sudo group with sudo privileges
User can log in to the system via all the usual channels like console and SSH (see page 101)
Along with the cumulus group, has both show and edit rights for NCLU (see page 85)
The root account:
Default password is disabled by default
Has the standard Linux root user access to everything on the switch
Disabled password prohibits login to the switch by SSH, telnet, FTP, and so forth
For best security, you should change the default password (using the passwd command) before you
configure Cumulus Linux on the switch.

You can add more user accounts as needed. Like the cumulus account, these accounts must use sudo to
[Link] 103
Cumulus Linux 3.5 User Guide

You can add more user accounts as needed. Like the cumulus account, these accounts must use sudo to
execute privileged commands (see page 105), so be sure to include them in the sudo group.
To access the switch without any password requires booting into a single shell/user mode (see page 766).
You can add and configure user accounts in Cumulus Linux with read-only or edit permissions for NCLU.
For more information, see Configuring User Accounts (see page 85).

Enabling Remote Access for the root User

As mentioned above, the root user does not have a password set for it, and it cannot log in to a switch via
SSH. This default account behavior is consistent with Debian. In order to connect to a switch using the root
account, you can do one of two things for the account:
Generate an SSH key
Set a password

Generating an SSH Key for the root Account

1. First, in a terminal on your host system (not the switch), check to see if a key already exists:

root@host:~# ls -al ~/.ssh/

The key is named something like id_dsa.pub, id_rsa.pub or id_ecdsa.pub.

2. If a key doesn't exist, generate a new one by first creating the RSA key pair:

root@host:~# ssh-keygen -t rsa

3. A prompt appears, asking you to Enter file in which to save the key (/root/.ssh/id_rsa):. Press Enter to use
the root user's home directory, or else provide a different destination.
4. You are prompted to Enter passphrase (empty for no passphrase):. This is optional but it does provide
an extra layer of security.
5. The public key is now located in /root/.ssh/id_rsa.pub. The private key (identification) is now
located in /root/.ssh/id_rsa.
6. Copy the public key to the switch. SSH to the switch as the cumulus user, then run:

cumulus@switch:~$ sudo mkdir -p /root/.ssh

cumulus@switch:~$ echo <SSH public key string> | sudo tee -a
/root/.ssh/authorized_keys

Setting the root User Password

1. Run:

cumulus@switch:~$ sudo passwd root

104 23 January 2018

 1.

Cumulus Networks

2. Change the /etc/ssh/sshd_config file's PermitRootLogin setting from without-password to yes

.

cumulus@switch:~$ sudo nano /etc/ssh/sshd_config

...

# Authentication:
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes

...

3. Restart the ssh service:

cumulus@switch:~$ sudo systemctl reload [Link]

Using sudo to Delegate Privileges

By default, Cumulus Linux has two user accounts: root and cumulus. The cumulus account is a normal user
and is in the group sudo.
You can add more user accounts as needed. Like the cumulus account, these accounts must use sudo to
execute privileged commands.

Contents
This chapter covers ...
Using sudo (see page 105)
sudoers Examples (see page 106)
Related Information (see page 111)

Using sudo
sudo allows you to execute a command as superuser or another user as specified by the security policy.
See man sudo(8) for details.
The default security policy is sudoers, which is configured using /etc/sudoers. Use /etc/sudoers.d/ to
add to the default sudoers policy. See man sudoers(5) for details.

Use visudo only to edit the sudoers file; do not use another editor like vi or emacs. See man
visudo(8) for details.
Errors in the sudoers file can result in losing the ability to elevate privileges to root. You can fix
this issue only by power cycling the switch and booting into single user mode. Before modifying
sudoers, enable the root user by setting a password for the root user.

[Link] 105
Cumulus Linux 3.5 User Guide

By default, users in the sudo group can use sudo to execute privileged commands. To add users to the
sudo group, use the useradd(8) or usermod(8) command. To see which users belong to the sudo
group, see /etc/group (man group(5)).
Any command can be run as sudo, including su. A password is required.
The example below shows how to use sudo as a non-privileged user cumulus to bring up an interface:

cumulus@switch:~$ ip link show dev swp1

3: swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master br0
state DOWN mode DEFAULT qlen 500
link/ether [Link] brd [Link]

cumulus@switch:~$ ip link set dev swp1 up

RTNETLINK answers: Operation not permitted

cumulus@switch:~$ sudo ip link set dev swp1 up

Password:

cumulus@switch:~$ ip link show dev swp1

3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master br0 state UP mode DEFAULT qlen 500
link/ether [Link] brd [Link]

sudoers Examples
The following examples show how you grant as few privileges as necessary to a user or group of users to
allow them to perform the required task. For each example, the system group noc is used; groups are
prefixed with an %.
When executed by an unprivileged user, the example commands below must be prefixed with sudo.

Category Privilege Example Command sudoers Entry

Monitoring Switch
port info
ethtool -m swp1 %noc ALL=(ALL) NOPASSWD:
/sbin/ethtool

Monitoring System
diagnostics
cl-support %noc ALL=(ALL) NOPASSWD:/usr
/cumulus/bin/cl-support

Monitoring

106 23 January 2018

 Cumulus Networks

Category Privilege Example Command sudoers Entry

cl-resource- %noc ALL=(ALL) NOPASSWD:/usr

Routing
diagnostics query /cumulus/bin/cl-resource-
query

Image Install
management images
onie-select %noc ALL=(ALL) NOPASSWD:/usr
[Link] /cumulus/bin/onie-select
/[Link]

Package Any apt-

management get
command apt-get update %noc ALL=(ALL) NOPASSWD:/usr
or apt-get /bin/apt-get
install

Package Just apt-

management get update
apt-get update %noc ALL=(ALL) NOPASSWD:/usr
/bin/apt-get update

Package Install
management packages
apt-get install %noc ALL=(ALL) NOPASSWD:/usr
vim /bin/apt-get install *

Package Upgrading
management
apt-get upgrade %noc ALL=(ALL) NOPASSWD:/usr
/bin/apt-get upgrade

Netfilter Install ACL

policies
cl-acltool -i %noc ALL=(ALL) NOPASSWD:/usr
/cumulus/bin/cl-acltool

[Link] 107
Cumulus Linux 3.5 User Guide
Category Privilege Example Command sudoers Entry

Netfilter List
iptables
rules iptables -L %noc ALL=(ALL) NOPASSWD:
/sbin/iptables

L1 + 2 features Any LLDP

command
lldpcli show %noc ALL=(ALL) NOPASSWD:/usr
neighbors / /sbin/lldpcli
configure

L1 + 2 features Just show

neighbors
lldpcli show %noc ALL=(ALL) NOPASSWD:/usr
neighbors /sbin/lldpcli show
neighbours*

Interfaces Modify any

interface
ip link set dev %noc ALL=(ALL) NOPASSWD:
swp1 {up|down} /sbin/ip link set *

Interfaces Up any
interface
ifup swp1 %noc ALL=(ALL) NOPASSWD:
/sbin/ifup

Interfaces Down any

interface
ifdown swp1 %noc ALL=(ALL) NOPASSWD:
/sbin/ifdown

Interfaces Up/down
only swp2
ifup swp2 /
ifdown swp2

108 23 January 2018

 Cumulus Networks

Category Privilege Example Command sudoers Entry

%noc ALL=(ALL) NOPASSWD:

/sbin/ifup swp2,/sbin
/ifdown swp2

Interfaces Any IP
address
chg ip addr %noc ALL=(ALL) NOPASSWD:
{add|del} /sbin/ip addr *
[Link]/30
dev swp1

Interfaces Only set IP

address
ip addr add %noc ALL=(ALL) NOPASSWD:
[Link]/30 /sbin/ip addr add *
dev swp1

Ethernet Any bridge

bridging command
brctl addbr br0 %noc ALL=(ALL) NOPASSWD:
/ brctl delif /sbin/brctl
br0 swp1

Ethernet Add
bridging bridges
and ints brctl addbr br0 %noc ALL=(ALL) NOPASSWD:
/ brctl addif /sbin/brctl addbr *,/sbin
br0 swp1 /brctl addif *

Spanning tree Set STP

properties
mstpctl %noc ALL=(ALL) NOPASSWD:
setmaxage br2 20 /sbin/mstpctl

[Link] 109
Cumulus Linux 3.5 User Guide

Category Privilege Example Command sudoers Entry

Troubleshooting Restart
switchd
systemctl %noc ALL=(ALL) NOPASSWD:/usr
restart switchd. /sbin/service switchd *
service

Troubleshooting Restart
any service
systemctl cron %noc ALL=(ALL) NOPASSWD:/usr
[Link] /sbin/service

Troubleshooting Packet
capture
tcpdump %noc ALL=(ALL) NOPASSWD:/usr
/sbin/tcpdump

L3 Add static
routes
ip route add %noc ALL=(ALL) NOPASSWD:/bin
[Link]/16 via /ip route add *
[Link]

L3 Delete
static
routes ip route del %noc ALL=(ALL) NOPASSWD:/bin
[Link]/16 via /ip route del *
[Link]

L3 Any static
route chg
ip route * %noc ALL=(ALL) NOPASSWD:/bin
/ip route *

L3 Any
iproute
command ip *

110 23 January 2018

 Cumulus Networks

Category Privilege Example Command sudoers Entry

%noc ALL=(ALL) NOPASSWD:/bin

/ip

L3 Non-
modal
OSPF cl-ospf area %noc ALL=(ALL) NOPASSWD:/usr
[Link] range /bin/cl-ospf
[Link]/24

Related Information
sudo
Adding Yourself to sudoers

LDAP Authentication and Authorization

Cumulus Linux uses Pluggable Authentication Modules (PAM) and Name Service Switch (NSS) for user
authentication.
NSS specifies the order of information sources used to resolve names for each service. Using this with
authentication and authorization, it provides the order and location used for user lookup and group
mapping on the system. PAM handles the interaction between the user and the system, providing login
handling, session setup, authentication of users and authorization of a user actions.
NSS enables PAM to use LDAP for providing user authentication, group mapping and information for other
services on the system.

Contents
This chapter covers ...
Configuring LDAP Authentication (see page 112)
Installing libnss-ldapd (see page 112)
Configuring [Link] (see page 113)
Connection (see page 113)
Search Function (see page 113)
Search Filters (see page 114)
Attribute Mapping (see page 114)
Example Configuration (see page 114)
Troubleshooting (see page 114)

Using nslcd Debug Mode (see page 114)

[Link] 111
Cumulus Linux 3.5 User Guide

Using nslcd Debug Mode (see page 114)

Common Problems (see page 116)
Configuring LDAP Authorization (see page 117)
Active Directory Configuration (see page 117)
LDAP Verification Tools (see page 118)
Identifying a User with the id Command (see page 118)
Using getent (see page 118)
Using LDAP search (see page 118)
LDAP Browsers (see page 119)
Related Information (see page 120)

Configuring LDAP Authentication

There are 3 common ways of configuring LDAP authentication on Linux:
libnss-ldap
libnss-ldapd
libnss-sss
This chapter covers using libnss-ldapd only. From internal testing, this library worked best with Cumulus
Linux and was the easiest to configure, automate and troubleshoot.

Installing libnss-ldapd
The libpam-ldapd package depends on nslcd, so to install libnss-ldapd, libpam-ldapd and ldap-
utils, you must run:

cumulus@switch:~$ sudo apt-get install libnss-ldapd libpam-ldapd ldap-

utils nslcd

This brings up an interactive prompt asking questions about the LDAP URI, search base distinguished name
(DN) and services that should have LDAP lookups enabled. This creates a very basic LDAP configuration,
using anonymous bind, and initiating the search for a user under the base DN specified.

Alternatively, these parameters can be pre-seeded using the debconf-utils. To use this
method, run apt-get install debconf-utils and create the pre-seeded parameters using
debconf-set-selections with the appropriate answers. Run debconf-show <pkg> to
check the settings. Here is an example of how to preseed answers to the installer questions using
debconf-set-selections .

Once the install is complete, the name service LDAP caching daemon (nslcd) will be running. This is the
service that handles all of the LDAP protocol interactions, and caches the information returned from the
LDAP server. In /etc/[Link], ldap has been appended and is the secondary information
source for passwd, group and shadow. The local files (/etc/passwd, /etc/groups and /etc/shadow) are
used first, as specified by the compat source.

112 23 January 2018

 Cumulus Networks

passwd: compat ldap

group: compat ldap
shadow: compat ldap

You are strongly advised to keep compat as the first source in NSS for passwd, group and shadow.
This prevents you from getting locked out of the system.

Configuring [Link]
You need to update the main configuration file (/etc/[Link]) after installation to accommodate the
expected LDAP server settings. The [Link] man page details all the available configuration options.
Some of the more important options are related to security and how the queries are handled.

Connection
The LDAP client starts a session by connecting to the LDAP server, by default, on TCP and UDP port 389, or
on port 636 for LDAPS. Depending on the configuration, this connection may be unauthenticated
(anonymous bind); otherwise, the client must provide a bind user and password. The variables used to
define the connection to the LDAP server are the URI and bind credentials.
The URI is mandatory, and specifies the LDAP server location using the FQDN or IP address. It also
designates whether to use ldap:// for clear text transport, or ldaps:// for SSL/TLS encrypted transport.
Optionally, an alternate port may also be specified in the URI. Typically, in production environments, it is
best to utilize the LDAPS protocol. Otherwise all communications are clear text and not secure.
After the connection to the server is complete, the BIND operation authenticates the session. The BIND
credentials are optional, and if not specified, an anonymous bind is assumed. This is typically not allowed in
most production environments. Configure authenticated (Simple) BIND by specifying the user ( binddn) and
password (bindpw) in the configuration. Another option is to use SASL (Simple Authentication and Security
Layer) BIND, which provides authentication services using other mechanisms, like Kerberos. Contact your
LDAP server administrator for this information since it depends on the configuration of the LDAP server
and what credentials are created for the client device.

# The location at which the LDAP server(s) should be reachable.

uri ldaps://[Link]
# The DN to bind with for normal lookups.
binddn cn=CLswitch,ou=infra,dc=example,dc=com
bindpw CuMuLuS

Search Function
When an LDAP client requests information about a resource, it must connect and bind to the server. Then
it performs one or more resource queries depending on what it is looking up. All search queries sent to the
LDAP server are created using the configured search base, filter, and the desired entry (uid=myuser) being
searched for. If the LDAP directory is large, this search may take a significant amount of time. It is a good
idea to define a more specific search base for the common maps (passwd and group).

[Link] 113
Cumulus Linux 3.5 User Guide

# The search base that will be used for all queries.

base dc=example,dc=com
# Mapped search bases to speed up common queries.
base passwd ou=people,dc=example,dc=com
base group ou=groups,dc=example,dc=com

Search Filters
It is also common to use search filters to specify criteria used when searching for objects within the
directory. This is used to limit the search scope when authenticating users. The default filters applied are:

filter passwd (objectClass=posixAccount)

filter group (objectClass=posixGroup)

Attribute Mapping
The map configuration allows for overriding the attributes pushed from LDAP. To override an attribute for a
given map*, specify the attribute name and the new value. One example of how this is useful is ensuring
the shell is bash and the home directory is /home/cumulus:

map passwd homeDirectory "/home/cumulus"

map passwd shell "/bin/bash"

*In LDAP, the map refers to one of the supported maps specified in the manpage for nslcd.
conf (such as passwd or group).

Example Configuration
Here is an example configuration using Cumulus Linux.

Troubleshooting

Using nslcd Debug Mode

When setting up LDAP authentication for the first time, Cumulus Networks recommends you turn off this
service using systemctl stop [Link] and run it in debug mode. Debug mode works whether
you are using LDAP over SSL (port 636) or an unencrypted LDAP connection (port 389).

cumulus@switch:~$ sudo systemctl stop [Link]

cumulus@switch:~$ sudo nslcd -d

Once you enable debug mode, run the following command to test LDAP queries:

114 23 January 2018

 Cumulus Networks

cumulus@switch:~$ sudo getent myuser

If LDAP is configured correctly, the following messages appear after you run the getent command:

nslcd: DEBUG: accept() failed (ignored): Resource temporarily

unavailable
nslcd: [8e1f29] DEBUG: connection from pid=11766 uid=0 gid=0
nslcd: [8e1f29] <passwd(all)> DEBUG: myldap_search(base="dc=example,
dc=com", filter="(objectClass=posixAccount)")
nslcd: [8e1f29] <passwd(all)> DEBUG: ldap_result(): uid=myuser,
ou=people,dc=example,dc=com
nslcd: [8e1f29] <passwd(all)> DEBUG: ldap_result(): ... 152 more
results
nslcd: [8e1f29] <passwd(all)> DEBUG: ldap_result(): end of results
(162 total)

In the output above, <passwd(all)> indicates that the entire directory structure was queried.
A specific user can be queried using the command:

cumulus@switch:~$ sudo getent passwd myuser

You can replace myuser with any username on the switch. The following debug output indicates that user
myuser exists:

nslcd: DEBUG: add_uri(ldap://[Link])

nslcd: version 0.8.10 starting
nslcd: DEBUG: unlink() of /var/run/nslcd/socket failed (ignored): No
such file or directory
nslcd: DEBUG: setgroups(0,NULL) done
nslcd: DEBUG: setgid(110) done
nslcd: DEBUG: setuid(107) done
nslcd: accepting connections
nslcd: DEBUG: accept() failed (ignored): Resource temporarily
unavailable
nslcd: [8b4567] DEBUG: connection from pid=11369 uid=0 gid=0
nslcd: [8b4567] <passwd="myuser"> DEBUG: myldap_search(base="
dc=cumulusnetworks,dc=com", filter="(&(objectClass=posixAccount)
(uid=myuser))")
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_initialize
(ldap://<ip_address>)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_rebind_proc()
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option
(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option
(LDAP_OPT_DEREF,0)

[Link] 115
Cumulus Linux 3.5 User Guide

nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option

(LDAP_OPT_TIMELIMIT,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option
(LDAP_OPT_TIMEOUT,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option
(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option
(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option
(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_simple_bind_s(NULL,
NULL) (uri="ldap://<ip_address>")
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_result(): end of
results (0 total)

Notice how the <passwd="myuser"> shows that the specific myuser user was queried.

Common Problems

SSL/TLS
The FQDN of the LDAP server URI does not match the FQDN in the CA-signed server certificate
exactly.
nslcd cannot read the SSL certificate, and will report a "Permission denied" error in the debug
during server connection negotiation. Check the permission on each directory in the path of the
root SSL certificate. Ensure that it is readable by the nslcd user.

NSCD
If the nscd cache daemon is also enabled and you make some changes to the user from LDAP,
you may want to clear the cache using the commands:

nscd --invalidate = passwd

nscd --invalidate = group

The nscd package works with nslcd to cache name entries returned from the LDAP server. This
may cause authentication failures. To work around these issues:
1. Disable nscd by running:

cumulus@switch:~$ sudo nscd -K

2. Restart the nslcd service:

cumulus@switch:~$ sudo systemctl restart [Link]

3. Try the authentication again.

116 23 January 2018
 Cumulus Networks

3. Try the authentication again.

LDAP
The search filter returns wrong results. Check for typos in the search filter. Use ldapsearch to test
your filter.
Optionally, configure the basic LDAP connection and search parameters in /etc/ldap/[Link]
.

# ldapsearch -D 'cn=CLadmin' -w 'CuMuLuS' "(&

(ObjectClass=inetOrgUser)(uid=myuser))"

When a local username also exists in the LDAP database, the order of the information sources in
/etc/nsswitch can be updated to query LDAP before the local user database. This is generally
not recommended. For example, the configuration below ensures that LDAP is queried before the
local database.

# /etc/[Link]
passwd: ldap compat

Configuring LDAP Authorization

Linux uses the sudo command to allow non-administrator users — like the default cumulus user account —
to perform privileged operations. To control the users authorized to use sudo, the /etc/sudoers file and
files located in the /etc/sudoers.d/ directory have a series of rules defined. Typically, the rules are
based on groups, but can also be defined for specific users. Therefore, sudo rules can be added using the
group names from LDAP. For example, if a group of users were associated with the group netadmin, a rule
can be added to give those users sudo privileges. Refer to the sudoers manual ( man sudoers) for a
complete usage description. Here's an illustration of this in /etc/sudoers:

# The basic structure of a user specification is “who where =

(as_whom) what”.
%sudo ALL=(ALL:ALL) ALL
%netadmin ALL=(ALL:ALL) ALL

Active Directory Configuration

Active Directory (AD) is a fully featured LDAP-based NIS server created by Microsoft. It offers unique
features that classic OpenLDAP servers lack. Therefore, it can be more complicated to configure on the
client and each version of AD is a little different in how it works with Linux-based LDAP clients. Some more
advanced configuration examples, from testing LDAP clients on Cumulus Linux with Active Directory (AD
/LDAP), are available in our knowledge base.

[Link] 117
Cumulus Linux 3.5 User Guide

LDAP Verification Tools

Typically, password and group information is retrieved from LDAP and cached by the LDAP client daemon.
To test the LDAP interaction, these command line tools can be used to trigger an LDAP query from the
device. This helps to create the best filters and verify the information sent back from the LDAP server.

Identifying a User with the id Command

The id command performs a username lookup by following the lookup information sources in NSS for the
passwd service. This simply returns the user ID, group ID and the group list retrieved from the information
source. In the following example, the user cumulus is locally defined in /etc/passwd, and myuser is on
LDAP. The NSS configuration has the passwd map configured with the sources compat ldap:

cumulus@switch:~$ id cumulus
uid=1000(cumulus) gid=1000(cumulus) groups=1000(cumulus),24(cdrom),25
(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev)
cumulus@switch:~$ id myuser
uid=1230(myuser) gid=3000(Development) groups=3000(Development),500
(Employees),27(sudo)

Using getent
The getent command retrieves all records found via NSS for a given map. It can also get a specific entry
under that map. Tests can be done with the passwd, group, shadow or any other map configured in /etc
/[Link]. The output from this command is formatted according to the map requested. Thus, for
the passwd service, the structure of the output is the same as the entries in /etc/passwd. The same can
be said for the group map will output the same as /etc/group. In this example, looking up a specific user
in the passwd map, the user cumulus is locally defined in /etc/passwd, and myuser is only in LDAP.

cumulus@switch:~$ getent passwd cumulus

cumulus:x:1000:1000::/home/cumulus:/bin/bash
cumulus@switch:~$ getent passwd myuser
myuser:x:1230:3000:My Test User:/home/myuser:/bin/bash

In the next example, looking up a specific group in the group service, the group cumulus is locally defined in
/etc/groups, and netadmin is on LDAP.

cumulus@switch:~$ getent group cumulus

cumulus:x:1000:
cumulus@switch:~$ getent group netadmin
netadmin:*:502:larry,moe,curly,shemp

Running the command getent passwd or getent group without a specific request, returns all local
and LDAP entries for the passwd and group maps, respectively.

Using LDAP search

118 23 January 2018
 Cumulus Networks

Using LDAP search

The ldapsearch command performs LDAP operations directly on the LDAP server. This does not interact
with NSS. This command helps display what the LDAP daemon process is receiving back from the server.
The command has many options. The simplest uses anonymous bind to the host and specifies the search
DN and what attribute to lookup.

cumulus@switch:~$ ldapsearch -H ldap://[Link] -b dc=example,

dc=com -x uid=myuser

Click to expand the command output ...

# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: uid=myuser
# requesting: ALL
#
# myuser, people, [Link]
dn: uid=myuser,ou=people,dc=example,dc=com
cn: My User
displayName: My User
gecos: myuser
gidNumber: 3000
givenName: My
homeDirectory: /home/myuser
initials: MU
loginShell: /bin/bash
mail: myuser@[Link]
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
objectClass: top
shadowExpire: -1
shadowFlag: 0
shadowMax: 999999
shadowMin: 8
shadowWarning: 7
sn: User
uid: myuser
uidNumber: 1234
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1

LDAP Browsers
[Link] 119
Cumulus Linux 3.5 User Guide

LDAP Browsers
There are some GUI LDAP clients that help to work with LDAP servers. These are free tools to help
graphically show the structure of the LDAP database.
Apache Directory Studio
LDAPManager

Related Information
Debian - configuring LDAP authentication
Debian - configuring PAM to use LDAP
GitHub - Arthur de Jong [Link] file
Debian backports

TACACS Plus
Cumulus Linux implements TACACS+ client AAA (Accounting, Authentication, and Authorization) in a
transparent way with minimal configuration. There is no need to create accounts or directories on the
switch. Accounting records are sent to all configuredTACACS+ servers by default. Use of per-command
authorization requires additional setup on the switch.

Contents
This chapter covers ...
Supported Features (see page 121)
Installing the TACACS+ Client Packages (see page )
Configuring the TACACS+ Client (see page )
TACACS+ Authentication (login) (see page )
TACACS+ Accounting (see page )
Configuring NCLU for TACACS+ Users (see page )
TACACS+ Per-command Authorization (see page )
Command Options (see page 125)
NSS Plugin (see page 126)
TACACS Configuration Parameters (see page 126)
Removing the TACACS+ Client Packages (see page )
Troubleshooting TACACS+ (see page )
Debugging Basic Server Connectivity or NSS Issues (see page 128)
Debugging Issues with Per-command Authorization (see page 129)
Debug Issues with Accounting Records (see page 130)
TACACS Component Software Descriptions (see page 130)
Limitations (see page 131)
TACACS+ Client Is only Supported through the Management Interface (see page )
Multiple TACACS+ Users (see page )

Issues with deluser Command (see page 132)

120 23 January 2018
 Cumulus Networks

Issues with deluser Command (see page 132)

Supported Features
Authentication via PAM; includes login, ssh, sudo and su
Runs over the eth0 management interface
Ability to run in the management VRF (see page 716)
TACACS+ privilege 15 users can run any command with sudo via the /etc/sudoers.d/tacplus
file that is installed by the libtacplus-map1 package
Up to 7 TACACS+ servers

Installing the TACACS+ Client Packages

TACACS+ requires the following packages to be installed on Cumulus Linux. They are not part of the base
Cumulus Linux image installation. All required packages can be installed easily with these commands:

cumulus@switch:~$ sudo -E apt-get update

cumulus@switch:~$ sudo -E apt-get install tacplus-client

Configuring the TACACS+ Client

Post-installation TACACS+ configuration requires (at minimum) editting only one file, /etc/tacplus_servers. It
is necessary add at least one server, and usually one shared secret (key). The server and secret
parameters can be given in any order, and must not include any whitespace (spaces or tabs), and can be
added anywhere in the file. For example, if your TACACS+ server IP address is [Link], and your
shared secret is tacacskey then you would add these parameters to /etc/tacplus_servers:

secret=tacacskey
server=[Link]

Up to 7 TACACS+ servers are supported. Connections are made in the order in which they are listed in this
file. In most cases, no other parameters need to be changed. All parameters used by any of the packages
can be added to this file, and will affect all the TACACS+ client software. It is also possible to configure some
of the packages through individual configuration files. For example, the timeout value (see description
below) is set to 5 seconds by default for NSS lookups in /etc/tacplus_nss.conf, while other packages
use a value of 10 seconds, set in /etc/tacplus_servers.
When TACACS+ servers or secrets are added or removed, auditd must be restarted (with systemctl
restart auditd) or a signal must be sent (with killall -HUP audisp-tacplus) before audisp-
tacplus will reread the configuration to see the changed server list. Usually this is an issue only at first
change to the configuration.
At this point, the Cumulus Linux switch should be able to query the TACACS server.
This is the complete list of the TACACS+ client configuration files, and their use. The full list of TACACS+
parameters is below at TACACS Parameters below.

[Link] 121
Cumulus Linux 3.5 User Guide

Filename Description

/etc This is the primary file that requires configuration post-installation, and is used by all
/tacplus_servers packages via include=/etc/tacplus_servers parameters in the other
configuration files, when installed. Since this is usually the file with the shared
secrets, it should not be world readable (should be Linux file mode 600).

/etc/nsswitch. When the libnss_tacplus package is installed, this file is configured to enable
conf tacplus lookups via libnss_tacplus. If this file is replaced by automation or other
means, you will need to add tacplus as the first lookup method for the passwd
database line.

/etc/tacplus_nss. This file sets the basic parameters for libnss_tacplus. It includes a debug variable
conf for debugging NSS lookups separately from other client packages.

/usr/share/pam- Configuration file for pam-auth-update to generate the files in the next row. These
configs/tacplus configurations are used at login, by su, and by ssh.

/etc/pam.d The /etc/pam.d/common-* files are updated for tacplus authentication. The files
/common-* are updated with pam-auth-update, when libpam-tacplus is installed or
removed.

/etc/sudoers.d This file allows TACACS+ privilege level 15 users to run commands with sudo. It also
/tacplus includes an example (commented out) showing how to enable privilege level 15
TACACS users to use sudo without having to enter a password, and an example of
how to enable all TACACS users to run specific commands via sudo. It should only be
editted with the command: visudo -f /etc/sudoers.d/tacplus

audisp-tacplus. audisp plugin configuration file. In general, no modifications are required.

conf

/etc/audisp TACACS+ server configuration file for accounting. In general, no modifications are
/audisp- required. It may be useful to use this configuration file when you only want to debug
tac_plus.conf TACACS+ accounting issues, not all TACACS+ users.

/etc/audit/rules. The auditd rules for TACACS+ accounting. The augenrules command uses all rules
d/audisp- files to generate the file rules file (below)
[Link]

/etc/audit/audit. Audit rules file generated during auditd installation.

rules

The /etc/pam.d/common-* files can be edited manually. However, if pam-auth-update is run

again after the changes are made, the update will fail. Configuration should be done in /usr
/share/pam-configs/tacplus instead, followed by running pam-auth-update.

122 23 January 2018

 Cumulus Networks

TACACS+ Authentication (login)

The initial authentication configuration is done through the PAM modules, and an updated version of the
libpam-tacplus package. When the package is installed, the PAM configuration is updated in /etc/pam.
d with the pam-auth-update command. If you have made changes to your PAM configuration, you may
need to integrate these changes yourself. If you are also using LDAP with the libpam-ldap package, you
will need to edit the PAM configuration to ensure the LDAP and TACACS ordering that you prefer. The
libpam-tacplus are configured to skip over rules, and the values in the success=2 may require
adjustments to skip over LDAP rules.

TACACS+ users at privilege levels other than 15 are not allowed to run sudo commands by
default, and are limited to commands that can be run with standard Linux user permissions.

TACACS+ Accounting
TACACS+ accounting is implemented with the audisp module, with an additional plugin for auditd/
audisp. The plugin maps the auid in the accounting record to a TACACS login, based on the auid and
sessionid. The audisp module requires libnss_tacplus, and uses the libtacplus_map.so library
interfaces as part of the modified lipam_tacplus package.
Communication with the TACACS+ servers is done via the libsimple-tacact1 library, through dlopen()
. A maximum of 240 bytes of command name and arguments are sent in the accounting record, due to the
TACACS+ field length limitation of 255 bytes.

All Linux commands result in an accounting record, including commands run as part of the login
process or as a sub-processes of other commands. This can sometimes generate a large number
of accounting records.

The IP address and encryption key of the server should be configured in the /etc/tacplus_servers file.
Minimal configuration to auditd and audisp is necessary to enable the audit records necessary for
accounting. These records are installed as part of the package.
audisp-tacplus installs the audit rules for command accounting. Modifying the configuration files is not
usually necessary. However, when a management VRF (see page 716) is configured, the accounting
configuration does need special modification, because the auditd service starts prior to networking. It is
necessary add the vrf parameter, and to signal the audisp-tacplus process to reread the configuration.
The example below shows that the management VRF is named mgmt. The vrf parameter can be placed in
either /etc/tacplus_servers or in /etc/audisp/audisp-tac_plus.conf.

vrf=mgmt

After editing the configuration file, notify the accounting process to reread it by sending the HUP signal:
killall -HUP audisp-tacplus.

All sudo commands run by TACACS+ users generate accounting records against the original
TACACS+ login name.

[Link] 123
Cumulus Linux 3.5 User Guide

For more information, refer to the audisp.8 and auditd.8 man pages.

Configuring NCLU for TACACS+ Users

NCLU (see page 85) has its own configuration file to enable TACACS+ privilege level 0 users to run the net
command. Edit the /etc/[Link] file, then:
To give a TACACS+ user access to the show commands, add that user to the users_with_show
line, then add the tacacs group to the groups_with_show line.
To give a TACACS+ user access to the edit commands (for all the NCLU write commands and to
restart services with NCLU), add that user to the users_with_edit line.

cumulus@switch:~$ sudo nano /etc/[Link]

...

# Control which users/groups are allowed to run 'add', 'del',

# 'clear', 'net abort', 'net commit' and restart services
# to apply those changes
users_with_edit = root, cumulus, TACACS_USER
groups_with_edit = netedit

# Control which users/groups are allowed to run 'show' commands

users_with_show = root, cumulus, TACACS_USER
groups_with_show = netshow, tacacs

...

TACACS_USER in the above output is actually the username of the account logged in via TACACS.

Do not add the tacacs group to the groups_with_edit line, as this is dangerous and can
potentially enable any user to log into the switch as the root user.

If the user/command combination is not authorized by the TACACS+ server, a message similar to the
following gets displayed:

tacuser0@switch:~$ net show version

net not authorized by TACACS+ with given arguments, not executing

124 23 January 2018

 Cumulus Networks

TACACS+ Per-command Authorization

Per-command authorization is handled with the tacplus-auth command. To make this an enforced
authorization, the TACACS+ login must be changed to use a restricted shell, with a very limited executable
search path. Otherwise, the user can bypass the authorization. The tacplus-restrict utility simplifies
the setup of the restricted environment. The example below initializes the environment for the tacacs0 user
account. This is the account used for TACACS+ users at privilege level 0.

tacuser0@switch:~$ sudo tacplus-restrict -i -u tacacs0 -a command1

command2 ... commandN

Command Options

Option Description

-i Initializes the environment. It only needs to be issued once per username.

-a The utility can be invoked with the -a option as many times as desired. For each command in
the -a list, a symbolic link is created from tacplus-auth to the relative portion of the
command name in the local bin subdirectory. These commands also need to be enabled on
the TACACS+ server. See the server documentation for how to do that. It is common to have
the server allow some options to a command, but not others.

-f Re-initializes the environment. If you need to start over, issue the -f option with the -i to
force the re-initialization; otherwise, repeated use of -i is ignored. As part of the initialization:
The user's shell is changed to /bin/rbash.
Any existing dot files are saved.
A limited environment is set up that does not allow general command execution, but
instead allows only commands from the user's local bin subdirectory.

As a full example, if you want to allow the user to be able to run the net and ip commands (potentially, if
the TACACS+ server authorizes), use the command:

cumulus@switch:~$ sudo tacplus-restrict -i -u tacacs0 -a ip net

After running this command, examining the tacacs0 directory should show something similar to the
following:

cumulus@switch:~$ sudo ls -lR ~tacacs0

total 12
lrwxrwxrwx 1 root root 22 Nov 21 22:07 ip -> /usr/sbin/tacplus-auth
lrwxrwxrwx 1 root root 22 Nov 21 22:07 net -> /usr/sbin/tacplus-auth

Other than shell built-ins, the only two commands the privilege level 0 TACACS users can run are the ip
[Link] 125
Cumulus Linux 3.5 User Guide

Other than shell built-ins, the only two commands the privilege level 0 TACACS users can run are the ip
and net commands.
If you mistakenly add potential commands with the -a option, you can remove the commands that you
don't want (the example below shows the net command):

cumulus@switch:~$ sudo rm ~tacacs0/bin/net

Or you can remove all the commands with:

cumulus@switch:~$ sudo rm ~tacacs0/bin/*

Use the man command on the switch for more information on tacplus-auth and tacplus-restrict.

cumulus@switch:~$ man tacplus-auth tacplus-restrict

NSS Plugin
When used with pam_tacplus, TACACS+ authenticated users are able to log in without a local account on
the system via the NSS plugin that comes with the tacplus_nss package. The plugin uses the mapped
tacplus information if the user is not found in the local password file, provides the getpwnam() and
getpwuid()entry point,s and uses the TACACS+ authentication functions.
The plugin asks the TACACS+ server if the user is known, and then for relevant attributes to determine the
user’s privilege level. When the libnss_tacplus package is installed, [Link] is be modified to
set tacplus as the first lookup method for passwd. If the order is changed, lookups will return the local
accounts such as tacacs0
If the user is not found, a mapped lookup is performed using the libtacplus_map.so exported
functions. The privilege level is appended to “tacacs”, and the lookup searches for the name in the local
password file. For example, privilege level 15 will search for the tacacs15 user. If the user is found, the
password structure is filled in with the user’s information.
If it is not found, the privilege level is decremented and checked again, until privilege level 0 (user t acacs0)
is reached. This allows use of only the two local users tacacs0 and tacacs15, if minimal configuration is
desired.

TACACS Configuration Parameters

The recognized configuration options are the same as the libpam_tacplus command line arguments;
not all pam_tacplus options are supported, however. These configuration parameters are documented in
the tacplus_servers.5 man page, which is part of the libpam-tacplus package.
The table below describes the configuration options available:

Configuration Description
Option

debug Output debugging information via syslog(3).

126 23 January 2018

 Cumulus Networks

Configuration Description
Option

Debugging is heavy, including passwords. Do not leave debugging

enabled on a production switch once you have completed
troubleshooting.

secret=STRING Secret key used to encrypt/decrypt packets sent to/received from the server. Can
be specified more than once, and can be in any order with respect to the server=
parameter. When fewer secret= parameters are specified, the last secret given is
used for the remaining servers. This parameter should only be put into files such as
/etc/tacplus_servers that are not world readable.

server=HOSTNAME Adds a TACACS+ server to the servers list. Servers will be queried in turn until a
match is found, or no servers remain in the list. Can be specified up to 7 times.
server=IP_ADDR
When the IP_ADDR form is used, it can be optionally followed by a port number,
preceded by a ":". The default port is 49.

When sending accounting records, the record is sent to all servers in the
list if acct_all=1, which is the default.

timeout=SECONDS TACACS+ server(s) communication timeout. The default value is 5 seconds.

login=STRING TACACS+ authentication service (pap, chap, or login). The default value is pap.

user_homedir=1 This is not enabled by default. When enabled, a separate home directory for each
TACACS+ user is created when the TACACS+ user first logs in. By default, the home
directory in the mapping accounts in /etc/passwd (/home/tacacs0 ... /home
/tacacs15) is used. If the home directory does not exist, it is created with the
mkhomedir_helper program, in the same manner as pam_mkhomedir.
This option is not honored for accounts with restricted shells when per-command
authorization is enabled.

acct_all=1 Configuration option for audisp_tacplus and pam_tacplus sending accounting

records to all supplied servers (1), or the first server to respond (0). The default
value is 1.

timeout=SECS Sets the timeout in seconds for connections to each TACACS+ server. The default is
10 seconds for all lookups except that NSS lookups use a 5 second timeout.

vrf=VRFNAME If the management network is in a VRF, set this variable to the VRF name. This
would usually be "mgmt". When this variable is set, the connection to the TACACS+
accounting servers is made through the named VRF.

service TACACS+ accounting and authorization service. Examples include shell, pap,
raccess, ppp, and slip.

[Link] 127
Cumulus Linux 3.5 User Guide

Configuration Description
Option

The default value is shell.

protocol TACACS+ protocol field. This option is use dependent.

PAM uses the SSH protocol.

Removing the TACACS+ Client Packages

To remove all of the TACACS+ client packages, use the following commands:

cumulus@switch:~$ sudo -E apt-get remove tacplus-client

cumulus@switch:~$ sudo -E apt-get autoremove

To remove the TACACS+ client configuration files as well as the packages (recommended), use this
command:

cumulus@switch:~$ sudo -E apt-get autoremove --purge

Troubleshooting TACACS+

Debugging Basic Server Connectivity or NSS Issues

The getent command can be used to determine whether TACACS+ is configured correctly, and the local
password is stored in the configuration files. In the example commands below, the cumulus user
represents the local user, while cumulusTAC represents the TACACS user.
To look up the username within all NSS methods:

cumulus@switch:~$ sudo getent passwd cumulusTAC

cumulusTAC:x:1016:1001:TACACS+ mapped user at privilege level 15,,,:
/home/tacacs15:/bin/bash

To look up the user within the local database only:

cumulus@switch:~$ sudo getent -s compat passwd cumulus

cumulus:x:1000:1000:cumulus,,,:/home/cumulus:/bin/bash

To look up the user within the TACACS+ database only:

cumulus@switch:~$ sudo getent -s tacplus passwd cumulusTAC

128 23 January 2018

 Cumulus Networks

cumulusTAC:x:1016:1001:TACACS+ mapped user at privilege level 15,,,:

/home/tacacs15:/bin/bash

If TACACS does not appear to be working correctly, the following configuration files should be debugged by
adding the debug=1 parameter to one or more of these files:
/etc/tacplus_servers
/etc/tacplus_nss.conf

debug=1 can also be added to individual pam_tacplus lines in /etc/pam.d/common*.

All log messages are stored in /var/log/syslog.

Incorrect Shared Key

The TACACS client on the switch and the TACACS server should have the same shared secret key. If this key
is incorrect, the following messages is printed to syslog:

2017-09-05T[Link].356520+00:00 leaf01 sshd[3176]: nss_tacplus:

TACACS+ server [Link]:49 read failed with protocol error
(incorrect shared secret?) user cumulus

Debugging Issues with Per-command Authorization

To debug TACACS user command authorization, have the TACACS+ user enter the following command at a
shell prompt, and then try the command again:

tacuser0@switch:~$ export TACACSAUTHDEBUG=1

When this debugging is enabled, additional information is shown for the command authorization
conversation with the TACACS+ server:

tacuser0@switch:~$ net pending

tacplus-auth: found matching command (/usr/bin/net) request
authorization
tacplus-auth: error connecting to [Link]:49 to request
authorization for net: Transport endpoint is not connected
tacplus-auth: cmd not authorized (16)
tacplus-auth: net not authorized from [Link]:49
net not authorized by TACACS+ with given arguments, not executing

tacuser0@switch:~$ net show version

tacplus-auth: found matching command (/usr/bin/net) request
authorization
tacplus-auth: error connecting to [Link]:49 to request
authorization for net: Transport endpoint is not connected

[Link] 129
Cumulus Linux 3.5 User Guide

tacplus-auth: [Link]:49 authorized command net

tacplus-auth: net authorized, executing
DISTRIB_ID="Cumulus Linux"
DISTRIB_RELEASE=3.4.0
DISTRIB_DESCRIPTION="Cumulus Linux 3.4.0"

To disable debugging:

tacuser0@switch:~$ export -n TACACSAUTHDEBUG

Debug Issues with Accounting Records

If TACACS+ servers have been added or deleted from the configuration files, make sure you notify the
audisp plugin with this command:

cumulus@switch:~$ sudo killall -HUP audisp-tacplus

If accounting records still are not being sent, add debug=1 to the file /etc/audisp/audisp-tac_plus.
conf, and then issue the command above to notify the plugin. Then have the TACACS+ user run a
command, and examine the end of /var/log/syslog for messages from the plugin. You can also check
the auditing log file /var/log/audit/[Link] to be sure the auditing records are being written. If
they are not, restart the audit daemon with:

cumulus@switch:~$ sudo systemctl restart [Link]

TACACS Component Software Descriptions

These different pieces of software are involved with delivering TACACS. Provided below is a brief description
of their functionalities.

Package Description
Name

audisp- This package uses auditing data from auditd to send accounting records to the TACACS+
tacplus_1. server and is started as part of auditd.
0.0-1-
cl3u3

libtac2_1. Basic TACACS+ server utility and communications routines.

4.0-cl3u2

libnss- Provides an interface between libc username lookups, the mapping functions, and the
tacplus_1. TACACS+ server.
0.1-cl3u3

130 23 January 2018

 Cumulus Networks

Package Description
Name

tacplus- This package provides the ability to do per-command TACACS+ authorization, and a setup
auth-1.0.0- utility tacplus-restrict to enable that. Per-command authorization is not done by default.
cl3u1

libpam- A modified version of the standard Debian package.

tacplus_1.
4.0-1-
cl3u2

libtacplus- The mapping functionality between local and TACACS+ users on the server. Sets the
map1_1. immutable sessionid and auditing UID to ensure the original user can be tracked through
0.0-cl3u2 multiple processes and privilege changes. Sets the auditing loginuid as immutable if
supported. Creates and maintains a status database in /run/tacacs_client_map to
manage and lookup mappings.

libsimple- Provides an interface for programs to send accounting records to the TACACS+ server. Used
tacacct1_1. by audisp-tacplus.
0.0-cl3u2

libtac2- Provides the “tacc” testing program and TACACS+ man page.
bin_1.4.0-
cl3u2

Limitations

TACACS+ Client Is only Supported through the Management Interface

The TACACS+ client is only supported through the switch's management interface, which could be eth0 or
eth1, or even the VRF management interface. The TACACS+ client is not supported through bonds, switch
virtual interfaces (SVIs) or switch port interfaces (swp).

Multiple TACACS+ Users

If two or more TACACS+ users are logged in simultaneously, with the same privilege level, while the
accounting records are maintained correctly, a lookup on either name will match both users, while a UID
lookup will only return the user that logged in first.
This means that any processes run by either user will be attributed to both, and all files created by either
user will be attributed to the first name matched. This is similar to adding two local users to the password
file with the same UID and GID, and is an inherent limitation of using the UID for the base user from the
password file.

The current algorithm returns the first name matching the UID from the mapping file; this could
be the first or second user that logged in.

To work around this issue, the switch’s audit log or the TACACS server accounting logs can be used to
[Link] 131
Cumulus Linux 3.5 User Guide

To work around this issue, the switch’s audit log or the TACACS server accounting logs can be used to
determine which processes and files were created by each user.
For commands that do not execute other commands (for example, changes to configurations in an
editor, or actions with tools like clagctl and vtysh), no additional accounting is done.
Per-command authorization is not implemented in this release except at the most basic level
(commands are permitted or denied based on the standard Linux user permissions for the local
TACACS users, and only privilege level 15 users can run sudo commands by default).
The Linux auditd system does not always generate audit events for processes when terminated with a
signal (via the kill system call or internal errors such as SIGSEGV). As a result, processes that exit on a
signal that isn’t caught and handled may not generate a STOP accounting record.

Issues with deluser Command

TACACS+ and other non-local users that run the deluser command with the --remove-home option will
see an error about not finding the user in /etc/passwd:

tacuser0@switch: deluser --remove-home USERNAME

userdel: cannot remove entry ‘USERNAME’ from /etc/passwd
/usr/sbin/deluser: `/usr/sbin/userdel USERNAME' returned error code
1. Exiting

However, the command does remove the home directory. The user can still log in on that account, but will
not have a valid home directory. This is a known upstream issue with the deluser command for all non-
local users.
The --remove-home option should only be used when the user_homedir=1 configuration command is
in use.

RADIUS AAA
Cumulus Networks offers add-on packages that provide the ability for RADIUS users to log in to Cumulus
Linux switches in a transparent way with minimal configuration. There is no need to create accounts or
directories on the switch. Authentication is handled via PAM, and includes login, ssh, sudo and su.

Installing and Configuring the RADIUS Packages

The plugin is installed from two RADIUS packages, which are not in the base Cumulus Linux image. There is
no RADIUS metapackage.
To install the plugins, run these commands:

cumulus@switch:~$ sudo apt-get update

cumulus@switch:~$ sudo apt-get install libnss-mapuser libpam-radius-
auth

To remove the plugins, see below (see page 135).

132 23 January 2018

 Cumulus Networks

During installation, the PAM configuration is modified automatically via pam-auth-update (8), and the
NSS configuration file /etc/[Link] is modified to add the mapuser and mapuid plugins. If you
remove or purge the packages, these files are modified to remove the configuration for these plugins.

Configuring the RADIUS Client

For common use cases, the only configuration file that needs to be modified is /etc/pam_radius_auth.
conf. You need to add the hostname or IP address of at least one RADIUS server (such as a freeradius
server on Linux), and the shared secret used to authenticate and encrypt communication with the server. If
you specify multiple server configuration lines, they are verified in the order listed. Other than memory,
there is no limit to the number of RADIUS servers that may be on a given switch. These fields are also
described in the pam_radius_auth (5) man page.
Consider the following example configuration in the /etc/pam_radius_auth.conf file:

# server[:port] shared_secret timeout (secs)

src_ip
aaa-testing radius321
[Link] secretkey

The timeout defaults to 3 seconds; you only need to change this setting if the server is slow or latencies
are high.
The src_ip option only needs to be specified if you want to use a specific interface to reach the server. It
can be set to the hostname of the interface, or an IPv4 or IPv6 address. If specified, the timeout option
must also be specified.
The other field that you may need to set is the vrf-name field. This is normally set to mgmt if you are using
a management VRF (see page 716). If you are specifying the src_ip field for a server entry, and that
interface is in a VRF, you may also need to set it. At this time, you cannot specify more than one VRF.
When first configuring the RADIUS client, it is a good idea to enable debugging. This can be done by
uncommenting the debug line in the configuration file. Debugging messages are written to syslog.
There are a number of PAM configuration keywords that you can set, but these do not normally need to be
configured. See the pam_radius_auth (8) man page for the variables that can be used. You can add
these either by editing:
The /etc/pam.d/common-auth lines for pam_radius_auth.so.
The /usr/share/pam-configs/radius file, then running pam-auth-update --package.
The latter method is recommended, because it is less likely to introduce errors.
The libpam-radius-auth package supplied with the Cumulus Linux RADIUS client is a newer version
than the one in Debian Jessie, and has added support for IPv6, the src_ip field described above, as well as
a number of bug fixes and minor features. Cumulus Linux further added VRF support, man pages
describing the PAM and RADIUS configuration, and the setting of the SUDO_PROMPT environment variable
to the login name as part of the mapping support described below.

Configuring NCLU for RADIUS Users

NCLU (see page 85) has its own configuration file to enable RADIUS users to run the net command. Edit
the /etc/[Link] file, then:

[Link] 133
Cumulus Linux 3.5 User Guide

To give a RADIUS user access to the show commands, add that user to the users_with_show line,
then add the radius_users group to the groups_with_show line.

To give a RADIUS user access to the edit commands (for all the NCLU write commands and to
restart services with NCLU), add that user to the users_with_edit line.

cumulus@switch:~$ sudo nano /etc/[Link]

...

# Control which users/groups are allowed to run 'add', 'del',

# 'clear', 'net abort', 'net commit' and restart services (quagga,
etc)
# to apply those changes
users_with_edit = root, cumulus, rad_user
groups_with_edit = netedit

# Control which users/groups are allowed to run 'show' commands

users_with_show = root, cumulus, rad_user
groups_with_show = netshow, radius_users

...

Do not add the radius_users group to the groups_with_edit line, as this is dangerous and can
potentially enable any user to log into the switch as the root user.

If the user/command combination is not authorized by the RADIUS server, a message similar to the
following gets displayed:

rad_user@switch:~$ net show version

net not authorized by TACACS+ with given arguments, not executing

Enabling Login without Local Accounts

The above description of the PAM interfaces is similar to the normal Debian libpam-radius-auth
package, which work without the libnss-mapuser package, if local or LDAP accounts are available for
each desired RADIUS user. Since LDAP is not commonly used with switches, and adding accounts locally is
cumbersome, Cumulus Linux added a mapping capability. This is enabled by the libnss-mapuser
package, which is specific to Cumulus Linux.
Mapping is done via two NSS (Name Service Switch) plugins, one for account name lookup, and one for UID
lookup. These are automatically configured in /etc/[Link] at installation, and are removed
when the package is removed. See the nss_mapuser (8) man page for the full description of this plugin.
Mapping is done on username at login to a fixed account specified in the configuration file, with the fields of
the fixed account used as a template for the user that is logging in.

134 23 January 2018

 Cumulus Networks

For example, if the name being looked up is dave and the fixed account in the configuration file is radius_user
, and that entry in /etc/passwd is:

radius_user:x:1017:1002:radius user:/home/radius_user:/bin/bash

then the matching line returned by running getent passwd dave would be:

cumulus@switch:~$ getent passwd dave

dave:x:1017:1002:dave mapped user:/home/dave:/bin/bash

and the home directory /home/dave would be created during the login process if it does not already exist,
and will be populated with the standard skeleton files by the mkhomedir_helper command.
The configuration file /etc/nss_mapuser.conf is used to configure the plugins. It does not normally
need to be changed. The nss_mapuser (5) man page fully describes the configuration file. Comments in
the file also describe the fields.
A flat file mapping is done based on the session number assigned during login, and it persists across su
and sudo. The mapping is removed at logout.

Enabling sudo Access for RADIUS Users

To allow RADIUS users to use sudo, you need to create a sudoers file authorizing them. The simplest case
is to give them full sudo access, similar to the cumulus user account. To do this, edit or create the /etc
/sudoers.d/radius file, and add the users to the file with an entry like this:

cumulus@switch:~$ sudo vi /etc/sudoers.d/radius

dave ALL=(ALL:ALL) ALL

where dave is the login account name of a RADIUS user. If you want all RADIUS users to have this ability, you
can enable sudo access for all members of the radius_users group:

cumulus@switch:~$ sudo vi /etc/sudoers.d/radius

%radius_users ALL=(ALL:ALL) ALL

Removing the RADIUS Client Packages

You can remove the RADIUS packages with the following command:

cumulus@switch:~$ sudo apt-get remove libnss-mapuser libpam-radius-

auth

When the packages are removed, the plugins are removed from the /etc/[Link] file and from
the PAM files, respectively.

To remove all configuration files for these packages, run:

[Link] 135
Cumulus Linux 3.5 User Guide

To remove all configuration files for these packages, run:

cumulus@switch:~$ sudo apt-get purge libnss-mapuser libpam-radius-auth

The RADIUS fixed account is not removed from /etc/passwd or /etc/group, and the home
directories are not removed either. They are left in case there are modifications to the account or
files in the home directories.

To remove the home directories of the RADIUS users, first get the list by running:

cumulus@switch:~$ sudo ls -l /home | grep radius

For all users listed other than the radius_user, run this command to remove the home directories:

cumulus@switch:~$ sudo deluser --remove-home USERNAME

where USERNAME is the account name (the home directory relative portion). This command gives the
following warning:

userdel: cannot remove entry 'USERNAME' from /etc/passwd

/usr/sbin/deluser: `/usr/sbin/userdel USERNAME' returned error code
1. Exiting.

Because the user is not listed in the /etc/passwd file. After removing all the RADIUS users, run the
command to remove the fixed account (the account may have been changed in /etc/nss_mapuser.conf
; if so, use that account name instead of radius_user).

cumulus@switch:~$ sudo deluser --remove-home radius_user

cumulus@switch:~$ sudo delgroup radius_users

Limitations

Multiple RADIUS Users

If two or more RADIUS users are logged in simultaneously, a UID lookup only returns the user that logged in
first. This means that any processes run by either user get attributed to both, and all files created by either
user get attributed to the first name matched. This is similar to adding two local users to the password file
with the same UID and GID, and is an inherent limitation of using the UID for the fixed user from the
password file.
The current algorithm returns the first name matching the UID from the mapping file; this could be the first
or second user that logged in.

136 23 January 2018

 Cumulus Networks

Related Information
TACACS+ client (see page 120)
Cumulus Networks RADIUS demo on GitHub
Cumulus Network TACACS demo on GitHub

Netfilter - ACLs
Netfilter is the packet filtering framework in Cumulus Linux as well as most other Linux distributions. There
are a number of tools available for configuring ACLs in Cumulus Linux, including:
iptables, ip6tables and ebtables are Linux userspace tools to administer filtering rules for
IPv4 packets, IPv6 packets and Ethernet frames (layer 2 using MAC addresses) respectively.
NCLU (see page 85), a Cumulus Linux-specific userspace tool for configuring custom ACLs.
cl-acltool, another Cumulus Linux-specific userspace tool to administer filtering rules and for
configuring the default ACLs.
NCLU and cl-acltool operate on various configuration files, and use iptables, ip6tables and
ebtables to install rules into the kernel. In addition to programming rules in the kernel, NCLU and cl-
acltool program rules in hardware for interfaces involving switch port interfaces, which iptables,
ip6tables and ebtables cannot do on their own.

In many instances, you can use NCLU to configure ACLs; however, cl-acltool must be used in
some cases. The examples below show when to use which tool.

If you need help getting started setting up ACLs, run net example acl to see a basic
configuration:
Click to see the example ...

cumulus@leaf01:~$ net example acl

Scenario
========
We would like to use access-lists on 'switch' to
- Restrict inbound traffic on swp1 to traffic from [Link]/24
destined for [Link]/24
- Restrict outbound traffic on swp2 to http, https, or ssh
*switch
/\
swp1 / \ swp2
/ \
/ \
host-11 host-12
switch net commands
====================
Create an ACL that accepts traffic from [Link]/24 destined
for [Link]/24

[Link] 137
Cumulus Linux 3.5 User Guide

and drops all other traffic

switch# net add acl ipv4 MYACL accept source-ip [Link]/24
dest-ip [Link]/24
switch# net add acl ipv4 MYACL drop source-ip any dest-ip any
Apply MYACL inbound on swp1
switch# net add interface swp1 acl ipv4 MYACL inbound
Create an ACL that accepts http, https, or ssh traffic and
drops all
other traffic.
switch# net add acl ipv4 WEB_OR_SSH accept tcp source-ip any
source-port any dest-ip any dest-port http
switch# net add acl ipv4 WEB_OR_SSH accept tcp source-ip any
source-port http dest-ip any dest-port any
switch# net add acl ipv4 WEB_OR_SSH accept tcp source-ip any
source-port any dest-ip any dest-port https
switch# net add acl ipv4 WEB_OR_SSH accept tcp source-ip any
source-port https dest-ip any dest-port any
switch# net add acl ipv4 WEB_OR_SSH accept tcp source-ip any
source-port any dest-ip any dest-port ssh
switch# net add acl ipv4 WEB_OR_SSH accept tcp source-ip any
source-port ssh dest-ip any dest-port any
switch# net add acl ipv4 WEB_OR_SSH drop source-ip any dest-ip
any
Apply WEB_OR_SSH outbound on swp2
switch# net add interface swp2 acl ipv4 WEB_OR_SSH outbound
commit the staged changes
switch# net commit
Verification
============
switch# net show configuration acl

Contents
This chapter covers ...
Understanding Traffic Rules In Cumulus Linux (see page 140)
Understanding Chains (see page 140)
Understanding Tables (see page 140)
Understanding Rules (see page 141)
How Rules Are Parsed and Applied (see page 142)
Rule Placement in Memory (see page 144)
Nonatomic Update Mode vs. Atomic Update Mode (see page 144)
Using iptables/ip6tables/ebtables Directly (see page 147)
Estimating the Number of Rules (see page 148)
Matching SVI/Bridged Interfaces in Rules (see page 149)
Installing and Managing ACL Rules with NCLU (see page 149)

Installing and Managing ACL Rules with cl-acltool (see page 151)
138 23 January 2018
 Cumulus Networks

Installing and Managing ACL Rules with cl-acltool (see page 151)
Installing Packet Filtering (ACL) Rules (see page 152)
Specifying which Policy Files to Install (see page 154)
Hardware Limitations on Number of Rules (see page 155)
Broadcom Tomahawk Limits (see page 155)
Broadcom Trident II+ Limits (see page )
Broadcom Trident II Limits (see page 156)
Broadcom Helix4 Limits (see page 156)
Mellanox Spectrum Limits (see page 157)
Supported Rule Types (see page 157)
iptables/ip6tables Rule Support (see page 158)
ebtables Rule Support (see page 158)
Other Unsupported Rules (see page 159)
Common Examples (see page 159)
Policing Control Plane and Data Plane Traffic (see page 159)
Setting DSCP on Transit Traffic (see page 161)
Verifying DSCP Values on Transit Traffic (see page 161)
Checking the Packet and Byte Counters for ACL Rules (see page 162)
Filtering Specific TCP Flags (see page 164)
Example Scenario (see page 164)
Switch 1 Configuration (see page 165)
Switch 2 Configuration (see page 166)
Egress Rule (see page 167)
Ingress Rule (see page 167)
Input Rule (see page 167)
Output Rule (see page 167)
Combined Rules (see page 167)
Layer 2-only Rules/ebtables (see page 168)
Useful Links (see page 168)
Caveats and Errata (see page 168)
Not All Rules Supported (see page 168)
ACL Log Policer Limits Traffic (see page 168)
Bridge Traffic Limitations (see page 168)
Log Actions Cannot Be Forwarded (see page 168)
Broadcom Range Checker Limitations (see page 168)
Inbound LOG Actions Only for Broadcom Switches (see page 169)
Tomahawk Hardware Limitations (see page 169)
Trident II+ Hardware Limitations (see page )

iptables Interactions with cl-acltool (see page 169)

[Link] 139
Cumulus Linux 3.5 User Guide

iptables Interactions with cl-acltool (see page 169)

Where to Assign Rules (see page 170)
Generic Error Message Displayed after ACL Rule Installation Failure (see page 170)
Dell S3048-ON Supports only 24K MAC Addresses (see page 170)

Understanding Traffic Rules In Cumulus Linux

Understanding Chains
Netfilter describes the mechanism for which packets are classified and controlled in the Linux kernel.
Cumulus Linux uses the Netfilter framework to control the flow of traffic to, from and across the switch.
Netfilter does not require a separate software daemon to run because it is part of the Linux kernel itself.
Netfilter asserts policies at layers 2, 3 and 4 of the OSI model by inspecting packet and frame headers
based on a list of rules. Rules are defined using syntax provided by the iptables, ip6tables and
ebtables userspace applications.
The rules created by these programs inspect or operate on packets at several points in the life of the
packet through the system. These five points are known as chains and are shown here:

The chains and their uses are:

PREROUTING: Touches packets before they are routed
INPUT: Touches packets once they are determined to be destined for the local system but before
they are received by the control plane software
FORWARD: Touches transit traffic as it moves through the box
OUTPUT: Touches packets that are sourced by the control plane software before they are put on
the wire
POSTROUTING: Touches packets immediately before they are put on the wire but after the routing
decision has been made

Understanding Tables
When building rules to affect the flow of traffic, the individual chains can be accessed by tables. Linux
provides three tables by default:
Filter: Classifies traffic or filters traffic
NAT: Applies Network Address Translation rules
140 23 January 2018
 Cumulus Networks

NAT: Applies Network Address Translation rules

Cumulus Linux does not support NAT.

Mangle: Alters packets as they move through the switch

Each table has a set of default chains that can be used to modify or inspect packets at different points of
the path through the switch. Chains contain the individual rules to influence traffic. Each table and the
default chains they support are shown below. Tables and chains in green are supported by Cumulus Linux,
those in red are not supported (that is, they are not hardware accelerated) at this time.

Understanding Rules
Rules are the items that actually classify traffic to be acted upon. Rules are applied to chains, which are
attached to tables, similar to the graphic below.

Rules have several different components; the examples below highlight those different components.

[Link] 141
Cumulus Linux 3.5 User Guide

Table: The first argument is the table. Notice the second example does not specify a table, that is
because the filter table is implied if a table is not specified.
Chain: The second argument is the chain. Each table supports several different chains. See
Understanding Tables above.
Matches: The third argument(s) are called the matches. You can specify multiple matches in a single
rule. However, the more matches you use in a rule, the more memory that rule consumes.
Jump: The jump specifies the target of the rule; that is, what action to take if the packet matches the
rule. If this option is omitted in a rule, then matching the rule will have no effect on the packet's fate,
but the counters on the rule will be incremented.
Target(s): The target can be a user-defined chain (other than the one this rule is in), one of the
special built-in targets that decides the fate of the packet immediately (like DROP), or an extended
target. See the Supported Rule Types and Common Usages (see page 157) section below for
examples of different targets.

How Rules Are Parsed and Applied

All the rules from each chain are read from iptables, ip6tables and ebtables and entered in order
into either the filter table or the mangle table. The rules are read from the kernel in the following order:
IPv6 (ip6tables)
IPv4 (iptables)
ebtables
When rules are combined and put into one table, the order determines the relative priority of the rules;
iptables and ip6tables have the highest precedence and ebtables has the lowest.
The Linux packet forwarding construct is an overlay for how the silicon underneath processes packets; to
that end, here are some things to be aware of:
The order of operations for how rules are processed is not perfectly maintained when you compare
how iptables and the switch silicon process packets. The switch silicon reorders rules when
switchd writes to the ASIC, whereas traditional iptables executes the list of rules in order.

142 23 January 2018

 Cumulus Networks

All rules are terminating. This means once a rule is matched, the action is carried out, and no more
rules are processed. The exception to this is when a SETCLASS rule is placed immediately before
another rule; this exists multiple times in the default ACL configuration.
In the example below, the SETCLASS action applied with the --in-interface option, creates the
internal ASIC classification, and continues to process the next rule, which does the rate-limiting for
the matched protocol:

-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p udp --dport

$BFD_ECHO_PORT -j SETCLASS --class 7
-A $INGRESS_CHAIN -p udp --dport $BFD_ECHO_PORT -j POLICE --set-
mode pkt --set-rate 2000 --set-burst 2000

If multiple contiguous rules with the same match criteria are applied to --in-interface
, only the first rule gets processed and then terminates processing. This would also be a
misconfiguration, because there is no reason to have duplicate rules with different
actions.

When processing traffic, rules affecting the FORWARD chain that specify an ingress interface are
performed prior to rules that match on an egress interface. As a workaround, rules that only affect
the egress interface can have an ingress interface wildcard (currently, only swp+ and bond+ are
supported as wildcard names; see below) that matches any interface applied so that you can
maintain order of operations with other input interface rules. Take the following rules, for example:

-A FORWARD -i $PORTA -j ACCEPT

-A FORWARD -o $PORTA -j ACCEPT <-- This rule is performed LAST
(because of egress interface matching)
-A FORWARD -i $PORTB -j DROP

If you modify the rules like this, they are performed in order:

-A FORWARD -i $PORTA -j ACCEPT

-A FORWARD -i swp+ -o $PORTA -j ACCEPT <-- These rules are
performed in order (because of wildcard match on ingress
interface)
-A FORWARD -i $PORTB -j DROP

When using rules that do a mangle and a filter lookup for a packet, Cumulus Linux does them in
parallel and combines the action.
If a switch port is assigned to a bond, any egress rules must be assigned to the bond.
When using the OUTPUT chain, rules must be assigned to the source. For example, if a rule is
assigned to the switch port in the direction of traffic but the source is a bridge (VLAN), the traffic
won't be affected by the rule and must be applied to the bridge.
If all transit traffic needs to have a rule applied, use the FORWARD chain, not the OUTPUT chain.

[Link] 143
Cumulus Linux 3.5 User Guide

ebtablesrules are put into either the IPv4 or IPv6 memory space depending on whether the rule
utilizes IPv4 or IPv6 to make a decision. Layer 2-only rules, which match the MAC address, are put
into the IPv4 memory space.

On Broadcom switches, the ingress INPUT chain rules match layer 2/layer 3 multicast packets
before multicast packet replication has occurred; hence, having a DROP rule affects all copies.

Rule Placement in Memory

INPUT and ingress (FORWARD -i) rules occupy the same memory space. A rule counts as ingress if the -i
option is set. If both input and output options (-i and -o) are set, the rule is considered as ingress and
occupies that memory space. For example:

-A FORWARD -i swp1 -o swp2 -s [Link] -d [Link] -p tcp -j ACCEPT

If you set an output flag with the INPUT chain you will get an error. For example, running cl-
acltool -i on the following rule:

-A FORWARD,INPUT -i swp1 -o swp2 -s [Link] -d [Link] -p

tcp -j ACCEPT

generates the following error:

error: line 2 : output interface specified with INPUT chain

error processing rule '-A FORWARD,INPUT -i swp1 -o swp2 -s
[Link] -d [Link] -p tcp -j ACCEPT'

However, simply removing the -o option and interface would make it a valid rule.

Nonatomic Update Mode vs. Atomic Update Mode

In Cumulus Linux, atomic update mode is enabled by default. However this mode limits the number of ACL
rules that may be configured by the user.

144 23 January 2018

 Cumulus Networks

To increase the number of ACL rules that may be configured, configure the switch to operate in nonatomic
mode.

How the Rules Get Installed

Instead of reserving 50% of your TCAM space for atomic updates, incremental update instead uses the
[Link] 145
Cumulus Linux 3.5 User Guide

Instead of reserving 50% of your TCAM space for atomic updates, incremental update instead uses the
available free space to write the new TCAM rules and swap over to the new rules once this is completed.
Cumulus Linux then deletes the old rules and frees up the original TCAM space. If there is insufficient free
space to complete this task, the original nonatomic update is performed, which interrupts traffic.

Enabling Nonatomic Update Mode

You can enable nonatomic updates for switchd, which offer better scaling because all TCAM resources
are used to actively impact traffic. With atomic updates, half of the hardware resources are on standby and
do not actively impact traffic.
Incremental nonatomic updates are table based, so they don't interrupt network traffic when new rules are
installed. The rules are mapped into the following tables and are updated in this order:
mirror (ingress only)
ipv4-mac (can be both ingress and egress)
ipv6 (ingress only)
The incremental nonatomic update operation follows this order:

1. Updates are performed incrementally, one table at a time without stopping traffic.
2. Cumulus Linux checks if a table's rules have changed since the last time they were installed; if a table
does not have any changes, then it is not reinstalled.
3. If there are changes in a table, then the new rules are populated in new groups or slices in hardware,
146 23 January 2018
 Cumulus Networks

3. If there are changes in a table, then the new rules are populated in new groups or slices in hardware,
then that table is switched over to the new groups or slices.
4. Finally, old resources for that table are freed. This process is repeated for each of the tables listed
above.
5. If there are not sufficient resources to hold both the new rule set and old rule set, then the regular
nonatomic mode is attempted. This will interrupt network traffic.
6. If the regular nonatomic update fails, Cumulus Linux reverts back to the previous rules.

To always start switchd with nonatomic updates:

1. Edit /etc/cumulus/[Link].
2. Add the following line to the file:

acl.non_atomic_update_mode = TRUE

3. Restart switchd:

cumulus@switch:~$ sudo systemctl restart [Link]

During regular non-incremental nonatomic updates, traffic is stopped first, and enabled after the
new configuration is written into the hardware completely.

Using iptables/ip6tables/ebtables Directly

Using iptables/ip6tables/ebtables directly is not recommended because any rules installed in these
cases only are applied to the Linux kernel and are not hardware accelerated via synchronization to the
switch silicon. Also running cl-acltool -i (the installation command) resets all rules and deletes
anything that is not stored in /etc/cumulus/acl/[Link].
For example, performing:

cumulus@switch:~$ sudo iptables -A INPUT -p icmp --icmp-type echo-

request -j DROP

Appears to work, and the rule appears when you run cl-acltool -L:

cumulus@switch:~$ sudo cl-acltool -L ip

-------------------------------
Listing rules of type iptables:
-------------------------------

TABLE filter :
Chain INPUT (policy ACCEPT 72 packets, 5236 bytes)

[Link] 147
Cumulus Linux 3.5 User Guide

pkts bytes target prot opt in out source destination

0 0 DROP icmp -- any any anywhere anywhere icmp echo-request

However, the rule is not synced to hardware when applied in this fashion and running cl-acltool -i or
reboot removes the rule without replacing it. To ensure all rules that can be in hardware are hardware
accelerated, place them in /etc/cumulus/acl/[Link] and install them by running cl-acltool
-i.

Estimating the Number of Rules

To estimate the number of rules that could be created from an ACL entry, first determine if that entry is an
ingress or an egress. Then determine if it is IPv4-mac or IPv6 type rule. This determines the slice to which
the rule belongs. Then use the following to determine how many entries are used up for each type.
By default, each entry occupies one double wide entry, except if the entry is one of the following:
An entry with multiple comma-separated input interfaces is split into one rule for each input
interface (listed after --in-interface below). For example, this entry splits into two rules:

-A FORWARD --in-interface swp1s0,swp1s1 -p icmp -j ACCEPT

An entry with multiple comma-separated output interfaces is split into one rule for each output
interface (listed after --out-interface below). This entry splits into two rules:

-A FORWARD --in-interface swp+ --out-interface swp1s0,swp1s1 -p

icmp -j ACCEPT

An entry with both input and output comma-separated interfaces is split into one rule for each
combination of input and output interface (listed after --in-interface and --out-interface
below). This entry splits into four rules:

-A FORWARD --in-interface swp1s0,swp1s1 --out-interface swp1s2,

swp1s3 -p icmp -j ACCEPT

An entry with multiple L4 port ranges is split into one rule for each range (listed after --dports
below). For example, this entry splits into two rules:

-A FORWARD --in-interface swp+ -p tcp -m multiport --dports 1050:

1051,1055:1056 -j ACCEPT

Port ranges are only allowed for ingress rules.

148 23 January 2018

 Cumulus Networks

Matching SVI/Bridged Interfaces in Rules

Cumulus Linux supports matching ACL rules for both ingress and egress interfaces on both VLAN-aware
(see page 327) and traditional mode (see page 338) bridges, including bridge SVIs (switch VLAN interfaces
(see page 323)) for input and output. However, keep the following in mind:
If a traditional mode bridge has a mix of different VLANs, or has both access and trunk members,
output interface matching is not supported.
For iptables rules, all IP packets in a bridge are matched, not just routed packets.
You cannot match both input and output interfaces in a rule.
For routed packets, Cumulus Linux cannot match the output bridge for SPAN/ERSPAN.
Following are example rules for a VLAN-aware bridge:

[ebtables]
-A FORWARD -i br0.100 -p IPv4 --ip-protocol icmp -j DROP
-A FORWARD -o br0.100 -p IPv4 --ip-protocol icmp -j ACCEPT

[iptables]
-A FORWARD -i br0.100 -p icmp -j DROP
-A FORWARD --out-interface br0.100 -p icmp -j ACCEPT
-A FORWARD --in-interface br0.100 -j POLICE --set-mode pkt --set-
rate 1 --set-burst 1 --set-class 0

And here are example rules for a traditional mode bridge:

[ebtables]
-A FORWARD -i br0 -p IPv4 --ip-protocol icmp -j DROP
-A FORWARD -o br0 -p IPv4 --ip-protocol icmp -j ACCEPT

[iptables]
-A FORWARD -i br0 -p icmp -j DROP
-A FORWARD --out-interface br0 -p icmp -j ACCEPT
-A FORWARD --in-interface br0 -j POLICE --set-mode pkt --set-rate
1 --set-burst 1 --set-class 0

Installing and Managing ACL Rules with NCLU

NCLU provides an easy way to create custom ACLs in Cumulus Linux. The rules you create live in the /var
/lib/cumulus/nclu/nclu_acl.conf file, which gets converted to a rules file, /etc/cumulus/acl
/policy.d/50_nclu_acl.rules. This way, the rules you create with NCLU are independent of the two
default files in /etc/cumulus/acl/policy.d/00control_plane.rules and
99control_plane_catch_all.rules, as the content in these files may get updated after you upgrade
Cumulus Linux.
Instead of crafting a rule by hand then installing it using cl-acltool, NCLU handles many of the options
automatically. For example, consider the following iptables rule:

[Link] 149
Cumulus Linux 3.5 User Guide

-A FORWARD -i swp1 -o swp2 -s [Link] -d [Link] -p tcp -j ACCEPT

You would create this rule, called EXAMPLE1, using NCLU like this:

cumulus@switch:~$ net add acl ipv4 EXAMPLE1 accept tcp source-ip

[Link]/32 source-port dest-ip [Link]/32 dest-port any
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

All options, such as the -j and -p, even FORWARD in the above rule, are added automatically when you
apply the rule to the control plane; NCLU figures it all out for you.
You can also set a priority value, which specifies the order in which the rules get executed, and the order in
which they appear in the rules file. Lower numbers are executed first. To add a new rule in the middle, first
run net show config acl, which displays the priority numbers. Otherwise, new rules get appended to
the end of the list of rules in the nclu_acl.conf and 50_nclu_acl.rules files.

If you need to hand edit a rule, don’t edit the 50_nclu_acl.rules file. Instead, edit the
nclu_acl.conf file.

After you add the rule, you need to apply it to an inbound or outbound interface using net add int acl;
swp1 is the inbound interface in our example:

cumulus@switch:~$ net add int swp1 acl ipv4 EXAMPLE1 inbound

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

After you commit your changes, you can verify the rule you created with NCLU by running net show
configuration acl:

cumulus@switch:~$ net show configuration acl

acl ipv4 EXAMPLEv4 priority 10 accept tcp source-ip [Link]/32
source-port any dest-ip [Link]/32 dest-port any

interface swp1
acl ipv4 EXAMPLE1 inbound

Or you can see all of the rules installed by running cat on the 50_nclu_acl.rules file:

cumulus@switch:~$ cat /etc/cumulus/acl/policy.d/50_nclu_acl.rules

[iptables]
# swp1: acl ipv4 EXAMPLE1 inbound

150 23 January 2018

 Cumulus Networks

-A FORWARD --in-interface swp1 --out-interface swp2 -j ACCEPT -p tcp -

s [Link]/32 -d [Link]/32 --dport 110

For INPUT and FORWARD rules, apply the rule to a control plane interface using net add control-
plane:

cumulus@switch:~$ net add control-plane acl ipv4 EXAMPLE1 inbound

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

To remove a rule, use net del acl ipv4|ipv6|mac RULENAME:

cumulus@switch:~$ net del acl ipv4 EXAMPLE1

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

This deletes all rules from the 50_nclu_acl.rules file with that name. It also deletes the interfaces
referenced in the nclu_acl.conf file.

Installing and Managing ACL Rules with cl-acltool

You can manage Cumulus Linux ACLs with cl-acltool. Rules are first written to the iptables chains, as
described above, and then synced to hardware via switchd.

Use iptables/ip6tables/ebtables and cl-acltool to manage rules in the default files,

00control_plane.rules and 99control_plane_catch_all.rules; they’re not aware of
rules created using NCLU.

To examine the current state of chains and list all installed rules, run:

cumulus@switch:~$ sudo cl-acltool -L all

-------------------------------
Listing rules of type iptables:
-------------------------------

TABLE filter :
Chain INPUT (policy ACCEPT 90 packets, 14456 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- swp+ any [Link]/5 anywhere
0 0 DROP all -- swp+ any loopback/8 anywhere
0 0 DROP all -- swp+ any [Link]/8 anywhere
0 0 DROP all -- swp+ any [Link] anywhere ...

To list installed rules using native iptables, ip6tables and ebtables, use the -L option with the
respective commands:

[Link] 151
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo iptables -L

cumulus@switch:~$ sudo ip6tables -L
cumulus@switch:~$ sudo ebtables -L

To flush all installed rules, run:

cumulus@switch:~$ sudo cl-acltool -F all

To flush only the IPv4 iptables rules, run:

cumulus@switch:~$ sudo cl-acltool -F ip

If the install fails, ACL rules in the kernel and hardware are rolled back to the previous state. Errors from
programming rules in the kernel or ASIC are reported appropriately.

Installing Packet Filtering (ACL) Rules

cl-acltool takes access control list (ACL) rules input in files. Each ACL policy file contains iptables,
ip6tables and ebtables categories under the tags [iptables], [ip6tables] and [ebtables]
respectively.
Each rule in an ACL policy must be assigned to one of the rule categories above.
See man cl-acltool(5) for ACL rule details. For iptables rule syntax, see man iptables(8). For
ip6tables rule syntax, see man ip6tables(8). For ebtables rule syntax, see man ebtables(8).
See man cl-acltool(5) and man cl-acltool(8) for further details on using cl-acltool; however,
some examples are listed here, and more are listed later in this chapter (see page ).

By default:
ACL policy files are located in /etc/cumulus/acl/policy.d/.
All *.rules files in this directory are included in /etc/cumulus/acl/[Link].
All files included in this [Link] file are installed when the switch boots up.
The [Link] file expects rules files to have a .rules suffix as part of the file name.

Here is an example ACL policy file:

[iptables]
-A INPUT --in-interface swp1 -p tcp --dport 80 -j ACCEPT
-A FORWARD --in-interface swp1 -p tcp --dport 80 -j ACCEPT

[ip6tables]
-A INPUT --in-interface swp1 -p tcp --dport 80 -j ACCEPT
-A FORWARD --in-interface swp1 -p tcp --dport 80 -j ACCEPT

152 23 January 2018

 Cumulus Networks

[ebtables]
-A INPUT -p IPv4 -j ACCEPT
-A FORWARD -p IPv4 -j ACCEPT

You can use wildcards or variables to specify chain and interface lists to ease administration of rules.

Interface Wildcards
Currently only swp+ and bond+ are supported as wildcard names. There may be kernel
restrictions in supporting more complex wildcards likes swp1+ etc.

INGRESS = swp+
INPUT_PORT_CHAIN = INPUT,FORWARD

[iptables]
-A $INPUT_PORT_CHAIN --in-interface $INGRESS -p tcp --dport 80 -j
ACCEPT

[ip6tables]
-A $INPUT_PORT_CHAIN --in-interface $INGRESS -p tcp --dport 80 -j
ACCEPT

[ebtables]
-A INPUT -p IPv4 -j ACCEPT

ACL rules for the system can be written into multiple files under the default /etc/cumulus/acl/policy.
d/ directory. The ordering of rules during installation follows the sort order of the files based on their file
names.
Use multiple files to stack rules. The example below shows two rules files separating rules for management
and datapath traffic:

cumulus@switch:~$ ls /etc/cumulus/acl/policy.d/
00sample_mgmt.rules 01sample_datapath.rules
cumulus@switch:~$ cat /etc/cumulus/acl/policy.d/00sample_mgmt.rules

INGRESS_INTF = swp+
INGRESS_CHAIN = INPUT

[iptables]
# protect the switch management
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -s [Link] -d
[Link] -p tcp -j ACCEPT
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -s [Link] -d
[Link] -p tcp -j ACCEPT
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -d [Link] -p udp -j
DROP

[Link] 153
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ cat /etc/cumulus/acl/policy.d/01sample_datapath.

rules
INGRESS_INTF = swp+
INGRESS_CHAIN = INPUT, FORWARD

[iptables]
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -s [Link] -p icmp -
j ACCEPT
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -s [Link] -d
[Link] -j DROP
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -s [Link] -d
[Link] -j DROP

Install all ACL policies under a directory:

cumulus@switch:~$ sudo cl-acltool -i -P ./rules

Reading files under rules
Reading rule file ./rules/01_http_rules.txt ...
Processing rules in file ./rules/01_http_rules.txt ...
Installing acl policy ...
Done.

Install all rules and policies included in /etc/cumulus/acl/[Link]:

cumulus@switch:~$ sudo cl-acltool -i

Specifying which Policy Files to Install

By default, any .rules file you configure in /etc/cumulus/acl/policy.d/ get installed by Cumulus
Linux. To add other policy files to an ACL, you need to include them in /etc/cumulus/acl/[Link]
. For example, in order for Cumulus Linux to install a rule in a policy file called 01_new.datapathacl, you
would add include /etc/cumulus/acl/policy.d/01_new.rules to [Link], as in this
example:

cumulus@switch:~$ sudo nano /etc/cumulus/acl/[Link]

#
# This file is a master file for acl policy file inclusion
#
# Note: This is not a file where you list acl rules.
#
# This file can contain:
# - include lines with acl policy files
# example:
# include <filepath>
#

154 23 January 2018

 Cumulus Networks

# see manpage cl-acltool(5) and cl-acltool(8) for how to write policy

files
#

include /etc/cumulus/acl/policy.d/01_new.datapathacl

Hardware Limitations on Number of Rules

The maximum number of rules that can be handled in hardware is a function of the following factors:
The platform type (switch silicon, like Tomahawk or Spectrum — see the HCL to determine which
platform type applies to a particular switch)
The mix of IPv4 and/or IPv6 rules; Cumulus Linux doesn't support the maximum number of rules for
both IPv4 and IPv6 simultaneously
The number of default rules provided by Cumulus Linux
Whether the rules are applied on ingress or egress
Whether the rules are in atomic or nonatomic mode; nonatomic mode rules are used when
nonatomic updates are enabled (see above (see page 144))
If the maximum number of rules for a particular table is exceeded, cl-acltool -i generates the
following error:

error: hw sync failed (sync_acl hardware installation failed) Rolling

back .. failed.

On Broadcom platforms, IPv6 egress rules are not supported. In certain cases the ebtables
egress rules can match switched IPv6 packets based on the etype field of the packet.

In the tables below, the default rules count toward the limits listed. The raw limits below assume only one
ingress and one egress table are present.

Broadcom Tomahawk Limits

Direction Atomic Mode Atomic Mode Nonatomic Mode Nonatomic Mode

IPv4 Rules IPv6 Rules IPv4 Rules IPv6 Rules

Ingress raw limit 512 512 1024 1024

Ingress limit with 256 (36 default) 256 (29 default) 768 (36 default) 768 (29 default)
default rules

Egress raw limit 256 0 512 0

Egress limit with 256 (29 default) 0 512 (29 default) 0

default rules

[Link] 155
Cumulus Linux 3.5 User Guide

Broadcom Trident II+ Limits

Direction Atomic Mode Atomic Mode Nonatomic Mode Nonatomic Mode

IPv4 Rules IPv6 Rules IPv4 Rules IPv6 Rules

Ingress raw limit 4096 4096 8192 8192

Ingress limit with 2048 (36 default) 3072 (29 default) 6144 (36 default) 6144 (29 default)
default rules

Egress raw limit 256 0 512 0

Egress limit with 256 (29 default) 0 512 (29 default) 0

default rules

Broadcom Trident II Limits

Direction Atomic Mode Atomic Mode Nonatomic Mode Nonatomic Mode

IPv4 Rules IPv6 Rules IPv4 Rules IPv6 Rules

Ingress raw limit 1024 1024 2048 2048

Ingress limit with 512 (36 default) 768 (29 default) 1536 (36 default) 1536 (29 default)
default rules

Egress raw limit 256 0 512 0

Egress limit with 256 (29 default) 0 512 (29 default) 0

default rules

Broadcom Helix4 Limits

Direction Atomic Mode Atomic Mode Nonatomic Mode Nonatomic Mode

IPv4 Rules IPv6 Rules IPv4 Rules IPv6 Rules

Ingress raw limit 1024 512 2048 1024

Ingress limit with 768 (36 default) 384 (29 default) 1792 (36 default) 896 (29 default)
default rules

Egress raw limit 256 0 512 0

Egress limit with 256 (29 default) 0 512 (29 default) 0

default rules

156 23 January 2018

 Cumulus Networks

Mellanox Spectrum Limits

The Mellanox Spectrum ASIC has one common TCAM for both ingress and egress, and it may be used for
other non-ACL-related resources. However, the number of supported rules varies with the TCAM profile
(see page 595) specified for the switch.

Profile Atomic Mode IPv4 Atomic Mode IPv6 Nonatomic Mode Nonatomic Mode
Rules Rules IPv4 Rules IPv6 Rules

default 1750 750 3500 1500

ipmc- 200 80 400 160

heavy

acl- 3000 2000 6000 2500

heavy

Supported Rule Types

The iptables/ip6tables/ebtables construct tries to layer the Linux implementation on top of the
underlying hardware but they are not always directly compatible. Here are the supported rules for chains in
iptables, ip6tables and ebtables.

To learn more about any of the options shown in the tables below, run iptables -h [name
of option]. The same help syntax works for options for ip6tables and ebtables.
Click to see an example of help syntax for an ebtables target

root@leaf1# ebtables -h tricolorpolice

<...snip...>
tricolorpolice option:
--set-color-mode STRING setting the mode in blind or aware
--set-cir INT setting committed information rate in kbits per
second
--set-cbs INT setting committed burst size in kbyte
--set-pir INT setting peak information rate in kbits per
second
--set-ebs INT setting excess burst size in kbyte
--set-conform-action-dscp INT setting dscp value if the
action is accept for conforming packets
--set-exceed-action-dscp INT setting dscp value if the action
is accept for exceeding packets
--set-violate-action STRING setting the action (accept/drop)
for violating packets
--set-violate-action-dscp INT setting dscp value if the
action is accept for violating packets
Supported chains for the filter table:
INPUT FORWARD OUTPUT

[Link] 157
Cumulus Linux 3.5 User Guide

iptables/ip6tables Rule Support

Rule Supported Unsupported

Element

Matches Src/Dst, IP protocol Rules with input/output

Ethernet interfaces are
In/out interface
ignored
IPv4: icmp, ttl,
Inverse matches
IPv6: icmp6, frag, hl,
IP common: tcp (with flags (see page 164)),
udp, multiport,TOS, DSCP, addrtype

Standard ACCEPT, DROP RETURN, QUEUE, STOP, Fall

Targets Thru, Jump

Extended LOG (IPv4/IPv6); UID is not supported for  

Targets LOG
TCP SEQ, TCP options or IP options
ULOG
SETQOS
DSCP
Unique to Cumulus Linux:
SPAN
ERSPAN (IPv4/IPv6)
POLICE
TRICOLORPOLICE
SETCLASS

ebtables Rule Support

Rule Supported Unsupported

Element

Matches ether type Inverse matches

input interface/wildcard Proto length
output interface/wildcard VLAN
src/dst MAC
IP: src, dest, tos, proto, sport, dport

158 23 January 2018

 Cumulus Networks

Rule Supported Unsupported

Element

IPv6: tclass, icmp6: type, icmp6: code range, src

/dst addr, sport, dport

802.1p (CoS)

Standard ACCEPT, DROP Return, Continue,

Targets Jump, Fall Thru

Extended Ulog  
Targets
log
Unique to Cumulus Linux:
span
erspan
police
tricolorpolice
setclass

Other Unsupported Rules

Rules that have no matches and accept all packets in a chain are currently ignored. This probably
has side effects in the sense that the rules below them do get hit, when normally they wouldn't.
Chain default rules (which are ACCEPT) are also ignored.

Common Examples

Policing Control Plane and Data Plane Traffic

You can configure quality of service for traffic on both the control plane and the data plane. By using QoS
policers, you can rate limit traffic so incoming packets get dropped if they exceed specified thresholds.

Counters on POLICE ACL rules in iptables do not currently show the packets that are dropped
due to those rules.

Use the POLICE target with iptables. POLICE takes these arguments:
--set-class value: Sets the system internal class of service queue configuration to value.
--set-rate value: Specifies the maximum rate in kilobytes (KB) or packets.
--set-burst value: Specifies the number of packets or kilobytes (KB) allowed to arrive
sequentially.
--set-mode string: Sets the mode in KB (kilobytes) or pkt (packets) for rate and burst size.

[Link] 159
Cumulus Linux 3.5 User Guide

For example, to rate limit the incoming traffic on swp1 to 400 packets/second with a burst of 100 packets
/second and set the class of the queue for the policed traffic as 0, set this rule in your appropriate .rules
file:

-A INPUT --in-interface swp1 -j POLICE --set-mode pkt --set-rate 400

--set-burst 100 --set-class 0

Here is another example of control plane ACL rules to lock down the switch. You specify them in /etc
/cumulus/acl/policy.d/00control_plane.rules:
View the contents of the file ...

INGRESS_INTF = swp+
INGRESS_CHAIN = INPUT
INNFWD_CHAIN = INPUT,FORWARD
MARTIAN_SOURCES_4 = "[Link]/5,[Link]/8,[Link]/8,
[Link]/32"
MARTIAN_SOURCES_6 = "ff00::/8,::/128,::ffff:[Link]/96,::1/128"
# Custom Policy Section
SSH_SOURCES_4 = "[Link]/24"
NTP_SERVERS_4 = "[Link]/32,[Link]/32"
DNS_SERVERS_4 = "[Link]/32,[Link]/32"
SNMP_SERVERS_4 = "[Link]/32"
[iptables]
-A $INNFWD_CHAIN --in-interface $INGRESS_INTF -s $MARTIAN_SOURCES_4 -
j DROP
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p ospf -j POLICE --
set-mode pkt --set-rate 2000 --set-burst 2000 --set-class 7
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p tcp --dport bgp -j
POLICE --set-mode pkt --set-rate 2000 --set-burst 2000 --set-class 7
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p tcp --sport bgp -j
POLICE --set-mode pkt --set-rate 2000 --set-burst 2000 --set-class 7
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p icmp -j POLICE --
set-mode pkt --set-rate 100 --set-burst 40 --set-class 2
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p udp --dport bootps:
bootpc -j POLICE --set-mode pkt --set-rate 100 --set-burst 100 --set-
class 2
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p tcp --dport bootps:
bootpc -j POLICE --set-mode pkt --set-rate 100 --set-burst 100 --set-
class 2
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p igmp -j POLICE --
set-mode pkt --set-rate 300 --set-burst 100 --set-class 6
# Custom policy
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p tcp --dport 22 -s
$SSH_SOURCES_4 -j ACCEPT
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p udp --sport 123 -s
$NTP_SERVERS_4 -j ACCEPT
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p udp --sport 53 -s
$DNS_SERVERS_4 -j ACCEPT

160 23 January 2018

 Cumulus Networks

-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p udp --dport 161 -s

$SNMP_SERVERS_4 -j ACCEPT
# Allow UDP traceroute when we are the current TTL expired hop
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -p udp --dport 1024:
65535 -m ttl --ttl-eq 1 -j ACCEPT
-A $INGRESS_CHAIN --in-interface $INGRESS_INTF -j DROP

Setting DSCP on Transit Traffic

The examples here use the mangle table to modify the packet as it transits the switch. DSCP is expressed in
decimal notation in the examples below.

[iptables]

#Set SSH as high priority traffic.

-t mangle -A FORWARD -p tcp --dport 22 -j DSCP --set-dscp 46

#Set everything coming in SWP1 as AF13

-t mangle -A FORWARD --in-interface swp1 -j DSCP --set-dscp 14

#Set Packets destined for [Link] as best effort

-t mangle -A FORWARD -d [Link]/32 -j DSCP --set-dscp 0

#Example using a range of ports for TCP traffic

-t mangle -A FORWARD -p tcp -s [Link]/32 --sport 10000:20000 -d
[Link]/32 --dport 10000:20000 -j DSCP --set-dscp 34

Verifying DSCP Values on Transit Traffic

The examples here use the DSCP match criteria in combination with other IP, TCP and interface matches to
identify traffic and count the number of packets.

[iptables]

#Match and count the packets that match SSH traffic with DSCP EF
-A FORWARD -p tcp --dport 22 -m dscp --dscp 46 -j ACCEPT

#Match and count the packets coming in SWP1 as AF13

-A FORWARD --in-interface swp1 -m dscp --dscp 14 -j ACCEPT
#Match and count the packets with a destination [Link] marked best
effort
-A FORWARD -d [Link]/32 -m dscp --dscp 0 -j ACCEPT

#Match and count the packets in a port range with DSCP AF41
-A FORWARD -p tcp -s [Link]/32 --sport 10000:20000 -d [Link]
/32 --dport 10000:20000 -m dscp --dscp 34 -j ACCEPT

[Link] 161
Cumulus Linux 3.5 User Guide

Checking the Packet and Byte Counters for ACL Rules

To verify the counters, using the above example rules, first send test traffic matching the patterns through
the network. The following example generates traffic with mz, which can be installed on host servers or
even on Cumulus Linux switches. Once traffic is sent to validate the counters, they are matched on switch1
use cl-acltool.

# Send 100 TCP packets on host1 with a DSCP value of EF with a

destination of host2 TCP port 22:

cumulus@host1$ mz eth1 -A [Link] -B [Link] -c 100 -v -t tcp

"dp=22,dscp=46"
IP: ver=4, len=40, tos=184, id=0, frag=0, ttl=255, proto=6, sum=0,
SA=[Link], DA=[Link],
payload=[see next layer]
TCP: sp=0, dp=22, S=42, A=42, flags=0, win=10000, len=20, sum=0,
payload=

# Verify the 100 packets are matched on switch1

cumulus@switch1$ sudo cl-acltool -L ip

-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 9314 packets, 753K bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
100 6400 ACCEPT tcp -- any any anywhere
anywhere tcp dpt:ssh DSCP match 0x2e
0 0 ACCEPT all -- swp1 any anywhere
anywhere DSCP match 0x0e
0 0 ACCEPT all -- any any [Link]
anywhere DSCP match 0x00
0 0 ACCEPT tcp -- any any [Link]
[Link] tcp spts:webmin:20000 dpts:webmin:2002

# Send 100 packets with a small payload on host1 with a DSCP value of
AF13 with a destination of host2:

cumulus@host1$ mz eth1 -A [Link] -B [Link] -c 100 -v -t ip

IP: ver=4, len=20, tos=0, id=0, frag=0, ttl=255, proto=0, sum=0,
SA=[Link], DA=[Link],
payload=

# Verify the 100 packets are matched on switch1

162 23 January 2018

 Cumulus Networks

cumulus@switch1$ sudo cl-acltool -L ip

-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 9314 packets, 753K bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
100 6400 ACCEPT tcp -- any any anywhere
anywhere tcp dpt:ssh DSCP match 0x2e
100 7000 ACCEPT all -- swp3 any anywhere
anywhere DSCP match 0x0e
100 6400 ACCEPT all -- any any [Link]
anywhere DSCP match 0x00
0 0 ACCEPT tcp -- any any [Link]
[Link] tcp spts:webmin:20000 dpts:webmin:2002

# Send 100 packets on host1 with a destination of host2:

cumulus@host1$ mz eth1 -A [Link] -B [Link] -c 100 -v -t ip

IP: ver=4, len=20, tos=56, id=0, frag=0, ttl=255, proto=0, sum=0,
SA=[Link], DA=[Link],
payload=

# Verify the 100 packets are matched on switch1

cumulus@switch1$ sudo cl-acltool -L ip

-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 9314 packets, 753K bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
100 6400 ACCEPT tcp -- any any anywhere
anywhere tcp dpt:ssh DSCP match 0x2e
100 7000 ACCEPT all -- swp3 any anywhere
anywhere DSCP match 0x0e
0 0 ACCEPT all -- any any [Link]
anywhere DSCP match 0x00
0 0 ACCEPT tcp -- any any [Link]
[Link] tcp spts:webmin:20000 dpts:webmin:2002Still
working

[Link] 163
Cumulus Linux 3.5 User Guide

Filtering Specific TCP Flags

The example solution below creates rules on the INPUT and FORWARD chains to drop ingress IPv4 and IPv6
TCP packets when the SYN bit is set and the RST, ACK and FIN bits are reset. The default for the INPUT and
FORWARD chains allows all other packets. The ACL is applied to ports swp20 and swp21. After configuring
this ACL, new TCP sessions that originate from ingress ports swp20 and swp21 will not be allowed. TCP
sessions that originate from any other port are allowed.

INGRESS_INTF = swp20,swp21

[iptables]
-A INPUT,FORWARD --in-interface $INGRESS_INTF -p tcp --syn -j DROP
[ip6tables]
-A INPUT,FORWARD --in-interface $INGRESS_INTF -p tcp --syn -j DROP

The --syn flag in the above rule matches packets with the SYN bit set and the ACK, RST and FIN bits are
cleared. It is equivalent to using -tcp-flags SYN,RST,ACK,FIN SYN. For example, the above rule could
be re-written as:

-A INPUT,FORWARD --in-interface $INGRESS_INTF -p tcp --tcp-flags SYN,

RST,ACK,FIN SYN -j DROP

Example Scenario
The following example scenario demonstrates where several different rules are applied to show what is
possible.

164 23 January 2018

 Cumulus Networks

Following are the configurations for the two switches used in these examples. The configuration for each
switch appears in /etc/network/interfaces on that switch.

Switch 1 Configuration

cumulus@switch1:~$ net show configuration files

...

/etc/network/interfaces
=======================

auto swp1
iface swp1

auto swp2
iface swp2

auto swp3
iface swp3

auto swp4
iface swp4

auto bond2

[Link] 165
Cumulus Linux 3.5 User Guide

iface bond2
bond-slaves swp3 swp4

auto br-untagged
iface br-untagged
address [Link]/24
bridge_ports swp1 bond2
bridge_stp on

auto br-tag100
iface br-tag100
address [Link]/24
bridge_ports swp2.100 bond2.100
bridge_stp on

...

Switch 2 Configuration

cumulus@switch2:~$ net show configuration files

...

/etc/network/interfaces
=======================

auto swp3
iface swp3

auto swp4
iface swp4

auto br-untagged
iface br-untagged
address [Link]/24
bridge_ports bond2
bridge_stp on

auto br-tag100
iface br-tag100
address [Link]/24
bridge_ports bond2.100
bridge_stp on

auto bond2
iface bond2
bond-slaves swp3 swp4

...

166 23 January 2018

 Cumulus Networks

Egress Rule
The following rule blocks any TCP with destination port 200 traffic going from host1 or host2 through the
switch (corresponding to rule 1 in the diagram above).

[iptables] -A FORWARD -o bond2 -p tcp --dport 200 -j DROP

Ingress Rule
The following rule blocks any UDP traffic with source port 200 going from host1 through the switch
(corresponding to rule 2 in the diagram above).

[iptables] -A FORWARD -i swp2 -p udp --sport 200 -j DROP

Input Rule
The following rule blocks any UDP traffic with source port 200 and destination port 50 going from host1 to
the switch (corresponding to rule 3 in the diagram above).

[iptables] -A INPUT -i swp1 -p udp --sport 200 --dport 50 -j DROP

Output Rule
The following rule blocks any TCP traffic with source port 123 and destination port 123 going from Switch 1
to host2 (corresponding to rule 4 in the diagram above).

[iptables] -A OUTPUT -o br-tag100 -p tcp --sport 123 --dport 123 -j

DROP

Combined Rules
The following rule blocks any TCP traffic with source port 123 and destination port 123 going from any
switch port egress or generated from Switch 1 to host1 or host2 (corresponding to rules 1 and 4 in the
diagram above).

[iptables] -A OUTPUT,FORWARD -o swp+ -p tcp --sport 123 --dport 123 -

j DROP

This also becomes 2 ACLs, and is effectively the same as:

[Link] 167
Cumulus Linux 3.5 User Guide

[iptables]
-A FORWARD -o swp+ -p tcp --sport 123 --dport 123 -j DROP
-A OUTPUT -o swp+ -p tcp --sport 123 --dport 123 -j DROP

Layer 2-only Rules/ebtables

The following rule blocks any traffic with source MAC address [Link] and destination MAC
address [Link] going from any switch port egress/ingress.

[ebtables] -A FORWARD -s [Link] -d [Link] -j

DROP

Useful Links
[Link]
[Link] packet filtering how-to

Caveats and Errata

Not All Rules Supported

As mentioned in the Supported Rules section (see page 157) above, not all iptables, ip6tables or
ebtables rules are supported. Refer to that section for specific rule support.

ACL Log Policer Limits Traffic

Traffic copied to the CPU is limited to 1 pkt/s by an ACL Log Policer, to protect the CPU from overloading.

Bridge Traffic Limitations

Bridge traffic that matches LOG ACTION rules are not logged in syslog, as the kernel and hardware identify
packets using different information.

Log Actions Cannot Be Forwarded

Logged packets cannot be forwarded. The hardware cannot both forward a packet and send the packet to
the control plane (or kernel) for logging. To emphasize this, a log action must also have a drop action.

Broadcom Range Checker Limitations

Broadcom platforms have only 24 range checkers. This is a separate resource from the total number of
ACLs allowed. If you are creating a large ACL configuration, use port ranges for large ranges of more than 5
ports.

168 23 January 2018

 Cumulus Networks

Inbound LOG Actions Only for Broadcom Switches

On Broadcom-based switches, LOG actions can only be done on inbound interfaces (the ingress direction),
not on outbound interfaces (the egress direction).

Tomahawk Hardware Limitations

Rate Limiting per Pipeline, Not Global

On Tomahawk switches, the field processor (FP) polices on a per-pipeline basis instead of globally, as with a
Trident II switch. If packets come in to different switch ports that are on different pipelines on the ASIC, they
may be rate limited differently.
For example, your switch is set so BFD if rate limited to 2000 packets per second. When the BFD packets
are received on port1/pipe1 and port2/pipe2, they would each be rate limited at 2000 pps, thus the switch
is rate limiting at 4000 pps overall. Since there are four pipelines on a Tomahawk switch, you could see a 4
increase of your configured rate limits.

Atomic Update Mode Enabled by Default

In Cumulus Linux, atomic update mode is enabled by default. If you have Tomahawk switches and plan to
use SPAN and/or mangle rules, you must disable atomic update mode.
To do so, enable non-atomic update mode by setting the value for acl.non_atomic_update_mode to
true in /etc/cumulus/[Link]:

acl.non_atomic_update_mode = TRUE

Then restart switchd.

Packets Undercounted during ACL Updates

On Tomahawk switches, when updating egress FP rules, some packets do not get counted. This results in
an underreporting of counts during ping-pong/incremental switchover.

Trident II+ Hardware Limitations

On a Trident II+ switch, the TCAM allocation for ACLs is limited to 2048 rules in atomic mode for a default
setup, instead of 4096 as advertised for ingress rules.

iptables Interactions with cl-acltool

Since Cumulus Linux is a Linux operating system, the iptables commands can be used directly and does
work. However, you should consider using cl-acltool instead because:
Without using cl-acltool, rules are not installed into hardware.
Running cl-acltool -i (the installation command) resets all rules and deletes anything that is
not stored in /etc/cumulus/acl/[Link].
For example, running the following command works:

[Link] 169
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo iptables -A INPUT -p icmp --icmp-type

echo-request -j DROP

And the rules appear when you run cl-acltool -L:

cumulus@switch:~$ sudo cl-acltool -L ip

-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 72 packets, 5236 bytes)
pkts bytes target prot opt in out source destination

0 0 DROP icmp -- any any anywhere anywhere

icmp echo-request

However, running cl-acltool -i or reboot removes them. To ensure all rules that can be in
hardware are hardware accelerated, place them in /etc/cumulus/acl/[Link] and run
cl-acltool -i.

Where to Assign Rules

If a switch port is assigned to a bond, any egress rules must be assigned to the bond.
When using the OUTPUT chain, rules must be assigned to the source. For example, if a rule is
assigned to the switch port in the direction of traffic but the source is a bridge (VLAN), the traffic won’
t be affected by the rule and must be applied to the bridge.
If all transit traffic needs to have a rule applied, use the FORWARD chain, not the OUTPUT chain.

Generic Error Message Displayed after ACL Rule Installation Failure

After an ACL rule installation failure, a generic error message like the following is displayed:

cumulus@switch:$ sudo cl-acltool -i -p 00control_plane.rules

Using user provided rule file 00control_plane.rules
Reading rule file 00control_plane.rules ...
Processing rules in file 00control_plane.rules ...
error: hw sync failed (sync_acl hardware installation failed)
Installing acl policy... Rolling back ..
failed.

Dell S3048-ON Supports only 24K MAC Addresses

The Dell S3048-ON has a limit of 24576 MAC address entries, instead of 32K for other 1G switches.

170 23 January 2018

 Cumulus Networks

Default Cumulus Linux ACL Configuration

The Cumulus Linux default ACL configuration is split into three parts, as outlined in the netfilter ACL
documentation (see page 137): IP tables, IPv6 tables, and EB tables. The sections below cover the default
configurations for each part, while the default file can be seen by clicking the Default ACL Configuration link:
Default ACL Configuration

cumulus@switch:~$ sudo cl-acltool -L all

-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 167 packets, 16481 bytes)
pkts bytes target prot opt in out source
destination
0 0 DROP all -- swp+ any [Link]/5
anywhere
0 0 DROP all -- swp+ any loopback/8
anywhere
0 0 DROP all -- swp+ any [Link]
/8 anywhere
0 0 DROP all -- swp+ any [Link]
anywhere
0 0 SETCLASS udp -- swp+ any anywhere
anywhere udp dpt:3785 SETCLASS class:7
0 0 POLICE udp -- any any anywhere
anywhere udp dpt:3785 POLICE mode:pkt rate:2000 burst:
2000
0 0 SETCLASS udp -- swp+ any anywhere
anywhere udp dpt:3784 SETCLASS class:7
0 0 POLICE udp -- any any anywhere
anywhere udp dpt:3784 POLICE mode:pkt rate:2000 burst:
2000
0 0 SETCLASS udp -- swp+ any anywhere
anywhere udp dpt:4784 SETCLASS class:7
0 0 POLICE udp -- any any anywhere
anywhere udp dpt:4784 POLICE mode:pkt rate:2000 burst:
2000
0 0 SETCLASS ospf -- swp+ any anywhere
anywhere SETCLASS class:7
0 0 POLICE ospf -- any any anywhere
anywhere POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS tcp -- swp+ any anywhere
anywhere tcp dpt:bgp SETCLASS class:7
0 0 POLICE tcp -- any any anywhere
anywhere tcp dpt:bgp POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS tcp -- swp+ any anywhere
anywhere tcp spt:bgp SETCLASS class:7

[Link] 171
Cumulus Linux 3.5 User Guide

0 0 POLICE tcp -- any any anywhere

anywhere tcp spt:bgp POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS tcp -- swp+ any anywhere
anywhere tcp dpt:5342 SETCLASS class:7
0 0 POLICE tcp -- any any anywhere
anywhere tcp dpt:5342 POLICE mode:pkt rate:2000 burst:
2000
0 0 SETCLASS tcp -- swp+ any anywhere
anywhere tcp spt:5342 SETCLASS class:7
0 0 POLICE tcp -- any any anywhere
anywhere tcp spt:5342 POLICE mode:pkt rate:2000 burst:
2000
0 0 SETCLASS icmp -- swp+ any anywhere
anywhere SETCLASS class:2
1 84 POLICE icmp -- any any anywhere
anywhere POLICE mode:pkt rate:100 burst:40
0 0 SETCLASS udp -- swp+ any anywhere
anywhere udp dpts:bootps:bootpc SETCLASS class:2
0 0 POLICE udp -- any any anywhere
anywhere udp dpt:bootps POLICE mode:pkt rate:100 burst:
100
0 0 POLICE udp -- any any anywhere
anywhere udp dpt:bootpc POLICE mode:pkt rate:100 burst:
100
0 0 SETCLASS tcp -- swp+ any anywhere
anywhere tcp dpts:bootps:bootpc SETCLASS class:2
0 0 POLICE tcp -- any any anywhere
anywhere tcp dpt:bootps POLICE mode:pkt rate:100 burst:
100
0 0 POLICE tcp -- any any anywhere
anywhere tcp dpt:bootpc POLICE mode:pkt rate:100 burst:
100
0 0 SETCLASS udp -- swp+ any anywhere
anywhere udp dpt:10001 SETCLASS class:3
0 0 POLICE udp -- any any anywhere
anywhere udp dpt:10001 POLICE mode:pkt rate:2000 burst:
2000
0 0 SETCLASS igmp -- swp+ any anywhere
anywhere SETCLASS class:6
1 32 POLICE igmp -- any any anywhere
anywhere POLICE mode:pkt rate:300 burst:100
0 0 POLICE all -- swp+ any anywhere
anywhere ADDRTYPE match dst-type LOCAL POLICE mode:pkt
rate:1000 burst:1000 class:0
0 0 POLICE all -- swp+ any anywhere
anywhere ADDRTYPE match dst-type IPROUTER POLICE mode:
pkt rate:400 burst:100 class:0
0 0 SETCLASS all -- swp+ any anywhere
anywhere SETCLASS class:0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination

172 23 January 2018

 Cumulus Networks

0 0 DROP all -- swp+ any [Link]/5

anywhere
0 0 DROP all -- swp+ any loopback/8
anywhere
0 0 DROP all -- swp+ any [Link]
/8 anywhere
0 0 DROP all -- swp+ any [Link]
anywhere
Chain OUTPUT (policy ACCEPT 107 packets, 12590 bytes)
pkts bytes target prot opt in out source
destination
TABLE mangle :
Chain PREROUTING (policy ACCEPT 172 packets, 17871 bytes)
pkts bytes target prot opt in out source
destination
Chain INPUT (policy ACCEPT 172 packets, 17871 bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 111 packets, 18134 bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 111 packets, 18134 bytes)
pkts bytes target prot opt in out source
destination
TABLE raw :
Chain PREROUTING (policy ACCEPT 173 packets, 17923 bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 112 packets, 18978 bytes)
pkts bytes target prot opt in out source
destination
--------------------------------
Listing rules of type ip6tables:
--------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
0 0 DROP all swp+ any ip6-mcastprefix/8
anywhere
0 0 DROP all swp+ any ::/128
anywhere
0 0 DROP all swp+ any ::ffff:[Link]/96
anywhere
0 0 DROP all swp+ any localhost/128
anywhere
0 0 POLICE udp swp+ any anywhere
anywhere udp dpt:3785 POLICE mode:pkt rate:2000 burst:
2000 class:7

[Link] 173
Cumulus Linux 3.5 User Guide

0 0 POLICE udp swp+ any anywhere

anywhere udp dpt:3784 POLICE mode:pkt rate:2000 burst:
2000 class:7
0 0 POLICE udp swp+ any anywhere
anywhere udp dpt:4784 POLICE mode:pkt rate:2000 burst:
2000 class:7
0 0 POLICE ospf swp+ any anywhere
anywhere POLICE mode:pkt rate:2000 burst:2000 class:7
0 0 POLICE tcp swp+ any anywhere
anywhere tcp dpt:bgp POLICE mode:pkt rate:2000 burst:
2000 class:7
0 0 POLICE tcp swp+ any anywhere
anywhere tcp spt:bgp POLICE mode:pkt rate:2000 burst:
2000 class:7
0 0 POLICE ipv6-icmp swp+ any
anywhere anywhere ipv6-icmp router-
solicitation POLICE mode:pkt rate:100 burst:100 class:2
0 0 POLICE ipv6-icmp swp+ any
anywhere anywhere ipv6-icmp router-
advertisement POLICE mode:pkt rate:500 burst:500 class:2
0 0 POLICE ipv6-icmp swp+ any
anywhere anywhere ipv6-icmp neighbour-
solicitation POLICE mode:pkt rate:400 burst:400 class:2
0 0 POLICE ipv6-icmp swp+ any
anywhere anywhere ipv6-icmp neighbour-
advertisement POLICE mode:pkt rate:400 burst:400 class:2
0 0 POLICE ipv6-icmp swp+ any
anywhere anywhere ipv6-icmptype 130 POLICE
mode:pkt rate:200 burst:100 class:6
0 0 POLICE ipv6-icmp swp+ any
anywhere anywhere ipv6-icmptype 131 POLICE
mode:pkt rate:200 burst:100 class:6
0 0 POLICE ipv6-icmp swp+ any
anywhere anywhere ipv6-icmptype 132 POLICE
mode:pkt rate:200 burst:100 class:6
0 0 POLICE ipv6-icmp swp+ any
anywhere anywhere ipv6-icmptype 143 POLICE
mode:pkt rate:200 burst:100 class:6
0 0 POLICE ipv6-icmp swp+ any
anywhere anywhere POLICE mode:pkt rate:64
burst:40 class:2
0 0 POLICE udp swp+ any anywhere
anywhere udp dpts:dhcpv6-client:dhcpv6-server POLICE
mode:pkt rate:100 burst:100 class:2
0 0 POLICE tcp swp+ any anywhere
anywhere tcp dpts:dhcpv6-client:dhcpv6-server POLICE
mode:pkt rate:100 burst:100 class:2
0 0 POLICE all swp+ any anywhere
anywhere ADDRTYPE match dst-type LOCAL POLICE mode:pkt
rate:1000 burst:1000 class:0

174 23 January 2018

 Cumulus Networks

0 0 POLICE all swp+ any anywhere

anywhere ADDRTYPE match dst-type IPROUTER POLICE mode:
pkt rate:400 burst:100 class:0
0 0 SETCLASS all swp+ any anywhere
anywhere SETCLASS class:0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
0 0 DROP all swp+ any ip6-mcastprefix/8
anywhere
0 0 DROP all swp+ any ::/128
anywhere
0 0 DROP all swp+ any ::ffff:[Link]/96
anywhere
0 0 DROP all swp+ any localhost/128
anywhere
Chain OUTPUT (policy ACCEPT 5 packets, 408 bytes)
pkts bytes target prot opt in out source
destination
TABLE mangle :
Chain PREROUTING (policy ACCEPT 7 packets, 718 bytes)
pkts bytes target prot opt in out source
destination
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
TABLE raw :
Chain PREROUTING (policy ACCEPT 7 packets, 718 bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
-------------------------------
Listing rules of type ebtables:
-------------------------------
TABLE filter :
Bridge table: filter
Bridge chain: INPUT, entries: 16, policy: ACCEPT
-d BGA -i swp+ -j setclass --class 7 , pcnt = 0 -- bcnt = 0
-d BGA -j police --set-mode pkt --set-rate 2000 --set-burst 2000 ,
pcnt = 0 -- bcnt = 0
-d [Link] -i swp+ -j setclass --class 7 , pcnt = 0 -- bcnt = 0

[Link] 175
Cumulus Linux 3.5 User Guide

-d [Link] -j police --set-mode pkt --set-rate 2000 --set-burst

2000 , pcnt = 0 -- bcnt = 0
-d [Link] -i swp+ -j setclass --class 6 , pcnt = 0 -- bcnt = 0
-d [Link] -j police --set-mode pkt --set-rate 200 --set-burst
200 , pcnt = 0 -- bcnt = 0
-d [Link] -i swp+ -j setclass --class 6 , pcnt = 0 -- bcnt = 0
-d [Link] -j police --set-mode pkt --set-rate 200 --set-burst
200 , pcnt = 0 -- bcnt = 0
-p ARP -i swp+ -j setclass --class 2 , pcnt = 0 -- bcnt = 0
-p ARP -j police --set-mode pkt --set-rate 400 --set-burst 100 , pcnt
= 0 -- bcnt = 0
-d [Link] -i swp+ -j setclass --class 7 , pcnt = 0 -- bcnt = 0
-d [Link] -j police --set-mode pkt --set-rate 2000 --set-
burst 2000 , pcnt = 0 -- bcnt = 0
-p IPv4 -i swp+ -j ACCEPT , pcnt = 0 -- bcnt = 0
-p IPv6 -i swp+ -j ACCEPT , pcnt = 0 -- bcnt = 0
-i swp+ -j setclass --class 0 , pcnt = 0 -- bcnt = 0
-j police --set-mode pkt --set-rate 100 --set-burst 100 , pcnt = 0 --
bcnt = 0
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

IP Tables

Action/Value Protocol/IP Address

Drop Source IPv4:

Destination IP: [Link]/5
Any
loopback/8
[Link]/4
[Link]

Set class: 7 Protocol:

Police: Packet UDP/BFD Echo
rate 2000 burst
UDP/BFD Control
2000
UDP BFD Multihop Control
Source IP: Any
OSPF
Destination IP:
Any TCP/BGP (spt dpt 179)
TCP/MLAG (spt dpt 5342)

Set Class: 6 Protocol:

Police: Rate 300 IGMP
burst 100
Source IP: Any

176 23 January 2018

 Cumulus Networks

Action/Value Protocol/IP Address

Destination IP:
Any

Set class: 2 Protocol:

Police: Rate 100 ICMP
burst 40
Source IP : Any
Destination IP:
Any

Set class: 2 Protocol:

Police: Rate 100 UDP/bootpc, bootps
burst 100
Source IP: Any
Destination IP:
Any

Set class: 3 Protocol:

Police: Rate 2000 UDP/LNV
burst:2000
Source IP: Any
Destination IP:
Any

Set class: 0 ADDRTYPE match dst-type LOCAL

Police: Rate 1000
burst 1000
LOCAL is any local address -> Receiving a packet with a destination
Source IP: Any matching a local IP address on the switch will go to the CPU.
Destination IP:
Any

Set class: 0 ADDRTYPE match dst-type IPROUTER

Police: Rate 400
burst 100
IPROUTER is any unresolved address -> On a l2/l3 boundary receiving a
Source IP: Any packet from L3 and needs to go to CPU in order to ARP for the destination.
Destination IP:
Any

Set class 0 All

Set class is internal to the switch - it does not set any precedence bits.

[Link] 177
Cumulus Linux 3.5 User Guide

IPv6 Tables

Action/Value Protocol/IP Address

Drop Source IPv6:

ff00::/8
::
::ffff:[Link]/96
localhost

Set class: 7 Protocol:

Police: Packet UDP/BFD Echo
rate 2000 burst
UDP/BFD Control
2000
UDP BFD Multihop Control
Source IPv6: Any
OSPF
Destination IPv6:
Any TCP/BGP (spt dpt 179)

Set class: 6 Protocol:

Police: Packet Rte: Multicast Listener Query (MLD)
200 burst 100
Multicast Listener Report (MLD)
Source IPv6: Any
Multicast Listener Done (MLD)
Destination IPv6:
Multicast Listener Report V2
Any

Set class: 2 Protocol:

Police: Packet ipv6-icmp router-solicitation
rate: 100 burst
100
Source IPv6: Any
Destination IPv6:
Any

Set class: 2 Protocol:

Police: Packet ipv6-icmp router-advertisement POLICE
rate: 500 burst
500
Source IPv6: Any
Destination IPv6:
Any

Set class: 2 Protocol:

178 23 January 2018

 Cumulus Networks

Action/Value Protocol/IP Address

Police: Packet ipv6-icmp neighbour-solicitation

rate: 400 burst
ipv6-icmp neighbour-advertisement
400
Source IPv6: Any
Destination IPv6:
Any

Set class: 2 Protocol:

Police: Packet Ipv6 icmp
rate: 64 burst: 40
Source IPv6: Any
Destination IPv6:
Any

Set class: 2 Protocol:

Police: Packet UDP/dhcpv6-client:dhcpv6-server (Spts & dpts)
rate: 100 burst:
100
Source IPv6: Any
Destination IPv6:
Any

Police: Packet ADDRTYPE match dst-type LOCAL

rate: 1000 burst
1000
Source IPv6: Any LOCAL is any local address -> Receiving a packet with a destination
matching a local IPv6 address on the switch will go to the CPU.
Destination IPv6:
Any

Set class: 0 ADDRTYPE match dst-type IPROUTER

Police: Packet
rate: 400 burst
100 IPROUTER is an unresolved address -> On a l2/l3 boundary receiving a
packet from L3 and needs to go to CPU in order to ARP for the destination.

Set class 0 All

Set class is internal to the switch - it does not set any precedence bits.

[Link] 179
Cumulus Linux 3.5 User Guide

EB Tables

Action/Value Protocol/MAC Address

Set Class: 7 BDPU

Police: packet rate: 2000 burst rate:2000 LACP
Any switchport input interface Cisco PVST

Set Class: 6 LLDP

Police: packet rate: 200 burst rate: 200 CDP
Any switchport input inteface

Set Class: 2 ARP

Police: packet rate: 400 burst rate: 100
Any switchport input interface

Catch All: IPv4

Allow all traffic IPv6
Any switchport input interface

Catch All (applied at end): ALL OTHER

Set class: 0
Police: packet rate 100 burst rate 100
Any switchport

Set class is internal to the switch. It does not set any precedence bits.

Filtering Learned MAC Addresses

On Broadcom switches, a MAC address is learned on a bridge regardless of whether or not a received
packet is dropped by an ACL (see page 137). This is due to how the hardware learns MAC addresses and
occurs before the ACL lookup. This can be a security or resource problem as the MAC address table has
the potential to get filled with bogus MAC addresses, so a malfunctioning host, network error, loop or
malicious attack on a shared L2 platform can create an outage for other hosts if the same MAC address is
learned on another port.
To prevent this from happening, Cumulus Linux filters frames before MAC learning occurs. Since the MAC
addresses and their port/VLAN associations are known at configuration time, you can create static MAC
addresses, then create ingress ACLs to whitelist traffic from these MAC addresses and drop traffic
otherwise.

180 23 January 2018

 Cumulus Networks

This feature is specific to switches on the Broadcom platform only; on Mellanox Spectrum
switches, the input port ACL does not have these issues when learning MAC addresses.

Create a configuration similar to the following, where you associate a port and VLAN with a given MAC
address, adding each one to the bridge:

cumulus@switch:~$ net add bridge bridge vids 100,200,300

cumulus@switch:~$ net add bridge bridge pvid 1
cumulus@switch:~$ net add bridge bridge ports swp1-3
cumulus@switch:~$ net add bridge pre-up bridge fdb add [Link]
11 dev swp1 master static vlan 100
cumulus@switch:~$ net add bridge pre-up bridge fdb add [Link]
22 dev swp2 master static vlan 200
cumulus@switch:~$ net add bridge pre-up bridge fdb add [Link]
33 dev swp3 master static vlan 300
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration in the /etc/network/interfaces file:

auto swp1
iface swp1

auto swp2
iface swp2

auto swp3
iface swp3

auto bridge
iface bridge
bridge-ports swp1 swp2 swp3
bridge-pvid 1
bridge-vids 100 200 300
bridge-vlan-aware yes
pre-up bridge fdb add [Link] dev swp1 master static
vlan 100
pre-up bridge fdb add [Link] dev swp2 master static
vlan 200
pre-up bridge fdb add [Link] dev swp3 master static
vlan 300

Alternately, if you need to list too many MAC addresses, you can run a script to create the same
configuration. For example, create a script called [Link] and put in the bridge fdb add commands
for each MAC address you need to configure:

cumulus@switch:~$ cat /etc/networks/[Link]

[Link] 181
Cumulus Linux 3.5 User Guide

#!/bin/bash
bridge fdb add [Link] dev swp1 master static vlan 100
bridge fdb add [Link] dev swp2 master static vlan 200
bridge fdb add [Link] dev swp3 master static vlan 300
bridge fdb add [Link] dev swp4 master static vlan 400
bridge fdb add [Link] dev swp5 master static vlan 500
bridge fdb add [Link] dev swp6 master static vlan 600

Then create the configuration using NCLU (see page 85):

cumulus@switch:~$ net add bridge bridge vids 100,200,300

cumulus@switch:~$ net add bridge bridge pvid 1
cumulus@switch:~$ net add bridge bridge ports swp1-3
cumulus@switch:~$ net add bridge pre-up /etc/networks/[Link]
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration in the /etc/network/interfaces file:

auto swp1
iface swp1

auto swp2
iface swp2

auto swp3
iface swp3

auto swp4
iface swp4

auto swp5
iface swp5

auto swp6
iface swp6

auto bridge
iface bridge
bridge-ports swp1 swp2 swp3 swp4 swp5 swp6
bridge-pvid 1
bridge-vids 100 200 300
bridge-vlan-aware yes
pre-up bridge fdb add [Link] dev swp1 master static
vlan 100
pre-up bridge fdb add [Link] dev swp2 master static
vlan 200

182 23 January 2018

 Cumulus Networks

pre-up bridge fdb add [Link] dev swp3 master static

vlan 300
pre-up bridge fdb add [Link] dev swp4 master static
vlan 400
pre-up bridge fdb add [Link] dev swp5 master static
vlan 500
pre-up bridge fdb add [Link] dev swp6 master static
vlan 600

Interactions with EVPN

If you are using EVPN (see page 465), local static MAC addresses added to the local FDB are exported as
static MAC addresses to remote switches. Remote MAC addresses are added as MAC addresses to the
remote FDB.

Managing Application Daemons

You manage application daemons (services) in Cumulus Linux in the following ways:
Identifying active listener ports
Identifying daemons currently active or stopped
Identifying boot time state of a specific daemon
Disabling or enabling a specific daemon

Contents
This chapter covers ...
Using systemd and the systemctl Command (see page 183)
Understanding the systemctl Subcommands (see page 184)
Ensuring a Service Starts after Multiple Restarts (see page 184)
Keeping systemd Services from Hanging after Starting (see page 185)
Identifying Active Listener Ports for IPv4 and IPv6 (see page 185)
Identifying Daemons Currently Active or Stopped (see page 186)
Identifying Essential Services (see page 190)

Using systemd and the systemctl Command

In general, you manage services using systemd via the systemctl command. You use it with any service
on the switch to start/stop/restart/reload/enable/disable/reenable or get the status of the service.

cumulus@switch:~$ sudo systemctl start | stop | restart | status |

reload | enable | disable | reenable [Link]

For example to restart networking, run the command:

[Link] 183
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo systemctl restart [Link]

Unlike the service command in Debian Wheezy, the service name is written after the systemctl
subcommand, not before it.

Understanding the systemctl Subcommands

systemctl has a number of subcommands that perform a specific operation on a given daemon.
status: Returns the status of the specified daemon.
start: Starts the daemon.
stop: Stops the daemon.
restart: Stops, then starts the daemon, all the while maintaining state. So if there are dependent
services or services that mark the restarted service as Required, the other services also get
restarted. For example, running systemctl restart [Link] restarts any of the routing
protocol daemons that are enabled and running, such as bgpd or ospfd.
reload: Reloads a daemon's configuration.
enable: Enables the daemon to start when the system boots, but does not start it unless you use
the systemctl start [Link] command or reboot the switch.
disable: Disables the daemon, but does not stop it unless you use the systemctl stop
[Link] command or reboot the switch. A disabled daemon can still be started or
stopped.
reenable: Disables, then enables a daemon. You might need to do this so that any new Wants or
WantedBy lines create the symlinks necessary for ordering. This has no side effects on other
daemons.

Ensuring a Service Starts after Multiple Restarts

By default, systemd is configured to try to restart a particular service only a certain number of times within
a given interval before the service fails to start at all. The settings for this are stored in the service script.
The settings are StartLimitInterval (which defaults to 10 seconds) and StartBurstLimit (which defaults to 5
attempts), but many services override these defaults, sometimes with much longer times. switchd.
service, for example, sets StartLimitInterval=10m and StartBurstLimit=3, which means if you restart switchd
more than 3 times in 10 minutes, it will not start.
When the restart fails for this reason, a message similar to the following appears:

Job for [Link] failed. See 'systemctl status switchd.

service' and 'journalctl -xn' for details.

And systemctl status [Link] shows output similar to:

Active: failed (Result: start-limit) since Thu 2016-04-07 [Link]

UTC; 15s ago

184 23 January 2018

 Cumulus Networks

To clear this error, run systemctl reset-failed [Link]. If you know you are going to
restart frequently (multiple times within the StartLimitInterval), you can run the same command before you
issue the restart request. This also applies to stop followed by start.

Keeping systemd Services from Hanging after Starting

If you start, restart or reload any systemd service that could be started from another systemd service, you
must use the --no-block option with systemctl. Otherwise, that service or even the switch itself may
hang after starting or restarting.

Identifying Active Listener Ports for IPv4 and IPv6

You can identify the active listener ports under both IPv4 and IPv6 using the netstat command:

cumulus@switch:~$ sudo netstat -nlp --inet --inet6

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 [Link]:53 [Link]:*
LISTEN 444/dnsmasq
tcp 0 0 [Link]:22 [Link]:*
LISTEN 874/sshd
tcp6 0 0 :::53 :::*
LISTEN 444/dnsmasq
tcp6 0 0 :::22 :::*
LISTEN 874/sshd
udp 0 0 [Link]:28450 [Link]:
* 839/dhclient
udp 0 0 [Link]:53 [Link]:
* 444/dnsmasq
udp 0 0 [Link]:68 [Link]:
* 839/dhclient
udp 0 0 [Link]:123 [Link]:
* 907/ntpd
udp 0 0 [Link]:123 [Link]:
* 907/ntpd
udp 0 0 [Link]:123 [Link]:
* 907/ntpd
udp 0 0 [Link]:4784 [Link]:
* 909/ptmd
udp 0 0 [Link]:3784 [Link]:
* 909/ptmd
udp 0 0 [Link]:3785 [Link]:
* 909/ptmd
udp6 0 0 :::58352 :::
* 839/dhclient
udp6 0 0 :::53 :::
* 444/dnsmasq
udp6 0 0 fe80::a200:ff:fe00::123 :::
* 907/ntpd

[Link] 185
Cumulus Linux 3.5 User Guide

udp6 0 0 [Link] :::

* 907/ntpd
udp6 0 0 :::123 :::
* 907/ntpd
udp6 0 0 :::4784 :::
* 909/ptmd
udp6 0 0 :::3784 :::
* 909/ptmd

Identifying Daemons Currently Active or Stopped

To determine which daemons are currently active or stopped, run cl-service-summary:

cumulus@switch:~$ cl-service-summary
Service cron enabled active
Service ssh enabled active
Service syslog enabled active
Service arp_refresh enabled active
Service clagd enabled active
Service lldpd enabled active
Service mstpd enabled active
Service poed inactive
Service portwd inactive
Service ptmd enabled active
Service pwmd enabled active
Service smond enabled active
Service switchd enabled active
Service vxrd disabled inactive
Service vxsnd disabled inactive
Service bgpd disabled inactive
Service isisd disabled inactive
Service ospf6d disabled inactive
Service ospfd disabled inactive
Service rdnbrd disabled inactive
Service ripd disabled inactive
Service ripngd disabled inactive
Service zebra disabled inactive

You can also run systemctl list-unit-files --type service to list all services on the switch and
see which ones are enabled:
Click here to see output of this command ...

cumulus@switch:~$ systemctl list-unit-files --type service

UNIT FILE STATE
[Link] enabled
[Link] enabled
[Link] disabled
arp_refresh.service enabled

186 23 January 2018

 Cumulus Networks

[Link] enabled
[email protected] disabled
[Link] enabled
[Link] masked
[Link] masked
[Link] masked
[Link] masked
[Link] masked
[Link] masked
[Link] enabled
[Link] enabled
[Link] disabled
[Link] disabled
[email protected] static
[Link] enabled
[Link] masked
[Link] masked
[Link] static
[Link] static
[Link] enabled
[Link] disabled
[Link] enabled
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] disabled
[Link] static
[Link] disabled
[Link] disabled
[email protected] disabled
[email protected] disabled
[Link] enabled
[Link] disabled
[email protected] disabled
[email protected] disabled
[Link] disabled
[Link] disabled
[Link] enabled
[Link] static
[Link] masked
[Link] static
[email protected] enabled
[Link] static
[Link] masked
[email protected] static
[Link] masked
[Link] enabled

[Link] 187
Cumulus Linux 3.5 User Guide

[email protected] enabled
[Link] enabled
[Link] masked
[Link] masked
[email protected] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] masked
[Link] static
[Link] static
[Link] enabled
[Link] enabled
[Link] enabled
[Link] enabled
[Link] enabled
[Link] static
[Link] enabled
[email protected] static
[Link] disabled
[Link] static
[Link] masked
[Link] masked
[Link] masked
[Link] masked
[Link] masked
[Link] masked
[Link] masked
[Link] enabled
[Link] enabled
[Link] disabled
[Link] enabled
[Link] enabled
[email protected] disabled
[Link] disabled
[Link] enabled
[Link] enabled
[Link] static
[Link] enabled
[Link] enabled
[Link] enabled
[Link] static
[Link] static
[Link] static
[Link] disabled
[Link] masked
[Link] static
[Link] masked
[Link] enabled
[Link] masked
[Link] masked

188 23 January 2018

 Cumulus Networks

[email protected] disabled
[Link] masked
[Link] enabled
[Link] disabled
[email protected] disabled
[Link] disabled
[email protected] disabled
[Link] enabled
[email protected] disabled
[Link] enabled
[Link] masked
[Link] masked
[Link] enabled
[Link] disabled
[Link] static
[Link] enabled
[Link] enabled
[Link] static
[Link] static
[Link] static
[email protected] static
[Link] static
[Link] static
[email protected] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] disabled
[Link] disabled
[email protected] disabled
[Link] static
[Link] static
[Link] static
[Link] disabled
[Link] static
[Link] disabled
[Link] disabled
[Link] static
[Link] static
[Link] disabled
[email protected] static
[Link] static
[Link] static

[Link] 189
Cumulus Linux 3.5 User Guide

[Link] static
[Link] static
[Link] static
[Link] disabled
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] static
[Link] masked
[Link] masked
[Link] masked
[Link] enabled
[Link] static
[email protected] static
[Link] static
[Link] enabled
[Link] enabled
[Link] enabled
[Link] disabled
[Link] disabled
wd_keepalive.service enabled
[Link] masked
[Link] enabled
[Link] disabled
191 unit files listed.
lines 147-194/194 (END)

Identifying Essential Services

If you need to know which services are required to run when the switch boots, run:

cumulus@switch:~$ sudo systemctl list-dependencies --before basic.

target

To see which services are needed for networking, run:

cumulus@switch:~$ sudo systemctl list-dependencies --after network.

target
[Link]

190 23 January 2018

 Cumulus Networks

[Link]
[Link]
wd_keepalive.service
[Link]

To identify the services needed for a multi-user environment, run:

cumulus@leaf01:~$ sudo systemctl list-dependencies --before multi-user.

target
[Link]

[Link]
[Link]
[Link]
[Link]
[Link]
[Link]

Configuring switchd
switchd is the daemon at the heart of Cumulus Linux. It communicates between the switch and Cumulus
Linux, and all the applications running on Cumulus Linux.
The switchd configuration is stored in /etc/cumulus/[Link].

Versions of Cumulus Linux prior to 2.1 stored the switchd configuration at /etc/default
/switchd.

Contents
This chapter covers ...
The switchd File System (see page 191)
Configuring switchd Parameters (see page 193)
Restarting switchd (see page 193)

The switchd File System

switchd also exports a file system, mounted on /cumulus/switchd, that presents all the switchd
configuration options as a series of files arranged in a tree structure. You can see the contents by parsing
the switchd tree; run tree /cumulus/switchd. The output below is for a switch with one switch port
configured:

[Link] 191
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo tree /cumulus/switchd/

/cumulus/switchd/
|-- config
| |-- acl
| | |-- non_atomic_update_mode
| | `-- optimize_hw
| |-- arp
| | `-- next_hops
| |-- buf_util
| | |-- measure_interval
| | `-- poll_interval
| |-- coalesce
| | |-- reducer
| | `-- timeout
| |-- disable_internal_restart
| |-- ignore_non_swps
| |-- interface
| | |-- swp1
| | | `-- storm_control
| | | |-- broadcast
| | | |-- multicast
| | | `-- unknown_unicast
| |-- logging
| |-- route
| | |-- host_max_percent
| | `-- table
| `-- stats
| `-- poll_interval
|-- ctrl
| |-- acl
| |-- hal
| | `-- resync
| |-- logger
| |-- netlink
| | `-- resync
| |-- resync
| `-- sample
| `-- ulog_channel
|-- run
| `-- route_info
| |-- ecmp_nh
| | |-- count
| | |-- max
| | `-- max_per_route
| |-- host
| | |-- count
| | |-- count_v4
| | |-- count_v6
| | `-- max
| |-- mac
| | |-- count
| | `-- max

192 23 January 2018

 Cumulus Networks

| `-- route
| |-- count_0
| |-- count_1
| |-- count_total
| |-- count_v4
| |-- count_v6
| |-- mask_limit
| |-- max_0
| |-- max_1
| `-- max_total
`-- version

Configuring switchd Parameters

You can use cl-cfg to configure many switchd parameters at runtime (like ACLs, interfaces, and route
table utilization), which minimizes disruption to your running switch. However, some options are read only
and cannot be configured at runtime.
For example, to see data related to routes, run:

cumulus@switch:~$ sudo cl-cfg -a switchd | grep route

[Link] = 254
route.host_max_percent = 50
cumulus@cumulus:~$

To modify the configuration, run cl-cfg -w. For example, to set the buffer utilization measurement
interval to 1 minute, run:

cumulus@switch:~$ sudo cl-cfg -w switchd buf_util.measure_interval=1

To verify that the value changed, use grep:

cumulus@switch:~$ cl-cfg -a switchd | grep buf

buf_util.poll_interval = 0
buf_util.measure_interval = 1

You can get some of this information by running cl-resource-query; though you cannot
update the switchd configuration with it.

Restarting switchd
Whenever you modify any switchd hardware configuration file (typically changing any *.conf file that
requires making a change to the switching hardware, like /etc/cumulus/datapath/[Link]),
you must restart switchd for the change to take effect:

[Link] 193
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo systemctl restart [Link]

You do not have to restart the switchd service when you update a network interface
configuration (that is, edit /etc/network/interfaces).

Restarting switchd causes all network ports to reset in addition to resetting the switch hardware
configuration.

Power over Ethernet - PoE

Cumulus Linux supports Power over Ethernet (PoE) and PoE+, so certain Cumulus Linux switches can
supply power from Ethernet switch ports to enabled devices over the Ethernet cables that connect them.
Power over Ethernet (PoE) is capable of powering devices up to 15W, while PoE+ can power devices up to
30W.
The currently supported platform is the Edge-Core AS4610-54P, which supports PoE and PoE+ and
configuration over Ethernet layer 2 LLDP for power negotiation.

Contents
This chapter covers ...
How It Works (see page 194)
About Link State and PoE State (see page 195)
Configuring PoE (see page 195)
poectl Arguments (see page 198)
Troubleshooting PoE and PoE+ (see page )
Verify the Link Is Up (see page 200)
View LLDP Information Using lldpcli (see page 200)
View LLDP Information Using tcpdump (see page 201)
Logging poed Events in syslog (see page 202)

How It Works
PoE functionality is provided by the cumulus-poe package. When a powered device is connected to the
switch via an Ethernet cable:
If the available power is greater than the power required by the connected device, power is supplied
to the switch port, and the device powers on
If available power is less than the power required by the connected device and the switch port's
priority is less than the port priority set on all powered ports, power is not supplied to the port

194 23 January 2018

 Cumulus Networks

If available power is less than the power required by the connected device and the switch port's
priority is greater than the priority of a currently powered port, power is removed from lower
priority port(s) and power is supplied to the port

If the total consumed power exceeds the configured power limit of the power source, low priority
ports are turned off. In the case of a tie, the port with the lower port number gets priority
Power is available as follows:

PSU 1 PSU 2 PoE Power Budget

920W x 750W

x 920W 750W

920W 920W 1650W

The AS4610-54P has an LED on the front panel to indicate PoE status:
Green: The poed daemon is running and no errors are detected
Yellow: One or more errors are detected or the poed daemon is not running

About Link State and PoE State

Link state and PoE state are completely independent of each other. When a link is brought down on a
particular port using ip link <port> down, power on that port is not turned off; however, LLDP
negotiation is not possible.

Configuring PoE
You use the poectl command utility to configure PoE on a switch that supports the feature. You can:
Enable or disable PoE for a given switch port
Set a switch port's PoE priority to one of three values: low, high or critical
The PoE configuration resides in /etc/cumulus/[Link]. The file lists all the switch ports, whether PoE
is enabled for those ports and the priority for each port.
Sample [Link] file ...

[enable]
swp1 = enable
swp2 = enable
swp3 = enable
swp4 = enable
swp5 = enable
swp6 = enable
swp7 = enable
swp8 = enable
swp9 = enable
swp10 = enable
swp11 = enable

[Link] 195
Cumulus Linux 3.5 User Guide

swp12 = enable
swp13 = enable
swp14 = enable
swp15 = enable
swp16 = enable
swp17 = enable
swp18 = enable
swp19 = enable
swp20 = enable
swp21 = enable
swp22 = enable
swp23 = enable
swp24 = enable
swp25 = enable
swp26 = enable
swp27 = enable
swp28 = enable
swp29 = enable
swp30 = enable
swp31 = enable
swp32 = enable
swp33 = enable
swp34 = enable
swp35 = enable
swp36 = enable
swp37 = enable
swp38 = enable
swp39 = enable
swp40 = enable
swp41 = enable
swp42 = enable
swp43 = enable
swp44 = enable
swp45 = enable
swp46 = enable
swp47 = enable
swp48 = enable
[priority]
swp1 = low
swp2 = low
swp3 = low
swp4 = low
swp5 = low
swp6 = low
swp7 = low
swp8 = low
swp9 = low
swp10 = low
swp11 = low
swp12 = low
swp13 = low
swp14 = low

196 23 January 2018

 Cumulus Networks

swp15 = low
swp16 = low
swp17 = low
swp18 = low
swp19 = low
swp20 = low
swp21 = low
swp22 = low
swp23 = low
swp24 = low
swp25 = low
swp26 = low
swp27 = low
swp28 = low
swp29 = low
swp30 = low
swp31 = low
swp32 = low
swp33 = low
swp34 = low
swp35 = low
swp36 = low
swp37 = low
swp38 = low
swp39 = low
swp40 = low
swp41 = low
swp42 = low
swp43 = low
swp44 = low
swp45 = low
swp46 = low
swp47 = low
swp48 = low

By default, PoE and PoE+ are enabled on all Ethernet/1G switch ports, and these ports are set with a low
priority. Switch ports can have low, high or critical priority.
There is no additional configuration for PoE+.
To change the priority for one or more switch ports, run poectl -p swp# [low|high|critical]. For
example:

cumulus@switch:~$ sudo poectl -p swp1-swp5,swp7 high

To disable PoE for one or more ports, run poectl -d [port_numbers]:

cumulus@switch:~$ sudo poectl -d swp1-swp5,swp7

To display PoE information for a set of switch ports, run poectl -i [port_numbers]:

[Link] 197
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo poectl -i swp10-swp13

Port Status Allocated Priority PD type
PD class Voltage Current Power
----- -------------------- ----------- -------- -----------
-------- ------- ------- ---------
swp10 connected negotiating low IEEE802.3at
4 53.5 V 25 mA 3.9 W
swp11 searching n/a low IEEE802.3at
none 0.0 V 0 mA 0.0 W
swp12 connected n/a low IEEE802.3at
2 53.5 V 25 mA 1.4 W
swp13 connected 51.0 W low IEEE802.3at
4 53.6 V 72 mA 3.8 W

The Status can be one of the following:

searching: PoE is enabled but no device has been detected.
disabled: The PoE port has been configured as disabled.
connected: A powered device is connected and receiving power.
power-denied: There is insufficient PoE power available to enable the connected device.
The Allocated column displays how much PoE power has been allocated to the port, which can be one of
the following:
n/a: No device is connected or the connected device does not support LLDP negotiation.
negotiating: An LLDP-capable device is connected and is negotiating for PoE power.
XX.X W: An LLDP-capable device has negotiated for XX.X watts of power (for example, 51.0 watts for
swp13 above).
To see all the PoE information for a switch, run poectl -s:

cumulus@switch:~$ poectl -s
System power:
Total: 730.0 W
Used: 11.0 W
Available: 719.0 W
Connected ports:
swp11, swp24, swp27, swp48

The set commands (priority, enable, disable) either succeed silently or display an error message if the
command fails.

poectl Arguments
The poectl command takes the following arguments:

198 23 January 2018

 Cumulus Networks

Argument Description

-h, --help Show this help message and exit

-i, --port- Returns detailed information for the specified ports. You can specify a range of ports. For
info example:
PORT_LIST -i swp1-swp5,swp10

On an Edge-Core AS4610-54P switch, the voltage reported by the poectl -i

command and measured through a power meter connected to the device varies
by 5V. The current and power readings are correct and no difference is seen for
them.

-a, --all Returns PoE status and detailed information for all ports.

-p, -- Sets priority for the specified ports: low, high, critical.
priority
PORT_LIST
PRIORITY

-d, -- Disables PoE operation on the specified ports.

disable-
ports
PORT_LIST

-e, -- Enables PoE operation on the specified ports.

enable-
ports
PORT_LIST

-s, -- Returns PoE status for the entire switch.

system

-r, --reset Performs a hardware reset on the specified ports. Use this if one or more ports are stuck in
PORT_LIST an error state. This does not reset any configuration settings for the specified ports.

-v, -- Displays version information.

version

-j, --json Displays output in JSON format.

--save Saves the current configuration. The saved configuration is automatically loaded on system
boot.

--load Loads and applies the saved configuration.

[Link] 199
Cumulus Linux 3.5 User Guide

Troubleshooting PoE and PoE+

You can troubleshoot PoE and PoE+ using the following utilities and files:
poectl -s, as described above.
The Cumulus Linux cl-support script, which includes PoE-related output from [Link],
syslog, poectl --diag-info and lldpctl.
lldpcli show neighbors ports <swp> protocol lldp hidden details
tcpdump -v -v -i <swp> ether proto 0x88cc
The contents of the PoE/PoE+ /etc/lldpd.d/[Link] configuration file, as described above.

Verify the Link Is Up

LLDP requires network connectivity, so verify that the link is up.

cumulus@switch:~$ net show interface swp20

Name MAC Speed MTU Mode
-- ------ ----------------- ------- ----- ---------
UP swp20 [Link] 1G 1500 Access/L2

View LLDP Information Using lldpcli

You can run lldpcli to view the LLDP information that has been received on a switch port. For example:

cumulus@switch:~$ sudo lldpcli show neighbors ports swp20 protocol

lldp hidden details
----------------------------------------------------------------------
---------
LLDP neighbors:
----------------------------------------------------------------------
---------
Interface: swp20, via: LLDP, RID: 2, Time: 0 day, [Link]
Chassis:
ChassisID: mac [Link]
SysName: ihm-ubuntu
SysDescr: Ubuntu 14.04.2 LTS Linux 3.14.4+ #1 SMP Thu Jun 26
[Link] UTC 2014 armv7l
MgmtIP: fe80::6ac9:bff:fe25:547c
Capability: Bridge, off
Capability: Router, off
Capability: Wlan, off
Capability: Station, on
Port:
PortID: mac [Link]
PortDescr: eth0
PMD autoneg: supported: yes, enabled: yes
Adv: 10Base-T, HD: yes, FD: yes

200 23 January 2018

 Cumulus Networks

Adv: 100Base-TX, HD: yes, FD: yes

MAU oper type: 100BaseTXFD - 2 pair category 5 UTP, full duplex
mode
MDI Power: supported: yes, enabled: yes, pair control: no
Device type: PD
Power pairs: spare
Class: class 4
Power type: 2
Power Source: Primary power source
Power Priority: low
PD requested power Value: 51000
PSE allocated power Value: 51000
UnknownTLVs:
TLV: OUI: 00,01,42, SubType: 1, Len: 1 05
TLV: OUI: 00,01,42, SubType: 1, Len: 1 0D
----------------------------------------------------------------------
---------

View LLDP Information Using tcpdump

You can use tcpdump to view the LLDP frames being transmitted and received. For example:

cumulus@switch:~$ sudo tcpdump -v -v -i swp20 ether proto 0x88cc

tcpdump: listening on swp20, link-type EN10MB (Ethernet), capture
size 262144 bytes
[Link].559022 LLDP, length 211
Chassis ID TLV (1), length 7
Subtype MAC address (4): [Link] (oui Unknown)
0x0000: 0400 30ab f2d7 a5
Port ID TLV (2), length 6
Subtype Interface Name (5): swp20
0x0000: 0573 7770 3230
Time to Live TLV (3), length 2: TTL 120s
0x0000: 0078
System Name TLV (5), length 13: dni-3048up-09
0x0000: 646e 692d 3330 3438 7570 2d30 39
System Description TLV (6), length 68
Cumulus Linux version 3.0.1~1466303042.2265c10 running on dni
3048up
0x0000: 4375 6d75 6c75 7320 4c69 6e75 7820 7665
0x0010: 7273 696f 6e20 332e 302e 317e 3134 3636
0x0020: 3330 3330 3432 2e32 3236 3563 3130 2072
0x0030: 756e 6e69 6e67 206f 6e20 646e 6920 3330
0x0040: 3438 7570
System Capabilities TLV (7), length 4
System Capabilities [Bridge, Router] (0x0014)
Enabled Capabilities [Router] (0x0010)
0x0000: 0014 0010
Management Address TLV (8), length 12

[Link] 201
Cumulus Linux 3.5 User Guide

Management Address length 5, AFI IPv4 (1): [Link]

Interface Index Interface Numbering (2): 2
0x0000: 0501 0a00 03be 0200 0000 0200
Management Address TLV (8), length 24
Management Address length 17, AFI IPv6 (2): fe80::230:abff:fef2:
d7a5
Interface Index Interface Numbering (2): 2
0x0000: 1102 fe80 0000 0000 0000 0230 abff fef2
0x0010: d7a5 0200 0000 0200
Port Description TLV (4), length 5: swp20
0x0000: 7377 7032 30
Organization specific TLV (127), length 9: OUI IEEE 802.3 Private
(0x00120f)
Link aggregation Subtype (3)
aggregation status [supported], aggregation port ID 0
0x0000: 0012 0f03 0100 0000 00
Organization specific TLV (127), length 9: OUI IEEE 802.3 Private
(0x00120f)
MAC/PHY configuration/status Subtype (1)
autonegotiation [supported, enabled] (0x03)
PMD autoneg capability [10BASE-T fdx, 100BASE-TX fdx,
1000BASE-T fdx] (0x2401)
MAU type 100BASEFX fdx (0x0012)
0x0000: 0012 0f01 0324 0100 12
Organization specific TLV (127), length 12: OUI IEEE 802.3
Private (0x00120f)
Power via MDI Subtype (2)
MDI power support [PSE, supported, enabled], power pair
spare, power class class4
0x0000: 0012 0f02 0702 0513 01fe 01fe
Organization specific TLV (127), length 5: OUI Unknown (0x000142)
0x0000: 0001 4201 0d
Organization specific TLV (127), length 5: OUI Unknown (0x000142)
0x0000: 0001 4201 01
End TLV (0), length 0

Logging poed Events in syslog

The poed service logs the following events to syslog:
When a switch provides power to a powered device
When a device that was receiving power is removed
When the power available to the switch changes
Errors

202 23 January 2018

 Cumulus Networks

Configuring a Global Proxy

You configure global HTTP and HTTPS proxies in the /etc/profile.d/ directory of Cumulus Linux. To do
so, set the http_proxy and https_proxy variables, which tells the switch the address of the proxy
server to use to fetch URLs on the command line. This is useful for programs such as apt/apt-get, curl
and wget, which can all use this proxy.

1. In a terminal, create a new file in the /etc/profile.d/ directory. In the code example below, the
file is called [Link], and is created using the text editor nano.

cumulus@switch:~$ sudo nano /etc/profile.d/[Link]

2. Add a line to the file to configure either an HTTP or an HTTPS proxy, or both:
HTTP proxy:

http_proxy=[Link]
export http_proxy

HTTPS proxy:

https_proxy=[Link]
export https_proxy

3. Create a file in the /etc/apt/[Link].d directory and add the following lines to the file for
acquiring the HTTP and HTTPS proxies; the example below uses http_proxy as the file name:

cumulus@switch:~$ sudo nano /etc/apt/[Link].d/http_proxy

Acquire::http::Proxy "[Link]
Acquire::https::Proxy "[Link]

4. Add the proxy addresses to /etc/wgetrc; you may have to uncomment the http_proxy and
https_proxy lines:

cumulus@switch:~$ sudo nano /etc/wgetrc

...

https_proxy = [Link]
http_proxy = [Link]

...

5. Run the source command, to execute the file in the current environment:

[Link] 203
Cumulus
5. Linux 3.5 User Guide

cumulus@switch:~$ source /etc/profile.d/[Link]

The proxy is now configured. The echo command can be used to confirm a proxy is set up correctly:
HTTP proxy:

cumulus@switch:~$ echo $http_proxy

[Link]

HTTPS proxy:

cumulus@switch:~$ echo $https_proxy

[Link]

Related Information
Setting up an apt package cache

HTTP API
Cumulus Linux 3.4+ implements an HTTP application programing interface to OpenStack ML2 driver (see
page 914) and NCLU (see page 85). Rather than accessing a Cumulus Linux host with SSH, you can
interact with a server with a HTTP client, such as cURL, HTTPie or a web browser.

The HTTP API service is enabled by default on chassis hardware only. However, the associated
server is configured to only listen to traffic originating from within the chassis.
The service is not enabled by default on non-chassis hardware.

Contents
This chapter covers ...
Getting Started (see page 204)
Configuration (see page 205)
Enable External Traffic on a Chassis (see page 206)
IP and Port Settings (see page 206)
Security (see page 207)
Authentication (see page 207)
Transport Layer Security (see page 207)
cURL Examples (see page 207)

Getting Started
204 23 January 2018
 Cumulus Networks

Getting Started

If upgrading from a version of Cumulus Linux before v3.4.0 the supporting software for the API
may not be installed. Install the required software with the following command.

cumulus@switch:~$ sudo apt-get install python-cumulus-restapi

To enable the HTTP API service, run the following systemd command:

cumulus@switch:~$ sudo systemctl enable restserver

If you are running Cumulus Linux 3.5.0 only, you also need to enable the restserver-
gunicorn service:

cumulus@switch:~$ sudo systemctl enable restserver-gunicorn

Use the systemctl start and systemctl stop commands to start/stop the HTTP API service:

cumulus@switch:~$ sudo systemctl start restserver

cumulus@switch:~$ sudo systemctl stop restserver

Each service runs as a background daemon once started.

Configuration
There are two configuration files associated with the HTTP API services:
/etc/[Link]
/etc/[Link]
The first configuration file is used for non-chassis hardware; the second, for chassis hardware.
Generally, only the configuration file relevant to your hardware needs to be edited, as the associated
services determine the appropriate configuration file to use at run time.

[Link] 205
Cumulus Linux 3.5 User Guide

Enable External Traffic on a Chassis

The HTTP API services are configured to listen on port 8080 for chassis hardware by default. However, only
HTTP traffic originating from internal link local management IPv6s will be allowed. To configure the services
to also accept HTTP requests originating from external sources:

1. Open /etc/[Link] in a text editor.

2. Uncomment the following line near the beginning of the file:

#user root;

3. Uncomment the server block lines near the end of the file.
4. Change the port on the now uncommented listen line if the default value, 8088, is not the
preferred port, and save the configuration file.
5. Verify the configuration file is still valid:

cumulus@switch:~$ sudo nginx -c /etc/[Link] -

t

If the configuration file is not valid, return to step 1; review any changes that were made, and correct
the errors.
6. Restart the daemons:

cumulus@switch:~$ sudo systemctl restart restserver

IP and Port Settings

The IP:port combinations that services listen to can be modified by changing the parameters of the listen
directive(s). By default, [Link] has only one listen parameter, whereas /etc/nginx-
[Link] has two independently configurable server blocks, each with a listen
directive. One server block is for external traffic, and the other for internal traffic.

All URLs must use HTTPS, rather than HTTP.

For more information on the listen directive, refer to the NGINX documentation.

Do not set the same listening port for internal and external chassis traffic.

206 23 January 2018

 Cumulus Networks

Security

Authentication
The default configuration requires all HTTP requests from external sources (not internal switch traffic) to set
the HTTP Basic Authentication header.
The user and password should correspond to a user on the host switch.

Transport Layer Security

All traffic must be secured in transport using TLSv1.2 by default. Cumulus Linux contains a self-signed
certificate and private key used server-side in this application so that it works out of the box, but Cumulus
Networks recommends you use your own certificates and keys. Certificates must be in the PEM format.
For step by step documentation for generating self-signed certificates and keys, and installing them to the
switch, refer to the Ubuntu Certificates and Security documentation.

Do not copy the [Link] or [Link] files. After installation, edit the “ssl_certificate”
and “ssl_certificate_key” values in the configuration file for your hardware.

cURL Examples
This section contains several example cURL commands for sending HTTP requests to a non-chassis host.
The following settings are used for these examples:
Username: user
Password: pw
IP: [Link]
Port: 8080

Requests for NCLU require setting the Content-Type request header to be set to application
/json.

cURL’s -k flag is necessary when the server uses a self-signed certificate. This is the default
configuration (see the Security section (see page 207)). To display the response headers, include -
D flag in the command.

To retrieve a list of all available HTTP endpoints:

cumulus@switch:~$ curl -X GET -k -u user:pw [Link]

To run net show counters on the host as a remote procedure call:

[Link] 207
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ curl -X POST -k -u user:pw -H "Content-Type:

application/json" -d '{"cmd": "show counters"}' [Link]
/nclu/v1/rpc

To add a bridge using ML2:

cumulus@switch:~$ curl -X PUT -k -u user:pw [Link]

/v1/bridge/"br1"/200

Interface
208 Configuration and 23 January 2018
 Cumulus Networks

Interface Configuration and

Management
ifupdown is the network interface manager for Cumulus Linux. Cumulus Linux 2.1 and later uses an
updated version of this tool, ifupdown2.
For more information on network interfaces, see Layer 1 and Switch Port Attributes (see page 225).

By default, ifupdown is quiet; use the verbose option -v when you want to know what is going
on when bringing an interface down or up.

Contents
This chapter covers ...
Basic Commands (see page 209)
Using NCLU to Set the Admin State of an Interface (see page 210)
ifupdown2 Interface Classes (see page 210)
Bringing All auto Interfaces Up or Down (see page 211)
Configuring a Loopback Interface (see page 212)
ifupdown Behavior with Child Interfaces (see page 212)
ifupdown2 Interface Dependencies (see page 213)
ifup Handling of Upper (Parent) Interfaces (see page 216)
Configuring IP Addresses (see page 217)
Specifying IP Address Scope (see page 218)
Purging Existing IP Addresses on an Interface (see page 219)
Specifying User Commands (see page 220)
Sourcing Interface File Snippets (see page 221)
Using Globs for Port Lists (see page 221)
Using Templates (see page 222)
Commenting out Mako Templates (see page 223)
Adding Descriptions to Interfaces (see page 223)
Caveats and Errata (see page 224)
Related Information (see page 225)

Basic Commands
To bring up an interface or apply changes to an existing interface, run:

[Link] 209
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo ifup <ifname>

To bring down a single interface, run:

cumulus@switch:~$ sudo ifdown <ifname>

ifdown always deletes logical interfaces after bringing them down. Use the --admin-state
option if you only want to administratively bring the interface up or down.

To see the link and administrative state, use the ip link show command:

cumulus@switch:~$ ip link show dev swp1

3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP mode DEFAULT qlen 500
link/ether [Link] brd [Link]

In this example, swp1 is administratively UP and the physical link is UP (LOWER_UP flag). More information
on interface administrative state and physical state can be found in this knowledge base article.

Using NCLU to Set the Admin State of an Interface

You can use NCLU (see page 85) to put an interface into an admin down state. The interface remains
down after any future reboots or applying configuration changes with ifreload -a. For example:

cumulus@switch:~$ net add interface swp1 link down

These commands create the following configuration in the /etc/network/interfaces file:

auto swp1
iface swp1
link-down yes

ifupdown2 Interface Classes

ifupdown2 provides for the grouping of interfaces into separate classes, where a class is simply a user-
defined label used to group interfaces that share a common function (like uplink, downlink or compute).
You specify classes in /etc/network/interfaces.
The most common class users are familiar with is auto, which you configure like this:

210 23 January 2018

 Cumulus Networks

auto swp1
iface swp1

You can add other classes using the allow prefix. For example, if you have multiple interfaces used for
uplinks, you can make up a class called uplinks:

auto swp1
allow-uplink swp1
iface swp1 inet static
address [Link]/31

auto swp2
allow-uplink swp2
iface swp2 inet static
address [Link]/31

This allows you to perform operations on only these interfaces using the --allow-uplinks option, or still use
the -a options since these interfaces are also in the auto class:

cumulus@switch:~$ sudo ifup --allow=uplinks

cumulus@switch:~$ sudo ifreload -a

Another example where this feature is useful is if you're using Management VRF (see page 716), you can
use the special interface class called mgmt, and put the management interface into that class.

The mgmt interface class is not supported if you are configuring Cumulus Linux using NCLU (see
page 85).

allow-mgmt eth0
iface eth0 inet dhcp
vrf mgmt

allow-mgmt mgmt
iface mgmt
address [Link]/8
vrf-table auto

All ifupdown2 commands (ifup, ifdown, ifquery, ifreload) can take a class. Include the --
allow=<class> option when you run the command. For example, to reload the configuration for the
management interface described above, run:

cumulus@switch:~$ sudo ifreload --allow=mgmt

Bringing All auto Interfaces Up or Down

[Link] 211
Cumulus Linux 3.5 User Guide

Bringing All auto Interfaces Up or Down

You can easily bring up or down all interfaces marked with the common auto class in /etc/network
/interfaces. Use the -a option. For further details, see individual man pages for ifup(8), ifdown(8),
ifreload(8).
To administratively bring up all interfaces marked auto, run:

cumulus@switch:~$ sudo ifup -a

To administratively bring down all interfaces marked auto, run:

cumulus@switch:~$ sudo ifdown -a

To reload all network interfaces marked auto, use the ifreload command, which is equivalent to running
ifdown then ifup, the one difference being that ifreload skips any configurations that didn't change):

cumulus@switch:~$ sudo ifreload -a

Configuring a Loopback Interface

Cumulus Linux has a loopback preconfigured in /etc/network/interfaces. When the switch boots up,
it has a loopback interface, called lo, which is up and assigned an IP address of [Link].

The loopback interface lo must always be specified in /etc/network/interfaces and must

always be up.

ifupdown Behavior with Child Interfaces

By default, ifupdown recognizes and uses any interface present on the system — whether a VLAN, bond
or physical interface — that is listed as a dependent of an interface. You are not required to list them in the
interfaces file unless they need a specific configuration, for MTU, link speed, and so forth (see page 225).
And if you need to delete a child interface, you should delete all references to that interface from the
interfaces file.
For this example, swp1 and swp2 below do not need an entry in the interfaces file. The following
stanzas defined in /etc/network/interfaces provide the exact same configuration:

With Child Interfaces Defined Without Child Interfaces Defined

auto swp1 auto bridge

iface swp1 iface bridge
bridge-vlan-aware yes

212 23 January 2018

 Cumulus Networks

auto swp2 bridge-ports swp1

iface swp2 swp2
bridge-vids 1-100
auto bridge bridge-pvid 1
iface bridge bridge-stp on
bridge-vlan-aware yes
bridge-ports swp1
swp2
bridge-vids 1-100
bridge-pvid 1
bridge-stp on

Bridge in Traditional Mode - Example

For this example, swp1.100 and swp2.100 below do not need an entry in the interfaces file. The
following stanzas defined in /etc/network/interfaces provide the exact same configuration:

With Child Interfaces Defined Without Child Interfaces Defined

auto swp1.100 auto br-100

iface swp1.100 iface br-100
auto swp2.100 address [Link]/24
iface swp2.100 address [Link]/64
auto br-100 bridge-ports swp1.100 swp2.
iface br-100 100
address [Link]/24 bridge-stp on
address [Link]/64
bridge-ports swp1.100 swp2.
100
bridge-stp on

For more information on the bridge in traditional mode vs the bridge in VLAN-aware mode, please read this
knowledge base article.

ifupdown2 Interface Dependencies

ifupdown2 understands interface dependency relationships. When ifup and ifdown are run with all
interfaces, they always run with all interfaces in dependency order. When run with the interface list on the
command line, the default behavior is to not run with dependents. But if there are any built-in dependents,
they will be brought up or down.
To run with dependents when you specify the interface list, use the --with-depends option. --with-
depends walks through all dependents in the dependency tree rooted at the interface you specify.
Consider the following example configuration:

auto bond1
iface bond1

[Link] 213
Cumulus Linux 3.5 User Guide

address [Link]/16
bond-slaves swp29 swp30

auto bond2
iface bond2
address [Link]/16
bond-slaves swp31 swp32

auto br2001
iface br2001
address [Link]/24
bridge-ports bond1.2001 bond2.2001
bridge-stp on

Using ifup --with-depends br2001 brings up all dependents of br2001: bond1.2001, bond2.2001,
bond1, bond2, bond1.2001, bond2.2001, swp29, swp30, swp31, swp32.

cumulus@switch:~$ sudo ifup --with-depends br2001

Similarly, specifying ifdown --with-depends br2001 brings down all dependents of br2001: bond1.
2001, bond2.2001, bond1, bond2, bond1.2001, bond2.2001, swp29, swp30, swp31, swp32.

cumulus@switch:~$ sudo ifdown --with-depends br2001

As mentioned earlier, ifdown2 always deletes logical interfaces after bringing them down. Use
the --admin-state option if you only want to administratively bring the interface up or down. In
terms of the above example, ifdown br2001 deletes br2001.

To guide you through which interfaces will be brought down and up, use the --print-dependency
option to get the list of dependents.
Use ifquery --print-dependency=list -a to get the dependency list of all interfaces:

cumulus@switch:~$ sudo ifquery --print-dependency=list -a

lo : None
eth0 : None
bond0 : ['swp25', 'swp26']
bond1 : ['swp29', 'swp30']
bond2 : ['swp31', 'swp32']
br0 : ['bond1', 'bond2']
bond1.2000 : ['bond1']
bond2.2000 : ['bond2']
br2000 : ['bond1.2000', 'bond2.2000']
bond1.2001 : ['bond1']
bond2.2001 : ['bond2']
br2001 : ['bond1.2001', 'bond2.2001']

214 23 January 2018

 Cumulus Networks

swp40 : None
swp25 : None
swp26 : None
swp29 : None
swp30 : None
swp31 : None
swp32 : None

To print the dependency list of a single interface, use:

cumulus@switch:~$ sudo ifquery --print-dependency=list br2001

br2001 : ['bond1.2001', 'bond2.2001']
bond1.2001 : ['bond1']
bond2.2001 : ['bond2']
bond1 : ['swp29', 'swp30']
bond2 : ['swp31', 'swp32']
swp29 : None
swp30 : None
swp31 : None
swp32 : None

To print the dependency information of an interface in dot format:

cumulus@switch:~$ sudo ifquery --print-dependency=dot br2001

/* Generated by GvGen v.0.9 ([Link]
*/
digraph G {
compound=true;
node1 [label="br2001"];
node2 [label="bond1.2001"];
node3 [label="bond2.2001"];
node4 [label="bond1"];
node5 [label="bond2"];
node6 [label="swp29"];
node7 [label="swp30"];
node8 [label="swp31"];
node9 [label="swp32"];
node1->node2;
node1->node3;
node2->node4;
node3->node5;
node4->node6;
node4->node7;
node5->node8;
node5->node9;
}

You can use dot to render the graph on an external system where dot is installed.

[Link] 215
Cumulus Linux 3.5 User Guide

To print the dependency information of the entire interfaces file:

cumulus@switch:~$ sudo ifquery --print-dependency=dot -a

>interfaces_all.dot

ifup Handling of Upper (Parent) Interfaces

When you run ifup on a logical interface (like a bridge, bond or VLAN interface), if the ifup resulted in the
creation of the logical interface, by default it implicitly tries to execute on the interface's upper (or parent)
interfaces as well. This helps in most cases, especially when a bond is brought down and up, as in the
example below. This section describes the behavior of bringing up the upper interfaces.
Consider this example configuration:

auto br100
iface br100
bridge-ports bond1.100 bond2.100

auto bond1
iface bond1
bond-slaves swp1 swp2

If you run ifdown bond1, ifdown deletes bond1 and the VLAN interface on bond1 (bond1.100); it also
removes bond1 from the bridge br100. Next, when you run ifup bond1, it creates bond1 and the VLAN
interface on bond1 (bond1.100); it also executes ifup br100 to add the bond VLAN interface (bond1.100)
to the bridge br100.

216 23 January 2018

 Cumulus Networks

As you can see above, implicitly bringing up the upper interface helps, but there can be cases where an
upper interface (like br100) is not in the right state, which can result in warnings. The warnings are mostly
harmless.
If you want to disable these warnings, you can disable the implicit upper interface handling by setting
skip_upperifaces=1 in /etc/network/ifupdown2/[Link].
With skip_upperifaces=1, you will have to explicitly execute ifup on the upper interfaces. In this case,
you will have to run ifup br100 after an ifup bond1 to add bond1 back to bridge br100.

Although specifying a subinterface like swp1.100 and then running ifup swp1.100 will also
result in the automatic creation of the swp1 interface in the kernel, Cumulus Networks
recommends you specify the parent interface swp1 as well. A parent interface is one where any
physical layer configuration can reside, such as link-speed 1000 or link-duplex full.
It's important to note that if you only create swp1.100 and not swp1, then you cannot run ifup
swp1 since you did not specify it.

Configuring IP Addresses
IP addresses are configured with the net add interface command.

Example IP Address Configuration

The following commands configure three IP addresses for swp1: two IPv4 addresses, and one
IPv6 address.

cumulus@switch:~$ net add interface swp1 ip address [Link]/30

cumulus@switch:~$ net add interface swp1 ip address [Link]/30
cumulus@switch:~$ net add interface swp1 ipv6 address 2001:
DB8::1/126
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

You can specify both IPv4 and IPv6 addresses for the same interface.

These commands create the following code snippet:

auto swp1
iface swp1
address [Link]/30
address [Link]/30
address [Link]/126

[Link] 217
Cumulus Linux 3.5 User Guide

The address method and address family are added by NCLU when needed, specifically
when you are creating DHCP or loopback interfaces.

auto lo
iface lo inet loopback

To show the assigned address on an interface, use ip addr show:

cumulus@switch:~$ ip addr show dev swp1

3: swp1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP qlen 500
link/ether [Link] brd [Link]
inet [Link]/30 scope global swp1
inet [Link]/30 scope global swp1
inet6 [Link]/126 scope global tentative
valid_lft forever preferred_lft forever

Specifying IP Address Scope

ifupdown2 does not honor the configured IP address scope setting in /etc/network/interfaces,
treating all addresses as global. It does not report an error. Consider this example configuration:

auto swp2
iface swp2
address [Link]/30
address [Link]/80
scope link

When you run ifreload -a on this configuration, ifupdown2 considers all IP addresses as global.

cumulus@switch:~$ ip addr show swp2

5: swp2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP group default qlen 1000
link/ether [Link] brd [Link]
inet [Link]/30 scope global swp2
valid_lft forever preferred_lft forever
inet6 [Link]/80 scope global
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef5:6282/64 scope link
valid_lft forever preferred_lft forever

To work around this issue, configure the IP address scope:

218 23 January 2018

 Cumulus Networks

Example post-up Configuration

cumulus@switch:~$ net add interface swp6 post-up ip address

add [Link]/32 dev swp6 scope site
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following code snippet in the /etc/network/interfaces file:

auto swp6
iface swp6
post-up ip address add [Link]/32 dev swp6 scope site

Now it has the correct scope:

cumulus@switch:~$ ip addr show swp6

9: swp6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP group default qlen 1000
link/ether [Link] brd [Link]
inet [Link]/32 scope site swp6
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef5:6286/64 scope link
valid_lft forever preferred_lft forever

Purging Existing IP Addresses on an Interface

By default, ifupdown2 purges existing IP addresses on an interface. If you have other processes that
manage IP addresses for an interface, you can disable this feature including the address-purge setting in
the interface's configuration.

cumulus@switch:~$ net add interface swp1 address-purge no

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration snippet in the /etc/network/interfaces file:

auto swp1
iface swp1
address-purge no

Purging existing addresses on interfaces with multiple iface stanzas is not supported. Doing so
[Link] 219
Cumulus Linux 3.5 User Guide

Purging existing addresses on interfaces with multiple iface stanzas is not supported. Doing so
can result in the configuration of multiple addresses for an interface after you change an
interface address and reload the configuration with ifreload -a. If this happens, you must
shut down and restart the interface with ifup and ifdown, or manually delete superfluous
addresses with ip address delete [Link]/mask dev DEVICE. See
also the Caveats and Errata (see page 224) section below for some cautions about using multiple
iface stanzas for the same interface.

Specifying User Commands

You can specify additional user commands in the interfaces file. As shown in the example below, the
interface stanzas in /etc/network/interfaces can have a command that runs at pre-up, up, post-up,
pre-down, down, and post-down:

cumulus@switch:~$ net add interface swp1 post-up /sbin/foo bar

cumulus@switch:~$ net add interface ip address [Link]/30
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration in the /etc/network/interfaces file:

auto swp1
iface swp1
address [Link]/30
post-up /sbin/foo bar

Any valid command can be hooked in the sequencing of bringing an interface up or down, although
commands should be limited in scope to network-related commands associated with the particular
interface.
For example, it wouldn't make sense to install some Debian package on ifup of swp1, even though that is
technically possible. See man interfaces for more details.

If your post-up command also starts, restarts or reloads any systemd service, you must use the
--no-block option with systemctl. Otherwise, that service or even the switch itself may hang
after starting or restarting.
For example, to restart the dhcrelay service after bringing up VLAN 100, first run:

cumulus@switch:~$ net add vlan 100 post-up systemctl --no-

block restart [Link]

This command creates the following configuration in the /etc/network/interfaces file:

220 23 January 2018

 Cumulus Networks

auto bridge
iface bridge
bridge-vids 100
bridge-vlan-aware yes

auto vlan100
iface vlan100
post-up systemctl --no-block restart [Link]
vlan-id 100
vlan-raw-device bridge

Sourcing Interface File Snippets

Sourcing interface files helps organize and manage the interfaces file. For example:

cumulus@switch:~$ cat /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface

auto eth0
iface eth0 inet dhcp

source /etc/network/interfaces.d/bond0

The contents of the sourced file used above are:

cumulus@switch:~$ cat /etc/network/interfaces.d/bond0

auto bond0
iface bond0
address [Link]/30
address [Link]/64
bond-slaves swp25 swp26

Using Globs for Port Lists

NCLU supports globs to define port lists (that is, a range of ports). The glob keyword is implied when you
specify bridge ports and bond slaves:

cumulus@switch:~$ net add bridge bridge ports swp1-4,6,10-12

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

[Link] 221
Cumulus Linux 3.5 User Guide

These commands produce the following snippet in the /etc/network/interfaces file:

...

auto swp1
iface swp1

auto swp2
iface swp2

auto swp3
iface swp3

auto swp4
iface swp4

auto swp6
iface swp6

auto swp10
iface swp10

auto swp11
iface swp11

auto swp12
iface swp12

Using Templates
ifupdown2 supports Mako-style templates. The Mako template engine is run over the interfaces file
before parsing.
Use the template to declare cookie-cutter bridges in the interfaces file:

%for v in [11,12]:
auto vlan${v}
iface vlan${v}
address 10.20.${v}.3/24
bridge-ports glob swp19-20.${v}
bridge-stp on
%endfor

And use it to declare addresses in the interfaces file:

222 23 January 2018

 Cumulus Networks

%for i in [1,12]:
auto swp${i}
iface swp${i}
address 10.20.${i}.3/24

Regarding Mako syntax, use square brackets ([1,12]) to specify a list of individual numbers (in
this case, 1 and 12). Use range(1,12) to specify a range of interfaces.

You can test your template and confirm it evaluates correctly by running mako-render /etc
/network/interfaces.

For more examples of configuring Mako templates, read this knowledge base article.

Commenting out Mako Templates

To comment out content in Mako templates, use double hash marks (##). For example:

## % for i in range(1, 4):

## auto swp${i}
## iface swp${i}
## % endfor
##

Adding Descriptions to Interfaces

You can add descriptions to the interfaces configured in /etc/network/interfaces by using the alias
keyword.

Example Alias Configuration

The following commands create an alias for swp1:

cumulus@switch:~$ net add interface swp1 alias

hypervisor_port_1
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following code snippet:

[Link] 223
Cumulus Linux 3.5 User Guide

auto swp1
iface swp1
alias hypervisor_port_1

You can query the interface description using NCLU. The alias appears in the Name column after the actual
interface name:

cumulus@switch$ net show interface swp1

Name MAC Speed MTU Mode
-- ------------------------ ----------------- ------- -----
---------
UP swp1 (hypervisor_port_1) [Link] 1G 1500
Access/L2

Interface descriptions also appear in the SNMP OID (see page 772) IF-MIB::ifAlias.

Aliases are limited to 256 characters.

Caveats and Errata

While ifupdown2 supports the inclusion of multiple iface stanzas for the same interface, Cumulus
Networks recommends you use a single iface stanza for each interface, if possible.
There are cases where you must specify more than one iface stanza for the same interface. For example,
the configuration for a single interface can come from many places, like a template or a sourced file.
If you do specify multiple iface stanzas for the same interface, make sure the stanzas do not specify the
same interface attributes. Otherwise, unexpected behavior can result.
For example, swp1 is configured in two places:

cumulus@switch:~$ cat /etc/network/interfaces

source /etc/interfaces.d/speed_settings

auto swp1
iface swp1
address [Link]/24

As well as /etc/interfaces.d/speed_settings

cumulus@switch:~$ cat /etc/interfaces.d/speed_settings

auto swp1
iface swp1

224 23 January 2018

 Cumulus Networks

link-speed 1000
link-duplex full

ifupdown2 correctly parses a configuration like this because the same attributes are not specified in
multiple iface stanzas.
And, as stated in the note above, you cannot purge existing addresses on interfaces with multiple iface
stanzas.

Related Information
Debian - Network Configuration
Linux Foundation - Bonds
Linux Foundation - Bridges
Linux Foundation - VLANs
man ifdown(8)
man ifquery(8)
man ifreload
man ifup(8)
man ifupdown-addons-interfaces(5)
man interfaces(5)

Layer 1 and Switch Port Attributes

This chapter discusses the various network interfaces on a switch running Cumulus Linux, how to configure
various interface-level settings (if needed) and some troubleshooting commands.

Contents
This chapter covers ...
Interface Types (see page 226)
Interface Settings (see page 226)
Differences between Broadcom-based and Mellanox-based Switches (see page 226)
Enabling Auto-negotiation (see page 227)
Default Interface Configuration Settings (see page 227)
Port Speed and Duplexing (see page 237)
MTU (see page 238)
Setting a Policy for Global System MTU (see page 240)
Creating a Default Policy for Various Interface Settings (see page 240)
Configuring Breakout Ports (see page 241)
Combining Four 10G Ports into One 40G Port (see page 244)
Logical Switch Port Limitations (see page 244)

Using ethtool to Configure Interfaces (see page 245)

[Link] 225
Cumulus Linux 3.5 User Guide

Using ethtool to Configure Interfaces (see page 245)

Verification and Troubleshooting Commands (see page 246)
Statistics (see page 246)
Querying SFP Port Information (see page 247)
Caveats and Errata (see page 247)
Timeout Error on Quanta LY8 and LY9 Switches (see page 247)
swp33 and swp34 Disabled on Some Switches (see page 248)
ethtool Shows Incorrect Port Speed on 100G Mellanox Switches (see page 248)
Related Information (see page 249)

Interface Types
Cumulus Linux exposes network interfaces for several types of physical and logical devices:
lo, network loopback device
ethN, switch management port(s), for out of band management only
swpN, switch front panel ports
(optional) brN, bridges (IEEE 802.1Q VLANs)
(optional) bondN, bonds (IEEE 802.3ad link aggregation trunks, or port channels)

Interface Settings
Each physical network interface has a number of configurable settings:
Auto-negotiation
Duplex
Forward error correction
Link speed
MTU, or maximum transmission unit
Almost all of these settings are configured automatically for you, depending upon your switch ASIC,
although you must always set MTU manually.

You can only set MTU for logical interfaces. If you try to set auto-negotiation, duplex mode or link
speed for a logical interface, an unsupported error gets returned.

Differences between Broadcom-based and Mellanox-based Switches

On a Broadcom-based switch, all you need to do is enable auto-negotiation. Once enabled, Cumulus Linux
automatically configures the link speed, duplex mode and forward error correction (FEC, if the cable or
optic requires it) for every switch port, based on the switch model and cable or optic used on the port, as
listed in the table below (see page 227).
Ports are always automatically configured on a Mellanox-based switch, with one exception — you only need
to configure is MTU (see page 238). You don't even need to enable auto-negotation, as the Mellanox
firmware configures everything for you.

226 23 January 2018

 Cumulus Networks

Enabling Auto-negotiation
To configure auto-negotiation for a Broadcom-based switch, set link-autoneg to on for all the switch
ports. For example, to enable auto-negotiation for swp1 through swp52:

cumulus@switch:~$ net add interface swp1-52 link autoneg on

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Any time you enable auto-negotiation, Cumulus Linux restores the default configuration settings specified
in the table below (see page ).
By default on a Broadcom-based switch, auto-negotiation is disabled — except on 10G and 1G BASE-T
switches, where it's required for links to work at all. And for RJ45-SFP converters, you need to manually
configure the settings as described in the default settings table below (see page ).
If you disable it later or never enable it, then you have to configure the duplex, FEC and link speed settings
manually using NCLU (see page 85) — see the relevant sections below. The default speed if you disable
auto-negotiation depends on the type of connector used with the port. For example, a QSFP28 optic
defaults to 100G, while a QSFP+ optic defaults to 40G and SFP+ defaults to 10G.

You cannot or should not disable auto-negotiation off for any type of copper cable, including:
10G BASE-T
10G DAC
40G DAC
100G DAC
However, RJ-45 (10/100/1000 BASE-T) adapters do not work with auto-negotiation enabled. You
must manually configure these ports using the settings below (link-autoneg=off, link-
speed=1000|100|10, link-duplex=full|half).

Depending upon the connector used for a port, enabling auto-negotiation also enables forward error
correction (FEC), if the cable requires it (see the table below (see page )). FEC always adjusts for the
speed of the cable. However, you cannot disable FEC separately using NCLU (see page 85).

Default Interface Configuration Settings

On a Broadcom-based switch, the configuration for each type of interface is described in the following
table. Except as noted below, the settings for both sides of the link are expected to be the same.

If the other side of the link is running a version of Cumulus Linux earlier than 3.2, depending up
on the interface type, auto-negotiation may not work on that switch. Cumulus Networks
recommends you use the default settings on this switch in this case.

For Mellanox-based switches, the Spectrum firmware decides on the best settings based on the switch
model and connector type.

[Link] 227
Cumulus Linux 3.5 User Guide

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

10/100 Off N/A (does The module has two

Base-T not apply at sets of electronics —
(RJ45 SFP) this speed) $ net add the port side, which
interface communicates to the
swp1 link switch ASIC, and the
speed 100 RJ45 side.
$ net add Auto-negotiation is
interface always used on the
swp1 link RJ45 side of the link by
autoneg off the PHY built into the
module. This is
independent of the
switch setting. Set
Configuration link-autoneg to off.
in /etc
Auto-negotiation needs
/network
to be enabled on the
/interfaces
server side in this
scenario.
auto swp1
iface swp1
link-
autoneg off
link-speed
100

10/100 Recommended N/A 10M or 100M speeds

Base-T on On are possible with
a 1G fixed $ net add autoneg OFF on both
copper interface sides. Testing on an
port swp1 link Edge-Core AS4610-54P
speed 100 revealed the ASIC
$ net add reporting autoneg as
interface ON.
swp1 link PoE may require
autoneg on autoneg to be ON.

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg on

228 23 January 2018

 Cumulus Networks

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

link-speed
100

1000BASE- Off N/A The module has two

T sets of electronics —
(RJ45 SFP) $ net add the port side, which
interface communicates to the
swp1 link switch ASIC, and the
speed 1000 RJ45 side.
$ net add Auto-negotiation is
interface always used on the
swp1 link RJ45 side of the link by
autoneg off the PHY built into the
module. This is
independent of the
switch setting. Set
Configuration link-autoneg to off.
in /etc
Auto-negotiation needs
/network
to be enabled on the
/interfaces
server side.

auto swp1
iface swp1
link-
autoneg off
link-speed
1000

1G BASE-T On N/A  
on a 1G
fixed $ net add
copper interface
port swp1 link
speed 1000
$ net add
interface
swp1 link
autoneg on

Configuration
in /etc
/network
/interfaces

[Link] 229
Cumulus Linux 3.5 User Guide

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

auto swp1
iface swp1
link-
autoneg on
link-speed
1000

1G BASE-T On N/A  
on a 10G
fixed $ net add
copper interface
port swp1 link
speed 1000
$ net add
interface
swp1 link
autoneg on

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg on
link-speed
1000

1000BASE- Recommended N/A Without auto-

SX, On negotiation, the link
1000BASE- $ net add stays up when there is
LX, interface a single fiber break.
1000BASE- swp1 link
CX speed 1000
(1G Fiber) $ net add
interface
swp1 link
autoneg on

230 23 January 2018

 Cumulus Networks

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg on
link-speed
1000

10G BASE- On N/A  

T fixed
copper $ net add
port interface
swp1 link
speed 10000
$ net add
interface
swp1 link
autoneg on

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg on
link-speed
10000

10G BASE- Off N/A  

CR,
10G BASE- $ net add
LR, interface
10G BASE- swp1 link
SR, speed 10000
10G AOC $ net add
interface

[Link] 231
Cumulus Linux 3.5 User Guide

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

swp1 link
autoneg off

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg off
link-speed
10000

40G BASE- Recommended Disable it 40G standards

CR4 On mandate auto-
$ net add negotiation should be
interface enabled for DAC
swp1 link connections.
speed 40000
$ net add
interface
swp1 link
autoneg on

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg on
link-speed
40000

Off Disable it  

$ net add
interface

232 23 January 2018

 Cumulus Networks

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

swp1 link
40G BASE-
SR4, speed 40000
40G BASE- $ net add
LR4, interface
40G AOC swp1 link
autoneg off

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg off
link-speed
40000

100G BASE- On auto-  

CR4 negotiated
$ net add
interface
swp1 link
speed 100000
$ net add
interface
swp1 link
autoneg on

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg on
link-speed
100000

[Link] 233
Cumulus Linux 3.5 User Guide

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

100G BASE- Off RS  

SR4,
100G AOC $ net add
interface
swp1 link
speed 100000
$ net add
interface
swp1 link
autoneg off
$ net add
interface
swp1 link fec
rs

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg off
link-speed
100000
link-fec rs

100G BASE- Off None  

LR4 stated
$ net add
interface
swp1 link
speed 100000
$ net add
interface
swp1 link
autoneg off
$ net add
interface
swp1 link fec
off

234 23 January 2018

 Cumulus Networks

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg off
link-speed
100000
link-fec
off

25G BASE- On auto-  

CR negotiated*
$ net add
interface
swp1 link
speed 25000
$ net add
interface
swp1 link
autoneg on

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg on
link-speed
25000

25G BASE- Off RS* Tomahawk cannot do

SR RS on a single channel,
$ net add only Base-R/FC
interface /FireCode/Type74,
swp1 link which violates the 802.3
speed 25000 by specification for
25G.

[Link] 235
Cumulus Linux 3.5 User Guide

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

$ net add
interface
swp1 link
autoneg off
$ net add
interface
swp1 link fec
baser

Configuration
in /etc
/network
/interfaces

auto swp1
iface swp1
link-
autoneg off
link-speed
25000
link-fec
baser

25G BASE- Off None  

LR stated
$ net add
interface
swp1 link
speed 25000
$ net add
interface
swp1 link
autoneg off
$ net add
interface
swp1 link fec
off

Configuration
in /etc
/network
/interfaces

236 23 January 2018

 Cumulus Networks

Speed Auto- FEC Manual Configuration Notes

negotiation Setting Steps

auto swp1
iface swp1
link-
autoneg off
link-speed
25000
link-fec
off

Port Speed and Duplexing

Cumulus Linux supports both half- and full-duplex configurations. Supported port speeds include 100M,
1G, 10G, 25G, 40G, 50G and 100G. If you need to manually set the speed on a Broadcom-based switch, set
it in terms of Mbps, where the setting for 1G is 1000, 40G is 40000 and 100G is 100000, for example.
The duplex mode setting defaults to full. You only need to specify link duplex if you want half-duplex
mode.

Example Port Speed and Duplexing Configuration

The following NCLU commands configure the port speed for the swp1 interface:

cumulus@switch:~$ net add interface swp1 link speed 10000

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The above commands create the following /etc/network/interfaces code snippet:

auto swp1
iface swp1
link-speed 10000

Port Speed Limitations

Ports can be configured to one speed less than their maximum speed.

Switch port Type Lowest Configurable Speed

1G 100 Mb

[Link] 237
Cumulus Linux 3.5 User Guide

Switch port Type Lowest Configurable Speed

10G 1 Gigabit (1000 Mb)

40G 10G*

100G 50G & 40G (with or without breakout port), 25G*, 10G*

*Requires the port to be converted into a breakout port. See below (see page 241).

MTU
Interface MTU (maximum transmission unit) applies to traffic traversing the management port, front panel
/switch ports, bridge, VLAN subinterfaces and bonds — in other words, both physical and logical interfaces.
MTU is the only interface setting that must be set manually.
In Cumulus Linux, ifupdown2 assigns 1500 as the default MTU setting. To change the setting, run:

cumulus@switch:~$ net add interface swp1 mtu 9000

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Some switches may not support the same maximum MTU setting in hardware for both the
management interface (eth0) and the data plane ports.

MTU for a Bridge

The MTU setting is the lowest MTU setting of any interface that is a member of that bridge (that is, every
interface specified in bridge-ports in the bridge configuration in the interfaces file), even if another
bridge member has a higher MTU value. There is no need to specify an MTU on the bridge. Consider this
bridge configuration:

auto bridge
iface bridge
bridge-ports bond1 bond2 bond3 bond4 peer5
bridge-vids 100-110
bridge-vlan-aware yes

In order for bridge to have an MTU of 9000, set the MTU for each of the member interfaces (bond1 to bond
4, and peer5), to 9000 at minimum.

Use MTU 9216 for a bridge

238 23 January 2018

 Cumulus Networks

Two common MTUs for jumbo frames are 9216 and 9000 bytes. The corresponding MTUs for the
VNIs would be 9166 and 8950.

When configuring MTU for a bond, configure the MTU value directly under the bond interface; the
configured value is inherited by member links/slave interfaces. If you need a different MTU on the bond, set
it on the bond interface, as this ensures the slave interfaces pick it up. There is no need to specify MTU on
the slave interfaces.
VLAN interfaces inherit their MTU settings from their physical devices or their lower interface; for example,
swp1.100 inherits its MTU setting from swp1. Hence, specifying an MTU on swp1 ensures that swp1.100
inherits swp1's MTU setting.
If you are working with VXLANs (see page 398), the MTU for a virtual network interface (VNI) must be 50
bytes smaller than the MTU of the physical interfaces on the switch, as those 50 bytes are required for
various headers and other data. You should also consider setting the MTU much higher than the default
1500.

Example MTU Configuration

In general, the policy file specified above handles default MTU settings for all interfaces on the
switch. If you need to configure a different MTU setting for a subset of interfaces, use NCLU (see
page 85).
The following commands configure an MTU minimum value of 9000 on swp1:

cumulus@switch:~$ net add interface swp1 mtu 9000

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following code snippet:

auto swp1
iface swp1
mtu 9000

You must take care to ensure there are no MTU mismatches in the conversation path.
MTU mismatches will result in dropped or truncated packets, degrading or blocking
network performance.

To view the MTU setting, use net show interface <interface>:

cumulus@switch:~$ net show interface swp1

Name MAC Speed MTU Mode
-- ------ ----------------- ------- ----- ---------
UP swp1 [Link] 1G 1500 Access/L2

[Link] 239
Cumulus Linux 3.5 User Guide

Setting a Policy for Global System MTU

For a global policy to set MTU, create a policy document (called [Link] here) like the following:

cat /etc/network/ifupdown2/policy.d/[Link]
{
"address": {"defaults": { "mtu": "9216" }
}
}

If your platform does not support a high MTU on eth0, you can set a lower MTU with the following
command:

net add interface eth0 mtu 1500

net commit

The policies and attributes in any file in /etc/network/ifupdown2/policy.d/ override the

default policies and attributes in /var/lib/ifupdown2/policy.d/.

Creating a Default Policy for Various Interface Settings

Instead of configuring these settings for each individual interface, you can specify a policy for all interfaces
on a switch, or tailor custom settings for each interface. Create a file in /etc/network/ifupdown2
/policy.d/, like in the following example (called [Link]), and populate the settings accordingly:

cumulus@switch:~$ cat /etc/network/ifupdown2/policy.d/[Link]

{
"ethtool": {
"defaults": {
"link-duplex": "full"
},
"iface_defaults": {
"swp1": {
"link-autoneg": "on",
"link-speed": "1000"
},
"swp50": {
"link-autoneg": "off",
"link-speed": "10000"
}
}
},

240 23 January 2018

 Cumulus Networks

"address": {
"defaults": { "mtu": "9000" }
}
}

Configuring Breakout Ports

Cumulus Linux has the ability to:
Break out 100G switch ports into the following with breakout cables:
2x50G, 4x25G, 4x10G
Break out 40G switch ports into four separate 10G ports for use with breakout cables.
Combine (also called aggregating or ganging) four 10G switch ports into one 40G port for use with a
breakout cable (not to be confused with a bond (see page 315)).
To configure a 4x25G breakout port, first configure the port to break out then set the link speed:

cumulus@switch:~$ net add interface swp3 breakout 4x

cumulus@switch:~$ net add interface swp3s0-3 link speed 25000
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

On Mellanox switches, you need to disable the next port (see below). In this example, you would
also run the following before committing the update:

cumulus@switch:~$ net add interface swp4 breakout disabled

These commands create 4 interfaces in the /etc/network/interfaces file named as follows:

cumulus@switch:~$ cat /etc/network/interfaces

...

auto swp3s0
iface swp3s0

auto swp3s1
iface swp3s1

auto swp3s2
iface swp3s2

auto swp3s3
iface swp3s3

[Link] 241
Cumulus Linux 3.5 User Guide

...

When you commit your change configuring the breakout ports, switchd restarts to apply the
changes. The restart interrupts network services (see page 193).

The breakout port configuration is stored in the /etc/cumulus/[Link] file.

/etc/cumulus/[Link] varies across different hardware platforms. Check the current list
of supported platforms on the hardware compatibility list.
A snippet from the /etc/cumulus/[Link] on a Dell S6000 switch (with a Trident II+ ASIC)
where swp3 is broken out as above looks like this:

cumulus@switch:~$ cat /etc/cumulus/[Link]

# [Link] --
#
# This file controls port aggregation and subdivision. For
example, QSFP+
# ports are typically configurable as either one 40G interface
or four
# 10G/1000/100 interfaces. This file sets the number of
interfaces per port
# while /etc/network/interfaces and ethtool configure the link
speed for each
# interface.
#
# You must restart switchd for changes to take effect.
#
# The DELL S6000 has:
# 32 QSFP ports numbered 1-32
# These ports are configurable as 40G, split into 4x10G
ports or
# disabled.
#
# The X pipeline covers QSFP ports 1 through 16 and the Y
pipeline
# covers QSFP ports 17 through 32.
#
# The Trident2 chip can only handle 52 logical ports per
pipeline.
#
# This means 13 is the maximum number of 40G ports you can
ungang
# per pipeline, with the remaining three 40G ports set to
# "disabled". The 13 40G ports become 52 unganged 10G
ports, which
# totals 52 logical ports for that pipeline.

242 23 January 2018

 Cumulus Networks

#
# QSFP+ ports
#
# <port label 1-32> = [4x10G|40G|disabled]
1=40G
2=40G
3=4x
4=40G
5=40G
6=40G
7=40G
8=40G
9=40G
10=40G
11=40G
12=40G
13=40G
14=40G
15=40G
16=40G
17=40G
18=40G
19=40G
20=40G
21=40G
22=40G
23=40G
24=40G
25=40G
26=40G
27=40G
28=40G
29=40G
30=40G
31=40G
32=40G

Notice that you can break out any of the 100G ports into a variety of options: four 10G ports, four
25G ports or two 50G ports. Keep in mind that you cannot have more than 128 total logical ports
on a Broadcom switch.

The Mellanox SN2700, SN2700B, SN2410, and SN2410B switches both have a limit of
64 logical ports in total. However, if you want to break out to 4x25G or 4x10G, you must
configure the logical ports as follows:
You can only break out odd-numbered ports into 4 logical ports.
You must disable the next even-numbered port.
These restrictions do not apply to a 2x50G breakout configuration.

[Link] 243
Cumulus Linux 3.5 User Guide

For example, if you have a 100G Mellanox SN2700 switch and break out port 11 into 4
logical ports, you must disable port 12 by running net add interface swp12
breakout disabled, which results in this configuration in /etc/cumulus/ports.
conf:

...

11=4x
12=disabled

...

There is no limitation on any port if interfaces are configured in 2x50G mode.

Here is an example showing how to configure breakout cables for the Mellanox Spectrum SN2700
.

Combining Four 10G Ports into One 40G Port

You can gang (aggregate) four 10G ports into one 40G port for use with a breakout cable, provided you
follow these requirements:
You must gang four 10G ports in sequential order. For example, you cannot gang swp1, swp10,
swp20 and swp40 together.
The ports must be in increments of four, with the starting port being swp1 (or swp5, swp9, or so
forth); so you cannot gang swp2, swp3, swp4 and swp5 together.
For example, to gang swp1 through swp4 into a 40G port, run:

cumulus@switch:~$ net add int swp1-4 breakout /4

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration snippet in the /etc/cumulus/[Link] file:

# SFP+ ports#
# <port label 1-48> = [10G|40G/4]
1=40G/4
2=40G/4
3=40G/4
4=40G/4
5=10G

Logical Switch Port Limitations

244 23 January 2018
 Cumulus Networks

Logical Switch Port Limitations

100G and 40G switches can support a certain number of logical ports, depending upon the manufacturer;
these include:
Mellanox SN2700 and SN2700B switches
Switches with Broadcom Tomahawk, Trident II and Trident II+ chipsets (check the HCL)
Before you configure any logical/unganged ports on a switch, check the limitations listed in /etc/cumulus
/[Link]; this file is specific to each manufacturer.
For example, the Dell S6000 [Link] file indicates the logical port limitation like this:

# [Link] --
#
# This file controls port aggregation and subdivision. For example,
QSFP+
# ports are typically configurable as either one 40G interface or four
# 10G/1000/100 interfaces. This file sets the number of interfaces
per port
# while /etc/network/interfaces and ethtool configure the link speed
for each
# interface.
#
# You must restart switchd for changes to take effect.
#
# The DELL S6000 has:
# 32 QSFP ports numbered 1-32
# These ports are configurable as 40G, split into 4x10G ports or
# disabled.
#
# The X pipeline covers QSFP ports 1 through 16 and the Y pipeline
# covers QSFP ports 17 through 32.
#
# The Trident2 chip can only handle 52 logical ports per pipeline.
#
# This means 13 is the maximum number of 40G ports you can ungang
# per pipeline, with the remaining three 40G ports set to
# "disabled". The 13 40G ports become 52 unganged 10G ports, which
# totals 52 logical ports for that pipeline.

The means the maximum number of ports for this Dell S6000 is 104.
Mellanox SN2700 and SN2700B switches have a limit of 64 logical ports in total. However, the logical ports
must be configured in a specific way. See the note (see page 241) above.

Using ethtool to Configure Interfaces

The Cumulus Linux ethtool command is an alternative for configuring interfaces as well as viewing and
troubleshooting them.
For example, to manually set link speed, auto-negotiation, duplex mode and FEC on swp1, run:

[Link] 245
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo ethtool -s swp1 speed 25000 autoneg off duplex

full
cumulus@switch:~$ sudo ethtool --set-fec swp1 encoding off

To view the FEC setting on an interface, run:

cumulus@switch:~$ sudo ethtool --show-fec swp1FEC parameters for swp1:

Auto-negotiation: off
FEC encodings : RS

Verification and Troubleshooting Commands

Statistics
High-level interface statistics are available with the net show interface command:

cumulus@switch:~$ net show interface swp1

Name MAC Speed MTU Mode

-- ------ ----------------- ------- ----- ---------
UP swp1 [Link] 1G 1500 Access/L2

Vlans in disabled State

-------------------------
br0

Counters TX RX
---------- ---- ----
errors 0 0
unicast 0 0
broadcast 0 0
multicast 0 0

LLDP
------ ---- ---------------------------
swp1 ==== [Link](server01)

Low-level interface statistics are available with ethtool:

cumulus@switch:~$ sudo ethtool -S swp1

NIC statistics:
HwIfInOctets: 21870
HwIfInUcastPkts: 0

246 23 January 2018

 Cumulus Networks

HwIfInBcastPkts: 0
HwIfInMcastPkts: 243
HwIfOutOctets: 1148217
HwIfOutUcastPkts: 0
HwIfOutMcastPkts: 11353
HwIfOutBcastPkts: 0
HwIfInDiscards: 0
HwIfInL3Drops: 0
HwIfInBufferDrops: 0
HwIfInAclDrops: 0
HwIfInBlackholeDrops: 0
HwIfInDot3LengthErrors: 0
HwIfInErrors: 0
SoftInErrors: 0
SoftInDrops: 0
SoftInFrameErrors: 0
HwIfOutDiscards: 0
HwIfOutErrors: 0
HwIfOutQDrops: 0
HwIfOutNonQDrops: 0
SoftOutErrors: 0
SoftOutDrops: 0
SoftOutTxFifoFull: 0
HwIfOutQLen: 0

Querying SFP Port Information

You can verify SFP settings using ethtool -m . The following example shows the output for 1G and 10G
modules:

cumulus@switch:~# sudo ethtool -m | egrep '(swp|RXPower :|TXPower :

|EthernetComplianceCode)'

swp1: SFP detected

EthernetComplianceCodes : 1000BASE-LX
RXPower : -10.4479dBm
TXPower : 18.0409dBm
swp3: SFP detected
10GEthernetComplianceCode : 10G Base-LR
RXPower : -3.2532dBm
TXPower : -2.0817dBm

Caveats and Errata

Timeout Error on Quanta LY8 and LY9 Switches

On Quanta T5048-LY8 and T3048-LY9 switches, an "Operation timed out" error occurs while removing and
reinserting QSFP module.
The QSPFx2 module cannot be removed while the switch is powered on, as it is not hot-swappable.
[Link] 247
Cumulus Linux 3.5 User Guide

The QSPFx2 module cannot be removed while the switch is powered on, as it is not hot-swappable.
However, if this occurs, you can get the link to come up; however, this involves restarting switchd (see
page 193), which disrupts your network.
On the T3048-LY9, run the following commands:

cumulus@switch:~$ sudo echo 0 > qsfpd_power_enable/value

cumulus@switch:~$ sudo rmmod quanta_ly9_rangeley_platform
cumulus@switch:~$ sudo modprobe quanta_ly9_rangeley_platform
cumulus@switch:~$ sudo systemctl restart [Link]

On the T5048-LY8, run the following commands:

cumulus@switch:~$ sudo echo 0 > qsfpd_power_enable/value

cumulus@switch:~$ sudo systemctl restart [Link]

swp33 and swp34 Disabled on Some Switches

The front SFP+ ports (swp33 and swp34) are disabled in Cumulus Linux on the following switches:
Dell Z9100-ON
Penguin Arctica 3200-series switches (the 3200C, 3200XL and 3200XLP)
Supermicro SSE-C3632S
These ports appear as disabled in the /etc/cumulus/[Link] file.

ethtool Shows Incorrect Port Speed on 100G Mellanox Switches

After setting interface speed to 40G by editing the [Link] file on a Mellanox switch, ethtool still
shows the speed as 100G.
This is a known issue whereby ethtool does not update after restarting switchd, so it continues to
display the outdated port speed.
To correctly set the port speed, use NCLU (see page 85) or ethtool to set the speed instead of hand
editing the [Link] file.
For example, to set the speed to 40G using NCLU:

cumulus@switch:~$ net add interface swp1 link speed 40000

Or using ethtool:

cumulus@switch:~$ sudo ethtool -s swp1 speed 40000

248 23 January 2018

 Cumulus Networks

Related Information
Debian - Network Configuration
Linux Foundation - VLANs
Linux Foundation - Bridges
Linux Foundation - Bonds

Buffer and Queue Management

Hardware datapath configuration manages packet buffering, queueing and scheduling in hardware. There
are two configuration input files:
/etc/cumulus/datapath/[Link], which describes priority groups and assigns the
scheduling algorithm and weights
/usr/lib/python2.7/dist-packages/cumulus/__chip_config/[bcm|mlx]/datapath.
conf, which assigns buffer space and egress queues
Each packet is assigned to an ASIC Class of Service (CoS) value based on the packet's priority value stored in
the 802.1p (Class of Service) or DSCP (Differentiated Services Code Point) header field. The choice to
schedule packets based on COS or DSCP is a configurable option in the /etc/cumulus/datapath
/[Link] file.
Priority groups include:
Control: Highest priority traffic
Service: Second-highest priority traffic
Bulk: All remaining traffic
The scheduler is configured to use a hybrid scheduling algorithm. It applies strict priority to control traffic
queues and a weighted round robin selection from the remaining queues. Unicast packets and multicast
packets with the same priority value are assigned to separate queues, which are assigned equal scheduling
weights.
Datapath configuration takes effect when you initialize switchd. Changes to the [Link] file
require you to restart the switchd (see page 193)service.

You can configure Quality of Service (QoS) for switches on the Broadcom Helix4, Tomahawk,
Trident II+ and Trident II platforms, and the Mellanox Spectrum platform only.

Contents
This chapter covers ...
Commands (see page 250)
Example Configuration File (see page 250)
Configuring Traffic Marking through ACL Rules (see page 254)
Configuring Priority Flow Control (see page 255)
Understanding Port Groups (see page 257)

Configuring Link Pause (see page 258)

[Link] 249
Cumulus Linux 3.5 User Guide

Configuring Link Pause (see page 258)

Configuring Cut-through Mode and Store and Forward Switching (see page 259)
Configuring Explicit Congestion Notification (see page 260)
Related Information (see page 261)

Commands
If you modify the configuration in the /etc/cumulus/datapath/[Link] file, you must restart
switchd (see page 193)for the changes to take effect:

cumulus@switch:~$ sudo systemctl restart [Link]

Example Configuration File

The following example /etc/cumulus/datapath/[Link] datapath configuration file applies to
10G, 40G, and 100G switches on Broadcom Tomahawk, Trident II+ or Trident II and Mellanox Spectrum
platforms only. However, see the note above for all the supported ASICs and speeds.
Keep in mind the following about the configuration:
Regarding the default source packet fields and mapping, each selected packet field should have a
block of mapped values. Any packet field value that is not specified in the configuration is assigned
to a default internal switch priority. The configuration applies to every forwarding port unless a
custom remark configuration is defined for that port (see below).
Regarding the default remark packet fields and mapping, each selected packet field should have a
block of mapped values. Any internal switch priority value that is not specified in the configuration is
assigned to a default packet field value. The configuration applies to every forwarding port unless a
custom remark configuration is defined for that port (see below).
Per-port source packet fields and mapping apply to the designated set of ports.
Per-port remark packet fields and mapping apply to the designated set of ports.
Click to view sample [Link] file ...

cumulus@switch:~$ cat /etc/cumulus/datapath/[Link]

#
# /etc/cumulus/datapath/[Link]
#
# packet header field used to determine the packet priority
level
# fields include {802.1p, dscp}
traffic.packet_priority_source_set = [802.1p,
dscp]
# remark packet priority
value
# fields include {802.1p,
none}
# remark packet priority value
# fields include {802.1p, dscp}
traffic.packet_priority_remark_set = [802.1p,dscp]

250 23 January 2018

 Cumulus Networks

# packet priority remark values assigned from each internal cos value
# internal cos values {cos_0..cos_7}
# (internal cos 3 has been reserved for CPU-generated traffic)
#
# 802.1p values = {0..7}
traffic.cos_0.priority_remark.8021p = [1]
traffic.cos_1.priority_remark.8021p = [0]
traffic.cos_2.priority_remark.8021p = [3]
traffic.cos_3.priority_remark.8021p = [2]
traffic.cos_4.priority_remark.8021p = [4]
traffic.cos_5.priority_remark.8021p = [5]
traffic.cos_6.priority_remark.8021p = [7]
traffic.cos_7.priority_remark.8021p = [6]
# dscp values = {0..63}
traffic.cos_0.priority_remark.dscp = [1]
traffic.cos_1.priority_remark.dscp = [9]
traffic.cos_2.priority_remark.dscp = [17]
traffic.cos_3.priority_remark.dscp = [25]
traffic.cos_4.priority_remark.dscp = [33]
traffic.cos_5.priority_remark.dscp = [41]
traffic.cos_6.priority_remark.dscp = [49]
traffic.cos_7.priority_remark.dscp = [57]
# Per-port remark packet fields and mapping: applies to the
designated set of ports.
remark.port_group_list = [remark_port_group]
remark.remark_port_group.packet_priority_remark_set = [802.1p,dscp]
remark.remark_port_group.port_set = swp1-swp4,swp6
remark.remark_port_group.cos_0.priority_remark.dscp = [2]
remark.remark_port_group.cos_1.priority_remark.dscp = [10]
remark.remark_port_group.cos_2.priority_remark.dscp = [18]
remark.remark_port_group.cos_3.priority_remark.dscp = [26]
remark.remark_port_group.cos_4.priority_remark.dscp = [34]
remark.remark_port_group.cos_5.priority_remark.dscp = [42]
remark.remark_port_group.cos_6.priority_remark.dscp = [50]
remark.remark_port_group.cos_7.priority_remark.dscp =
[58]
# packet priority values assigned to each internal cos
value
# internal cos values {cos_0..
cos_7}
# (internal cos 3 has been reserved for CPU-generated traffic)
#
# 802.1p values = {0..7}
traffic.cos_0.priority_source.8021p = [0]
traffic.cos_1.priority_source.8021p = [1]
traffic.cos_2.priority_source.8021p = [2]
traffic.cos_3.priority_source.8021p = []
traffic.cos_4.priority_source.8021p = [3,4]
traffic.cos_5.priority_source.8021p = [5]
traffic.cos_6.priority_source.8021p = [6]
traffic.cos_7.priority_source.8021p = [7]
# dscp values = {0..63}

[Link] 251
Cumulus Linux 3.5 User Guide

traffic.cos_0.priority_source.dscp = [0,1,2,3,4,5,6,7]
traffic.cos_1.priority_source.dscp = [8,9,10,11,12,13,14,15]
traffic.cos_2.priority_source.dscp = []
traffic.cos_3.priority_source.dscp = []
traffic.cos_4.priority_source.dscp = []
traffic.cos_5.priority_source.dscp = []
traffic.cos_6.priority_source.dscp = []
traffic.cos_7.priority_source.dscp =
[56,57,58,59,60,61,62,63]
# Per-port source packet fields and mapping: applies to the
designated set of ports.
source.port_group_list = [source_port_group]
source.source_port_group.packet_priority_source_set = [802.1p,dscp]
source.source_port_group.port_set = swp1-swp4,swp6
source.source_port_group.cos_0.priority_source.8021p = [7]
source.source_port_group.cos_1.priority_source.8021p = [6]
source.source_port_group.cos_2.priority_source.8021p = [5]
source.source_port_group.cos_3.priority_source.8021p = [4]
source.source_port_group.cos_4.priority_source.8021p = [3]
source.source_port_group.cos_5.priority_source.8021p = [2]
source.source_port_group.cos_6.priority_source.8021p = [1]
source.source_port_group.cos_7.priority_source.8021p = [0]
# priority groups
traffic.priority_group_list = [control, service, bulk]
# internal cos values assigned to each priority group
# each cos value should be assigned exactly once
# internal cos values {0..7}
priority_group.control.cos_list = [7]
priority_group.service.cos_list = [2]
priority_group.bulk.cos_list = [0,1,3,4,5,6]
# to configure priority flow control on a group of ports:
# -- assign cos value(s) to the cos list
# -- add or replace a port group names in the port group list
# -- for each port group in the list
# -- populate the port set, e.g.
# swp1-swp4,swp8,swp50s0-swp50s3
# -- set a PFC buffer size in bytes for each port in the group
# -- set the xoff byte limit (buffer limit that triggers PFC frame
transmit to start)
# -- set the xon byte delta (buffer limit that triggers PFC frame
transmit to stop)
# -- enable PFC frame transmit and/or PFC frame receive
# priority flow control
# pfc.port_group_list = [pfc_port_group]
# pfc.pfc_port_group.cos_list = []
# pfc.pfc_port_group.port_set = swp1-swp4,swp6
# pfc.pfc_port_group.port_buffer_bytes = 25000
# pfc.pfc_port_group.xoff_size = 10000
# pfc.pfc_port_group.xon_delta = 2000
# pfc.pfc_port_group.tx_enable = true
# pfc.pfc_port_group.rx_enable = true
# to configure pause on a group of ports:

252 23 January 2018

 Cumulus Networks

# -- add or replace port group names in the port group list

# -- for each port group in the list
# -- populate the port set, e.g.
# swp1-swp4,swp8,swp50s0-swp50s3
# -- set a pause buffer size in bytes for each port in the group
# -- set the xoff byte limit (buffer limit that triggers pause
frames transmit to start)
# -- set the xon byte delta (buffer limit that triggers pause
frames transmit to stop)
# link pause
# link_pause.port_group_list = [pause_port_group]
# link_pause.pause_port_group.port_set = swp1-swp4,swp6
# link_pause.pause_port_group.port_buffer_bytes = 25000
# link_pause.pause_port_group.xoff_size = 10000
# link_pause.pause_port_group.xon_delta = 2000
# link_pause.pause_port_group.rx_enable = true
# link_pause.pause_port_group.tx_enable = true
# scheduling algorithm: algorithm values = {dwrr}
[Link] = dwrr
# traffic group scheduling weight
# weight values = {0..127}
# '0' indicates strict priority
priority_group.[Link] = 0
priority_group.[Link] = 32
priority_group.[Link] = 16
# To turn on/off Denial of service (DOS) prevention checks
dos_enable = false
# Cut-through is disabled by default on all chips with the exception
of
# Spectrum. On Spectrum cut-through cannot be disabled.
#cut_through_enable = false
# Enable resilient hashing
#resilient_hash_enable = FALSE
# Resilient hashing flowset entries per ECMP group
# Valid values - 64, 128, 256, 512, 1024
#resilient_hash_entries_ecmp = 128
# Enable symmetric hashing
#symmetric_hash_enable = TRUE
# Set sflow/sample ingress cpu packet rate and burst in packets/sec
# Values: {0..16384}
#[Link] = 16384
#[Link] = 16384
#Specify the maximum number of paths per route entry.
# Maximum paths supported is 200.
# Default value 0 takes the number of physical ports as the max path
size.
#ecmp_max_paths = 0
#Specify the hash seed for Equal cost multipath entries
# Default value 0
# Value Rang: {0..4294967295}
#ecmp_hash_seed = 42
# Specify the forwarding table resource allocation profile, applicable

[Link] 253
Cumulus Linux 3.5 User Guide

# only on platforms that support universal forwarding resources.

#
# /usr/cumulus/sbin/cl-rsource-query reports the allocated table sizes
# based on the profile setting.
#
# Values: one of {'default', 'l2-heavy', 'v4-lpm-heavy', 'v6-lpm-
heavy'}
# Default value: 'default'
# Note: some devices may support more modes, please consult user
# guide for more details
#
#forwarding_table.profile = default

On Mellanox Spectrum switches, packet priority remark must be enabled on the ingress port. A
packet received on a remark-enabled port is remarked according to the priority mapping
configured on the egress port. If packet priority remark is configured the same way on every port,
the default configuration example above is correct. However, per-port customized configurations
require two port groups: one for the ingress ports and one for the egress ports, as below:

remark.port_group_list = [ingress_remark_group,
egress_remark_group]
remark.ingress_remark_group.packet_priority_remark_set = [dscp]
remark.remark_port_group.port_set = swp1-swp4,swp6
remark.egress_remark_group.port_set = swp10-swp20
remark.egress_remark_group.cos_0.priority_remark.dscp = [2]
remark.egress_remark_group.cos_1.priority_remark.dscp = [10]
remark.egress_remark_group.cos_2.priority_remark.dscp = [18]
remark.egress_remark_group.cos_3.priority_remark.dscp = [26]
remark.egress_remark_group.cos_4.priority_remark.dscp = [34]
remark.egress_remark_group.cos_5.priority_remark.dscp = [42]
remark.egress_remark_group.cos_6.priority_remark.dscp = [50]
remark.egress_remark_group.cos_7.priority_remark.dscp = [58]

Configuring Traffic Marking through ACL Rules

You can mark traffic for egress packets through iptables or ip6tables rule classifications. To enable
these rules, you do one of the following:
Mark DSCP values in egress packets.
Mark 802.1p CoS values in egress packets.
To enable traffic marking, use cl-acltool. Add the -p option to specify the location of the policy file. By
default, if you don't include the -p option, cl-acltool looks for the policy file in /etc/cumulus/acl
/policy.d/.
The iptables-/ip6tables-based marking is supported via the following action extension:

254 23 January 2018

 Cumulus Networks

-j SETQOS --set-dscp 10 --set-cos 5

For ebtables, the setqos keyword must be in lowercase, as in:

[ebtables]
-A FORWARD -o swp5 -j setqos --set-cos 5

You can specify one of the following targets for SETQOS/setqos:

Option Description

--set-cos Sets the datapath resource/queuing class value. Values are defined in IEEE_P802.1p.
INT

--set-dscp Sets the DSCP field in packet header to a value, which can be either a decimal or hex value.
value

--set-dscp- Sets the DSCP field in the packet header to the value represented by the DiffServ class
class class value. This class can be EF, BE or any of the CSxx or AFxx classes.

You can specify either --set-dscp or --set-dscp-class, but not both.

Here are two example rules:

[iptables]
-t mangle -A FORWARD --in-interface swp+ -p tcp --dport bgp -j SETQOS
--set-dscp 10 --set-cos 5

[ip6tables]
-t mangle -A FORWARD --in-interface swp+ -j SETQOS --set-dscp 10

You can put the rule in either the mangle table or the default filter table; the mangle table and filter table
are put into separate TCAM slices in the hardware.
To put the rule in the mangle table, include -t mangle; to put the rule in the filter table, omit -t mangle.

Configuring Priority Flow Control

Priority flow control, as defined in the IEEE 802.1Qbb standard, provides a link-level flow control mechanism
that can be controlled independently for each Class of Service (CoS) with the intention to ensure no data
frames are lost when congestion occurs in a bridged network.

PFC is not supported on switches with the Helix4 ASIC.

[Link] 255
Cumulus Linux 3.5 User Guide

PFC is a layer 2 mechanism that prevents congestion by throttling packet transmission. When PFC is
enabled for received packets on a set of switch ports, the switch detects congestion in the ingress buffer of
the receiving port and signals the upstream switch to stop sending traffic. If the upstream switch has PFC
enabled for packet transmission on the designated priorities, it responds to the downstream switch and
stops sending those packets for a period of time.
PFC operates between two adjacent neighbor switches; it does not provide end-to-end flow control.
However, when an upstream neighbor throttles packet transmission, it could build up packet congestion
and propagate PFC frames further upstream: eventually the sending server could receive PFC frames and
stop sending traffic for a time.
The PFC mechanism can be enabled for individual switch priorities on specific switch ports for RX and/or TX
traffic. The switch port’s ingress buffer occupancy is used to measure congestion. If congestion is present,
the switch transmits flow control frames to the upstream switch. Packets with priority values that do not
have PFC configured are not counted during congestion detection; neither do they get throttled by the
upstream switch when it receives flow control frames.
PFC congestion detection is implemented on the switch using xoff and xon threshold values for the specific
ingress buffer which is used by the targeted switch priorities. When a packet enters the buffer and the
buffer occupancy is above the xoff threshold, the switch transmits an Ethernet PFC frame to the upstream
switch to signal packet transmission should stop. When the buffer occupancy drops below the xon
threshold, the switch sends another PFC frame upstream to signal that packet transmission can resume.
(PFC frames contain a quanta value to indicate a timeout value for the upstream switch: packet
transmission can resume after the timer has expired, or when a PFC frame with quanta == 0 is received
from the downstream switch.)
After the downstream switch has sent a PFC frame upstream, it continues to receive packets until the
upstream switch receives and responds to the PFC frame. The downstream ingress buffer must be large
enough to store those additional packets after the xoff threshold has been reached.

Before Cumulus Linux 3.1.1, PFC was designated as a lossless priority group. The lossless priority
group has been removed from Cumulus Linux.

Priority flow control is fully supported on both Broadcom and Mellanox switches.
PFC is disabled by default in Cumulus Linux. Enabling priority flow control (PFC) requires configuring the
following settings in /etc/cumulus/datapath/[Link] on the switch:
Specifying the name of the port group in pfc.port_group_list in brackets; for example, pfc.
port_group_list = [pfc_port_group].
Assigning a CoS value to the port group in pfc.pfc_port_group.cos_list setting. Note that
pfc_port_group is the name of a port group you specified above and is used throughout the following
settings.
Populating the port group with its member ports in pfc.pfc_port_group.port_set.
Setting a PFC buffer size in pfc.pfc_port_group.port_buffer_bytes. This is the maximum
number of bytes allocated for storing bursts of packets, guaranteed at the ingress port. The default
is 25000 bytes.
Setting the xoff byte limit in pfc.pfc_port_group.xoff_size. This is a threshold for the PFC
buffer; when this limit is reached, an xoff transition is initiated, signaling the upstream port to stop
sending traffic, during which time packets continue to arrive due to the latency of the
communication. The default is 10000 bytes.

256 23 January 2018

 Cumulus Networks

Setting the xon delta limit in pfc.pfc_port_group.xon_delta. This is the number of bytes to
subtract from the xoff limit, which results in a second threshold at which the egress port resumes
sending traffic. After the xoff limit is reached and the upstream port stops sending traffic, the buffer
begins to drain. When the buffer reaches 8000 bytes (assuming default xoff and xon settings), the
egress port signals that it can start receiving traffic again. The default is 2000 bytes.

Enabling the egress port to signal the upstream port to stop sending traffic (pfc.
pfc_port_group.tx_enable). The default is true.
Enabling the egress port to receive notifications and act on them (pfc.pfc_port_group.
rx_enable). The default is true.
The switch priority value(s) are mapped to the specific ingress buffer for each targeted switch port.
Cumulus Linux looks at either the 802.1p bits or the IP layer DSCP bits depending on which is
configured in the [Link] file to map packets to internal switch priority values.
The following configuration example shows PFC configured for ports swp1 through swp4 and swp6:

# to configure priority flow control on a group of ports:

# -- assign cos value(s) to the cos list
# -- add or replace a port group names in the port group list
# -- for each port group in the list
# -- populate the port set, e.g.
# swp1-swp4,swp8,swp50s0-swp50s3
# -- set a PFC buffer size in bytes for each port in the group
# -- set the xoff byte limit (buffer limit that triggers PFC frame
transmit to start)
# -- set the xon byte delta (buffer limit that triggers PFC frame
transmit to stop)
# -- enable PFC frame transmit and/or PFC frame receive
# priority flow control
pfc.port_group_list = [pfc_port_group]
pfc.pfc_port_group.cos_list = []
pfc.pfc_port_group.port_set = swp1-swp4,swp6
pfc.pfc_port_group.port_buffer_bytes = 25000
pfc.pfc_port_group.xoff_size = 10000
pfc.pfc_port_group.xon_delta = 2000
pfc.pfc_port_group.tx_enable = true
pfc.pfc_port_group.rx_enable = true

Understanding Port Groups

A port group refers to one or more sequences of contiguous ports. Multiple port groups can be defined by:
Adding a comma-separated list of port group names to the port_group_list.
Adding the port_set, rx_enable, and tx_enable configuration lines for each port group.
You can specify the set of ports in a port group in comma-separated sequences of contiguous ports; you
can see which ports are contiguous in /var/lib/cumulus/porttab. The syntax supports:
A single port (swp1s0 or swp5)
A sequence of regular swp ports (swp2-swp5)

A sequence within a breakout swp port (swp6s0-swp6s3)

[Link] 257
Cumulus Linux 3.5 User Guide

A sequence within a breakout swp port (swp6s0-swp6s3)

A sequence of regular and breakout ports, provided they are all in a contiguous range. For example:

...
swp2
swp3
swp4
swp5
swp6s0
swp6s1
swp6s2
swp6s3
swp7
...

Restart switchd (see page 193)to allow the PFC configuration changes to take effect:

cumulus@switch:~$ sudo systemctl restart [Link]

Configuring Link Pause

The PAUSE frame is a flow control mechanism that halts the transmission of the transmitter for a specified
period of time. A server or other network node within the data center may be receiving traffic faster than it
can handle it, thus the PAUSE frame. In Cumulus Linux, individual ports can be configured to execute link
pause by:
Transmitting pause frames when its ingress buffers become congested (TX pause enable) and/or
Responding to received pause frames (RX pause enable).
Link pause is disabled by default. Enabling link pause requires configuring settings in /etc/cumulus
/datapath/[Link], similar to how you configure priority flow control (see page 249). The settings
are explained in that section as well.
Here is an example configuration which turns of both types of link pause for swp1 through swp4 and swp6:

# to configure pause on a group of ports:

# -- add or replace port group names in the port group list
# -- for each port group in the list
# -- populate the port set, e.g.
# swp1-swp4,swp8,swp50s0-swp50s3
# -- set a pause buffer size in bytes for each port in the group
# -- set the xoff byte limit (buffer limit that triggers pause
frames transmit to start)
# -- set the xon byte delta (buffer limit that triggers pause
frames transmit to stop)

# link pause
link_pause.port_group_list = [pause_port_group]

258 23 January 2018

 Cumulus Networks

link_pause.pause_port_group.port_set = swp1-swp4,swp6
link_pause.pause_port_group.port_buffer_bytes = 25000
link_pause.pause_port_group.xoff_size = 10000
link_pause.pause_port_group.xon_delta = 2000
link_pause.pause_port_group.rx_enable = true
link_pause.pause_port_group.tx_enable = true

Restart switchd (see page 193)to allow link pause configuration changes to take effect:

cumulus@switch:~$ sudo systemctl restart [Link]

Configuring Cut-through Mode and Store and Forward Switching

Cut-through mode is disabled in Cumulus Linux by default on switches with Broadcom ASICs. With cut-
though mode enabled and link pause is asserted, Cumulus Linux generates a TOVR and TUFL ERROR;
certain error counters increment on a given physical port.

cumulus@switch:~$ sudo ethtool -S swp49 | grep Error

HwIfInDot3LengthErrors: 0
HwIfInErrors: 0
HwIfInDot3FrameErrors: 0
SoftInErrors: 0
SoftInFrameErrors: 0
HwIfOutErrors: 35495749
SoftOutErrors: 0

cumulus@switch:~$ sudo ethtool -S swp50 | grep Error

HwIfInDot3LengthErrors: 3038098
HwIfInErrors: 297595762
HwIfInDot3FrameErrors: 293710518

To work around this issue, disable link pause or disable cut-through mode in /etc/cumulus/datapath
/[Link].
To disable link pause, comment out the link_pause* section in /etc/cumulus/datapath/traffic.
conf:

cumulus@switch:~$ sudo nano /etc/cumulus/datapath/[Link]

#link_pause.port_group_list = [port_group_0]
#link_pause.port_group_0.port_set = swp45-swp54
#link_pause.port_group_0.rx_enable = true
#link_pause.port_group_0.tx_enable = true

To enable store and forward switching, set cut_through_enable to false in /etc/cumulus/datapath

/[Link]:

[Link] 259
Cumulus Linux 3.5 User Guide

cumulus@switch:~$ sudo nano /etc/cumulus/datapath/[Link]

cut_through_enable = false

Configuring Explicit Congestion Notification

Explicit Congestion Notification (ECN) is defined by RFC 3168. ECN gives a Cumulus Linux switch the ability to
mark a packet to signal impending congestion instead of dropping the packet outright, which is how TCP
typically behaves when ECN is not enabled.
ECN is a layer 3 end-to-end congestion notification mechanism only. Packets can be marked as ECN-capable
transport (ECT) by the sending server. If congestion is observed by any switch while the packet is getting
forwarded, the ECT-enabled packet can be marked by the switch to indicate the congestion. The end
receiver can respond to the ECN-marked packets by signaling the sending server to slow down
transmission. The sending server marks a packet ECT by setting the least 2 significant bits in an IP header
DiffServ (ToS) field to 01 or 10. A packet that has the least 2 significant bits set to 00 indicates a non-ECT-
enabled packet.
The ECN mechanism on a switch only marks packets to notify the end receiver. It does not take any other
action or change packet handling in any way, nor does it respond to packets that have already been marked
ECN by an upstream switch.

On Trident II switches only, if ECN is enabled on a specific queue, the ASIC also enables WRED
on the same queue. If the packet is ECT marked (the ECN bits are 01 or 10), the ECN mechanism
executes as described above. However, if it is entering an ECN-enabled queue but is not ECT
marked (the ECN bits are 00), then the WRED mechanism uses the same threshold and
probability values to decide whether to drop the packet. Packets entering a non-ECN-enabled
queue do not get marked or dropped due to ECN or WRED in any case.

ECN is implemented on the switch using minimum and maximum threshold values for the egress queue
length. When a packet enters the queue and the average queue length is between the minimum and
maximum threshold values, a configurable probability value will determine whether the packet will be
marked. If the average queue length is above the maximum threshold value, the packet is always marked.
The downstream switches with ECN enabled perform the same actions as the traffic is received. If the ECN
bits are set, they remain set. The only way to overwrite ECN bits is to enable it — that is, set the ECN bits to
11.
ECN is supported on Broadcom Tomahawk, Trident II+ and Trident II, and Mellanox Spectrum switches only.
Click to learn how to configure ECN ...
ECN is disabled by default in Cumulus Linux. You can enable ECN for individual switch priorities on specific
switch ports. ECN requires configuring the following settings in /etc/cumulus/datapath/traffic.
conf on the switch:
Specifying the name of the port group in ecn.port_group_list in brackets; for example, ecn.
port_group_list = [ecn_port_group].
Assigning a CoS value to the port group in ecn.ecn_port_group.cos_list. If the CoS value of a
packet matches the value of this setting, then ECN is applied. Note that ecn_port_group is the name
of a port group you specified above.
Populating the port group with its member ports (ecn.ecn_port_group.port_set), where
ecn_port_group is the name of the port group you specified above. Congestion is measured on the
egress port queue for the ports listed here, using the average queue length: if congestion is present,
a packet entering the queue may be marked to indicate that congestion was observed. Marking a
packet involves setting the least 2 significant bits in the IP header DiffServ (ToS) field to 11.

260 23 January 2018

 Cumulus Networks

The switch priority value(s) are mapped to specific egress queues for the target switch ports.
The ecn.ecn_port_group.probability value indicates the probability of a packet being
marked if congestion is experienced.
The following configuration example shows ECN configured for ports swp1 through swp4 and swp6:

# Explicit Congestion Notification

# to configure ECN on a group of ports:
# -- add or replace port group names in the port group list
# -- assign cos value(s) to the cos list *ECN will only be applied
to traffic matching this COS*
# -- for each port group in the list
# -- populate the port set, e.g.
# swp1-swp4,swp8,swp50s0-swp50s3
ecn.port_group_list = [ecn_port_group]
ecn.ecn_port_group.cos_list = [0]
ecn.ecn_port_group.port_set = swp1-swp4,swp6
ecn.ecn_port_group.min_threshold_bytes = 40000
ecn.ecn_port_group.max_threshold_bytes = 200000
ecn.ecn_port_group.probability = 100

Restart switchd (see page 193)to allow the ECN configuration changes to take effect:

cumulus@switch:~$ sudo systemctl restart [Link]

Related Information
iptables-extensions man page

Configuring Hardware-enabled DDOS Protection

The DDOS protection mechanism protects data plane, control plane and management plane traffic in the
switch. It drops any packets that match one or more of the following criteria while incurring no
performance impact:
Source IP address matches the destination address for IPv4 and IPv6 packets
Source MAC address matches the destination MAC address
Unfragmented or first fragment SYN packets with a source port of 0-1023
TCP packets with control flags =0 and seq number == 0
TCP packets with FIN, URG and PSH bits set and seq number == 0
TCP packets with both SYN and FIN bits set
TCP source PORT matches the destination PORT
UDP source PORT matches the destination PORT
First TCP fragment with partial TCP header
TCP header has fragment offset value of 1

ICMPv6 ping packets payload larger than programmed value of ICMP max size
[Link] 261
Cumulus Linux 3.5 User Guide

ICMPv6 ping packets payload larger than programmed value of ICMP max size
ICMPv4 ping packets payload larger than programmed value of ICMP max size
Fragmented ICMP packet
IPv6 fragment lower than programmed minimum IPv6 packet size

This configuration option is only available for Broadcom Trident, Trident II, and Tomahawk
chipsets.

Cumulus Networks recommends enabling this feature when deploying a switch with the above mentioned
ASICs, as hardware-based DDOS protection is disabled by default. Although Cumulus recommends
enabling all of the above criteria, they can be individually enabled if desired.

Configure Persistent DDOS Protection

1. Open the /etc/cumulus/datapath/[Link] file in a text editor.
2. Enable DOS prevention checks by changing the following value to true, and save the file:

# To turn on/off Denial of Service (DOS) prevention checks

dos_enable = true

3. Open the /usr/lib/python2.7/dist-packages/cumulus/__chip_config/bcm/datapath.

conf file in a text editor.
4. Set the following checks to true, and save the file:

# Enabling/disabling Denial of service (DOS) prevetion checks

# To change the default configuration:
# enable/disable the individual DOS checks.
dos.sip_eq_dip = true
dos.smac_eq_dmac = true
dos.tcp_hdr_partial = true
dos.tcp_syn_frag = true
dos.tcp_ports_eq = true
dos.tcp_flags_syn_fin = true
dos.tcp_flags_fup_seq0 = true
dos.tcp_offset1 = true
dos.tcp_ctrl0_seq0 = true
dos.udp_ports_eq = true
dos.icmp_frag = true
dos.icmpv4_length = true
dos.icmpv6_length = true
dos.ipv6_min_frag = true

5. Restart switchd to enable DOS protection:

262 23 January 2018

 5.

Cumulus Networks

cumulus@switch:~$ sudo systemctl restart [Link]

Configuring DHCP Relays

You can configure DHCP relays for IPv4 and IPv6.
To run DHCP for both IPv4 and IPv6, initiate the DHCP relay once for IPv4 and once for IPv6. Following are
the configurations on the server hosts, DHCP relay and DHCP server using the following topology:

The dhcpd and dhcrelay services are disabled by default. After you finish configuring the DHCP
relays and servers, you need to start those services.

Contents
This chapter covers ...
Configuring IPv4 DHCP Relays (see page 263)
Using DHCP Option 82 (see page 265)
Configuring IPv6 DHCP Relays (see page 265)
Configuring Multiple DHCP Relays (see page 265)
Configuring a DHCP Relay with VRR (see page 266)
Configuring the DHCP Relay Service Manually (Advanced) (see page 268)
Troubleshooting the DHCP Relays (see page 268)
Looking at the Log on Switch where DHCP Relay Is Configured (see page 269)

Configuring IPv4 DHCP Relays

Configure isc-dhcp-relay using NCLU (see page 85), specifying the IP addresses to each DHCP server
and the interfaces that are used as the uplinks.
In the examples below, the DHCP server IP address is [Link], VLAN 1 (the SVI is vlan1) and the
uplinks are swp51 and swp52.

[Link] 263
Cumulus Linux 3.5 User Guide

You configure a DHCP relay on a per-VLAN basis, specifying the SVI, not the parent bridge — in
our example, you would specify vlan1 as the SVI for VLAN 1; do not specify the bridge named
bridge in this case.
As per RFC 3046, you can specify as many server IP addresses that can fit in 255 octets, specifying
each address only once.

cumulus@leaf01:~$ net add dhcp relay interface swp51

cumulus@leaf01:~$ net add dhcp relay interface swp52
cumulus@leaf01:~$ net add dhcp relay interface vlan1
cumulus@leaf01:~$ net add dhcp relay server [Link]
cumulus@leaf01:~$ net pending
cumulus@leaf01:~$ net commit

These commands create the following configuration in the /etc/default/isc-dhcp-relay file:

cumulus@leaf01:~$ cat /etc/default/isc-dhcp-relay

SERVERS="[Link]"
INTF_CMD="-i vlan1 -i swp51 -i swp52"
OPTIONS=""

After you've finished configuring the DHCP relay, restart then enable the dhcrelay service so the
configuration persists between reboots:

cumulus@leaf01:~$ sudo systemctl restart [Link]

cumulus@leaf01:~$ sudo systemctl enable [Link]

To see the status of the DHCP relay, use the systemctl status [Link] command:

cumulus@leaf01:~$ sudo systemctl status [Link]

[Link] - DHCPv4 Relay Agent Daemon
Loaded: loaded (/lib/systemd/system/[Link]; enabled)
Active: active (running) since Fri 2016-12-02 [Link] UTC; 2min
16s ago
Docs: man:dhcrelay(8)
Main PID: 1997 (dhcrelay)
CGroup: /[Link]/[Link]
1997 /usr/sbin/dhcrelay --nl -d -q -i vlan1 -i swp51 -i
swp52 [Link]

264 23 January 2018

 Cumulus Networks

Using DHCP Option 82

DHCP relays can be configured to inject the circuit-id field with the -a option, which you add to the
OPTIONS line in /etc/default/isc-dhcp-relay. By default, the ingress SVI interface that the relayed
DHCP discover packet is processed against is injected into this field. You can change this behavior by
adding the --use-pif-circuit-id option. With this option, the physical switch port (swp) that the
discover packet arrives on is placed in the circuit-id field.

Configuring IPv6 DHCP Relays

If you're configuring IPv6, the /etc/default/isc-dhcp-relay6 variables file has a different format than
the /etc/default/isc-dhcp-relay file for IPv4 DHCP relays. Make sure to configure the variables
appropriately by editing this file.

You cannot use NCLU to configure IPv6 relays.

cumulus@leaf01:$ sudo nano /etc/default/isc-dhcp-relay6

SERVERS=" -u [Link]%swp51 -u [Link]%swp52"
INTF_CMD="-l vlan1"

After you've finished configuring the DHCP relay, save your changes, restart the dhcrelay6 service, then
enable the dhcrelay6 service so the configuration persists between reboots:

cumulus@leaf01:~$ sudo systemctl restart [Link]

cumulus@leaf01:~$ sudo systemctl enable [Link]

To see the status of the IPv6 DHCP relay, use the systemctl status [Link] command:

cumulus@leaf01:~$ sudo systemctl status [Link]

[Link] - DHCPv6 Relay Agent Daemon
Loaded: loaded (/lib/systemd/system/[Link]; disabled)
Active: active (running) since Fri 2016-12-02 [Link] UTC; 1s ago
Docs: man:dhcrelay(8)
Main PID: 6152 (dhcrelay)
CGroup: /[Link]/[Link]
6152 /usr/sbin/dhcrelay -6 --nl -d -q -l vlan1 -u [Link]
100::2 swp51 -u [Link] swp52

Configuring Multiple DHCP Relays

Cumulus Linux supports configuring multiple DHCP relay daemons on a switch, to enable relaying of
packets from different bridges to different upstreams.

1. As the sudo user, open /etc/vrf/[Link] in a text editor, and remove dhcrelay.
[Link] 265
Cumulus Linux 3.5 User Guide

1. As the sudo user, open /etc/vrf/[Link] in a text editor, and remove dhcrelay.
2. Run the following command to reload the systemd files:

cumulus@switch:~$ sudo systemctl daemon-reload

3. Create a config file in /etc/default using the following format for each dhcrelay: isc-dhcp-
relay-<dhcp-name>. An example file can be seen below:

# Defaults for isc-dhcp-relay initscript# sourced by /etc/init.d

/isc-dhcp-relay
# installed at /etc/default/isc-dhcp-relay by the maintainer
scripts
#
# This is a POSIX shell fragment
#
# What servers should the DHCP relay forward requests to?
SERVERS="[Link]"
# On what interfaces should the DHCP relay (dhrelay) serve DHCP
requests?
# Always include the interface towards the DHCP server.
# This variable requires a -i for each interface configured
above.
# This will be used in the actual dhcrelay command
# For example, "-i eth0 -i eth1"
INTF_CMD="-i swp2s2 -i swp2s3"
# Additional options that are passed to the DHCP relay daemon?
OPTIONS=""

4. Run the following command to start a dhcrelay instance, replacing dhcp-name with the instance
name or number:

cumulus@switch:~$ sudo systemctl start dhcrelay@<dhcp-name>

Configuring a DHCP Relay with VRR

If a DHCP relay is configured and you want to enable virtual router redundancy (VRR) (see page 385) on the
SVI, then you must include the VRR interface in the INTF_CMD field in the /etc/default/isc-dhcp-
relay file. For example:

cumulus@switch:~$ net add bridge

cumulus@switch:~$ net add vlan 500 ip address [Link]/24
cumulus@switch:~$ net add vlan 500 ip address-virtual [Link]
01 [Link]/24
cumulus@switch:~$ net add dhcp relay interface vlan500
cumulus@switch:~$ net add dhcp relay interface vlan500-v0

266 23 January 2018

 Cumulus Networks

cumulus@switch:~$ net add dhcp relay server [Link]

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration in the /etc/network/interfaces file:

cumulus@switch:~$ cat /etc/network/interfaces

...

auto bridge
iface bridge
bridge-vids 500
bridge-vlan-aware yes

auto vlan500
iface vlan500
address [Link]/24
address-virtual [Link] [Link]/24
vlan-id 500
vlan-raw-device bridge

auto vlan500-v0
iface vlan500-v0

They also create the following configuration in the /etc/default/isc-dhcp-relay file:

cumulus@leaf02:mgmt-vrf:~$ cat /etc/default/isc-dhcp-relay

# Defaults for isc-dhcp-relay initscript
# sourced by /etc/init.d/isc-dhcp-relay
# installed at /etc/default/isc-dhcp-relay by the maintainer scripts
#
# This is a POSIX shell fragment
#

# What servers should the DHCP relay forward requests to?

SERVERS="[Link]"

# On what interfaces should the DHCP relay (dhrelay) serve DHCP

requests?
# Always include the interface towards the DHCP server.
# This variable requires a -i for each interface configured above.
# This will be used in the actual dhcrelay command
# For example, "-i eth0 -i eth1"

[Link] 267
Cumulus Linux 3.5 User Guide

INTF_CMD="-i vlan500 -i vlan500-v0"

# Additional options that are passed to the DHCP relay daemon?

OPTIONS=""

Configuring the DHCP Relay Service Manually (Advanced)

Configuring the DHCP service manually ...
By default, Cumulus Linux configures the DHCP relay service automatically. However, in older versions of
Cumulus Linux, you needed to edit the [Link] file as described below. The IPv4 dhcrelay.
service Unit script calls /etc/default/isc-dhcp-relay to find launch variables.

cumulus@switch:~$ cat /lib/systemd/system/[Link]

[Unit]
Description=DHCPv4 Relay Agent Daemon
Documentation=man:dhcrelay(8)
After=[Link] [Link] [Link]
[Service]
Type=simple
EnvironmentFile=-/etc/default/isc-dhcp-relay
# Here, we are expecting the INTF_CMD to contain
# the -i for each interface specified,
# e.g. "-i eth0 -i swp1"
ExecStart=/usr/sbin/dhcrelay -d -q $INTF_CMD $SERVERS $OPTIONS
[Install]
WantedBy=[Link]

The /etc/default/isc-dhcp-relay variables file needs to reference both interfaces participating in

DHCP relay (facing the server and facing the client) and the IP address of the server. If the client-facing
interface is a bridge port, specify the switch virtual interface (SVI) name if using a VLAN-aware bridge (see
page 327) (for example, vlan100), or the bridge name if using traditional bridging (for example, br100).

Troubleshooting the DHCP Relays

If you are experiencing issues with the DHCP relay, you can run the following commands to determine
whether or not the issue is with systemd. The following commands manually activate the DHCP relay
process, and they do not persist when you reboot the switch:

cumulus@switch:~$ /usr/sbin/dhcrelay -4 -i <interface_facing_host>

<ip_address_dhcp_server> -i <interface_facing_dhcp_server>
cumulus@switch:~$ /usr/sbin/dhcrelay -6 -l <interface_facing_host> -u
<ip_address_dhcp_server>%<interface_facing_dhcp_server>

For example:

cumulus@leaf01:~$ /usr/sbin/dhcrelay -4 -i vlan1 [Link] -i swp51

268 23 January 2018

 Cumulus Networks

cumulus@leaf01:~$ /usr/sbin/dhcrelay -6 -l vlan1 -u [Link]%

swp51

See man dhcrelay for more information.

Looking at the Log on Switch where DHCP Relay Is Configured

Use the journalctl command to look at the behavior on the Cumulus Linux switch that is providing the
DHCP relay functionality:

cumulus@leaf01:~$ sudo journalctl -l -n 20 | grep dhcrelay

Dec 05 [Link] leaf01 dhcrelay[6152]: sending upstream swp52
Dec 05 [Link] leaf01 dhcrelay[6152]: sending upstream swp51
Dec 05 [Link] leaf01 dhcrelay[6152]: Relaying Reply to fe80::4638:
[Link] port 546 down.
Dec 05 [Link] leaf01 dhcrelay[6152]: Relaying Reply to fe80::4638:
[Link] port 546 down.
Dec 05 [Link] leaf01 dhcrelay[6152]: Relaying Renew from fe80::4638:
[Link] port 546 going up.
Dec 05 [Link] leaf01 dhcrelay[6152]: sending upstream swp52
Dec 05 [Link] leaf01 dhcrelay[6152]: sending upstream swp51
Dec 05 [Link] leaf01 dhcrelay[6152]: Relaying Reply to fe80::4638:
[Link] port 546 down.
Dec 05 [Link] leaf01 dhcrelay[6152]: Relaying Reply to fe80::4638:
[Link] port 546 down.

You can run the command journalctl command with the --since flag to specify a time period:

cumulus@leaf01:~$ sudo journalctl -l --since "2 minutes ago" | grep

dhcrelay
Dec 05 [Link] leaf01 dhcrelay[6152]: Relaying Renew from fe80::4638:
[Link] port 546 going up.
Dec 05 [Link] leaf01 dhcrelay[6152]: sending upstream swp52
Dec 05 [Link] leaf01 dhcrelay[6152]: sending upstream swp51

Configuring DHCP Servers

To run DHCP for both IPv4 and IPv6, you need to initiate the DHCP server twice: once for IPv4 and once for
IPv6. The following configuration uses the following topology for the host, DHCP relay and DHCP server:

[Link] 269
Cumulus Linux 3.5 User Guide

For the configurations used in this chapter, the DHCP server is a switch running Cumulus Linux; however,
the DHCP server can also be located on a dedicated server in your environment.

The dhcpd and dhcrelay services are disabled by default. After you finish configuring the DHCP
relays and servers, you need to start those services.

Contents
This chapter covers ...
Configuring DHCP Server on Cumulus Linux Switches (see page 270)
Configuring the IPv4 DHCP Server (see page 270)
Configuring the IPv6 DHCP Server (see page 271)
Troubleshooting the Log from a DHCP Server (see page 272)

Configuring DHCP Server on Cumulus Linux Switches

You can use the following sample configurations for [Link] and [Link] to start both an IPv4
and an IPv6 DHCP server. The configuration files for the two DHCP server instances need to have two pools:
Pool 1: Subnet overlaps interfaces
Pool 2: Subnet that includes the addresses

Configuring the IPv4 DHCP Server

In a text editor, edit the [Link] file with a configuration similar to the following:

cumulus@switch:~$ cat /etc/dhcp/[Link]

ddns-update-style none;

default-lease-time 600;
max-lease-time 7200;

270 23 January 2018

 Cumulus Networks

subnet [Link] netmask [Link] {

}
subnet [Link] netmask [Link] {
range [Link] [Link];
}

Just as you did with the DHCP relay scripts, edit the DHCP server configuration file so it can launch the
DHCP server when the system boots. Here is a sample configuration:

cumulus@switch:~$ cat /etc/default/isc-dhcp-server

DHCPD_CONF="-cf /etc/dhcp/[Link]"

INTERFACES="swp1"

After you've finished configuring the DHCP server, enable the dhcpd service immediately:

cumulus@switch:~$ sudo systemctl enable [Link]

Configuring the IPv6 DHCP Server

In a text editor, edit the [Link] file with a configuration similar to the following:

cumulus@switch:~$ cat /etc/dhcp/[Link]

ddns-update-style none;

default-lease-time 600;
max-lease-time 7200;

subnet6 [Link]/64 {
}
subnet6 [Link]/64 {
range6 [Link] [Link];
}

Just as you did with the DHCP relay scripts, edit the DHCP server configuration file so it can launch the
DHCP server when the system boots. Here is a sample configuration:

cumulus@switch:~$ cat /etc/default/isc-dhcp-server6

DHCPD_CONF="-cf /etc/dhcp/[Link]"

INTERFACES="swp1"

You cannot use NCLU to configure IPv6 DHCP servers.

[Link] 271
Cumulus Linux 3.5 User Guide

After you've finished configuring the DHCP server, enable the dhcpd6 service immediately:

cumulus@switch:~$ sudo systemctl enable [Link]

Troubleshooting the Log from a DHCP Server

The DHCP server knows whether a DHCP request is a relay or a non-relay DHCP request. On isc-dhcp-
server, for example, it is possible to tail the log and look at the behavior firsthand:

cumulus@server02:~$ sudo tail /var/log/syslog | grep dhcpd

2016-12-05T[Link].379633+00:00 server02 dhcpd: Relay-forward
message from [Link] port 547, link address [Link],
peer address fe80::4638:39ff:fe00:3
2016-12-05T[Link].380081+00:00 server02 dhcpd: Advertise NA:
address [Link] to client with duid [Link]
[Link] iaid = 956301315 valid for 600 seconds
2016-12-05T[Link].380470+00:00 server02 dhcpd: Sending Relay-reply
to [Link] port 547

802.1X Interfaces
The IEEE 802.1X protocol provides a method of authenticating a client (called a supplicant) over wired
media. It also provides access for individual MAC addresses on a switch (called the authenticator) after those
MAC addresses have been authenticated by an authentication server — typically a RADIUS (see page 132)
(Remote Authentication Dial In User Service, defined by RFC 2865) server.
A Cumulus Linux switch acts as an intermediary between the clients connected to the wired ports and the
authentication server, which is reachable over the existing network. EAPOL (Extensible Authentication
Protocol (EAP) over LAN — EtherType value of 0x888E, defined by RFC 3748) operates on top of the data
link layer; it is used by the switch to communicate with supplicants connected to the switch ports.
Cumulus Linux implements 802.1X through the Debian hostapd package, which has been modified to
provide the PAE (port access entity).

272 23 January 2018

 Cumulus Networks

Contents
This chapter covers ...
Supported Features and Limitations (see page 273)
Installing the 802.1X Package (see page 274)
Configuring 802.1X Interfaces (see page 274)
Configuring Accounting and Authentication Ports (see page 276)
Configuring MAC Authentication Bypass (see page 276)
Configuring a Parking VLAN (see page 276)
Verifying Connections from Linux Supplicants (see page 277)
Configuring Dynamic VLAN Assignments (see page 277)
Troubleshooting (see page 278)
Advanced Troubleshooting (see page 281)
Configuring the RADIUS Server (see page 281)

Supported Features and Limitations

This feature is supported on 1G Broadcom-based platforms only.
The protocol is supported on physical interfaces only (bridged/access only and routed interfaces) —
such as swp1 or swp2s0; these interfaces cannot be part of a bond.
However, 802.1X is not supported on eth0.
Enabling or disabling 802.1X capability on ports results in hostapd reloading. Existing authorized
sessions will not be reset.
MAC authentication bypass (MAB) and parking VLAN require a bridge access port interface.
Changing any of the following RADIUS parameters restarts hostapd, which forces existing,
authorized users to re-authenticate:
The RADIUS server IP address, shared secret, authentication port or accounting port
Parking VLAN ID
MAB activation delay

EAP reauthentication period

[Link] 273
Cumulus Linux 3.5 User Guide

EAP reauthentication period

Removing all 802.1X interfaces

Changing the interface dot1x, dot1x mab, or dot1x parking-vlan settings do not
reset existing authorized user ports.

Up to three RADIUS servers can be configured, for failover purposes.

Do not use a Cumulus Linux switch as the RADIUS server.

This has been tested with only a few wpa_supplicant (Debian) and Windows7 supplicants.
RADIUS authentication is supported with FreeRadius and Cisco ACS.
Supports simple login/password, PEAP/MSCHAPv2 (Win7) and EAP-TLS (Debian).

Installing the 802.1X Package

If you upgraded Cumulus Linux from a version earlier than 3.3.0 instead of performing a full binary install,
you need to install the hostapd package on your switch:

cumulus@switch:~$ sudo -E apt-get update

cumulus@switch:~$ sudo -E apt-get install hostapd
cumulus@switch:~$ sudo -E apt-get upgrade

Configuring 802.1X Interfaces

NCLU (see page 85) handles all the configuration of 802.1X interfaces, updating hostapd and other
components so you don't have to manually modify configuration files. All the interfaces share the same
RADIUS server settings.
The 802.1X-specific settings are:
accounting-port: RADIUS accounting parameters, which defaults to 1813.
authentication-port: RADIUS authentication port, which defaults to 1812.
server-ip: RADIUS Server IPv4 or IPv6 address, which has no default, but is required.
shared-secret: RADIUS shared secret, which has no default, but is required.
Make sure you configure the RADIUS server before the interfaces. See below (see page 281) for details.

1. Create a simple interface bridge configuration on the switch and add the switch ports that are
members of the bridge. You can use glob syntax to add a range of interfaces. The MAB and parking
VLAN configurations require interfaces to be bridge access ports.

cumulus@switch:~$ net add bridge bridge ports swp1-4

2. Configure the settings for the 802.1X RADIUS server, including its IP address and shared secret:

274 23 January 2018

 2. Cumulus Networks

cumulus@switch:~$ net add dot1x radius server-ip [Link]

cumulus@switch:~$ net add dot1x radius shared-secret testing123

3. Enable 802.1X on interfaces, then review and commit the new configuration:

cumulus@switch:~$ net add interface swp1-4 dot1x

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration snippet in the /etc/network/interfaces file:

cumulus@switch:~$ cat /etc/network/interfaces

...

auto swp1
iface swp1
bridge-learning off

auto swp2
iface swp2
bridge-learning off

auto swp3
iface swp3
bridge-learning off

auto swp4
iface swp4
bridge-learning off
...

auto bridge
iface bridge
bridge-ports swp1 swp2 swp3 swp4
bridge-vlan-aware yes

Verify the 802.1X configuration, showing the configuration and its status:

cumulus@switch:~$ net show configuration commands | grep dot1x

dot1x radius server-ip [Link]
dot1x radius authentication-port 1812
dot1x radius accounting-port 1813
dot1x radius shared-secret testing123

[Link] 275
Cumulus Linux 3.5 User Guide

interface swp2,swp3,swp1,swp4 dot1x

cumulus@switch:~$ net show dot1x status

IEEE802.1X Enabled Status: enabled
IEEE802.1X Active Status: active

Configuring Accounting and Authentication Ports

You can configure the accounting and authentication ports in Cumulus Linux. The default values are 1813
for the accounting port and 1812 for the authentication port.
You can also change the reauthentication period for Extensible Authentication Protocol (EAP). The period
defaults to 0 (no re-authentication is performed by the switch).
To use different ports, do the following:

cumulus@switch:~$ net add dot1x radius authentication-port 2812

cumulus@switch:~$ net add dot1x radius accounting-port 2813
cumulus@switch:~$ net add dot1x eap-reauth-period 86400
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Configuring MAC Authentication Bypass

MAC authentication bypass (MAB) enables bridge ports to allow devices to bypass authentication based on
their MAC address. This is useful for devices that do not support PAE, such as printers or phones. You can
change the MAB activation delay from the default of 30 seconds, but the delay must be between 5 and 30
seconds. The switch port must be part of bridge named bridge.

MAB supports one authenticated MAC address per port only.

You must configure MAB on the RADIUS server.

To enable a bridge port for MAB and to change the MAB activation delay, do the following on the RADIUS
client (that is, the Cumulus Linux switch):

cumulus@switch:~$ net add dot1x mab-activation-delay 20

cumulus@switch:~$ net add interface swp1 dot1x mab
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Configuring a Parking VLAN

If a non-authorized supplicant tries to communicate with the switch, you can route traffic from that device
to a different VLAN and associate that VLAN with one of the switch ports. The switch port must be part of
bridge named bridge.

276 23 January 2018

 Cumulus Networks

cumulus@switch:~$ net add dot1x parking-vlan-id 777

cumulus@switch:~$ net add interface swp1 dot1x parking-vlan
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Verifying Connections from Linux Supplicants

To test that a supplicant (client) can communicate with the Cumulus Linux Authenticator switch, run the
following command from the supplicant:

root@host1:/home/cumulus# wpa_supplicant -c /etc/wpa_supplicant.conf -

D wired -i swp1
Successfully initialized wpa_supplicant
swp1: Associated with [Link]
swp1: CTRL-EVENT-EAP-STARTED EAP authentication started
swp1: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=4
swp1: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 4 (MD5) selected
swp1: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
swp1: CTRL-EVENT-CONNECTED - Connection to [Link] compl

Or from another supplicant:

root@host2:/home/cumulus# wpa_supplicant -c /etc/wpa_supplicant.conf -

D wired -i swp1
Successfully initialized wpa_supplicant
swp1: Associated with [Link]
swp1: CTRL-EVENT-EAP-STARTED EAP authentication started
swp1: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=4
swp1: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 4 (MD5) selected
swp1: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
swp1: CTRL-EVENT-CONNECTED - Connection to [Link] comp

Configuring Dynamic VLAN Assignments

A common requirement for campus networks is to assign dynamic VLANs to specific users in combination
with IEEE 802.1x. After authenticating a supplicant, the user is assigned a VLAN based on the RADIUS
configuration.
To enable dynamic VLAN assignment globally, where VLAN attributes sent from the RADIUS server are
applied to the bridge:

cumulus@switch:~$ net add dot1x dynamic-vlan

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

[Link] 277
Cumulus Linux 3.5 User Guide

You can specify the require option in the command so that VLAN attributes are required. If VLAN
attributes do not exist in the access response packet returned from the RADIUS server, the user is not
authorized and has no connectivity. If the RADIUS server returns VLAN attributes but the user has an
incorrect password, the user is placed in the parking VLAN (if you have configured parking VLAN).

cumulus@switch:~$ net add dot1x dynamic-vlan require

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The following example shows a typical RADIUS configuration (shown for Freeradius, not typically configured
or run on the Cumulus Linux device) for a user with dynamic VLAN assignment:

# # VLAN 100 Client Configuration for Freeradius RADIUS Server.

# # This is not part of the CL configuration.
vlan100client Cleartext-Password := "client1password"
Service-Type = Framed-User,
Tunnel-Type = VLAN,
Tunnel-Medium-Type = "IEEE-802",
Tunnel-Private-Group-ID = 100

To disable dynamic VLAN assignment, where VLAN attributes sent from the RADIUS server are ignored and
users are authenticated based on existing credentials:

cumulus@switch:~$ net del dot1x dynamic-vlan

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Enabling or disabling dynamic VLAN assignment restarts hostapd, which forces existing,
authorized users to re-authenticate.

Troubleshooting
To check connectivity between two supplicants, ping one host from the other:

root@host1:/home/cumulus# ping [Link]

PING [Link] ([Link]) 56(84) bytes of data.
64 bytes from [Link]: icmp_seq=1 ttl=64 time=0.604 ms
64 bytes from [Link]: icmp_seq=2 ttl=64 time=0.552 ms
^C
--- [Link] ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.552/0.578/0

You can run net show dot1x with the following options for more data:
278 23 January 2018
 Cumulus Networks

You can run net show dot1x with the following options for more data:
json: Prints the command output in JSON format.
macs: Displays MAC address information.
port-details: Shows counters from the IEEE8021-PAE-MIB for ports.
radius-details: Shows counters from the RADIUS-CLIENT MIB (RFC 2618) for ports.
status: Displays the status of the daemon.
To check to see which MAC addresses have been authorized by RADIUS:

cumulus@switch:~$ net show dot1x macs

Interface Attribute Value
----------- ------------- -----------------
swp1 MAC Addresses [Link]
swp2 No Data
swp3 No Data
swp4 No Data

To check the port detail counters:

cumulus@switch:~$ net show dot1x port-details

Interface Attribute Value

----------- ---------------------------------------- ---------
swp1 Mac Addresses [Link]
[Link]
authMultiSessionId
96703ADC82D77DF2
connected_time 182
dot1xAuthEapolFramesRx 3
dot1xAuthEapolFramesTx 3
dot1xAuthEapolLogoffFramesRx 0
dot1xAuthEapolReqFramesTx 2
dot1xAuthEapolReqIdFramesTx 1
dot1xAuthEapolRespFramesRx 2
dot1xAuthEapolRespIdFramesRx 1
dot1xAuthEapolStartFramesRx 1
dot1xAuthInvalidEapolFramesRx 0
dot1xAuthLastEapolFrameSource [Link]
00:01
dot1xAuthLastEapolFrameVersion 2
dot1xAuthPaeState 5
dot1xAuthQuietPeriod 60
dot1xAuthReAuthEnabled FALSE
dot1xAuthReAuthPeriod 0
dot1xAuthServerTimeout 30
dot1xAuthSessionAuthenticMethod 1
dot1xAuthSessionId
1B50FE8939FD9F5E

[Link] 279
Cumulus Linux 3.5 User Guide

dot1xAuthSessionTerminateCause 999
dot1xAuthSessionTime 182
dot1xAuthSessionUserName testing
dot1xPaePortProtocolVersion 2
last_eap_type_as 4 (MD5)
last_eap_type_sta 4
(MD5)

To check RADIUS counters:

cumulus@switch:~$ net show dot1x radius-details swp1

Interface Attribute Value

----------- ---------------------------------------- ---------
swp1 radiusAccClientRequests 1
radiusAccClientResponses 1
radiusAccClientServerPortNumber 1813
radiusAccServerAddress [Link]
radiusAuthClientAccessAccepts 1
radiusAuthClientAccessChallenges 1
radiusAuthClientAccessRejects 0
radiusAuthClientAccessRequests 0
radiusAuthClientServerPortNumber 1812
radiusAuthServerAddress [Link]
radiusAuthServerIndex 1

...

You can also check logging with journalctl:

cumulus@switch-01:~$ sudo journalctl –f –u hostapd

Apr 19 [Link] switch-01 hostapd[12462]: swp1: interface state
UNINITIALIZED->ENABLED
Apr 19 [Link] switch-01 hostapd[12462]: swp1: AP-ENABLED
Apr 19 [Link] switch-01 hostapd[12462]: Reading rule file /etc
/cumulus/acl/policy.d/00control_ps ...
Apr 19 [Link] switch-01 hostapd[12462]: Processing rules in file
/etc/cumulus/acl/policy.d/00...
Apr 19 [Link] switch-01 hostapd[12462]: Reading rule file /etc
/cumulus/acl/policy.d/100_dot1x...
Apr 19 [Link] switch-01 hostapd[12462]: Processing rules in file
/etc/cumulus/acl/policy.d/ ..
Apr 19 [Link] switch-01 hostapd[12462]: Reading rule file /etc
/cumulus/acl/policy.d/99control
Apr 19 [Link] switch-01 hostapd[12462]: Processing rules in file
/etc/cumulus/acl/policy.d/99
Apr 19 [Link] switch-01 hostapd[12462]: Installing acl policy
Apr 19 [Link] switch-01 hostapd[12462]: done.

280 23 January 2018

 Cumulus Networks

Advanced Troubleshooting
More advanced troubleshooting can be accomplished with the following commands.
You can increase the debug level in hostapd by copying over the hostapd service file, then adding -d, -dd
or -ddd to the ExecStart line in the [Link] file:

cumulus@switch:~$ cp /lib/systemd/system/[Link] /etc/systemd

/system/[Link]
cumulus@switch:~$ sudo nano /etc/systemd/system/[Link]
...

ExecStart=/usr/sbin/hostapd –ddd –c /etc/[Link]

...

You can watch debugs with journalctl as supplicants attempt to connect:

cumulus@switch:~$ sudo journalctl -n 1000 -u hostapd # see the

last 1000 lines of hostapd debug logging
cumulus@switch:~$ sudo journalctl -f -u hostapd #
continuous tail of the hostapd daemon debug logging

You can check ACL rules in /etc/cumulus/acl/policy.d/100_dot1x_swpX.rules before and after

a supplicant attempts to authenticate:

cumulus@switch:~$ sudo cl-acltool -L eb | grep swpXX

cumulus@switch:~$ sudo cl-netstat | grep swpXX # look at
interface counters

You can check tc rules in /var/lib/hostapd/acl/tc_swpX.rules with:

cumulus@switch:~$ sudo tc -s filter show dev swpXX parent 1:

cumulus@switch:~$ sudo tc -s filter show dev swpXX parent ffff:

Configuring the RADIUS Server

If you haven't done so already, you need to configure the RADIUS server — preferably not on the Cumulus
Linux switch — before configuring any interfaces for 802.1X.
To add a popular and freely available RADIUS server called FreeRADIUS on a Debian workstation, do the
following:

root@radius:~# apt-get update

root@radius:~# apt-get install freeradius-utils freeradius-common

[Link] 281
Cumulus Linux 3.5 User Guide

Once installed and configured, the FreeRADIUS server can serve Cumulus Linux running hostapd as a
RADIUS client.
For more information, see the FreeRADIUS documentation.

Layer
282 1 and 2 23 January 2018
 Cumulus Networks

Layer 1 and 2

Spanning Tree and Rapid Spanning Tree

Spanning tree protocol (STP) is always recommended in layer 2 topologies, as it prevents bridge loops and
broadcast radiation on a bridged network. STP also provides redundant links for automatic failover when
an active link fails. STP is disabled by default on bridges in Cumulus Linux.

Contents
This chapter covers ...
Supported Modes (see page 283)
STP for a VLAN-aware Bridge (see page 284)
STP within a Traditional Mode Bridge (see page 284)
Viewing Bridge and STP Status/Logs (see page 284)
Using Linux to Check Spanning Tree Status (Advanced) (see page 287)
Customizing Spanning Tree Protocol (see page 288)
Spanning Tree Priority (see page 288)
PortAdminEdge/PortFast Mode (see page 289)
PortAutoEdge (see page 290)
BPDU Guard (see page 290)
Bridge Assurance (see page 293)
BPDU Filter (see page 293)
Storm Control (see page 294)
Configuring Other Spanning Tree Parameters (see page 294)
Caveats and Errata (see page 297)
Related Information (see page 297)

Supported Modes
The STP modes Cumulus Linux supports vary depending upon whether the traditional or VLAN-aware
bridge driver mode (see page 321) is in use.
Bridges configured in VLAN-aware (see page 327) mode operate only in RSTP mode. NCLU (see page 85),
the network command line utility for configuring Cumulus Linux, only supports bridges in VLAN-aware
mode.
For a bridge configured in traditional mode, PVST and PVRST are supported, with the default set to PVRST.
Each traditional bridge has its own separate STP instance.
Since you cannot use NCLU to configure a traditional mode bridge, you must configure it directly in the
/etc/network/interfaces file.

[Link] 283
Cumulus Linux 3.5 User Guide

STP for a VLAN-aware Bridge

VLAN-aware (see page 327) bridges only operate in RSTP mode. STP bridge protocol data units (BPDUs) are
transmitted on the native VLAN.
If a bridge running RSTP (802.1w) receives a common STP (802.1D) BPDU, it will automatically fall back to
802.1D operation. RSTP interoperates with MST seamlessly, creating a single instance of spanning tree,
which transmits BPDUs on the native VLAN. RSTP treats the MST domain as if it were one giant switch.

As of version 3.2.1, STP is enabled by default in Cumulus Linux. There is no need to specify
bridge-stp on for the bridge any more.

STP within a Traditional Mode Bridge

Per VLAN Spanning Tree (PVST) creates a spanning tree instance for a bridge. Rapid PVST (PVRST) supports
RSTP enhancements for each spanning tree instance. In order to use PVRST with a traditional bridge, a
bridge corresponding to the untagged native VLAN must be created, and all the physical switch ports must
be part of the same VLAN.

When connected to a switch that has a native VLAN configuration, the native VLAN must be
configured to be VLAN 1 only for maximum interoperability.

Viewing Bridge and STP Status/Logs

To check STP status for a bridge, run net show bridge spanning-tree:
Click to reveal the output ...

cumulus@switch:~$ net show bridge spanning-tree

bridge CIST info
enabled yes
bridge id 1.000.[Link]
designated root 1.000.[Link]
regional root 1.000.[Link]
root port none
path cost 0 internal path cost 0
max age 20 bridge max age 20
forward delay 15 bridge forward delay 15
tx hold count 6 max hops 20
hello time 2 ageing time 300
force protocol version rstp
time since topology change 253343s
topology change count 4
topology change no
topology change port peerlink
last topology change port leaf03-04
bridge:exit01-02 CIST info

284 23 January 2018

 Cumulus Networks

enabled no role
Disabled
port id 8.004 state
discarding
external port cost 305 admin external cost 0
internal port cost 305 admin internal cost 0
designated root 1.000.[Link] dsgn external cost 0
dsgn regional root 1.000.[Link] dsgn internal cost 0
designated bridge 1.000.[Link] designated port
8.004
admin edge port no auto edge port yes
oper edge port no topology change ack no
point-to-point yes admin point-to-point auto
restricted role no restricted TCN no
port hello time 2 disputed no
bpdu guard port no bpdu guard error no
network port no BA inconsistent no
Num TX BPDU 2 Num TX TCN 0
Num RX BPDU 0 Num RX TCN 0
Num Transition FWD 0 Num Transition BLK 2
bpdufilter port no
clag ISL no clag ISL Oper UP no
clag role primary clag dual conn mac 00:
[Link]
clag remote portID [Link] clag system mac 44:
[Link]
bridge:leaf01-02 CIST info
enabled yes role
Designated
port id 8.003 state
forwarding
external port cost 10000 admin external cost 0
internal port cost 10000 admin internal cost 0
designated root 1.000.[Link] dsgn external cost 0
dsgn regional root 1.000.[Link] dsgn internal cost 0
designated bridge 1.000.[Link] designated port
8.003
admin edge port no auto edge port yes
oper edge port no topology change ack no
point-to-point yes admin point-to-point auto
restricted role no restricted TCN no
port hello time 2 disputed no
bpdu guard port no bpdu guard error no
network port no BA inconsistent no
Num TX BPDU 253558 Num TX TCN 2
Num RX BPDU 253373 Num RX TCN 4
Num Transition FWD 126675 Num Transition BLK
126694
bpdufilter port no
clag ISL no clag ISL Oper UP no
clag role primary clag dual conn mac 44:
[Link]

[Link] 285
Cumulus Linux 3.5 User Guide

clag remote portID [Link] clag system mac 44:

[Link]
bridge:leaf03-04 CIST info
enabled yes role
Designated
port id 8.001 state
forwarding
external port cost 10000 admin external cost 0
internal port cost 10000 admin internal cost 0
designated root 1.000.[Link] dsgn external cost 0
dsgn regional root 1.000.[Link] dsgn internal cost 0
designated bridge 1.000.[Link] designated port
8.001
admin edge port no auto edge port yes
oper edge port no topology change ack no
point-to-point yes admin point-to-point auto
restricted role no restricted TCN no
port hello time 2 disputed no
bpdu guard port no bpdu guard error no
network port no BA inconsistent no
Num TX BPDU 130960 Num TX TCN 6
Num RX BPDU 4 Num RX TCN 1
Num Transition FWD 2 Num Transition BLK 1
bpdufilter port no
clag ISL no clag ISL Oper UP no
clag role primary clag dual conn mac 44:
[Link]
clag remote portID [Link] clag system mac 44:
[Link]
bridge:peerlink CIST info
enabled yes role
Designated
port id F.002 state
forwarding
external port cost 10000 admin external cost 0
internal port cost 10000 admin internal cost 0
designated root 1.000.[Link] dsgn external cost 0
dsgn regional root 1.000.[Link] dsgn internal cost 0
designated bridge 1.000.[Link] designated port F.
002
admin edge port no auto edge port yes
oper edge port no topology change ack no
point-to-point yes admin point-to-point auto
restricted role no restricted TCN no
port hello time 2 disputed no
bpdu guard port no bpdu guard error no
network port no BA inconsistent no
Num TX BPDU 126700 Num TX TCN 2
Num RX BPDU 6 Num RX TCN 3
Num Transition FWD 2 Num Transition BLK 1
bpdufilter port no
clag ISL yes clag ISL Oper UP yes

286 23 January 2018

 Cumulus Networks

clag role primary clag dual conn mac 00:

[Link]
clag remote portID [Link] clag system mac 44:
[Link]

Using Linux to Check Spanning Tree Status (Advanced)

Using Linux to check STP status ...
mstpctl is the utility provided by the mstpd service to configure STP. The mstpd daemon is an open
source project used by Cumulus Linux to implement IEEE802.1D 2004 and IEEE802.1Q 2011.
mstpd is started by default when the switch boots. mstpd logs and errors are located in /var/log
/syslog.

mstpd is the preferred utility for interacting with STP on Cumulus Linux. brctl also provides
certain methods for configuring STP; however, they are not as complete as the tools offered in
mstpd and output from brctl can be misleading in some cases.

To get the bridge state, use:

cumulus@switch:~$ sudo brctl show

bridge name bridge id STP enabled interfaces
bridge 8000.001401010100 yes swp1
swp4
swp5

To get the mstpd bridge state, use:

cumulus@switch:~$ net show bridge spanning-tree

bridge CIST info
enabled yes
bridge id F.000.[Link]
designated root F.000.[Link]
regional root F.000.[Link]
root port none
path cost 0 internal path cost 0
max age 20 bridge max age 20
forward delay 15 bridge forward delay 15
tx hold count 6 max hops 20
hello time 2 ageing time 200
force protocol version rstp
time since topology change 90843s
topology change count 4
topology change no
topology change port swp4
last topology change port swp5

To get the mstpd bridge port state, use:

[Link] 287
Cumulus Linux 3.5 User Guide

To get the mstpd bridge port state, use:

cumulus@switch:~$ sudo mstpctl showport bridge

E swp1 8.001 forw F.000.[Link] F.000.[Link]
8.001 Desg
swp4 8.002 forw F.000.[Link] F.000.[Link]
8.002 Desg
E swp5 8.003 forw F.000.[Link] F.000.[Link]
8.003 Desg
cumulus@switch:~$ net show bridge spanning-tree
...
bridge:swp1 CIST info
enabled yes role
Designated
port id 8.001 state
forwarding
external port cost 2000 admin external cost 0
internal port cost 2000 admin internal cost 0
designated root F.000.[Link] dsgn external cost 0
dsgn regional root F.000.[Link] dsgn internal cost 0
designated bridge F.000.[Link] designated port
8.001
admin edge port no auto edge port yes
oper edge port yes topology change ack no
point-to-point yes admin point-to-point auto
restricted role no restricted TCN no
port hello time 2 disputed no
bpdu guard port no bpdu guard error no
network port no BA inconsistent no
Num TX BPDU 45772 Num TX TCN 4
Num RX BPDU 0 Num RX TCN 0
Num Transition FWD 2 Num Transition BLK 2

Customizing Spanning Tree Protocol

There are a number of ways you can customize STP in Cumulus Linux. You should exercise extreme caution
with many of the settings below to prevent malfunctions in STP's loop avoidance.

Spanning Tree Priority

If you have an MSTI (multiple spanning tree instance), you can set the tree priority for a bridge. The bridge
with the lowest priority is elected the root bridge. The priority must be a number between 0 and 65535 and
must be a multiple of 4096; the default is 32768.

For msti, only 0 is supported currently.

To set the tree priority, run:

288 23 January 2018

 Cumulus Networks

cumulus@switch:~$ net add bridge stp treeprio 8192

cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

PortAdminEdge/PortFast Mode
PortAdminEdge is equivalent to the PortFast feature offered by other vendors. It enables or disables the
initial edge state of a port in a bridge.
All ports configured with PortAdminEdge bypass the listening and learning states to move immediately to
forwarding.

Using PortAdminEdge mode has the potential to cause loops if it is not accompanied by the
BPDU guard (see page 290) feature.

While it is common for edge ports to be configured as access ports for a simple end host, this is not
mandatory. In the data center, edge ports typically connect to servers, which may pass both tagged and
untagged traffic.

Example VLAN-aware Bridge Configuration

To configure PortAdminEdge mode, use the bpduguard and portadminedge NCLU configuration
commands:

cumulus@switch:~$ net add interface swp5 stp bpduguard

cumulus@switch:~$ net add interface swp5 stp portadminedge
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The NCLU commands above create the following code snippet:

auto swp5
iface swp5
mstpctl-bpduguard yes
mstpctl-portadminedge yes

Example Traditional Bridge Configuration

For a bridge in traditional mode (see page 321), configure PortAdminEdge under the bridge stanza
in /etc/network/interfaces:

auto br2
iface br2 inet static

[Link] 289
Cumulus Linux 3.5 User Guide

bridge-ports swp1 swp2 swp3 swp4

mstpctl-bpduguard swp1=yes swp2=yes swp3=yes swp4=yes
mstpctl-portadminedge swp1=yes swp2=yes swp3=yes swp4=yes

To load the new configuration, run ifreload -a:

cumulus@switch:~$ sudo ifreload -a

PortAutoEdge
PortAutoEdge is an enhancement to the standard PortAdminEdge (PortFast) mode, which allows for the
automatic detection of edge ports. PortAutoE