Automated VMware Cloud Foundation Lab Deployment Branch Fork Vlan Management Domain

calendar Mar 7, 2025 · LAB AUTOMATION VCENTER ESXI NESTED ESXI CLOUD BUILDER SDDC MANAGER VCF NSX-T VCP-VCF VCF MANAGEMENT WORKLOAD DOMAIN VCF PLANNING AND PREPARATION WORKBOOK NSX REFERENCE DESIGN GUIDE VYOS VLAN DNS NTP  ·
Share on: linkedin copy
VCF 2 Node VSAN with VLANS

Overview

Laying Plan with tools at hand, namely VMware Cloud Foundation 5.2 Planning and Preparation Workbook, VMware NSX Reference Design Guide 4.2, Automated VMware Cloud Foundation Lab Deployment Branch Fork Vlan and Vyos

At a starting point we'll use this logical design from VMware NSX Reference Design Guide 4.2 (download it) and we'll look at Virtual Distributed Switch profile 3,
then finally we'll map every information from Automated VMware Cloud Foundation Lab Deployment Branch Fork Vlan (download it) and Vyos,
to fill VMware Cloud Foundation 5.2 Planning and Preparation Workbook again (download it).

This step might help getting an understanding but not require to deploy the lab.

VMware NSX Reference Design Guide 4.2 - VCF Management Domain Logical Design

Here i give a step by step example how to fill the workbook based on the value vcf-automated-lab-deployment vlan version and Vyos router

Planning and Preparation VMware Cloud Foundation 5.2.1 Management Domain Deployment

We are looking at deploying VCF with absolute minimal resources to give a chance to more people a way of VCF hands-on practice in addition to VMware Hands-On Lab to learn and prepare for attending VCP-VCF Administrator

On that note Franky and team are holding multiple VCP-VCF Administrator Exam Study Groups. The remaining study group dates are below and I suggest that you sign up and attend these.

Mar 11, 2025 @ 07:00 AM PTC
Mar 18, 2025 @ 07:00 AM PTC
Mar 25, 2025 @ 07:00 AM PTC

You can sign up for the tree remaining study groups at https://broadcom.zoom.us/meeting/register/jqEifVifQc-w6O1ZqbGydw#/registration?utm_medium=CMTYsocial&utm_source=CMTYTwitter

Discord Channel - https://discord.gg/cAgPKfAPnd

Community Exam Guide - https://docs.google.com/document/d/1k5bo1HuDXVVYckb_lv1irkdPzR2xhL1O-iRxE5HIRxA/edit?usp=sharing

Blog covering the recordings - https://blogs.vmware.com/code/2025/02/13/join-the-vmware-code-vcp-vcf-study-group/

Prepare physical ESXi Host/Cluster

ESXi Physical Host vSwitches Portgroups VLAN

Note:

  • DRS must be enabled on the Cluster to run Automated VMware Cloud Foundation Lab Deployment because it is required to create a vApp that contain the lab VMs.
  • vSwitch0 has all security to "refused" and MTU 1500
  • vSwitch1 has all security to "accept" and MTU 9000
  • vcf-dc1 a windows 2019 act as DNS server with single vNIC
  • Vyos-a act has - TOR (Top Of Rack) L3Switch/Router - and serve NTP on VCF VM VLAN 10 and ESXi VLAN 11 Networks with reverse DNS

assuming
vswitch0 or VDS already created with uplink vmnic0
with portgroup 'VM Network' with access to internet

the following is using Standard vSwitch because the goal was to allow VCF to run on 2 Nested ESXi with 52GB each on a 128GB of RAM physical ESXi host

Create new vSwitch1
MTU 9000
Add an uplink vmnic1
Portgroup 'VMTRUNK'
VLAN 4095
Add networking
Create a second portgroup '101010-Network'
VLAN 10
Edit vSwitch1
Enable promiscuous mode

ssh to the Physical ESXi and prevent Duplicate packets with this command:
esxcli system settings advanced set -o /Net/ReversePathFwdCheckPromisc -i 1

Deploy and Configure Vyos router

download Vyos 1.3 iso

create a vm debian 64bit 1cpu 1gb ram 2vNIC

Physical ESXi host VM Network on both vNIC

boot
login
vyos
vyos

sudo loadkeys fr #your country code keymap if different that us

install image
[Enter] everytime
or Yes to confirm if needed
Vyos new password twice

1config
2set interface ethernet eth0 address 192.168.1.253/24
3set protocols static routing 0.0.0.0/0 next-hop #your default gateway to internet
4set service ssh
5commit
6save

shutdown
deconnect CDROM from Vyos VM and from the ESXi Host if any
connect Vyos vm 1st vNIC to portgroup 'VM Network' #access to internet
connect Vyos vm 2nd vNIC to portgroup 'VMTRUNK'

Configure Vyos Interfaces

ssh Vyos 

 1config
 2set interfaces ethernet eth0 ipv6 address no-default-link-local
 3set interfaces ethernet eth1 ipv6 address no-default-link-local
 4set interfaces ethernet eth1 mtu '9000'
 5set interfaces ethernet eth1 vif 10 address '10.10.10.1/24'
 6set interfaces ethernet eth1 vif 10 description 'VLAN 10 for VM MGMT'
 7set interfaces ethernet eth1 vif 10 ipv6 address no-default-link-local
 8set interfaces ethernet eth1 vif 10 mtu '1500'
 9set interfaces ethernet eth1 vif 11 address '10.10.11.1/24'
10set interfaces ethernet eth1 vif 11 description 'VLAN 11 for ESXi MGMT'
11set interfaces ethernet eth1 vif 11 ipv6 address no-default-link-local
12set interfaces ethernet eth1 vif 11 mtu '1500'
13set interfaces ethernet eth1 vif 12 address '10.10.12.1/24'
14set interfaces ethernet eth1 vif 12 description 'VLAN 12 for VMOTION'
15set interfaces ethernet eth1 vif 12 ipv6 address no-default-link-local
16set interfaces ethernet eth1 vif 12 mtu '9000'
17set interfaces ethernet eth1 vif 13 address '10.10.13.1/24'
18set interfaces ethernet eth1 vif 13 description 'VLAN 13 for VSAN'
19set interfaces ethernet eth1 vif 13 ipv6 address no-default-link-local
20set interfaces ethernet eth1 vif 13 mtu '9000'
21set interfaces ethernet eth1 vif 14 address '10.10.14.1/24'
22set interfaces ethernet eth1 vif 14 description 'VLAN 14 for HOST VTEP'
23set interfaces ethernet eth1 vif 14 ipv6 address no-default-link-local
24set interfaces ethernet eth1 vif 14 mtu '9000'

Configure Vyos NAT

1set nat source rule 10 outbound-interface 'eth0'
2set nat source rule 10 source address '10.10.10.0/24'
3set nat source rule 10 translation address 'masquerade'
4set nat source rule 11 outbound-interface 'eth0'
5set nat source rule 11 source address '10.10.11.0/24'
6set nat source rule 11 translation address 'masquerade'

Configure Vyos DNS Forwarding

 1set service dns forwarding allow-from '127.0.0.0/8'
 2set service dns forwarding allow-from 169.254.0.0/16
 3set service dns forwarding allow-from '10.0.0.0/8'
 4set service dns forwarding allow-from '172.16.0.0/12'
 5set service dns forwarding allow-from '192.168.0.0/16'
 6set service dns forwarding domain 10.10.10.in-addr.arpa. server '192.168.1.100'
 7set service dns forwarding domain 11.10.10.in-addr.arpa. server '192.168.1.100'
 8set service dns forwarding domain abidi.systems server '192.168.1.100'
 9set service dns forwarding listen-address '10.10.10.1'
10set service dns forwarding listen-address '10.10.11.1'
11set service dns forwarding listen-address '192.168.1.253'
12set service dns forwarding name-server '192.168.1.100'

Configure Vyos Hostname (could be ntp and then make FQDN as ntp.your.domain like in VCF Planning Workbook Prerequisite)

1set system host-name Vyos-a

In your DNS Server

ADD Vyos A/PTR entry, before that create DNS reverse lookup zone for the subnet if not already done

Configure Vyos DNS Server

1set system name-server '192.168.1.100'

Configure Vyos NTP

 1set system ntp allow-clients address '127.0.0.0/8'
 2set system ntp allow-clients address '169.254.0.0/16'
 3set system ntp allow-clients address '10.0.0.0/8'
 4set system ntp allow-clients address '172.16.0.0/12'
 5set system ntp allow-clients address '192.168.0.0/16'
 6set system ntp allow-clients address '::1/128'
 7set system ntp allow-clients address 'fe80::/10'
 8set system ntp allow-clients address 'fc00::/7'
 9set system ntp interface 'eth1.10'
10set system ntp interface 'eth1.11'
11set system ntp listen-address '192.168.1.253'
12set system ntp server pool.ntp.org

Create DNS Reverse Lookup Zone fo each subnets first (for PTR records) then Create DNS Forward lookup Hosts A records entry

DNS RECORDS FOR VCF

Configure Automated VMware Cloud Foundation Lab Deployment Branch Fork Vlan

Download Vlan commit Automated VMware Cloud Foundation Lab Deployment Branch Fork Vlan

sample-vcf-mgmt-variables.ps1

$VMNetwork = "VMTRUNK" #portgroup vlan 4095 on vSwitch or
vlan trunking (10-14) on VDS
(Vlan Guest Tagging VGT = Trunk, the guest are NestedEsxi VMs with VMKernel VLAN VMK0)
on Virtual Switch with MTU 9000 on physical ESXi/Cluster not shown here, with promiscuous mode enabled
and prevent duplicate packets with following
esxcli system settings advanced set -o /Net/ReversePathFwdCheckPromisc -i 1
$CBVMNetwork = "101010-Network" #portgroup vlan $NestedVMNetworkVLanId (10) because CB OVA doesn't expose $ovfconfig.common.guestinfo.vlan,
no VGT but Virtual Switch Tagging VST = Access,
on same on Virtual Switch

Because VM and NestedEsxi use there own vlan, they need an interface vlan gateway on a virtual or physical upstream top of rack (tor L3switch/router Vyos in this case)
$VMGateway = "10.10.10.1" #interface vlan for $NestedVmManagementNetworkCidr on upstream tor (Vyos)
$vmk0Gateway = "10.10.11.1" #interface vlan for $NestedESXiManagementNetworkCidr on upstream tor (Vyos)

$VMNTP = "192.168.1.253" #upstream tor Vyos 1.3.x
because ntpd allowed multiple "interface" (like 1.10 for VM and 1.11 for ESXi)
but it migrated to chrony Vyos 1.4 which support listening only a single interface
NTP "listen address/fqdn" (DNS A/PTR records required),
example: A FQDN Vyos-a.abidi.systems <-> IP 192.168.1.253 / PTR FQDN 253.1.168.192.in-addr.arpa <-> Hostname Vyos-a.abidi.systems.),
to upstream NTP server like pool.ntp.org

Because to run less than 4 VSAN nodes (not supported), we needed to add this in the resulting json
$hostFailuresToTolerate = 0 #1 default value for required 4 VSAN Ready Nodes

$CloudbuilderIP = "10.10.10.180" #must be on same subnet as $NestedVmManagementNetworkCidr (10.10.10.0/24)

uncomment for default VCF required 4 VSAN Ready Nodes

1**$NestedESXiHostnameToIPsForManagementDomain** = @{
2	"vcf-m01-esx01"   = "10.10.11.185"
3	"vcf-m01-esx02"   = "10.10.11.186"
4	#"vcf-m01-esx03"   = "10.10.11.187"
5	#"vcf-m01-esx04"   = "10.10.11.188"
6}

$NestedESXiMGMTvCPU = "8" #12 default value
$NestedESXiMGMTvMEM = "52" #GB #96 default value, going below 52GB would get memory issue during NSX host prepartion on VLCM images based Cluster

Because VM has there own network
$NestedVmManagementNetworkCidr = "10.10.10.0/24" #gateway editable here $VMGateway (10.10.10.1)

Because Host has there own networks
$NestedESXiManagementNetworkCidr = "10.10.11.0/24" #gateway editable here $vmk0Gateway (10.10.11.1)
$NestedESXivMotionNetworkCidr = "10.10.12.0/24" #gateway not editable here also is .1 $esxivMotionGateway (10.10.12.1) on upstream tor (Vyos eth1.12)
$NestedESXivSANNetworkCidr = "10.10.13.0/24" #gateway not editable here also is .1 $esxivSANGateway (10.10.13.1) on upstream tor (Vyos eth1.13)
$NestedESXiNSXTepNetworkCidr = "10.10.14.0/24" #gateway not editable here also is .1 $esxiNSXTepGateway (10.10.14.1) on upstream tor (Vyos eth1.14)

Because every networks has there own vlan
$NestedVMNetworkVLanId = "10"
$vmk0VLanId = "11"
$vmotionVLanId = "12"
$vsanVLanId = "13"
$HostTepVLanId = "14"

vcf-automated-lab-deployment.ps1

Because we want to set vlan on NestedEsxi (VM Network vlan 10)
$setVLanId = 1

Because Wld is not supported yet on this commit
$deployNestedESXiVMsForWLD = 0

Because to run less than 4 VSAN nodes (not officially supported but working for lab), to SSH into Cloud Builder to reconfigure the size of the Cluster as steps described above
$startVCFBringup = 0

Not applicable in current commit
$generateWldHostCommissionJson = 0

Not enabled
$uploadVCFNotifyScript = 0

The resulting deployment variables should look like this:

$preCheck = 1
$confirmDeployment = 1
$deployNestedESXiVMsForMgmt = 1
$setVLanId = 1
$deployNestedESXiVMsForWLD = 0
$deployCloudBuilder = 1
$moveVMsIntovApp = 1
$generateMgmJson = 1
$startVCFBringup = 0
$generateWldHostCommissionJson = 0
$uploadVCFNotifyScript = 0

Moved start time after the deployment is confirmed with the Y to be more accruate
$StartTime = Get-Date

The new syntaxe to start the deployment script with the variables file use the argument -EnvConfigFile which you found by typing - then tab completion.

1.\vcf-automated-lab-deployment.ps1 -EnvConfigFile .\sample-vcf-mgmt-variables.ps1

sum

exec

Because to run less than 4 VSAN nodes (not supported), after deployment 15min or so, tweak the VCF mgmt cluster minimum size as following steps and follow the vcf-bringup-debug.log
- ssh Cloud Builder as admin
- su - $CloudbuilderRootPassword
- echo "bringup.mgmt.cluster.minimum.size=1" >> /etc/vmware/vcf/bringup/application.properties
- systemctl restart vcf-bringup.service
- tail -f /opt/vmware/bringup/logs/vcf-bringup-debug.log
- wait a minute to begin SDDC deployment on Cloud Builder UI

VMware Cloud Builder bringup.mgmt.cluster.minimum.size=1 and follow vcf-bringup-debug log

VMware Cloud Builder mystere NTP warnings explanations:

VMware Cloud Builder mystere NTP warnings needed both forward and reverse DNS as noted by William Lam and can also be found in VCF workbook in prerequisite sheet
It can also be confirmed in the debug log that CB try both forward and reverse DNS resolution:
NTP both forward and reverse DNS lookup succeed because A and PTR records for Vyos-a were created

VMware Cloud Builder validations all green

VMware Cloud Builder You have successfully deployed VMware Cloud Foundation

VMware Cloud Builder report bringup with VLANS

At this point can safely shudown Cloud Builder VM and move it out of the lab vApp.

VMware Cloud Foundation Setup Guide - Plan Upgrade

Next we will do deployment of Edge Cluster, AVNs, and Aria Lifecycle on X Region

Here a post on how to use Planning Workbook, Configure VLANs, DNS, NTP, BGP and Deploy VMware NSX Edge Cluster via VCF API with minimal resources

Stay tuned and join the zoom meeting VCP-VCF Administrator Exam Study Groups mentioned above

You can sign up for the remaining study groups at https://broadcom.zoom.us/meeting/register/jqEifVifQc-w6O1ZqbGydw#/registration?utm_medium=CMTYsocial&utm_source=CMTYTwitter

Discord Channel - https://discord.gg/cAgPKfAPnd

Community Exam Guide - https://docs.google.com/document/d/1k5bo1HuDXVVYckb_lv1irkdPzR2xhL1O-iRxE5HIRxA/edit?usp=sharing

Blog covering the recordings - https://blogs.vmware.com/code/2025/02/13/join-the-vmware-code-vcp-vcf-study-group/