Race condition when automatically assigning IPs to VMs #7907
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
Core/Network
CLOUDSTACK VERSION
4.17.2
CONFIGURATION
N/A
OS / ENVIRONMENT
N/A
SUMMARY
When creating 2 virtual machines in the same guest network at the exact same second (in our case 23 Aug 2023 13:34:11) with automatic IP assignment, both VMs can reserve a single IP address by race condition.
STEPS TO REPRODUCE
With Terraform, create 2 VMs in the same Guest network
resource "cloudstack_instance" "terraform_vm" {
name = "vm-created-using-terraform"
service_offering = "Dynamic"
template = "Debian 11"
network_id = resource.cloudstack_network.testnet.id
keypair = "[name_of_ssh_keypair]"
details = {"cpuNumber" = 4,"memory" = 2048}
expunge = "true"
zone = "PA4"
}
resource "cloudstack_instance" "terraform_vm_2" {
name = "vm-created-using-terraform-2"
service_offering = "Dynamic"
template = "Debian 11"
network_id = resource.cloudstack_network.testnet.id
keypair = "[name_of_ssh_keypair]"
details = {"cpuNumber" = 4,"memory" = 2048}
expunge = "true"
zone = "PA4"
}
Then apply the manifest several times (until the race condition triggers), and check their IPs
EXPECTED RESULTS
The IPs are always different
ACTUAL RESULTS
In some iterations, here is what we saw (sensitive information redacted):
Both VMs have reserved the exact same IP, the same Zone and guest network
Here is the relevant entries from the cloud database:
> select * from nics where ip4_address='10.4.1.144' and state='Allocated' \G;
*************************** 1. row ***************************
id: 7225
uuid: eab5163e-b88b-43a0-b6a3-ac02fd5a6037
instance_id: 5471
mac_address: 02:00:7c:99:00:19
ip4_address: 10.4.1.144
netmask: 255.255.255.0
gateway: 10.4.1.1
ip_type: Ip4
broadcast_uri: NULL
network_id: 627
mode: Dhcp
state: Allocated
strategy: Start
reserver_name: VxlanGuestNetworkGuru
reservation_id: a3a9b296-e962-49f4-8c8f-3a3eef9128da
device_id: 0
update_time: 2023-08-23 15:34:11
isolation_uri: NULL
ip6_address: NULL
default_nic: 1
vm_type: User
created: 2023-08-23 13:34:11
removed: NULL
ip6_gateway: NULL
ip6_cidr: NULL
secondary_ip: 0
display_nic: 1
*************************** 2. row ***************************
id: 7227
uuid: 49af630f-9075-40f3-ae95-3d5a28c19ec2
instance_id: 5475
mac_address: 02:00:6e:f1:00:1d
ip4_address: 10.4.1.144
netmask: 255.255.255.0
gateway: 10.4.1.1
ip_type: Ip4
broadcast_uri: NULL
network_id: 627
mode: Dhcp
state: Allocated
strategy: Start
reserver_name: VxlanGuestNetworkGuru
reservation_id: 5dc0a9e8-b4c5-4f8e-a883-00178984c09a
device_id: 0
update_time: 2023-08-23 15:34:11
isolation_uri: NULL
ip6_address: NULL
default_nic: 1
vm_type: User
created: 2023-08-23 13:34:11
removed: NULL
ip6_gateway: NULL
ip6_cidr: NULL
secondary_ip: 0
display_nic: 1