Instance types and preferences
FEATURE STATE:
instancetype.kubevirt.io/v1alpha1(Experimental) as of the v0.56.0 KubeVirt releaseinstancetype.kubevirt.io/v1alpha2(Experimental) as of the v0.58.0 KubeVirt releaseinstancetype.kubevirt.io/v1beta1as of the v1.0.0 KubeVirt release
See the Version History section for more details.
Introduction
KubeVirt’s VirtualMachine API contains many advanced options for tuning the performance of a VM that goes beyond what typical users need to be aware of. Users have previously been unable to simply define the storage/network they want assigned to their VM and then declare in broad terms what quality of resources and kind of performance characteristics they need for their VM.
Instance types and preferences provide a way to define a set of resource, performance and other runtime characteristics, allowing users to reuse these definitions across multiple VirtualMachines.
VirtualMachineInstancetype
---apiVersion: instancetype.kubevirt.io/v1beta1kind: VirtualMachineInstancetypemetadata:name: example-instancetypespec:cpu:guest: 1memory:guest: 128Mi
KubeVirt provides two CRDs for instance types, a cluster wide VirtualMachineClusterInstancetype and a namespaced VirtualMachineInstancetype. These CRDs encapsulate the following resource related characteristics of a VirtualMachine through a shared VirtualMachineInstancetypeSpec:
- CPU : Required number of vCPUs presented to the guest
- Memory : Required amount of memory presented to the guest
- GPUs : Optional list of vGPUs to passthrough
- HostDevices : Optional list of
HostDevicesto passthrough IOThreadsPolicy: OptionalIOThreadsPolicyto be used- LaunchSecurity: Optional
LaunchSecurityto be used
Anything provided within an instance type cannot be overridden within the VirtualMachine. For example, as CPU and Memory are both required attributes of an instance type, if a user makes any requests for CPU or Memory resources within the underlying VirtualMachine, the instance type will conflict and the request will be rejected during creation.
VirtualMachinePreference
---apiVersion: instancetype.kubevirt.io/v1beta1kind: VirtualMachinePreferencemetadata:name: example-preferencespec:devices:preferredDiskBus: virtiopreferredInterfaceModel: virtio
KubeVirt also provides two further preference based CRDs, again a cluster wide VirtualMachineClusterPreference and namespaced VirtualMachinePreference. These CRDsencapsulate the preferred value of any remaining attributes of a VirtualMachine required to run a given workload, again this is through a shared VirtualMachinePreferenceSpec.
Unlike instance types, preferences only represent the preferred values and as such, they can be overridden by values in the VirtualMachine provided by the user.
In the example shown below, a user has provided a VirtualMachine with a disk bus already defined within a DiskTarget and has also selected a set of preferences with DevicePreference and preferredDiskBus , so the user’s original choice within the VirtualMachine and DiskTarget are used:
$ kubectl apply -f - << EOF---apiVersion: instancetype.kubevirt.io/v1beta1kind: VirtualMachinePreferencemetadata:name: example-preference-disk-virtiospec:devices:preferredDiskBus: virtio---apiVersion: kubevirt.io/v1kind: VirtualMachinemetadata:name: example-preference-user-overridespec:preference:kind: VirtualMachinePreferencename: example-preference-disk-virtiorunning: falsetemplate:spec:domain:memory:guest: 128Midevices:disks:- disk:bus: sataname: containerdisk- disk: {}name: cloudinitdiskresources: {}terminationGracePeriodSeconds: 0volumes:- containerDisk:image: registry:5000/kubevirt/cirros-container-disk-demo:develname: containerdisk- cloudInitNoCloud:userData: |#!/bin/shecho 'printed from cloud-init userdata'name: cloudinitdiskEOFvirtualmachinepreference.instancetype.kubevirt.io/example-preference-disk-virtio createdvirtualmachine.kubevirt.io/example-preference-user-override configured$ virtctl start example-preference-user-overrideVM example-preference-user-override was scheduled to start# We can see the original request from the user within the VirtualMachine lists `containerdisk` with a `SATA` bus$ kubectl get vms/example-preference-user-override -o json | jq .spec.template.spec.domain.devices.disks[{"disk": {"bus": "sata"},"name": "containerdisk"},{"disk": {},"name": "cloudinitdisk"}]# This is still the case in the VirtualMachineInstance with the remaining disk using the `preferredDiskBus` from the preference of `virtio`$ kubectl get vmis/example-preference-user-override -o json | jq .spec.domain.devices.disks[{"disk": {"bus": "sata"},"name": "containerdisk"},{"disk": {"bus": "virtio"},"name": "cloudinitdisk"}]
VirtualMachine
---apiVersion: kubevirt.io/v1kind: VirtualMachinemetadata:name: example-vmspec:instancetype:kind: VirtualMachineInstancetypename: example-instancetypepreference:kind: VirtualMachinePreferencename: example-preference
The previous instance type and preference CRDs are matched to a given VirtualMachine through the use of a matcher. Each matcher consists of the following:
Name(string): Name of the resource being referencedKind(string): Optional, defaults to the cluster wide CRD kinds ofVirtualMachineClusterInstancetypeorVirtualMachineClusterPreferenceif not providedRevisionName(string) : Optional, name of a ControllerRevision containing a copy of theVirtualMachineInstancetypeSpecorVirtualMachinePreferenceSpectaken when theVirtualMachineis first created. See the Versioning section below for more details on how and why this is captured.InferFromVolume(string): Optional, see the Inferring defaults from a Volume section below for more details.
Creating InstanceTypes, Preferences and VirtualMachines
It is possible to streamline the creation of instance types, preferences, and virtual machines with the usage of the virtctl command-line tool. To read more about it, please see the Creating VirtualMachines.
Versioning
Versioning of these resources is required to ensure the eventual VirtualMachineInstance created when starting a VirtualMachine does not change between restarts if any referenced instance type or set of preferences are updated during the lifetime of the VirtualMachine.
This is currently achieved by using ControllerRevision to retain a copy of the VirtualMachineInstancetype or VirtualMachinePreference at the time the VirtualMachine is created. A reference to these ControllerRevisions are then retained in the InstancetypeMatcher and PreferenceMatcher within the VirtualMachine for future use.
$ kubectl apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yamlvirtualmachineinstancetype.instancetype.kubevirt.io/csmall createdvirtualmachine.kubevirt.io/vm-cirros-csmall created$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype{"kind": "VirtualMachineInstancetype","name": "csmall","revisionName": "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1"}$ kubectl get controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1 -o json | jq .{"apiVersion": "apps/v1","data": {"apiVersion": "instancetype.kubevirt.io/v1beta1","kind": "VirtualMachineInstancetype","metadata": {"creationTimestamp": "2022-09-30T12:20:19Z","generation": 1,"name": "csmall","namespace": "default","resourceVersion": "10303","uid": "72c3a35b-6e18-487d-bebf-f73c7d4f4a40"},"spec": {"cpu": {"guest": 1},"memory": {"guest": "128Mi"}}},"kind": "ControllerRevision","metadata": {"creationTimestamp": "2022-09-30T12:20:19Z","name": "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1","namespace": "default","ownerReferences": [{"apiVersion": "kubevirt.io/v1","blockOwnerDeletion": true,"controller": true,"kind": "VirtualMachine","name": "vm-cirros-csmall","uid": "5216527a-1d31-4637-ad3a-b640cb9949a2"}],"resourceVersion": "10307","uid": "a7bc784b-4cea-45d7-8432-15418e1dd7d3"},"revision": 0}$ kubectl delete vm/vm-cirros-csmallvirtualmachine.kubevirt.io "vm-cirros-csmall" deleted$ kubectl get controllerrevision/controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1Error from server (NotFound): controllerrevisions.apps "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1" not found
Users can opt in to moving to a newer generation of an instance type or preference by removing the referenced revisionName from the appropriate matcher within the VirtualMachine object. This will result in fresh ControllerRevisions being captured and used.
The following example creates a VirtualMachine using an initial version of the csmall instance type before increasing the number of vCPUs provided by the instance type:
$ kubectl apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yamlvirtualmachineinstancetype.instancetype.kubevirt.io/csmall createdvirtualmachine.kubevirt.io/vm-cirros-csmall created$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype{"kind": "VirtualMachineInstancetype","name": "csmall","revisionName": "vm-cirros-csmall-csmall-3e86e367-9cd7-4426-9507-b14c27a08671-1"}$ virtctl start vm-cirros-csmallVM vm-cirros-csmall was scheduled to start$ kubectl get vmi/vm-cirros-csmall -o json | jq .spec.domain.cpu{"cores": 1,"model": "host-model","sockets": 1,"threads": 1}$ kubectl patch VirtualMachineInstancetype/csmall --type merge -p '{"spec":{"cpu":{"guest":2}}}'virtualmachineinstancetype.instancetype.kubevirt.io/csmall patched
In order for this change to be picked up within the VirtualMachine, we need to stop the running VirtualMachine and clear the revisionName referenced by the InstancetypeMatcher:
$ virtctl stop vm-cirros-csmallVM vm-cirros-csmall was scheduled to stop$ kubectl patch vm/vm-cirros-csmall --type merge -p '{"spec":{"instancetype":{"revisionName":""}}}'virtualmachine.kubevirt.io/vm-cirros-csmall patched$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype{"kind": "VirtualMachineInstancetype","name": "csmall","revisionName": "vm-cirros-csmall-csmall-3e86e367-9cd7-4426-9507-b14c27a08671-2"}
As you can see above, the InstancetypeMatcher now references a new ControllerRevision containing generation 2 of the instance type. We can now start the VirtualMachine again and see the new number of vCPUs being used by the VirtualMachineInstance:
$ virtctl start vm-cirros-csmallVM vm-cirros-csmall was scheduled to start$ kubectl get vmi/vm-cirros-csmall -o json | jq .spec.domain.cpu{"cores": 1,"model": "host-model","sockets": 2,"threads": 1}
inferFromVolume
The inferFromVolume attribute of both the InstancetypeMatcher and PreferenceMatcher allows a user to request that defaults are inferred from a volume. When requested, KubeVirt will look for the following labels on the underlying PVC, DataSource or DataVolume to determine the default name and kind:
instancetype.kubevirt.io/default-instancetypeinstancetype.kubevirt.io/default-instancetype-kind(optional, defaults toVirtualMachineClusterInstancetype)instancetype.kubevirt.io/default-preferenceinstancetype.kubevirt.io/default-preference-kind(optional, defaults toVirtualMachineClusterPreference)
These values are then written into the appropriate matcher by the mutation webhook and used during validation before the VirtualMachine is formally accepted.
Failure to find the referenced Volume or labels on an underlying resource will cause the request to be rejected.
$ kubectl kustomize https://github.com/kubevirt/common-instancetypes.git | kubectl apply -f -[..]$ virtctl image-upload pvc cirros-pvc --size=1Gi --image-path=./cirros-0.5.2-x86_64-disk.img[..]$ kubectl label pvc/cirros-pvc \instancetype.kubevirt.io/default-instancetype=server.tiny \instancetype.kubevirt.io/default-preference=cirros[..]$ kubectl apply -f - << EOF---apiVersion: cdi.kubevirt.io/v1beta1kind: DataSourcemetadata:name: cirros-datasourcespec:source:pvc:name: cirros-pvcnamespace: default---apiVersion: kubevirt.io/v1kind: VirtualMachinemetadata:name: cirrosspec:instancetype:inferFromVolume: cirros-volumepreference:inferFromVolume: cirros-volumerunning: falsedataVolumeTemplates:- metadata:name: cirros-datavolumespec:pvc:accessModes:- ReadWriteOnceresources:requests:storage: 1GistorageClassName: localsourceRef:kind: DataSourcename: cirros-datasourcenamespace: defaulttemplate:spec:domain:devices: {}volumes:- dataVolume:name: cirros-datavolumename: cirros-volumeEOF[..]kubectl get vms/cirros -o json | jq '.spec.instancetype, .spec.preference'{"kind": "virtualmachineclusterinstancetype","name": "server.tiny","revisionName": "cirros-server.tiny-76454433-3d82-43df-a7e5-586e48c71f68-1"}{"kind": "virtualmachineclusterpreference","name": "cirros","revisionName": "cirros-cirros-85823ddc-9e8c-4d23-a94c-143571b5489c-1"}
common-instancetypes
A set of common instance types and preferences are provided through the kubevirt/common-instancetypes project. To install all resources provided by the project, simply build with kustomize and apply:
$ kubectl kustomize https://github.com/kubevirt/common-instancetypes.git | kubectl apply -f -
Alternatively targets for each of the available custom resource types are available for example for VirtualMachineClusterInstancetype:
$ kubectl kustomize https://github.com/kubevirt/common-instancetypes.git/VirtualMachineClusterInstancetypes | kubectl apply -f -
Examples
Various examples are available within the kubevirt repo under /examples. The following uses an example VirtualMachine provided by the containerdisk/fedora repo and replaces much of the DomainSpec with the equivalent instance type and preferences:
$ kubectl apply -f - << EOF---apiVersion: instancetype.kubevirt.io/v1beta1kind: VirtualMachineInstancetypemetadata:name: cmediumspec:cpu:guest: 1memory:guest: 1Gi---apiVersion: instancetype.kubevirt.io/v1beta1kind: VirtualMachinePreferencemetadata:name: fedoraspec:devices:preferredDiskBus: virtiopreferredInterfaceModel: virtiopreferredRng: {}features:preferredAcpi: {}preferredSmm: {}firmware:preferredUseEfi: truepreferredUseSecureBoot: true---apiVersion: kubevirt.io/v1kind: VirtualMachinemetadata:creationTimestamp: nullname: fedoraspec:instancetype:name: cmediumkind: virtualMachineInstancetypepreference:name: fedorakind: virtualMachinePreferencerunStrategy: Alwaystemplate:metadata:creationTimestamp: nullspec:domain:devices: {}volumes:- containerDisk:image: quay.io/containerdisks/fedora:latestname: containerdisk- cloudInitNoCloud:userData: |-#cloud-configssh_authorized_keys:- ssh-rsa AAAA...name: cloudinitEOF
Version History
instancetype.kubevirt.io/v1alpha1 (Experimental)
- Initial development version.
instancetype.kubevirt.io/v1alpha2 (Experimental)
This version captured complete
VirtualMachine{Instancetype,ClusterInstancetype,Preference,ClusterPreference}objects within the createdControllerRevisionsThis version is backwardly compatible with
instancetype.kubevirt.io/v1alpha1.
instancetype.kubevirt.io/v1beta1
- The following instance type attribute has been added:
The following preference attributes have been added:
- Spec.Devices.PreferredInterfaceMasquerade
- Spec.PreferredSubdomain
- Spec.PreferredTerminationGracePeriodSeconds
This version is backwardly compatible with
instancetype.kubevirt.io/v1alpha1andinstancetype.kubevirt.io/v1alpha2objects, no modifications are required to existingVirtualMachine{Instancetype,ClusterInstancetype,Preference,ClusterPreference}orControllerRevisions.As with the migration to kubevirt.io/v1 it is recommend previous users of
instancetype.kubevirt.io/v1alpha1orinstancetype.kubevirt.io/v1alpha2use kube-storage-version-migrator to upgrade any stored objects toinstancetype.kubevirt.io/v1beta1.