The TrueNAS Storage driver allows an Incus node to utilise a remote TrueNAS storage server to host one or more incus storage pools.

When the node is in a cluster, then all members of the cluster can use the storage pool simultaneously. This works well , for example, when live migrating a VM from one Incus node to another.

The driver is block-based, ie all Incus volumes are created as zvols on the remote server, and the zvol block devices are accessed on the local incus node via iscsi

The driver is modelled after the existing ZFS driver, and supports most existing ZFS functionality, but on a remote TrueNAS server.

For example, a local VM can be snapshotted and then cloned. The snapshot and cloning will be performed on the remote server after synchronizing the local filesystem, and the clone will then be activated via iSCSI as necessary.

The driver uses the truenas_incus_ctl tool, which provides a simple CLI interface to the TrueNAS API to effect changes on the remote server. Additionally, the tool can also activate/deactivate remote zvols using open-iscsi.

If the tool is not found, then the driver is disabled.

The latest version of the truenas tool can be obtained from : https://github.com/truenas/truenas_incus_ctl, and must be installed in the PATH supplied to incusd (v0.7.2+ is required)

open-iscsi can be installed with apt install open-iscsi

To login to the remote TrueNAS host first run the command:

sudo truenas_incus_ctl config login

Example:

$ sudo truenas_incus_ctl config login
[sudo] password for user: 
Enter a name for this connection: example
Enter the TrueNAS hostname or IP address: server.example.com
Setting up connection to TrueNAS host: server.example.com
Choose authentication method (1 for API Key, 2 for Username/Password): 2
Testing connection to wss://server.example.com/api/current...
Enter your TrueNAS username: truenas_admin
Enter your TrueNAS password: 
Generating API key...
API key successfully generated
Successfully connected to wss://server.example.com/api/current
Configuration for 'example' (connecting to server.example.com) saved to /root/.truenas_incus_ctl/config.json

After logging in, ensure iSCSI is configured and working with the following command:

sudo truenas_incus_ctl share iscsi setup --test

As an example of the functionality of the TrueNAS tool, this command can be used to list all datasets on the remote server

sudo truenas_incus_ctl dataset ls

Once logged in and iSCSI has been configured, the following command can be used to create a remote storage pool

incus storage create <poolname> truenas source=[host:]<pool>/<dataset>/[remote-poolname]

source is used to describe the location where the storage pool will be created on the remote TrueNAS host
host is optional, and remote-poolname will be set to poolname if not supplied.

truenas.config can be used to specify a specific login config
truenas.host can be used to specify a specific host, which the tool will lookup in the config.json

Running Tests

There are a number of INCUS_TRUENAS_ environment variables that can be used to configure the remote TrueNAS that will be used by the driver during the test. The only environment variable that is required is INCUS_TRUENAS_DATASET

I use the following commands in a script called start-tests.sh to run the suite using the default host configured in truenas_incus_ctl config file

sudo -E env PATH=$PATH INCUS_BACKEND=truenas INCUS_TRUENAS_DATASET=tank/se/incus-tests ./main-tn.sh $1

Alternatively, I specify a config for a different host. Hosts and API keys can also be specified manually to avoid the use of a config file. Alternate config files can be specified.

sudo -E env PATH=$PATH INCUS_VERBOSE=0 INCUS_TMPFS=0 INCUS_BACKEND=truenas INCUS_TRUENAS_DATASET=puddle/se/incus-tests INCUS_TRUENAS_CONFIG=goldeye2 ./main-tn.sh $1

I run the standalone tests using ./start-tests standalone, as the cluster tests require net ns support from truenas_incus_ctl which is not implemented.

main-tn.sh is the same as main.sh, except some tests which are problematic on my testing hosts are disabled.

main-tn.sh should probably be deleted once the ./main.sh standalone tests are verified

When running the test truenas_incus_ctl will be launched in daemon mode. It can be useful to prelaunch the daemon to view its console as the test suite runs.

for example:
sudo /home/<user>/go/bin/truenas_incus_ctl daemon /home/<user>/tncdaemon.sock

Incus CI Integration

I would suggest that a VM with a "golden" image of GoldEye with the middleware defer patch is used as a target. The truenas_incus_ctl tool should be built from master and the standalone tests used. A preconfigured config.json for the tool could be used to provide the host/api-key to use for the tests. A container dataset should be created and removed after the test suite is run.

Known Issues

• Creating and Deleting remote Volumes is relatively slow due to current TrueNAS bottlenecks. Work is ongoing on resolving and hopefully will be incorporated in a future builds of Goldeye. Add a defer flag to iscsi target/extent/targetextent create/update/delete truenas/middleware#16614
• Requires TrueNAS Goldeye for snapshot rename capability (currently in Nightlies), although there is a patch that can be easily installed in Fangtooth (25.04) https://github.com/truenas/truenas_incus_ctl#zfssnapshotrename-backport
• Cluster tests are not currently passing (requires net ns work-arounds), but cluster usage is supported. Use standalone tests
• TrueNAS hosts don't currently support zvol-shrinking via API. The driver attempts to zvol-shrink via the tool, and then ignores the inevitable "shrink failure", but warns with the correct zfs set command required. If the API is ever changed, the tool will be updated to add support.
• Optimized migrations between storage pools are not currently implemented.