Repository navigation
Expand file tree
/
Copy pathsandboxing.qmd
More file actions
498 lines (353 loc) · 25.9 KB
/
Copy pathsandboxing.qmd
File metadata and controls
498 lines (353 loc) · 25.9 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
---
title: Sandboxing
llms-description: Isolating code generated by models and setting up complex computing environments for tasks.
---
## Overview
Model tool calls are executed within the main process running the evaluation task. However, the work that tools perform — such as running shell commands or executing Python code — very often needs to happen in a dedicated, isolated environment (for example, `bash()` runs its shell commands in a sandbox). This might be the case if:
- You are creating tools, agents or scorers that execute arbitrary code (e.g. shell commands or Python code).
- You need to provision per-sample filesystem resources.
- You want to provide access to a more sophisticated evaluation environment (e.g. creating network hosts for a cybersecurity eval).
To accommodate these scenarios, Inspect provides support for *sandboxing*, which typically involves provisioning containers in which tools, agents and scorers can execute commands and code. Several of Inspect's [standard tools](tools-standard.qmd) require a sandbox, including `bash()`, `python()`, and `text_editor()`. Support for Docker sandboxes is built in, and the [Extension API](extensions-sandboxes.qmd#sec-sandbox-environment-extensions) enables the creation of additional sandbox types.
## Example: File Listing
Let's take a look at a simple example to illustrate. First, we'll define a `list_files()` tool. This tool need to access the `ls` command—it does so by calling the `sandbox()` function to get access to the `SandboxEnvironment` instance for the currently executing `Sample`:
``` python
from inspect_ai.tool import ToolError, tool
from inspect_ai.util import sandbox
@tool
def list_files():
async def execute(dir: str):
"""List the files in a directory.
Args:
dir: Directory
Returns:
File listing of the directory
"""
result = await sandbox().exec(["ls", dir])
if result.success:
return result.stdout
else:
raise ToolError(result.stderr)
return execute
```
The `exec()` function is used to list the directory contents. Note that its not immediately clear where or how `exec()` is implemented (that will be described shortly!).
Here's an evaluation that makes use of this tool:
``` python
from inspect_ai import task, Task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import generate, use_tools
dataset = [
Sample(
input='Is there a file named "bar.txt" '
+ 'in the current directory?',
target="Yes",
files={"bar.txt": "hello"},
)
]
@task
def file_probe():
return Task(
dataset=dataset,
solver=[
use_tools([list_files()]),
generate()
],
sandbox="docker",
scorer=includes(),
)
```
We've included `sandbox="docker"` so operations requested through `sandbox()` run in a Docker container. A sandbox environment must be specified at the sample, task, or evaluation level if any tools, agents or scorers call the `sandbox()` function.
Provisioning a sandbox does not automatically run tools, agents or scorers in the container. That code still runs as part of the evaluation; only work that it requests through the `sandbox()` interface runs in the container.
Note that `files` are specified as part of the `Sample`. Files can be specified inline using plain text (as depicted above), inline using a base64-encoded data URI, or as a path to a file or remote resource (e.g. S3 bucket). Relative file paths are resolved according to the location of the underlying dataset file.
## Environment Interface
The following instance methods are available to tools, agents and scorers that need to interact with a `SandboxEnvironment`:
{{< include _sandboxenv-exec.md >}}
{{< include _sandboxenv-exec-remote.md >}}
{{< include _sandboxenv-interface.md >}}
## Environment Binding
There are two sandbox environments built in to Inspect and six available as external packages. Dockerfile-compatible sandboxes accept standard `Dockerfile` and `compose.yaml` configuration files.
| Environment Type | Package | Dockerfile | Description |
|------------------|------------------|------------------|------------------|
| `docker` | Built-in | Yes | [Docker](#sec-docker-configuration) local installation. |
| `k8s` | [inspect-k8s-sandbox](https://pypi.org/project/inspect-k8s-sandbox/) | Yes | [Kubernetes](https://k8s-sandbox.aisi.org.uk/) cluster. |
| `daytona` | [inspect-sandboxes](https://pypi.org/project/inspect-sandboxes/) | Yes | [Daytona](https://meridianlabs-ai.github.io/inspect_sandboxes/daytona.html) cloud sandbox. |
| `modal` | [inspect-sandboxes](https://pypi.org/project/inspect-sandboxes/) | Yes | [Modal](https://meridianlabs-ai.github.io/inspect_sandboxes/modal.html) cloud sandbox. |
| `proxmox` | [inspect_proxmox_sandbox](https://github.com/UKGovernmentBEIS/inspect_proxmox_sandbox) | No | [Proxmox](https://github.com/UKGovernmentBEIS/inspect_proxmox_sandbox) with virtual machines. |
| `vagrant` | [inspect_vagrant_sandbox](https://github.com/jasongwartz/inspect_vagrant_sandbox) | No | [Vagrant](https://github.com/jasongwartz/inspect_vagrant_sandbox) virtual machines on any Vagrant-supported hypervisor. |
| `openshell` | [inspect-openshell-sandbox](https://pypi.org/project/inspect-openshell-sandbox/) | Yes | [NVIDIA OpenShell](https://github.com/32bitsret/inspect-openshell-sandbox) sandboxes with Landlock filesystem and network policies. |
| `local` | Built-in | No | Local file system (no sandbox). |
The `local` environment always executes as the current effective user. On POSIX systems, `exec(user=...)` accepts that user's name or UID; other identities raise `SandboxUserUnsupportedError` before execution. On other platforms, omit `user`.
Sandbox environment definitions can be bound at the `Sample`, `Task`, or `eval()` level. Binding precedence goes from `eval()`, to `Task` to `Sample`, however sandbox config files defined on the `Sample` always take precedence when the sandbox type for the `Sample` is the same as the enclosing `Task` or `eval()`.
Here is a `Task` that defines a `sandbox`:
``` python
Task(
dataset=dataset,
plan([
use_tools([read_file(), list_files()])),
generate()
]),
scorer=match(),
sandbox="docker"
)
```
By default, any `Dockerfile` and/or `compose.yaml` file within the task directory will be automatically discovered and used. If your compose file has a different name then you can provide an override specification as follows:
``` python
sandbox=("docker", "attacker-compose.yaml")
```
### Programmatic Configuration {#programmatic-configuration}
For more dynamic scenarios, you can construct a `ComposeConfig` object programmatically rather than using a static YAML file. This is useful when you need to vary container configuration based on task parameters:
``` python
from inspect_ai.util import ComposeConfig, ComposeService, SandboxEnvironmentSpec
@task
def my_task(cpus: float = 1.0):
config = ComposeConfig(
services={
"default": ComposeService(
image="python:3.12-bookworm",
init=True,
command="tail -f /dev/null",
mem_limit="512m",
cpus=cpus,
network_mode="none",
)
}
)
return Task(
dataset=dataset,
solver=[use_tools([read_file()]), generate()],
scorer=match(),
sandbox=SandboxEnvironmentSpec("docker", config),
)
```
The `ComposeConfig` and `ComposeService` classes mirror the structure of Docker Compose files, supporting fields like `image`, `build`, `command`, `environment`, `volumes`, `ports`, `mem_limit`, `cpus`, and more. Extension fields (prefixed with `x-`) are also supported.
Bind-mount sources and `env_file` paths in a `ComposeConfig` must be absolute (or start with `~/`). The generated Compose file is stored outside the task directory, so Inspect rejects a relative bind-mount source or `env_file` rather than let Compose resolve it there. Build such paths from the task file, for example `Path(__file__).parent / "data"`.
## Sandbox Limits {#sec-sandbox-limits}
By default, sandboxes limit the size of file reads to 100MB and execution output to 10MB. These limits exist to prevent boundary cases of outputs or executions that don't terminate and result in OOM or hung evaluations (i.e. they usually indicate an error by the model).
The two limits are not enforced in the same way. `read_file()` always raises `OutputLimitExceededError` when the limit is exceeded. For `exec()` the behaviour depends on the sandbox provider: the built-in providers front-truncate each output stream (keeping only its trailing portion), so `exec()` returns a successful `ExecResult` whose `stdout` or `stderr` may be silently incomplete rather than raising. Don't rely on an exception to detect `exec()` overflow—this matters in particular when parsing structured output such as JSON. If a command can produce output near the limit and you need all of it, have the command write to a file and read it with `read_file()`.
You can however increase these limits using environment variables. For example, here we set the read file limit to 200MB and the exec output size to 20MB:
``` bash
export INSPECT_SANDBOX_MAX_READ_FILE_SIZE=209715200
export INSPECT_SANDBOX_MAX_EXEC_OUTPUT_SIZE=20971520
```
## Per Sample Setup
The `Sample` class includes `sandbox`, `files` and `setup` fields that are used to specify per-sample sandbox config, file assets, and setup logic.
### Sandbox {#sec-per-sample-sandbox}
You can either define a default `sandbox` for an entire `Task` as illustrated above, or alternatively define a per-sample `sandbox`. For example, you might want to do this if each sample has its own Dockerfile and/or custom compose configuration file. (Note, each sample gets its own sandbox *instance*, even if the sandbox is defined at Task level. So samples do not interfere with each other's sandboxes.)
The `sandbox` can be specified as a string (e.g. `"docker`"), a tuple of sandbox type and config file (e.g. `("docker", "compose.yaml")`), or a `SandboxEnvironmentSpec` with a `ComposeConfig` for [Programmatic Configuration](#programmatic-configuration). This last option is particularly useful when you need to vary container configuration (e.g. docker image) on a per-sample basis.
### Files
Sample `files` is a `dict[str,str]` that specifies files to copy into sandbox environments. The key of the `dict` specifies the name of the file to write. By default files are written into the default sandbox environment but they can optionally include a prefix indicating that they should be written into a specific sandbox environment (e.g. `"victim:flag.txt": "flag.txt"`).
The value of the `dict` can be either the file contents, a file path, or a base64 encoded [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs).
### Script
If there is a Sample `setup` bash script it will be executed within the default sandbox environment after any Sample `files` are copied into the environment. The `setup` field can be either the script contents, a file path containing the script, or a base64 encoded [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs).
## Docker Configuration {#sec-docker-configuration}
### Installation
Before using Docker sandbox environments, please be sure to install [Docker Engine](https://docs.docker.com/engine/install/). Inspect checks the connected Docker daemon and requires version 24.0.6 or greater.
Docker Compose is checked separately: Inspect requires version 2.21.0 or greater, and version 2.22.0 or greater when Inspect pulls sandbox images from a registry.
Inspect's prerequisite checks confirm that Docker supports the features Inspect needs. They do not check whether security updates are installed. You are responsible for using vendor-supported releases and applying security updates to Docker Engine or Docker Desktop, Compose, the container runtime (such as runc), and the Docker host's operating system. This also applies when the Docker host is remote.
Don't count on a non-root container user to contain an adversarial model, as a capable model may get root in its container (see [Adversarial Models](agent-bridge.qmd#adversarial-models)).
If you plan on running evaluations with large numbers of concurrent containers (\> 30) you should also configure Docker's [default address pools](https://straz.to/2021-09-08-docker-address-pools/) to accommodate this.
### File Access
File reads require a running Linux container with `/bin/sh`, `/usr/bin/timeout`
and `tar` supporting `--no-recursion`. Reads enforce the requested user's
filesystem permissions, including for symlink targets. Pass `user="root"`
when privileged access is intended.
Reads follow kernel permissions, so root can normally read files with no read
bits (mode `000`). Reads of concurrently modified files return best-effort
contents or fail, including when a file shrinks during the read.
### Task Configuration
You can use the Docker sandbox environment without any special configuration, however most commonly you’ll provide explicit configuration via either a `Dockerfile` or a [Docker Compose](https://docs.docker.com/compose/compose-file/) configuration file (`compose.yaml`).
Here is how Docker sandbox environments are created based on the presence of `Dockerfile` and/or `compose.yml` in the task directory:
| Config Files | Behavior |
|---------------------------|---------------------------------------------|
| None | Creates a sandbox environment based on the official [`python:3.12-bookworm`](https://hub.docker.com/_/python) image. |
| `Dockerfile` | Creates a sandbox environment by building the image. |
| `compose.yaml` | Creates sandbox environment(s) based on `compose.yaml`. |
The default Docker sandbox and several examples use mutable image tags. A tag such as `python:3.12-bookworm` can refer to a different image when the publisher releases updates. To control when you adopt updates, pin an image digest in your own `compose.yaml`, for example `image: python:3.12-bookworm@sha256:<digest>`. You'll then need to update that digest to receive newer image versions. See Docker's [guidance on pinning images](https://docs.docker.com/build/building/best-practices/#pin-base-image-versions).
Providing a `compose.yaml` is not strictly required, as Inspect will automatically generate one as needed. The generated Compose configuration sets `network_mode: none`, which prevents network access at container runtime.
Supplying a custom Compose configuration — whether a Compose file or `ComposeConfig` — *replaces* the generated configuration rather than extending it, including its `network_mode: none`. Docker Compose's default is a project-scoped network with outbound Internet access, so include `network_mode: none` in your own configuration unless the evaluation requires networking.
If the evaluation does need network access, omit `network_mode` — that provides outbound access while keeping the project's network isolated (`network_mode: bridge` would instead join Docker's shared built-in bridge, so prefer omitting it). When services only need to communicate with each other, use an [internal network](https://docs.docker.com/reference/compose-file/networks/#internal), which allows service-to-service traffic without external connectivity.
Here's an example of a `compose.yaml` file that sets container resource limits and isolates it from all network interactions including internet access:
``` {.yaml filename="compose.yaml"}
services:
default:
build: .
init: true
command: tail -f /dev/null
cpus: 1.0
mem_limit: 0.5gb
network_mode: none
```
The `network_mode: none` entry applies only to processes inside the container. It does not restrict network access from the evaluation process or model provider. Code in custom tools, agents or scorers may therefore access the internet outside the Docker sandbox. Tools that do not use the Inspect sandbox, such as `web_search()`, are also unaffected. The `init: true` entry enables the container to respond to shutdown requests. The `command` is provided to prevent the container from exiting after it starts.
Here is what a simple `compose.yaml` would look like for a local pre-built image named `ctf-agent-environment` (resource limits excluded for brevity):
``` {.yaml filename="compose.yaml"}
services:
default:
image: ctf-agent-environment
x-local: true
init: true
command: tail -f /dev/null
network_mode: none
```
The `ctf-agent-environment` is not an image that exists on a remote registry, so we add the `x-local: true` to indicate that it should not be pulled. If local images are tagged, they also will not be pulled by default (so `x-local: true` is not required). For example:
``` {.yaml filename="compose.yaml"}
services:
default:
image: ctf-agent-environment:1.0.0
init: true
command: tail -f /dev/null
network_mode: none
```
If we are using an image from a remote registry we similarly don't need to include `x-local`:
``` {.yaml filename="compose.yaml"}
services:
default:
image: python:3.12-bookworm
init: true
command: tail -f /dev/null
network_mode: none
```
See the [Docker Compose](https://docs.docker.com/compose/compose-file/) documentation for information on all available container options.
### Prebuilt Images
If the images for your sandboxes are already present in the local Docker image store (for example, on an air-gapped machine that cannot reach a registry), pass `--sandbox-prebuilt` to skip image builds entirely:
``` bash
inspect eval ctf.py --sandbox-prebuilt
```
Or from Python:
``` python
eval("ctf.py", sandbox_prebuilt=True)
```
With this option set, task startup verifies images instead of building them. Every service with a `build` section or `x-local: true` must name an `image` that already exists locally, and Inspect's internal images (e.g. for the computer tool) must also be present. Other services use their local image when present, and a failed pull of a missing one is also an error. A missing image results in an error before any samples run. The option can also be set via the `INSPECT_EVAL_SANDBOX_PREBUILT` environment variable.
Inspect injects sandbox tools binaries into sandboxes for some tools. An install built from the sdist (e.g. conda-forge or `pip install --no-binary inspect-ai`) does not bundle them and downloads them at first use. For offline / air-gapped use, run `inspect download sandbox-tools` while online to fetch them ahead of time. The download writes into the `inspect_ai` package directory. For a read-only shared install, a user who can write to that directory should run `inspect download sandbox-tools` once.
### Multiple Environments
In some cases you may want to create multiple sandbox environments (e.g. if one environment has complex dependencies that conflict with the dependencies of other environments). To do this specify multiple named services:
``` {.yaml filename="compose.yaml"}
services:
default:
image: ctf-agent-environment
x-local: true
init: true
cpus: 1.0
mem_limit: 0.5gb
network_mode: none
victim:
image: ctf-victim-environment
x-local: true
init: true
cpus: 1.0
mem_limit: 1gb
network_mode: none
```
The first environment listed is the “default” environment, and can be accessed from within a tool with a normal call to `sandbox()`. Other environments would be accessed by name, for example:
``` python
sandbox() # default sandbox environment
sandbox("victim") # named sandbox environment
```
If you define multiple sandbox environments the default sandbox environment will be determined as follows:
1. First, take any sandbox environment named `default`;
2. Then, take any environment with the `x-default` key set to `true`;
3. Finally, use the first sandbox environment as the default.
You can use the `sandbox_default()` context manager to temporarily change the default sandbox (for example, if you have tools that always target the default sandbox that you want to temporarily redirect):
``` python
with sandbox_default("victim"):
# call tools, etc.
```
### Infrastructure
Note that in many cases you’ll want to provision additional infrastructure (e.g. other hosts or volumes). For example, here we define an additional container (“writer”) as well as a volume shared between the default container and the writer container:
``` yaml
services:
default:
image: ctf-agent-environment
x-local: true
init: true
network_mode: none
volumes:
- ctf-challenge-volume:/shared-data
writer:
image: ctf-challenge-writer
x-local: true
init: true
network_mode: none
volumes:
- ctf-challenge-volume:/shared-data
volumes:
ctf-challenge-volume:
```
See the documentation on [Docker Compose](https://docs.docker.com/compose/compose-file/) files for information on their full schema and feature set.
### Sample Metadata
You might want to interpolate Sample metadata into your Docker compose files. You can do this using the standard compose environment variable syntax, where any metadata in the Sample is made available with a `SAMPLE_METADATA_` prefix. For example, you might have a per-sample memory limit (with a default value of 0.5gb if unspecified):
``` yaml
services:
default:
image: ctf-agent-environment
x-local: true
init: true
cpus: 1.0
mem_limit: ${SAMPLE_METADATA_MEMORY_LIMIT-0.5gb}
network_mode: none
```
Note that `-` suffix that provides the default value of 0.5gb. This is important to include so that when the compose file is read *without* the context of a Sample (for example, when pulling/building images at startup) that a default value is available.
### Mounting Host Directories
A malicious agent with write access to your host machine through a volume mount can change files outside the sandbox. Even if it is restricted to a single directory, it could rewrite `compose.yml` to give future samples or evaluations more privileges than you intended, or redirect a later sample’s mount with a symlink, exposing host files you did not intend to share.
For security, Inspect blocks volume mounts that give Docker sandboxes write access to files or directories on the host.
To share a host directory read-only, add `:ro` to the mount:
```yaml
services:
default:
image: ubuntu:24.04
volumes:
- ./host-dir:/container-dir:ro
```
Read-only sharing requires Docker Engine 25+, Compose 2.24+, and Linux kernel 5.12+ on the Docker host. Inspect also checks that the container runtime can keep mounts inside the shared directory read-only. These checks apply to Compose `configs` and `secrets` too.
Inspect also rejects read-only host mounts if the service's privileges or mount options could make them writable.
These checks only cover mounts; privileged containers, host devices or Docker sockets can still give an agent access to the host.
For writable data, you can copy inputs with [`Sample.files`](#files) or use a [Docker-managed volume](#infrastructure).
::: {.callout-caution appearance="simple" collapse="true" title="Unsafe host access"}
Should your eval require unsafe host write access, you may allow unsafe host access for all services in your Compose file:
```yaml
x-inspect_docker_sandbox:
unsafe_allow_writable_host_mounts: true
```
This opt-in is also required for `volumes_from` and volumes with custom names or drivers, driver options, or external storage. It also bypasses the read-only compatibility checks above.
:::
## Environment Cleanup
When a task is completed, Inspect will automatically cleanup resources associated with the sandbox environment (e.g. containers, images, and networks). If for any reason resources are not cleaned up (e.g. if the cleanup itself is interrupted via Ctrl+C) you can globally cleanup all environments with the `inspect sandbox cleanup` command. For example, here we cleanup all environments associated with the `docker` provider:
``` bash
$ inspect sandbox cleanup docker
```
In some cases you may *prefer* not to cleanup environments. For example, you might want to examine their state interactively from the shell in order to debug an agent. Use the `--no-sandbox-cleanup` argument to do this:
``` bash
$ inspect eval ctf.py --no-sandbox-cleanup
```
You can also do this when using `eval(`):
``` python
eval("ctf.py", sandbox_cleanup = False)
```
When you do this, you'll see a list of sandbox containers printed out which includes the ID of each container. You can then use this ID to get a shell inside one of the containers:
``` bash
docker exec -it inspect-task-iqes2sgrg8tkq5q5w-default-1 bash -l
```
When you no longer need the environments, you can clean them up either all at once or individually by project name. The printed list ends with the cleanup command for each environment:
``` bash
# cleanup all environments
inspect sandbox cleanup docker
# cleanup single environment
inspect sandbox cleanup docker inspect-task-iqes2sgrg8tkq5q5w
```
## Resource Management
Creating and executing code within Docker containers can be expensive both in terms of memory and CPU utilisation. Inspect provides some automatic resource management to keep usage reasonable in the default case. This section describes that behaviour as well as how you can tune it for your use-cases.
{{< include _container_limits.md >}}
### Container Resources
Use a `compose.yaml` file to limit the resources consumed by each running container. For example:
``` {.yaml filename="compose.yaml"}
services:
default:
image: ctf-agent-environment
x-local: true
command: tail -f /dev/null
cpus: 1.0
mem_limit: 0.5gb
network_mode: none
```
## Troubleshooting
To diagnose sandbox execution issues (e.g. commands that don't terminate properly, container lifecycle issues, etc.) you should use Inspect's [Tracing](tracing.qmd) facility.
Trace logs record the beginning and end of calls to `subprocess()` (e.g. tool calls that run commands in sandboxes) as well as control commands sent to Docker Compose. The `inspect trace anomalies` subcommand then enables you to query for commands that don't terminate, timeout, or have errors. See the article on [Tracing](tracing.qmd) for additional details.