Skip to content

Commit 40b8371

Browse files
committed
feat(gateway): identify gateways in exported traces
Attach the configured gateway installation name and compute driver to the gateway OpenTelemetry resource so operators can filter traces from multiple installations that share a collector. Forward the gateway name and OTLP endpoint to managed external drivers so their distinct service resources carry the same installation identity. Keep service.name stable per process type, omit blank resource values, and leave per-span operation names and request attributes unchanged. Refs #2507 Signed-off-by: Kris Hicks <[email protected]>
1 parent 7f97c3b commit 40b8371

23 files changed

Lines changed: 332 additions & 108 deletions

File tree

‎architecture/gateway.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -730,12 +730,14 @@ between a trace and its log lines. Store and compute-driver spans become
730730
children of the request span. Reconciliation, provider refresh, and
731731
driver-watch loops create their own operation spans because they have no
732732
inbound request to provide a parent. gRPC status is recorded when response
733-
trailers arrive.
734-
735-
The gateway forwards OTLP configuration and W3C trace context to managed
736-
external drivers. Built-in drivers use dedicated in-process providers that
737-
preserve the same RPC trace boundary. Each driver exports to the configured
738-
collector under its own service name.
733+
trailers arrive. Gateway spans carry resource attributes for the gateway
734+
identity and configured compute driver.
735+
736+
The gateway forwards OTLP configuration, its configured gateway name, and W3C
737+
trace context to managed external drivers. Built-in drivers use dedicated
738+
in-process providers that preserve the same RPC trace boundary. Each driver
739+
exports to the configured collector under its own service name and carries the
740+
gateway name as a resource attribute.
739741

740742
Two invariants shape the failure behavior. Telemetry is diagnostic, so no OTLP
741743
failure stops the gateway from serving: a malformed endpoint is logged at

‎crates/openshell-driver-docker/src/main.rs‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,13 +38,18 @@ struct Args {
3838

3939
#[arg(long, env = "OPENSHELL_OTLP_ENDPOINT")]
4040
otlp_endpoint: Option<String>,
41+
42+
#[arg(long, env = "OPENSHELL_GATEWAY_NAME")]
43+
gateway_name: Option<String>,
4144
}
4245

4346
#[tokio::main]
4447
async fn main() -> Result<()> {
4548
let args = Args::parse();
46-
let (tracer_provider, setup_error) =
47-
openshell_driver_docker::otel_tracing::provider_for(args.otlp_endpoint.as_deref());
49+
let (tracer_provider, setup_error) = openshell_driver_docker::otel_tracing::provider_for(
50+
args.otlp_endpoint.as_deref(),
51+
args.gateway_name.as_deref(),
52+
);
4853
tracing_subscriber::registry()
4954
.with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)))
5055
.with(tracing_subscriber::fmt::layer())

‎crates/openshell-driver-docker/src/otel_tracing.rs‎

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -90,13 +90,28 @@ pub(crate) fn compute_driver_rpc_operation(path: &str) -> (&'static str, &'stati
9090
}
9191
}
9292

93+
/// Build a tracer provider for the configured OTLP/gRPC endpoint and gateway.
9394
#[must_use]
94-
pub fn provider_for(endpoint: Option<&str>) -> (Option<SdkTracerProvider>, Option<SetupError>) {
95-
openshell_otel::provider_for(endpoint.map(|endpoint| OtlpTraceConfig {
96-
endpoint,
97-
service_name: ServiceName::Fixed(SERVICE_NAME),
98-
service_version: Some(openshell_core::VERSION),
99-
resource_attributes: Vec::new(),
95+
pub fn provider_for(
96+
endpoint: Option<&str>,
97+
gateway_name: Option<&str>,
98+
) -> (Option<SdkTracerProvider>, Option<SetupError>) {
99+
openshell_otel::provider_for(endpoint.map(|endpoint| {
100+
OtlpTraceConfig {
101+
endpoint,
102+
service_name: ServiceName::Fixed(SERVICE_NAME),
103+
service_version: Some(openshell_core::VERSION),
104+
resource_attributes: gateway_name
105+
.map(str::trim)
106+
.filter(|name| !name.is_empty())
107+
.map(|name| {
108+
vec![opentelemetry::KeyValue::new(
109+
"openshell.gateway.name",
110+
name.to_string(),
111+
)]
112+
})
113+
.unwrap_or_default(),
114+
}
100115
}))
101116
}
102117

@@ -151,11 +166,11 @@ mod tests {
151166
}
152167

153168
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
154-
async fn tracing_docker_driver_spans_reach_otlp_collector_with_distinct_service_name() {
169+
async fn tracing_docker_driver_spans_reach_otlp_collector_with_resource_identity() {
155170
let _tracing_lock = super::test_lock().await;
156171
let collector = OtlpTestServer::start().await;
157172

158-
let (provider, error) = super::provider_for(Some(collector.endpoint()));
173+
let (provider, error) = super::provider_for(Some(collector.endpoint()), Some("docker-dev"));
159174
assert!(error.is_none());
160175
let provider = provider.expect("provider");
161176
let subscriber = tracing_subscriber::registry().with(super::layer(&provider));
@@ -175,6 +190,7 @@ mod tests {
175190
.iter()
176191
.any(|span| span.name == "docker.schedule_sandbox")
177192
);
193+
assert_eq!(received.gateway_names, ["docker-dev"]);
178194
assert!(
179195
received
180196
.service_names

‎crates/openshell-driver-kubernetes/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ openshell-core = { path = "../openshell-core", default-features = false }
1919
openshell-otel = { path = "../openshell-otel" }
2020
openshell-policy = { path = "../openshell-policy" }
2121

22+
opentelemetry = { workspace = true }
2223
tokio = { workspace = true }
2324
tonic = { workspace = true, features = ["transport"] }
2425
prost = { workspace = true }
@@ -39,7 +40,6 @@ notify = "8"
3940

4041
[dev-dependencies]
4142
openshell-otel-test-support = { path = "../openshell-otel-test-support" }
42-
opentelemetry = { workspace = true }
4343
opentelemetry_sdk = { workspace = true, features = ["testing"] }
4444
bytes = { workspace = true }
4545
http = { workspace = true }

‎crates/openshell-driver-kubernetes/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,9 @@ When the gateway configures `[openshell.gateway.otlp]`, Kubernetes
2323
compute-driver spans export to the same OTLP/gRPC collector with the service
2424
name `openshell-driver-kubernetes`. The driver preserves the gateway trace
2525
context and uses the same compute-driver RPC span names in its in-process and
26-
standalone forms.
26+
standalone forms. Standalone deployments set `--gateway-name` or
27+
`OPENSHELL_GATEWAY_NAME` so exported spans carry the same
28+
`openshell.gateway.name` resource attribute as gateway spans.
2729

2830
When it creates an Agent Sandbox resource, the driver serializes the active W3C
2931
trace context into the controller-reserved `opentelemetry.io/trace-context`

‎crates/openshell-driver-kubernetes/src/main.rs‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,9 @@ struct Args {
4242
#[arg(long, env = "OPENSHELL_OTLP_ENDPOINT")]
4343
otlp_endpoint: Option<String>,
4444

45+
#[arg(long, env = "OPENSHELL_GATEWAY_NAME")]
46+
gateway_name: Option<String>,
47+
4548
#[arg(long, env = "OPENSHELL_WORKSPACE_MODE", default_value = "shared")]
4649
workspace_mode: WorkspaceMode,
4750

@@ -215,8 +218,10 @@ async fn shutdown_signal() {
215218
#[tokio::main]
216219
async fn main() -> Result<()> {
217220
let args = Args::parse();
218-
let (tracer_provider, setup_error) =
219-
openshell_driver_kubernetes::otel_tracing::provider_for(args.otlp_endpoint.as_deref());
221+
let (tracer_provider, setup_error) = openshell_driver_kubernetes::otel_tracing::provider_for(
222+
args.otlp_endpoint.as_deref(),
223+
args.gateway_name.as_deref(),
224+
);
220225
tracing_subscriber::registry()
221226
.with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)))
222227
.with(tracing_subscriber::fmt::layer())
@@ -349,17 +354,20 @@ mod tests {
349354
use super::*;
350355

351356
#[test]
352-
fn accepts_gateway_otlp_endpoint() {
357+
fn accepts_gateway_otlp_configuration() {
353358
let args = Args::try_parse_from([
354359
"openshell-driver-kubernetes",
355360
"--otlp-endpoint",
356361
"http://collector.example:4317",
362+
"--gateway-name",
363+
"kubernetes-dev",
357364
])
358365
.expect("OTLP endpoint should parse");
359366

360367
assert_eq!(
361368
args.otlp_endpoint.as_deref(),
362369
Some("http://collector.example:4317")
363370
);
371+
assert_eq!(args.gateway_name.as_deref(), Some("kubernetes-dev"));
364372
}
365373
}

‎crates/openshell-driver-kubernetes/src/otel_tracing.rs‎

Lines changed: 25 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,28 @@ const SERVICE_NAME: &str = "openshell-driver-kubernetes";
1212
const INSTRUMENTATION_SCOPE: &str = "openshell-driver-kubernetes";
1313
pub const IN_PROCESS_TARGET_PREFIX: &str = "openshell_driver_kubernetes";
1414

15+
/// Build a tracer provider for the configured OTLP/gRPC endpoint and gateway.
1516
#[must_use]
16-
pub fn provider_for(endpoint: Option<&str>) -> (Option<SdkTracerProvider>, Option<SetupError>) {
17-
openshell_otel::provider_for(endpoint.map(|endpoint| OtlpTraceConfig {
18-
endpoint,
19-
service_name: ServiceName::Fixed(SERVICE_NAME),
20-
service_version: Some(openshell_core::VERSION),
21-
resource_attributes: Vec::new(),
17+
pub fn provider_for(
18+
endpoint: Option<&str>,
19+
gateway_name: Option<&str>,
20+
) -> (Option<SdkTracerProvider>, Option<SetupError>) {
21+
openshell_otel::provider_for(endpoint.map(|endpoint| {
22+
OtlpTraceConfig {
23+
endpoint,
24+
service_name: ServiceName::Fixed(SERVICE_NAME),
25+
service_version: Some(openshell_core::VERSION),
26+
resource_attributes: gateway_name
27+
.map(str::trim)
28+
.filter(|name| !name.is_empty())
29+
.map(|name| {
30+
vec![opentelemetry::KeyValue::new(
31+
"openshell.gateway.name",
32+
name.to_string(),
33+
)]
34+
})
35+
.unwrap_or_default(),
36+
}
2237
}))
2338
}
2439

@@ -75,10 +90,11 @@ mod tests {
7590
}
7691

7792
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
78-
async fn tracing_kubernetes_driver_spans_reach_otlp_collector_with_distinct_service_name() {
93+
async fn tracing_kubernetes_driver_spans_reach_otlp_collector_with_resource_identity() {
7994
let _tracing_lock = super::test_lock().await;
8095
let collector = OtlpTestServer::start().await;
81-
let (provider, error) = super::provider_for(Some(collector.endpoint()));
96+
let (provider, error) =
97+
super::provider_for(Some(collector.endpoint()), Some("kubernetes-dev"));
8298
assert!(error.is_none());
8399
let provider = provider.expect("provider");
84100
let subscriber = tracing_subscriber::registry().with(super::layer(&provider));
@@ -98,6 +114,7 @@ mod tests {
98114
.iter()
99115
.any(|span| span.name == "kubernetes.create_sandbox")
100116
);
117+
assert_eq!(received.gateway_names, ["kubernetes-dev"]);
101118
assert!(
102119
received
103120
.service_names

‎crates/openshell-driver-podman/src/main.rs‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@ struct Args {
4040
#[arg(long, env = "OPENSHELL_OTLP_ENDPOINT")]
4141
otlp_endpoint: Option<String>,
4242

43+
#[arg(long, env = "OPENSHELL_GATEWAY_NAME")]
44+
gateway_name: Option<String>,
45+
4346
/// Path to the Podman API Unix socket.
4447
#[arg(long, env = "OPENSHELL_PODMAN_SOCKET")]
4548
podman_socket: Option<PathBuf>,
@@ -174,8 +177,10 @@ struct Args {
174177
#[tokio::main]
175178
async fn main() -> Result<()> {
176179
let args = Args::parse();
177-
let (tracer_provider, setup_error) =
178-
openshell_driver_podman::otel_tracing::provider_for(args.otlp_endpoint.as_deref());
180+
let (tracer_provider, setup_error) = openshell_driver_podman::otel_tracing::provider_for(
181+
args.otlp_endpoint.as_deref(),
182+
args.gateway_name.as_deref(),
183+
);
179184
tracing_subscriber::registry()
180185
.with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level)))
181186
.with(tracing_subscriber::fmt::layer())
@@ -306,17 +311,20 @@ mod tests {
306311
}
307312

308313
#[test]
309-
fn accepts_gateway_otlp_endpoint() {
314+
fn accepts_gateway_otlp_configuration() {
310315
let args = Args::try_parse_from([
311316
"openshell-driver-podman",
312317
"--otlp-endpoint",
313318
"http://collector.internal:4317",
319+
"--gateway-name",
320+
"production-us-west",
314321
])
315-
.expect("OTLP endpoint should be accepted");
322+
.expect("OTLP configuration should be accepted");
316323

317324
assert_eq!(
318325
args.otlp_endpoint.as_deref(),
319326
Some("http://collector.internal:4317")
320327
);
328+
assert_eq!(args.gateway_name.as_deref(), Some("production-us-west"));
321329
}
322330
}

‎crates/openshell-driver-podman/src/otel_tracing.rs‎

Lines changed: 25 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,28 @@ const SERVICE_NAME: &str = "openshell-driver-podman";
1212
const INSTRUMENTATION_SCOPE: &str = "openshell-driver-podman";
1313
pub const IN_PROCESS_TARGET_PREFIX: &str = "openshell_driver_podman";
1414

15+
/// Build a tracer provider for the configured OTLP/gRPC endpoint and gateway.
1516
#[must_use]
16-
pub fn provider_for(endpoint: Option<&str>) -> (Option<SdkTracerProvider>, Option<SetupError>) {
17-
openshell_otel::provider_for(endpoint.map(|endpoint| OtlpTraceConfig {
18-
endpoint,
19-
service_name: ServiceName::Fixed(SERVICE_NAME),
20-
service_version: Some(openshell_core::VERSION),
21-
resource_attributes: Vec::new(),
17+
pub fn provider_for(
18+
endpoint: Option<&str>,
19+
gateway_name: Option<&str>,
20+
) -> (Option<SdkTracerProvider>, Option<SetupError>) {
21+
openshell_otel::provider_for(endpoint.map(|endpoint| {
22+
OtlpTraceConfig {
23+
endpoint,
24+
service_name: ServiceName::Fixed(SERVICE_NAME),
25+
service_version: Some(openshell_core::VERSION),
26+
resource_attributes: gateway_name
27+
.map(str::trim)
28+
.filter(|name| !name.is_empty())
29+
.map(|name| {
30+
vec![opentelemetry::KeyValue::new(
31+
"openshell.gateway.name",
32+
name.to_string(),
33+
)]
34+
})
35+
.unwrap_or_default(),
36+
}
2237
}))
2338
}
2439

@@ -114,11 +129,12 @@ mod tests {
114129
}
115130

116131
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
117-
async fn podman_driver_spans_reach_otlp_collector_with_distinct_service_name() {
132+
async fn podman_driver_spans_reach_otlp_collector_with_resource_identity() {
118133
let _tracing_lock = super::test_lock().await;
119134
let collector = OtlpTestServer::start().await;
120135

121-
let (provider, error) = super::provider_for(Some(collector.endpoint()));
136+
let (provider, error) =
137+
super::provider_for(Some(collector.endpoint()), Some("production-us-west"));
122138
assert!(error.is_none());
123139
let provider = provider.expect("provider");
124140
let subscriber = tracing_subscriber::registry().with(super::layer(&provider));
@@ -138,6 +154,7 @@ mod tests {
138154
.iter()
139155
.any(|span| span.name == "podman.create")
140156
);
157+
assert_eq!(received.gateway_names, ["production-us-west"]);
141158
assert!(
142159
received
143160
.service_names

‎crates/openshell-driver-vm/README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,10 @@ First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw
4040
By default `mise run gateway:vm`:
4141

4242
- Listens on plaintext HTTP at `127.0.0.1:18081`.
43-
- Registers the CLI gateway `vm-dev` by writing `~/.config/openshell/gateways/vm-dev/metadata.json`. It does not modify the workspace `.env`.
43+
- Configures the gateway installation name as `vm-dev` and registers the same
44+
name with the CLI by writing
45+
`~/.config/openshell/gateways/vm-dev/metadata.json`. It does not modify the
46+
workspace `.env`.
4447
- Persists the gateway SQLite DB under `.cache/gateway-vm/gateway.db`.
4548
- Places the VM driver state (per-sandbox `overlay.ext4`, image cache, and `run/compute-driver.sock`) under `/tmp/openshell-vm-driver-$USER-vm-dev/` so the AF_UNIX socket path stays under macOS `SUN_LEN`.
4649
- Writes `.cache/gateway-vm/gateway.toml` with `[openshell.drivers.vm].driver_dir = "$PWD/target/debug"` so the freshly built `openshell-driver-vm` is used instead of an older installed copy from `~/.local/libexec/openshell`, `/usr/libexec/openshell`, or `/usr/local/libexec`.
@@ -68,7 +71,7 @@ Override defaults via environment:
6871
# custom port (fails fast if in use)
6972
OPENSHELL_SERVER_PORT=18091 mise run gateway:vm
7073

71-
# custom CLI gateway name + namespace
74+
# custom gateway installation/CLI name + namespace
7275
OPENSHELL_VM_GATEWAY_NAME=vm-feature-a \
7376
OPENSHELL_SANDBOX_NAMESPACE=vm-feature-a \
7477
mise run gateway:vm

0 commit comments

Comments
 (0)