Skip to content

Commit b8c31bf

Browse files
committed
feat(server): write gateway OCSF events to JSONL
Previously, gateway security activity was available only in diagnostic output, and events not associated with a sandbox, such as TLS certificate reloads, had no independent structured record. Now, configuring `openshell.gateway.ocsf_log` writes every gateway-produced OCSF record to a bounded JSONL destination independently of `RUST_LOG`. The destination supports daily or disabled rotation, retention limits, queue bounds, and optional schema downgrade to OCSF 1.1 or 1.3. Additionally, existing gateway emitters (TLS reloads, service routing, and policy approval and auto-approval audits) emit structured events, so they reach the JSONL destination, console shorthand, and the affected sandbox's log stream. Records identify the gateway by its configured name in `device.uid` and `device.name`, shared across replicas, with `device.hostname` identifying the replica and `device.os` the gateway's operating system. Metrics and warnings expose known best-effort losses. Refs #2762 Signed-off-by: Kris Hicks <[email protected]>
1 parent 45e3308 commit b8c31bf

30 files changed

Lines changed: 1617 additions & 107 deletions

File tree

‎architecture/gateway.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1301,6 +1301,43 @@ startup and disables export. Export is best-effort — the SDK logs runtime
13011301
failures, and a failed batch is dropped rather than retried. Buffered spans
13021302
flush after the server loop exits so `SIGTERM` does not drop in-flight traces.
13031303

1304+
### OCSF file collection
1305+
1306+
The gateway owns one optional native OCSF JSONL output, configured through
1307+
`openshell.gateway.ocsf_log`. An in-process bounded queue captures
1308+
gateway-produced OCSF records independently of diagnostic filtering and sandbox
1309+
log subscriptions, including events without a sandbox association. It preserves
1310+
event identity and does not translate records into destination-specific schemas.
1311+
An optional sink-level schema target applies the shared lossy downgrade transform
1312+
immediately before queueing, so queue byte limits describe the encoded records
1313+
that reach disk. Internal events remain in the native schema.
1314+
1315+
Console formatting and sandbox log subscriptions also read the structured event:
1316+
the console renders shorthand, and the log bus routes by the affected container's
1317+
UID. Gateway-wide events have no sandbox subscription. These consumers remain
1318+
available without enabling JSONL output. Event origin identifies the producer
1319+
(`Gateway` or `Supervisor`), independently of the affected sandbox.
1320+
1321+
Gateway `device.uid` and `device.name` use the operator-configured gateway name,
1322+
shared by all replicas. `device.hostname` identifies the emitting replica.
1323+
Operators must assign distinct names across installations collected together;
1324+
the default name is not globally unique. Renaming changes the audit identity.
1325+
1326+
Tracing callbacks queue serialized records with count and byte bounds; overflow
1327+
drops incoming records. A dedicated filesystem worker owns appends, recovery of
1328+
incomplete tails, daily rename-based rotation, and retention. Failed writes are
1329+
not replayed, and subsequent records are discarded during bounded reopen backoff.
1330+
Metrics expose writer failures and known or uncertain losses outside the output.
1331+
Tail recovery reports discarded bytes, not an invented missing-record count.
1332+
1333+
Each replica requires its own file. An external shipper owns rotation discovery,
1334+
checkpoints, remote retries, credentials, and SIEM or OTLP conversion. Retention
1335+
must exceed expected shipper outages to avoid pruning unread segments. Concurrent
1336+
writers and external copy-truncate rotation are unsupported. Output failures do
1337+
not block sandbox execution. Shutdown allows five seconds to drain; an outstanding
1338+
OS write cannot be cancelled and remains uncertain. Flush is not fsync, and neither
1339+
the queue nor append results promise durable acceptance or audit completeness.
1340+
13041341
### Package-managed gateway registry
13051342

13061343
The CLI reads its active-gateway and per-gateway metadata from

‎architecture/sandbox.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -749,6 +749,9 @@ failures retain their known peer or listening endpoint.
749749
Configuration diagnostics use Config State Change.
750750
Unix socket relay and relay-control notifications, plus proxy and mediation
751751
failures without an observed network endpoint, use Base Event.
752+
Gateway-origin events describe the gateway process's OS, independently of any
753+
associated sandbox. Linux supervisor events retain Linux as their device OS,
754+
including when a native macOS or Windows gateway manages the sandbox.
752755

753756
## Policy Proposals
754757

‎crates/openshell-driver-mxc/src/etw_consumer.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1885,6 +1885,7 @@ fn etw_ctx(sandbox_id: &str, sandbox_name: &str) -> EventContext {
18851885
sandbox_id: sandbox_id.to_string(),
18861886
sandbox_name: sandbox_name.to_string(),
18871887
container_image: "mxc/appcontainer".to_string(),
1888+
origin: openshell_ocsf::EventOrigin::Supervisor,
18881889
hostname: gateway_hostname().to_string(),
18891890
product_version: env!("CARGO_PKG_VERSION").to_string(),
18901891
proxy_ip: std::net::IpAddr::V4(std::net::Ipv4Addr::LOCALHOST),

‎crates/openshell-ocsf/src/builders/mod.rs‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,18 @@ use crate::enums::StatusId;
161161
use crate::events::base_event::BaseEventData;
162162
use crate::objects::{Container, Device, Endpoint, Image, Metadata, Product};
163163

164+
/// Which `OpenShell` component produced an event.
165+
#[derive(Debug, Clone, PartialEq, Eq)]
166+
pub enum EventOrigin {
167+
/// The supervisor process enforcing sandbox policy.
168+
Supervisor,
169+
/// The gateway process, which has no sandbox or container of its own.
170+
Gateway {
171+
/// Operator-assigned gateway name (`[openshell.gateway] name`).
172+
name: String,
173+
},
174+
}
175+
164176
/// Immutable context created once at sandbox startup.
165177
///
166178
/// Passed to every event builder to populate shared OCSF fields
@@ -181,15 +193,21 @@ pub struct EventContext {
181193
pub proxy_ip: IpAddr,
182194
/// Proxy listen port.
183195
pub proxy_port: u16,
196+
/// Which component is emitting.
197+
pub origin: EventOrigin,
184198
}
185199

186200
impl EventContext {
187201
/// Build the OCSF `Metadata` object for any event.
188202
#[must_use]
189203
pub fn metadata(&self, profiles: &[&str]) -> Metadata {
204+
let product = match self.origin {
205+
EventOrigin::Supervisor => Product::openshell_sandbox(&self.product_version),
206+
EventOrigin::Gateway { .. } => Product::openshell_gateway(&self.product_version),
207+
};
190208
Metadata {
191209
version: OCSF_VERSION.to_string(),
192-
product: Product::openshell_sandbox(&self.product_version),
210+
product,
193211
profiles: profiles.iter().map(|s| (*s).to_string()).collect(),
194212
uid: Some(uuid::Uuid::new_v4().to_string()),
195213
log_source: None,
@@ -215,7 +233,10 @@ impl EventContext {
215233
/// on (Linux for the in-sandbox supervisor, Windows for the MXC gateway).
216234
#[must_use]
217235
pub fn device(&self) -> Device {
218-
Device::for_current_os(&self.hostname)
236+
match &self.origin {
237+
EventOrigin::Supervisor => Device::for_current_os(&self.hostname),
238+
EventOrigin::Gateway { name } => Device::gateway(&self.hostname, name),
239+
}
219240
}
220241

221242
/// Build the `proxy_endpoint` object for the Network Proxy profile.
@@ -255,6 +276,7 @@ pub(crate) fn test_sandbox_context() -> EventContext {
255276
product_version: "0.1.0".to_string(),
256277
proxy_ip: "10.42.0.1".parse().unwrap(),
257278
proxy_port: 3128,
279+
origin: EventOrigin::Supervisor,
258280
}
259281
}
260282

‎crates/openshell-ocsf/src/ctx.rs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
//! not been set (e.g. unit tests that exercise builders without booting the
99
//! sandbox).
1010
11-
use crate::EventContext;
11+
use crate::{EventContext, EventOrigin};
1212
use std::sync::{LazyLock, OnceLock};
1313

1414
static OCSF_CTX: OnceLock<EventContext> = OnceLock::new();
@@ -21,6 +21,7 @@ static OCSF_CTX_FALLBACK: LazyLock<EventContext> = LazyLock::new(|| EventContext
2121
product_version: env!("CARGO_PKG_VERSION").to_string(),
2222
proxy_ip: std::net::IpAddr::from([127, 0, 0, 1]),
2323
proxy_port: 3128,
24+
origin: EventOrigin::Supervisor,
2425
});
2526

2627
/// Initialise the process-wide OCSF sandbox context.

‎crates/openshell-ocsf/src/enums/device_type.rs‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ use serde_repr::{Deserialize_repr, Serialize_repr};
1515
pub enum DeviceTypeId {
1616
/// 0 — Unknown
1717
Unknown = 0,
18+
/// 1 — Server
19+
Server = 1,
1820
/// 99 — Other
1921
Other = 99,
2022
}
@@ -30,6 +32,7 @@ impl std::fmt::Display for DeviceTypeId {
3032
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
3133
formatter.write_str(match self {
3234
Self::Unknown => "Unknown",
35+
Self::Server => "Server",
3336
Self::Other => "Other",
3437
})
3538
}
@@ -43,6 +46,7 @@ mod tests {
4346
fn device_type_display_uses_schema_labels() {
4447
for (device_type, expected) in [
4548
(DeviceTypeId::Unknown, "Unknown"),
49+
(DeviceTypeId::Server, "Server"),
4650
(DeviceTypeId::Other, "Other"),
4751
] {
4852
assert_eq!(device_type.to_string(), expected);
@@ -52,7 +56,11 @@ mod tests {
5256

5357
#[test]
5458
fn device_type_json_roundtrip() {
55-
for (device_type, expected) in [(DeviceTypeId::Unknown, 0), (DeviceTypeId::Other, 99)] {
59+
for (device_type, expected) in [
60+
(DeviceTypeId::Unknown, 0),
61+
(DeviceTypeId::Server, 1),
62+
(DeviceTypeId::Other, 99),
63+
] {
5664
let json = serde_json::to_value(device_type).unwrap();
5765
assert_eq!(json, serde_json::json!(expected));
5866
let decoded: DeviceTypeId = serde_json::from_value(json).unwrap();

‎crates/openshell-ocsf/src/lib.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,8 @@ pub use objects::{
5858
// --- Builders ---
5959
pub use builders::{
6060
ApiActivityBuilder, AppLifecycleBuilder, BaseEventBuilder, ConfigStateChangeBuilder,
61-
DetectionFindingBuilder, EventContext, HttpActivityBuilder, NetworkActivityBuilder,
62-
ProcessActivityBuilder, SshActivityBuilder,
61+
DetectionFindingBuilder, EventContext, EventOrigin, HttpActivityBuilder,
62+
NetworkActivityBuilder, ProcessActivityBuilder, SshActivityBuilder,
6363
};
6464

6565
// --- Tracing layers ---

‎crates/openshell-ocsf/src/objects/device.rs‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,19 @@ pub struct OsInfo {
4040
pub name: String,
4141
}
4242

43+
impl OsInfo {
44+
/// Display name for a `std::env::consts::OS` value.
45+
#[must_use]
46+
pub fn pretty_name(os: &str) -> &str {
47+
match os {
48+
"linux" => "Linux",
49+
"windows" => "Windows",
50+
"macos" => "macOS",
51+
other => other,
52+
}
53+
}
54+
}
55+
4356
impl Device {
4457
/// Create a Linux sandbox device with the given hostname.
4558
#[must_use]
@@ -87,6 +100,22 @@ impl Device {
87100
Self::linux(hostname)
88101
}
89102
}
103+
104+
/// Create the device for a gateway replica.
105+
#[must_use]
106+
pub fn gateway(hostname: &str, name: &str) -> Self {
107+
Self {
108+
hostname: hostname.to_string(),
109+
type_id: DeviceTypeId::Server,
110+
type_label: DeviceTypeId::Server.to_string(),
111+
name: Some(name.to_string()),
112+
// Operators assign a unique name per installation; replicas share this UID.
113+
uid: Some(name.to_string()),
114+
os: Some(OsInfo {
115+
name: OsInfo::pretty_name(std::env::consts::OS).to_string(),
116+
}),
117+
}
118+
}
90119
}
91120

92121
#[cfg(test)]
@@ -140,4 +169,45 @@ mod tests {
140169
assert_eq!(decoded, device);
141170
assert_eq!(serde_json::to_value(&decoded).unwrap(), json);
142171
}
172+
173+
#[test]
174+
fn os_pretty_names_capitalize_known_platforms() {
175+
for (os, expected) in [
176+
("linux", "Linux"),
177+
("windows", "Windows"),
178+
("macos", "macOS"),
179+
("freebsd", "freebsd"),
180+
] {
181+
assert_eq!(OsInfo::pretty_name(os), expected);
182+
}
183+
}
184+
185+
#[test]
186+
fn gateway_device_does_not_inherit_the_sandbox_type() {
187+
let json = serde_json::to_value(Device::gateway("gateway-0", "production")).unwrap();
188+
189+
assert_eq!(json["type_id"], 1);
190+
assert_eq!(json["type"], "Server");
191+
}
192+
193+
#[test]
194+
fn gateway_installations_with_identical_hostnames_have_distinct_uids() {
195+
let first = Device::gateway("openshell-gateway-0", "production");
196+
let second = Device::gateway("openshell-gateway-0", "staging");
197+
198+
assert_eq!(first.uid.as_deref(), Some("production"));
199+
assert_eq!(second.uid.as_deref(), Some("staging"));
200+
assert_ne!(first.uid, second.uid);
201+
assert_eq!(first.hostname, second.hostname);
202+
}
203+
204+
#[test]
205+
fn gateway_replicas_share_the_installation_uid() {
206+
let first = Device::gateway("openshell-gateway-0", "production");
207+
let second = Device::gateway("openshell-gateway-1", "production");
208+
209+
assert_eq!(first.uid.as_deref(), Some("production"));
210+
assert_eq!(first.uid, second.uid);
211+
assert_ne!(first.hostname, second.hostname);
212+
}
143213
}

‎crates/openshell-ocsf/src/objects/metadata.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,16 @@ impl Product {
5151
version: Some(version.to_string()),
5252
}
5353
}
54+
55+
/// Create the `OpenShell` Gateway product, for control-plane events.
56+
#[must_use]
57+
pub fn openshell_gateway(version: &str) -> Self {
58+
Self {
59+
name: "OpenShell Gateway".to_string(),
60+
vendor_name: "OpenShell".to_string(),
61+
version: Some(version.to_string()),
62+
}
63+
}
5464
}
5565

5666
#[cfg(test)]

‎crates/openshell-ocsf/tests/event_identity.rs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
use std::net::{IpAddr, Ipv4Addr};
77

88
use openshell_ocsf::{
9-
ActivityId, Endpoint, EventContext, NetworkActivityBuilder, OcsfEvent, SeverityId,
9+
ActivityId, Endpoint, EventContext, EventOrigin, NetworkActivityBuilder, OcsfEvent, SeverityId,
1010
};
1111

1212
fn sandbox_ctx(container_image: &str) -> EventContext {
@@ -18,6 +18,7 @@ fn sandbox_ctx(container_image: &str) -> EventContext {
1818
product_version: "0.42.1".to_string(),
1919
proxy_ip: IpAddr::V4(Ipv4Addr::LOCALHOST),
2020
proxy_port: 8888,
21+
origin: EventOrigin::Supervisor,
2122
}
2223
}
2324

0 commit comments

Comments
 (0)