Skip to content

Instantly share code, notes, and snippets.

@zanetworker
Last active October 17, 2026 18:58
Show Gist options
  • Select an option

  • Save zanetworker/dd3bf999d1f18e6557fa5b95bf87712d to your computer and use it in GitHub Desktop.

Select an option

Save zanetworker/dd3bf999d1f18e6557fa5b95bf87712d to your computer and use it in GitHub Desktop.
How SIEMs handle OpenShell's OCSF 1.8 events: spike results

How SIEMs handle OpenShell's OCSF 1.8 events: spike results

Amazon Security Lake and Google Security Operations (SecOps) are documented as accepting only older Open Cybersecurity Schema Framework (OCSF) versions, 1.3 and earlier and 1.1 respectively. Checked against real OpenShell 1.8.0 records, through their published code, mapping tables, and documentation, neither one gates on the version.

OpenShell emits OCSF 1.8.0 events by default. It also has an in-process downgrade to 1.1 and 1.3 for security information and event management (SIEM) systems documented as supporting only older versions. #4284 has to settle two questions: which single version OpenShell emits (1.1 or 1.8), and how the downgrade goes away. This spike tested what ten SIEMs, data lakes, and pipelines do with real OpenShell records at each version, because their published version tables turned out to be a poor guide.

Summary

Five of the ten sinks ran live in local containers. The five cloud services, including Security Lake and Google SecOps, were not run live; their verdicts come from published code, mapping tables, and documentation. The split is listed under Live and inferred results.

  • OCSF 1.8.0 records were accepted by every sink tested, and none of them gates on metadata.version.
  • The 1.1 and 1.3 downgrade didn't improve compatibility in these tests, because none of the sinks needed the older version. Two of its details affect Tenzir and the XSIAM pack.
  • What does break is independent of the version: top-level container, which nano rejects on every record, and class coverage that differs per sink.
  • Recommendation: keep emitting 1.8.0, remove the downgrade (#4284), and fix container (#4283) together with the XSIAM pack author.

Findings

No sink tested rejects a record because its version is 1.8.0. None of them reads metadata.version to decide anything. They route on class_uid, category_uid, or class_name, and map fields by path. That holds for the two sinks documented as older-only, too.

  • Amazon Security Lake custom sources do not check the version anywhere on the write, crawl, or query path, according to the API reference, the user guide, and AWS's own sample writers. This wasn't observed on a live account. The "1.3 and earlier" statement sits next to a client-side validation tool, and that tool accepts only 1.1.0 and 1.0.0-rc.2, so it rejects 1.3 as well.
  • The Google SecOps OCSF parser copies the version into a label and branches on class name.

In these tests the downgrade didn't improve compatibility over native 1.8, because none of the sinks needed the older version. Two details matter if it stays. It writes 1.1 and 1.3 rather than 1.1.0 and 1.3.0, which Tenzir's exact version match doesn't recognize, so Tenzir drops those records. It also removes container, which the Cortex XSIAM content pack uses for sandbox identity. The downgraded records still carry a few fields the older versions don't define: checked against the official schemas, 61 of 63 records don't validate as 1.1.0 and 43 of 63 don't validate as 1.3.0.

Two things do break, and neither depends on the version.

  • Top-level container is undefined in OCSF 1.8.0 and 1.9.0. nano's strict validator rejects every native record on it, and Tenzir and Cribl drop it.
  • Class coverage differs per sink. SSH Activity, Device Config State Change, Application Lifecycle, and Base Event are unmapped in Google SecOps and in Splunk's OCSF to Common Information Model (OCSF-CIM) add-on, and Security Lake has no event class for Base Event.

How it was tested

The spike ran in three steps. It captured real OpenShell output, turned it into three corpora that differ only in schema version handling, and put every corpus through each sink the same way.

flowchart LR
    subgraph capture["1. Capture"]
        direction TB
        SB["Three Podman sandboxes built from #4288<br/>(no target, 1.1 target, 1.3 target)"]
        TR["Same traffic in each:<br/>denied DNS, TCP and HTTPS, SSH exec"]
        BL["Production builders<br/>for classes the traffic misses"]
        SB --- TR
    end
    subgraph corpus["2. Corpus: 63 records per version"]
        direction TB
        N18["native 1.8.0"]
        D11["main's downgrade to 1.1"]
        D13["main's downgrade to 1.3"]
    end
    subgraph run["3. Each sink"]
        direction TB
        LIVE["Live local run<br/>Elastic, Splunk, nano, Tenzir, Cribl"]
        STATIC["Published code or mapping tables<br/>Security Lake, SecOps, XSIAM,<br/>Sentinel, Datadog"]
    end
    capture --> corpus --> run --> V["Verdict per sink and version:<br/>accepted, degraded, or rejected"]
Loading

The corpus is real OpenShell output, 63 records per version and 189 in total:

  • 57 records captured from supervisor JSON Lines (JSONL) files in Podman sandboxes built from the #4288 branch. Each sandbox got the same traffic: denied DNS, TCP, and HTTPS connections plus SSH exec. The records cover Base Event, Network Activity, SSH Activity, and Device Config State Change.
  • Six records from the production builders for the classes the capture didn't produce: HTTP Activity (denied and allowed), Process Activity, Detection Finding, Application Lifecycle, and API Activity with ai_model.

Each record exists in three forms: native 1.8.0, the current downgrade to 1.1, and the current downgrade to 1.3. The downgrade forms come from main's downgrade_event, which #4288 leaves unchanged. Several sinks also got control copies that differ from native records only in metadata.version, which separates the effect of the label from the effect of content.

Every sink got the same rules. Nothing used a vendor account or cloud resources, and local containers were capped to fit a 4 GiB Podman virtual machine. Where a sink could run locally, it was tested live. Where it couldn't, its published parser or pipeline code was run or analyzed against the corpus, and documentation was the last resort. A shared script also checked every attribute path in the corpus against the official schema.ocsf.io definitions for 1.1.0, 1.3.0, 1.5.0, and 1.8.0.

Live and inferred results

These five ran live, as real software in local Podman containers:

  • Elastic: Elasticsearch 9.2.0 with the real amazon_security_lake integration pipelines. Fleet and the Parquet read from Security Lake were not part of the run.
  • Splunk: Splunk Enterprise 10.6.0.5. The OCSF-CIM add-on itself needs a Splunkbase login, so an app rebuilt from its published mappings stood in for it.
  • nano: nano's own strict OCSF 1.9 validator, built from source. The full nano stack needs more memory than the test machine had.
  • Tenzir: Tenzir 6.19.1.
  • Cribl: Cribl Stream 4.20.2. Its Security Lake destination needs AWS, so that part is inferred from its bundled documentation and settings.

These five were not run live, and their verdicts are inferred:

  • Amazon Security Lake: AWS's OCSF validation tool ran locally, and pyarrow and DuckDB stood in for the Glue crawler and Athena. A run in a real account was attempted and blocked (see Not yet confirmed).
  • Google SecOps: records walked through the parser's published mapping tables, because the parser code isn't public.
  • Cortex XSIAM: the content pack's modeling rule re-implemented and run over the corpus.
  • Microsoft Sentinel and Datadog: published parser code, pipeline configuration, and documentation.

The table lists what ran for each sink.

Sink What ran Versions tested
Amazon Security Lake AWS's OCSF validation tool run locally on the corpus; one Parquet file per class written the way AWS's samples write them; pyarrow schema inference standing in for the Glue crawler and DuckDB for Athena; API reference, user guide, and AWS sample code read pyarrow 26.0.0, DuckDB 1.5.6
Google SecOps A script walked every record through the parser's 12 published mapping tables (about 4,700 rows) Parser docs as published
Elastic Elasticsearch in Podman with the integration's 22 ingest pipelines and index templates; verbose _simulate on every record, then bulk indexing of 1.8 and 1.1 records into one index in both orders Elasticsearch 9.2.0, amazon_security_lake 2.10.1
Splunk Splunk Enterprise in Podman on its local trial license; all records sent through the HTTP Event Collector; field extraction and filter searches; an app rebuilt from the OCSF-CIM add-on's published mappings Splunk Enterprise 10.6.0.5
Cortex XSIAM The OpenShell pack's modeling rule, from its open pull requests, re-implemented and run over the corpus demisto/content PRs #46184 and #46275
Microsoft Sentinel Azure-Sentinel searched for OCSF parsers; the Halcyon DCR projection applied to the corpus by script Azure-Sentinel master
Datadog Built-in pipeline configs read (90 OCSF schema processors); status remapper documented behavior checked against the corpus integrations-core pipelines
nano nano's own strict OCSF 1.9 validator built from source in Podman; corpus run as sent and after nano's metadata stamping nano at commit b8f16e91
Tenzir ocsf_cast on every corpus and on relabelled copies; to_amazon_security_lake against a local S3 stand-in Tenzir 6.19.1
Cribl Pipelines configured through the REST API; Parquet writer run with 1.1.0 and 1.3.0 schemas and with the criblio pack schemas Cribl Stream 4.20.2

Results by sink

Two places in the path decide a record's fate, and only Tenzir's depends on the version string. The flowchart traces a record through them.

flowchart TD
    R["OpenShell record"] --> Q{"How does the sink treat<br/>metadata.version?"}
    Q -->|"Ignores it<br/>Security Lake, SecOps, Elastic, Splunk,<br/>XSIAM, Sentinel, Datadog, Cribl"| C
    Q -->|"Exact match<br/>Tenzir"| T{"Known version<br/>string?"}
    T -->|"1.8.0"| C
    T -->|"1.1 or 1.3<br/>(the downgrade's labels)"| DROP["Dropped"]
    Q -->|"Overwrites with 1.9.0, then strict check<br/>nano"| NV{"Top-level<br/>container?"}
    NV -->|"Yes: every native record"| REJ["Rejected"]
    NV -->|"No: downgraded records"| C
    C{"Does the sink map<br/>this class?"} -->|"Yes"| OK["Normalized; fields outside<br/>the sink's mapping kept or dropped"]
    C -->|"No: for example SSH, Config State Change,<br/>Base Event in SecOps and Splunk"| RAW["Stored but not normalized<br/>(SecOps outcome unconfirmed)"]
Loading
Sink Method Native 1.8.0 Downgrade 1.1 / 1.3 Reads metadata.version? Confidence
Amazon Security Lake (custom source) Docs, AWS sample code, a local Parquet and DuckDB test, live run of AWS's validation tool Accepted Accepted, loses the fields the downgrade drops No; only AWS's separate validation tool does Medium-high; not run on a live account
Google SecOps Corpus walked through the published per-field mapping tables Accepted but degraded Same result as 1.8 Copied into a label only Medium
Elastic (amazon_security_lake 2.10.1) Live: Elasticsearch 9.2.0, the integration's 22 ingest pipelines Accepted, 0 errors Accepted, identical ECS output, fewer fields No High for Elasticsearch; Parquet leg not tested
Splunk Enterprise 10.6 Live indexing; OCSF-CIM add-on rebuilt from its published mappings Accepted Accepted, fewer fields No High for core, medium for the add-on
Cortex XSIAM (OpenShell content pack) The pack's modeling rule emulated over the corpus Accepted, fully modeled Accepted, loses sandbox and host IDs on every record No High
Microsoft Sentinel Vendor parsers and DCR patterns in Azure-Sentinel Accepted; kept fields depend on the DCR Same No Medium
Datadog Published pipeline config and docs Accepted; OCSF mapping is done at 1.5.0 in the pipeline Accepted, fewer fields No Medium
nano Live: nano's own strict OCSF 1.9 validator Rejected, 63 of 63, all on container Accepted, 63 of 63 Overwrites it with 1.9.0 High
Tenzir 6.19.1 Live: ocsf_cast and to_amazon_security_lake Accepted, 63 of 63 Rejected, 63 of 63: unknown OCSF version Yes, exact string match High
Cribl Stream 4.20.2 Live: pipelines and Parquet writer Accepted; Parquet schema decides what's kept Accepted, labels written unchanged No High; Security Lake destination inferred

Amazon Security Lake

The CreateCustomLogSource API takes no OCSF version. Writing to a custom source is a plain S3 PutObject, and the Glue crawler infers the table schema from the Parquet files, so 1.8-only fields become ordinary columns. The "1.3 and earlier" note on the custom sources page points to the OCSF validation tool, which accepts only 1.1.0 and 1.0.0-rc.2 and exits on anything else. AWS's own transformation library writes version strings such as v1.1.0, which would fail if Security Lake validated the field.

Query quality depends on the Parquet layout far more than on the version. Each object needs one event class with stable column types, and each class has to be registered on the custom source. The class list has no Base Event, so OpenShell's 12 relay records (class_uid: 0) have no class to register under.

A local stand-in for the query path (pyarrow schema inference for the crawler, DuckDB for Athena) showed that 1.8 files only add columns over 1.1 and 1.3 files, with no type conflicts. Mixing 1.8 and 1.1 files in one table queried fine after a crawl, and only queries naming a field the table had never seen failed. Batches of the same version differed in columns about as much as 1.8 and 1.1 did, so column drift comes from batch content rather than from the version.

No subscriber found branches on the version. IBM QRadar is limited by its own parser, which documents OCSF 1.0.0-rc.2 only.

Google SecOps

The OCSF parser routes on the class_name string and maps fields by path. Its code isn't public, so the spike walked all 189 records through the published mapping tables (12 class tables, about 4,700 rows). The parsed output is identical for 1.8, 1.1, and 1.3, because the fields that differ between them (container, observation_point_id, ai_model) aren't mapped.

Some fields are lost at every version: proxy_endpoint (the parser only knows the older proxy object), http_request.url.path, firewall_rule, and action, status, and status_detail on Network Activity. Records that fail to parse stay in Raw Log Search with a drop reason in the ingestion metrics, according to Google's documentation.

Elastic

All 189 records went through the amazon_security_lake integration's ingest pipelines with no errors, and indexing 1.8 and 1.1 records into one index gave no mapping conflicts in either order. The Elastic Common Schema (ECS) output is identical across versions. Fields the integration doesn't define, such as ai_model, stay searchable under ocsf.*, so 1.8 keeps strictly more data. The test didn't cover Security Lake's Parquet write or Elastic Agent's Parquet decode.

Splunk

Splunk core does no schema validation at ingest, and nested fields extract the same way for every version. The OCSF-CIM add-on needs a Splunkbase login to download, so it was rebuilt from its published mappings. It selects data by the ocsf: sourcetype and maps by class_uid, and its 1.0, 1.1, and 1.2 mapping tables are identical. It doesn't map Base Event, Detection Finding, SSH Activity, Device Config State Change, or Application Lifecycle at any version.

Cortex XSIAM

The OpenShell content pack is in open pull requests demisto/content#46184 and #46275, built against native 1.8 output. It never reads the version. It reads top-level container.uid and container.name for sandbox attribution, which is why the downgrade costs it the sandbox ID and host device ID on every record.

Microsoft Sentinel and Datadog

Neither has a generic OCSF ingestion path that knows about versions. In Sentinel, what's kept depends on the data collection rule (DCR) the customer writes, and a DCR that stores the whole record keeps everything. The OCSF parsers in Azure-Sentinel all come from vendors that send their own OCSF, and a projection like Halcyon's drops container, http_request, finding_info, and others at any version. Datadog builds its own OCSF tree in a pipeline at a version set in the pipeline config, 1.5.0 for every built-in pipeline. Separately, Datadog's default status remapper turns status: "Failure" (28 of 63 records) into log status emerg.

nano, Tenzir, and Cribl

nano stores only OCSF 1.9.0, overwrites metadata.version before validating, and rejects rows one at a time. Every native record fails on top-level container. With container removed, all 63 pass. Moving it under device.container fails on image.uid, which the image object requires.

Tenzir knows OCSF 1.0.0 through 1.9.0 and casts native 1.8 records cleanly. Its version lookup is an exact string match, so 1.1 and 1.3 fail and the record is dropped. Relabelling 1.8 records as 1.1.0 and casting them gives nearly the same result as the in-process downgrade, so a pipeline can do the downgrade if a sink ever needs one.

Cribl never reads the version. Its Parquet writer silently drops fields that aren't in the selected schema and rejects a whole row on a type mismatch. The criblio OCSF pack schemas date from 1.0.0-rc.2 and type time as a 32-bit integer, which rejects every OpenShell row because millisecond timestamps overflow it.

What this means for #4284

Keeping 1.8.0 and removing the downgrade doesn't reduce compatibility with any sink tested. Users who set a downgrade target would get records that are at least as usable, and in Tenzir and XSIAM they would keep data the downgrade removes.

A switch to 1.1 doesn't buy compatibility with any sink tested here. It would cost ai_model, the typed trace context #2640 needs, and the XSIAM pack's current mapping. It also needs builder work. In an earlier Podman run, 12 of 21 real records couldn't be valid 1.1.

The work that helps strict consumers doesn't depend on the version, and the first item blocks one of them today.

  1. Fix top-level container (#4283). Agree the shape with the XSIAM pack author first, since the pack reads the current field.
  2. Document which classes each major sink maps, so operators know SSH and Config State Change events need custom mapping in some SIEMs.
  3. Document the Security Lake requirements that do apply. Each Parquet object holds one registered class, and Base Event has no class to register under.

Not yet confirmed

Four tests would settle what the local runs could only approximate. Each needs an account, so none was run. An attempt to run the Security Lake test in an AWS account that belongs to an AWS Organization failed at CreateDataLake, because Security Lake can only be enabled in a standalone account or a delegated Security Lake administrator account. That requirement applies to any OpenShell user who wants a Security Lake custom source.

Test Needs Estimated cost
Live Security Lake custom source, Glue crawler, and Athena queries AWS account Under $1
Splunk's real OCSF-CIM add-on Free Splunkbase login None
Google SecOps runParser dry run on the corpus SecOps tenant Near zero
Cribl's Security Lake destination AWS account Under $5

Upstream issues found

Three issues in other projects affect OpenShell records at every version and are worth raising upstream.

  • AWS's OCSF validation tool sets an empty class_name for Device Config State Change, so every record of that class fails its check.
  • The criblio OCSF pack types time as a 32-bit integer, which overflows on millisecond timestamps.
  • Datadog's default status remapper maps status: "Failure" to emerg, which mislabels failed operations as emergencies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment