Amazon Security Lake and Google Security Operations (SecOps) are documented as accepting only older Open Cybersecurity Schema Framework (OCSF) versions, 1.3 and earlier and 1.1 respectively. Checked against real OpenShell 1.8.0 records, through their published code, mapping tables, and documentation, neither one gates on the version.
OpenShell emits OCSF 1.8.0 events by default. It also has an in-process downgrade to 1.1 and 1.3 for security information and event management (SIEM) systems documented as supporting only older versions. #4284 has to settle two questions: which single version OpenShell emits (1.1 or 1.8), and how the downgrade goes away. This spike tested what ten SIEMs, data lakes, and pipelines do with real OpenShell records at each version, because their published version tables turned out to be a poor guide.
Five of the ten sinks ran live in local containers. The five cloud services, including Security Lake and Google SecOps, were not run live; their verdicts come from published code, mapping tables, and documentation. The split is listed under Live and inferred results.
- OCSF 1.8.0 records were accepted by every sink tested, and none of them gates on
metadata.version. - The 1.1 and 1.3 downgrade didn't improve compatibility in these tests, because none of the sinks needed the older version. Two of its details affect Tenzir and the XSIAM pack.
- What does break is independent of the version: top-level
container, which nano rejects on every record, and class coverage that differs per sink. - Recommendation: keep emitting 1.8.0, remove the downgrade (#4284), and fix
container(#4283) together with the XSIAM pack author.
No sink tested rejects a record because its version is 1.8.0. None of them reads metadata.version to decide anything. They route on class_uid, category_uid, or class_name, and map fields by path. That holds for the two sinks documented as older-only, too.
- Amazon Security Lake custom sources do not check the version anywhere on the write, crawl, or query path, according to the API reference, the user guide, and AWS's own sample writers. This wasn't observed on a live account. The "1.3 and earlier" statement sits next to a client-side validation tool, and that tool accepts only 1.1.0 and 1.0.0-rc.2, so it rejects 1.3 as well.
- The Google SecOps OCSF parser copies the version into a label and branches on class name.
In these tests the downgrade didn't improve compatibility over native 1.8, because none of the sinks needed the older version. Two details matter if it stays. It writes 1.1 and 1.3 rather than 1.1.0 and 1.3.0, which Tenzir's exact version match doesn't recognize, so Tenzir drops those records. It also removes container, which the Cortex XSIAM content pack uses for sandbox identity. The downgraded records still carry a few fields the older versions don't define: checked against the official schemas, 61 of 63 records don't validate as 1.1.0 and 43 of 63 don't validate as 1.3.0.
Two things do break, and neither depends on the version.
- Top-level
containeris undefined in OCSF 1.8.0 and 1.9.0. nano's strict validator rejects every native record on it, and Tenzir and Cribl drop it. - Class coverage differs per sink. SSH Activity, Device Config State Change, Application Lifecycle, and Base Event are unmapped in Google SecOps and in Splunk's OCSF to Common Information Model (OCSF-CIM) add-on, and Security Lake has no event class for Base Event.
The spike ran in three steps. It captured real OpenShell output, turned it into three corpora that differ only in schema version handling, and put every corpus through each sink the same way.
flowchart LR
subgraph capture["1. Capture"]
direction TB
SB["Three Podman sandboxes built from #4288<br/>(no target, 1.1 target, 1.3 target)"]
TR["Same traffic in each:<br/>denied DNS, TCP and HTTPS, SSH exec"]
BL["Production builders<br/>for classes the traffic misses"]
SB --- TR
end
subgraph corpus["2. Corpus: 63 records per version"]
direction TB
N18["native 1.8.0"]
D11["main's downgrade to 1.1"]
D13["main's downgrade to 1.3"]
end
subgraph run["3. Each sink"]
direction TB
LIVE["Live local run<br/>Elastic, Splunk, nano, Tenzir, Cribl"]
STATIC["Published code or mapping tables<br/>Security Lake, SecOps, XSIAM,<br/>Sentinel, Datadog"]
end
capture --> corpus --> run --> V["Verdict per sink and version:<br/>accepted, degraded, or rejected"]
The corpus is real OpenShell output, 63 records per version and 189 in total:
- 57 records captured from supervisor JSON Lines (JSONL) files in Podman sandboxes built from the #4288 branch. Each sandbox got the same traffic: denied DNS, TCP, and HTTPS connections plus SSH exec. The records cover Base Event, Network Activity, SSH Activity, and Device Config State Change.
- Six records from the production builders for the classes the capture didn't produce: HTTP Activity (denied and allowed), Process Activity, Detection Finding, Application Lifecycle, and API Activity with
ai_model.
Each record exists in three forms: native 1.8.0, the current downgrade to 1.1, and the current downgrade to 1.3. The downgrade forms come from main's downgrade_event, which #4288 leaves unchanged. Several sinks also got control copies that differ from native records only in metadata.version, which separates the effect of the label from the effect of content.
Every sink got the same rules. Nothing used a vendor account or cloud resources, and local containers were capped to fit a 4 GiB Podman virtual machine. Where a sink could run locally, it was tested live. Where it couldn't, its published parser or pipeline code was run or analyzed against the corpus, and documentation was the last resort. A shared script also checked every attribute path in the corpus against the official schema.ocsf.io definitions for 1.1.0, 1.3.0, 1.5.0, and 1.8.0.
These five ran live, as real software in local Podman containers:
- Elastic: Elasticsearch 9.2.0 with the real
amazon_security_lakeintegration pipelines. Fleet and the Parquet read from Security Lake were not part of the run. - Splunk: Splunk Enterprise 10.6.0.5. The OCSF-CIM add-on itself needs a Splunkbase login, so an app rebuilt from its published mappings stood in for it.
- nano: nano's own strict OCSF 1.9 validator, built from source. The full nano stack needs more memory than the test machine had.
- Tenzir: Tenzir 6.19.1.
- Cribl: Cribl Stream 4.20.2. Its Security Lake destination needs AWS, so that part is inferred from its bundled documentation and settings.
These five were not run live, and their verdicts are inferred:
- Amazon Security Lake: AWS's OCSF validation tool ran locally, and pyarrow and DuckDB stood in for the Glue crawler and Athena. A run in a real account was attempted and blocked (see Not yet confirmed).
- Google SecOps: records walked through the parser's published mapping tables, because the parser code isn't public.
- Cortex XSIAM: the content pack's modeling rule re-implemented and run over the corpus.
- Microsoft Sentinel and Datadog: published parser code, pipeline configuration, and documentation.
The table lists what ran for each sink.
| Sink | What ran | Versions tested |
|---|---|---|
| Amazon Security Lake | AWS's OCSF validation tool run locally on the corpus; one Parquet file per class written the way AWS's samples write them; pyarrow schema inference standing in for the Glue crawler and DuckDB for Athena; API reference, user guide, and AWS sample code read | pyarrow 26.0.0, DuckDB 1.5.6 |
| Google SecOps | A script walked every record through the parser's 12 published mapping tables (about 4,700 rows) | Parser docs as published |
| Elastic | Elasticsearch in Podman with the integration's 22 ingest pipelines and index templates; verbose _simulate on every record, then bulk indexing of 1.8 and 1.1 records into one index in both orders |
Elasticsearch 9.2.0, amazon_security_lake 2.10.1 |
| Splunk | Splunk Enterprise in Podman on its local trial license; all records sent through the HTTP Event Collector; field extraction and filter searches; an app rebuilt from the OCSF-CIM add-on's published mappings | Splunk Enterprise 10.6.0.5 |
| Cortex XSIAM | The OpenShell pack's modeling rule, from its open pull requests, re-implemented and run over the corpus | demisto/content PRs #46184 and #46275 |
| Microsoft Sentinel | Azure-Sentinel searched for OCSF parsers; the Halcyon DCR projection applied to the corpus by script | Azure-Sentinel master |
| Datadog | Built-in pipeline configs read (90 OCSF schema processors); status remapper documented behavior checked against the corpus | integrations-core pipelines |
| nano | nano's own strict OCSF 1.9 validator built from source in Podman; corpus run as sent and after nano's metadata stamping | nano at commit b8f16e91 |
| Tenzir | ocsf_cast on every corpus and on relabelled copies; to_amazon_security_lake against a local S3 stand-in |
Tenzir 6.19.1 |
| Cribl | Pipelines configured through the REST API; Parquet writer run with 1.1.0 and 1.3.0 schemas and with the criblio pack schemas | Cribl Stream 4.20.2 |
Two places in the path decide a record's fate, and only Tenzir's depends on the version string. The flowchart traces a record through them.
flowchart TD
R["OpenShell record"] --> Q{"How does the sink treat<br/>metadata.version?"}
Q -->|"Ignores it<br/>Security Lake, SecOps, Elastic, Splunk,<br/>XSIAM, Sentinel, Datadog, Cribl"| C
Q -->|"Exact match<br/>Tenzir"| T{"Known version<br/>string?"}
T -->|"1.8.0"| C
T -->|"1.1 or 1.3<br/>(the downgrade's labels)"| DROP["Dropped"]
Q -->|"Overwrites with 1.9.0, then strict check<br/>nano"| NV{"Top-level<br/>container?"}
NV -->|"Yes: every native record"| REJ["Rejected"]
NV -->|"No: downgraded records"| C
C{"Does the sink map<br/>this class?"} -->|"Yes"| OK["Normalized; fields outside<br/>the sink's mapping kept or dropped"]
C -->|"No: for example SSH, Config State Change,<br/>Base Event in SecOps and Splunk"| RAW["Stored but not normalized<br/>(SecOps outcome unconfirmed)"]
| Sink | Method | Native 1.8.0 | Downgrade 1.1 / 1.3 | Reads metadata.version? |
Confidence |
|---|---|---|---|---|---|
| Amazon Security Lake (custom source) | Docs, AWS sample code, a local Parquet and DuckDB test, live run of AWS's validation tool | Accepted | Accepted, loses the fields the downgrade drops | No; only AWS's separate validation tool does | Medium-high; not run on a live account |
| Google SecOps | Corpus walked through the published per-field mapping tables | Accepted but degraded | Same result as 1.8 | Copied into a label only | Medium |
Elastic (amazon_security_lake 2.10.1) |
Live: Elasticsearch 9.2.0, the integration's 22 ingest pipelines | Accepted, 0 errors | Accepted, identical ECS output, fewer fields | No | High for Elasticsearch; Parquet leg not tested |
| Splunk Enterprise 10.6 | Live indexing; OCSF-CIM add-on rebuilt from its published mappings | Accepted | Accepted, fewer fields | No | High for core, medium for the add-on |
| Cortex XSIAM (OpenShell content pack) | The pack's modeling rule emulated over the corpus | Accepted, fully modeled | Accepted, loses sandbox and host IDs on every record | No | High |
| Microsoft Sentinel | Vendor parsers and DCR patterns in Azure-Sentinel | Accepted; kept fields depend on the DCR | Same | No | Medium |
| Datadog | Published pipeline config and docs | Accepted; OCSF mapping is done at 1.5.0 in the pipeline | Accepted, fewer fields | No | Medium |
| nano | Live: nano's own strict OCSF 1.9 validator | Rejected, 63 of 63, all on container |
Accepted, 63 of 63 | Overwrites it with 1.9.0 | High |
| Tenzir 6.19.1 | Live: ocsf_cast and to_amazon_security_lake |
Accepted, 63 of 63 | Rejected, 63 of 63: unknown OCSF version |
Yes, exact string match | High |
| Cribl Stream 4.20.2 | Live: pipelines and Parquet writer | Accepted; Parquet schema decides what's kept | Accepted, labels written unchanged | No | High; Security Lake destination inferred |
The CreateCustomLogSource API takes no OCSF version. Writing to a custom source is a plain S3 PutObject, and the Glue crawler infers the table schema from the Parquet files, so 1.8-only fields become ordinary columns. The "1.3 and earlier" note on the custom sources page points to the OCSF validation tool, which accepts only 1.1.0 and 1.0.0-rc.2 and exits on anything else. AWS's own transformation library writes version strings such as v1.1.0, which would fail if Security Lake validated the field.
Query quality depends on the Parquet layout far more than on the version. Each object needs one event class with stable column types, and each class has to be registered on the custom source. The class list has no Base Event, so OpenShell's 12 relay records (class_uid: 0) have no class to register under.
A local stand-in for the query path (pyarrow schema inference for the crawler, DuckDB for Athena) showed that 1.8 files only add columns over 1.1 and 1.3 files, with no type conflicts. Mixing 1.8 and 1.1 files in one table queried fine after a crawl, and only queries naming a field the table had never seen failed. Batches of the same version differed in columns about as much as 1.8 and 1.1 did, so column drift comes from batch content rather than from the version.
No subscriber found branches on the version. IBM QRadar is limited by its own parser, which documents OCSF 1.0.0-rc.2 only.
The OCSF parser routes on the class_name string and maps fields by path. Its code isn't public, so the spike walked all 189 records through the published mapping tables (12 class tables, about 4,700 rows). The parsed output is identical for 1.8, 1.1, and 1.3, because the fields that differ between them (container, observation_point_id, ai_model) aren't mapped.
Some fields are lost at every version: proxy_endpoint (the parser only knows the older proxy object), http_request.url.path, firewall_rule, and action, status, and status_detail on Network Activity. Records that fail to parse stay in Raw Log Search with a drop reason in the ingestion metrics, according to Google's documentation.
All 189 records went through the amazon_security_lake integration's ingest pipelines with no errors, and indexing 1.8 and 1.1 records into one index gave no mapping conflicts in either order. The Elastic Common Schema (ECS) output is identical across versions. Fields the integration doesn't define, such as ai_model, stay searchable under ocsf.*, so 1.8 keeps strictly more data. The test didn't cover Security Lake's Parquet write or Elastic Agent's Parquet decode.
Splunk core does no schema validation at ingest, and nested fields extract the same way for every version. The OCSF-CIM add-on needs a Splunkbase login to download, so it was rebuilt from its published mappings. It selects data by the ocsf: sourcetype and maps by class_uid, and its 1.0, 1.1, and 1.2 mapping tables are identical. It doesn't map Base Event, Detection Finding, SSH Activity, Device Config State Change, or Application Lifecycle at any version.
The OpenShell content pack is in open pull requests demisto/content#46184 and #46275, built against native 1.8 output. It never reads the version. It reads top-level container.uid and container.name for sandbox attribution, which is why the downgrade costs it the sandbox ID and host device ID on every record.
Neither has a generic OCSF ingestion path that knows about versions. In Sentinel, what's kept depends on the data collection rule (DCR) the customer writes, and a DCR that stores the whole record keeps everything. The OCSF parsers in Azure-Sentinel all come from vendors that send their own OCSF, and a projection like Halcyon's drops container, http_request, finding_info, and others at any version. Datadog builds its own OCSF tree in a pipeline at a version set in the pipeline config, 1.5.0 for every built-in pipeline. Separately, Datadog's default status remapper turns status: "Failure" (28 of 63 records) into log status emerg.
nano stores only OCSF 1.9.0, overwrites metadata.version before validating, and rejects rows one at a time. Every native record fails on top-level container. With container removed, all 63 pass. Moving it under device.container fails on image.uid, which the image object requires.
Tenzir knows OCSF 1.0.0 through 1.9.0 and casts native 1.8 records cleanly. Its version lookup is an exact string match, so 1.1 and 1.3 fail and the record is dropped. Relabelling 1.8 records as 1.1.0 and casting them gives nearly the same result as the in-process downgrade, so a pipeline can do the downgrade if a sink ever needs one.
Cribl never reads the version. Its Parquet writer silently drops fields that aren't in the selected schema and rejects a whole row on a type mismatch. The criblio OCSF pack schemas date from 1.0.0-rc.2 and type time as a 32-bit integer, which rejects every OpenShell row because millisecond timestamps overflow it.
Keeping 1.8.0 and removing the downgrade doesn't reduce compatibility with any sink tested. Users who set a downgrade target would get records that are at least as usable, and in Tenzir and XSIAM they would keep data the downgrade removes.
A switch to 1.1 doesn't buy compatibility with any sink tested here. It would cost ai_model, the typed trace context #2640 needs, and the XSIAM pack's current mapping. It also needs builder work. In an earlier Podman run, 12 of 21 real records couldn't be valid 1.1.
The work that helps strict consumers doesn't depend on the version, and the first item blocks one of them today.
- Fix top-level
container(#4283). Agree the shape with the XSIAM pack author first, since the pack reads the current field. - Document which classes each major sink maps, so operators know SSH and Config State Change events need custom mapping in some SIEMs.
- Document the Security Lake requirements that do apply. Each Parquet object holds one registered class, and Base Event has no class to register under.
Four tests would settle what the local runs could only approximate. Each needs an account, so none was run. An attempt to run the Security Lake test in an AWS account that belongs to an AWS Organization failed at CreateDataLake, because Security Lake can only be enabled in a standalone account or a delegated Security Lake administrator account. That requirement applies to any OpenShell user who wants a Security Lake custom source.
| Test | Needs | Estimated cost |
|---|---|---|
| Live Security Lake custom source, Glue crawler, and Athena queries | AWS account | Under $1 |
| Splunk's real OCSF-CIM add-on | Free Splunkbase login | None |
Google SecOps runParser dry run on the corpus |
SecOps tenant | Near zero |
| Cribl's Security Lake destination | AWS account | Under $5 |
Three issues in other projects affect OpenShell records at every version and are worth raising upstream.
- AWS's OCSF validation tool sets an empty
class_namefor Device Config State Change, so every record of that class fails its check. - The criblio OCSF pack types
timeas a 32-bit integer, which overflows on millisecond timestamps. - Datadog's default status remapper maps
status: "Failure"toemerg, which mislabels failed operations as emergencies.