Skip to content

improved schema mapper #1427

@mccanne

Description

@mccanne

(We've talked about this at length and I couldn't find an issue, so sorry if this is redundant.)

The ingest path needs improved ways to transform loosely typed JSON to more structured (but un-rigid) zng schemas. A schema mapper coould be used in context of JSON ingest but the building blocks could be generally usefuly.

Currently, simple rules like field matches identify an entire schema to be used. This ends up generating a large number of fixed schemas that are mapped onto. Instead, the schema mapper should have a flexible set of composable building blocks that don't necessarily define a list of target schemas but rather compose dynamic schemas on the fly by blending the results of the rules.

Also, the JSON reader currently creates aliases for the zenum and port types use by zeek logs. These static aliases should be removed from the JSON reader and moved into the schema mapper.

Metadata

Metadata

Assignees

Labels

duplicateThis issue or pull request already exists

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions