This document summarizes the most important changes in the current Zeek

release. For an exhaustive list of changes, see the ``CHANGES`` file

(note that submodules, such as Broker, come with their own ``CHANGES``.)

Zeek 7.2.0

==========

Breaking Changes

----------------

- The ``is_remote_event()``, ``current_analyzer()`` and ``current_event_time()``

builtin functions do not return the previous event's values anymore when event

draining has completed. The same applies to the corresponding C++ accessors on

the ``EventMgr`` class. The functions now return false, 0 or the zero time instead.

New Functionality

-----------------

- Some DNS events are not raised when ``dns_skip_all_addl`` is set to true.

Zeek now raises a warning when a script declares these events while this

option is set to true.

- Types can now be used as constants in Zeek script. This allows types to be

directly passed into BIFs without aliasing.

- A new ``enc_part`` field was added to the Kerberos ``KRB_Response`` record

passed as part of the ``krb_as_response`` event. This field contains the

encrypted session information from a Kerberos response, including the cipher

and encrypted data.

- Geneve tunnel options of the current packet can be extracted from scripts

using the new PacketAnalyzer::Geneve::get_options() builtin function.

- The new ``is_valid_subnet()`` function mirrors ``is_valid_ip()``, for subnets.

- A new Storage framework was merged into the Zeek tree. The intention with this framework

is to eventually replace the storage functionality that Broker provides, including

direct storage via calls such as ``Cluster::create_store`` and ``Broker::put_unique`` as

well as storage-backed tables via the ``&backend`` attribute. This is an initial version

for testing, and will be expanded upon in the future. The current state of the framework

is as follows:

- A new API was added for storage backend plugins.

- Script-level functions for opening and closing backends, and insertion, retrieval, and

erasure of elements are available.

- Backends can support both asynchronous mode (using ``when`` statements) and

synchronous mode (blocking until the operation copmletes). BIF methods were added

under new ``Storage::Async`` and ``Storage::Sync`` modules for these two modes. The

modes can be used interchangeably with the same backend handle.

- SQLite and Redis backends exist in the Zeek tree by default. We are working on a

backend for NATS that will be available as an external plugin, but it is not quite

ready yet. Both of the existing backends support usage in a cluster environment.

- Broker now exposes more information through ``broker.log``. Broker generated

log messages are now propagated as events to Zeek. This allows exposing more

information for debugging and operational behavior of Broker via Zeek logs.

Two new script-level options ``Broker::log_severity_level`` and

``Broker::log_stderr_severity_level`` have been introduced to control

the which events to expose by default.

Changed Functionality

---------------------

- The ``service`` field in the connection log is now sorted in the order that

protocol analyzers raise their confirmation events.

Since the time at which the protocol confirmation is raised depends on the

individual implementation of each analyzer, there is no specific meaning

to the order that the services appear. However, the order should be

deterministic between runs. It also will in many cases represent

the order in which layered protocols are parsed (e.g. "quic,ssl").

- The way that protocol violations are handled by the dynamic protocol

detection (DPD) changed. Now, a violation that is raised by an analyzer

before it is confirmed will immediately disable the analyzer. This adjusts

the behavior back to the historically desired state, and aligns it with

the treatment of confirmed analyzers.

As a consequence of this, the option ``DPD::max_violations`` is no longer used.

It will be retained till Zeek 8.1 to prevent script errors, and raises a

deprecation warning.

To extend the visibility of protocol violations, a new option

``DPD::track_removed_services_in_connection`` was added. Enabling it causes

failed analyzers to no longer be removed from the ``service`` field of the

connection log. Instead, analyzers are never removed after they are

confirmed. Instead, failed analyzers are logged by additionally adding an

entry with a prepended "-". So a connection that attached the ``ssl``

analyzer which later failed due to a protocol error will be logged as

``ssl,-ssl``.

This change also adds a new policy script,

``protocols/conn/failed-service-logging.zeek``. Loading this script adds the

column ``failed_service`` to the connection.log. This column contains the

list of protocol analyzers that failed due to a protocol error.

- Command line options processing will no longer print usage whenever there

is an error. Instead, issues in command line processing will print an error,

then prompt to use --help. The --help usage will now print to standard output

rather than standard error.

- Saving seeds with ``--save-seeds`` will now put Zeek into deterministic mode.

A subsequent ``--load-seeds`` run with the same scripts and traces will produce

identical UID values as the original ``--save-seeds` run.

- The `policy/protocols/dns/detect-external-names.zeek` script now no longer logs

names that were found in mDNS broadcasts by default. This is configurable with

the new `DNS::skip_resp_host_port_pairs` option.

Furthermore, the script now supports and logs IPv6 results.

- The ``mkdir()``, ``rmdir()``, ``unlink()``, and ``rename()`` functions now

trigger reporter warnings instead of builtin errors when hitting trouble. This

allows Zeek to continue gracefully in case of such problems, particularly

during ``zeek_init()``.

- The RDP analyzer now also parses connections that do not contain the cookie

field, which were previously rejected.

Removed Functionality

---------------------

Deprecated Functionality

------------------------

- Support for DNS resolution of hostname literals in Zeek scripts has been

deprecated. If you've used this feature, use the new ``blocking_lookup_hostname()``

builtin function to populate sets or tables in a ``zeek_init()`` handler,

or with top-level statements.

Zeek 7.1.0

==========

We would like to thank Aashish Sharma (@initconf), Andras Gemes (@gemesa),

Anthony Kasza (@anthonykasza), Benjamin Grap (@blightzero), Chiragdeshlehra27,

@cooper-grill, Craig Leres (@leres), Eldon Koyle (@ekoyle), Emmanuele Zambon

(@zambo99), Fox-IT Data Science (@fox-ds), Fupeng Zhao (@AmazingPP), Jan

Grashöfer (@J-Gras), Jordan Barnartt (@JordanBarnartt), Jürgen Löhel (@jloehel),

Justin Azoff (@JustinAzoff), Lucas (@Lucasmeteenc), Martin van Hensbergen

(@martinvanhensbergen), Matti Bispham (@mbispham), Matteo (@skorpion98), Mike

Dopheide (@dopheide-esnet), Mike Peters (@MP-Corelight), Mohan Dhawan

(@Mohan-Dhawan), Pierre (@p-l-), @robinkou, Rodrigo Rojo (@r-rojo), @scyllaever,

Seth Hall (@sethhall), Simeon Miteff (@simeonmiteff), @Sonderino, @superzerosec,

Sven (@svenvanhal), Theo Buehler (@botovq), @timo-mue, @Zopazz, and

@zrobinette12 for their contributions to this release.

Breaking Changes

----------------

- The ``OpaqueVal::DoSerialize`` and ``OpaqueVal::DoUnserialize`` methods were

marked as deprecated in v7.0 and have now been removed as per the Zeek

deprecation policy. Plugins that were overriding these methods and were not

updated will fail to compile. Those plugins should be updated to override the

new ``OpaqueVal::DoSerializeData`` and ``OpaqueVal::DoUnserializeData``

methods.

- Certain internal methods on the broker and logging classes have been changed to

accept std::vector<threading::Value> parameters instead of threading::Value**

to leverage automatic memory management, reduce the number of allocations

and use move semantics to express ownership.

The DoWrite() and HookLogWrite() methods which can be provided by plugins

are not affected by this change, so we keep backwards compatibility with

existing log writers.

- ``Func::Name()`` was deprecated, use ``Func::GetName()`` instead.

New Functionality

-----------------

- The following dependencies have had updates:

- The bundled version of Spicy was updated to 1.12.0. See

https://github.com/zeek/spicy/releases/tag/v1.12.0 for notes on what's new

with Spicy.

- The bundled version of c-ares has been updated to v1.34.2, which required

some updates to Zeek's internal DNS resolver due to changes in the c-ares

API. At least version v1.28.0 is now required to build Zeek.

- Python 3.9 is now required for Zeek and all of it's associated subprojects.

- IP-based connections that were previously not logged due to using an unknown

IP protocol (e.g. not TCP, UDP, or ICMP) now appear in conn.log. All conn.log

entries have a new ``ip_proto`` column that indicates the numeric IP protocol

identifier used by the connection. A new policy script at

``policy/protocols/conn/ip-proto-name-logging.zeek`` can be loaded to also add

an ``ip_proto_name`` column with a string version of the ``ip_proto`` value.

This entire feature can be disabled by loading the new

``policy/protocols/conn/disable-unknown-ip-proto-support.zeek`` policy script.

- New ``Cluster::publish()``, ``Cluster::subscribe()`` and ``Cluster::unsubscribe()``

functions have been added. In contrast to their ``Broker`` counterparts, these

will operator on whichever cluster backend is enabled. Going forward, in-tree

``Broker::publish()`` usages will be replaced with ``Cluster::publish()`` and

script writers should opt to prefer these over the Broker-specific functions.

- Zeek now includes a PostgreSQL protocol analyzer. This analyzer is enabled

by default. The analyzer's events and its ``postgresql.log`` should be

considered preliminary and experimental until the arrival of Zeek's next

long-term-stable release (8.0).

If you observe unusually high CPU consumption or other issues due to this

analyzer being enabled by default, the easiest way to disable it is via the

``Analyzer::disabled_analyzers`` const as follows:

redef Analyzer::disabled_analyzers += {

Analyzer::ANALYZER_POSTGRESQL,

};

If you observe PostgreSQL traffic in your environment, please provide feedback

about the analyzer and structure of the new log.

- Broker's message I/O buffering now operates on per-peering granularity at the

sender (it was previously global) and provides configurable overflow handling

when a fast sender overwhelms a slow receiver, via the following new tunables

in the ``Broker`` module:

const peer_buffer_size = 2048 &redef;

const peer_overflow_policy = "disconnect" &redef;

const web_socket_buffer_size = 512 &redef;

const web_socket_overflow_policy = "disconnect" &redef;

When a send buffer overflows (i.e., it is full when a node tries to transmit

another message), the sender may drop the message and unpeer the slow receiver

(policy ``disconnect``, the default), drop the newest message in the buffer

(``drop_newest``), or drop the oldest (``drop_oldest``). Buffer sizes are

measured in number of messages, not bytes. Note that "sender" and "receiver"

are independent of the direction in which Zeek established the peering. After

disconnects Zeek automatically tries to re-establish peering with the slow

node, in case it recovers.

Zeek notifies you in two ways of such disconnects:

* A cluster.log entry for the sending node indicates that a slow peered node

has been removed. Here node ``worker01`` has removed a peered ``proxy01`:

1733468802.626622 worker01 removed due to backpressure overflow: 127.0.0.1:42204/tcp (proxy01)

* The labeled counter metric ``zeek_broker_backpressure_disconnects_total``

in the telemetry framework tracks the number of times such disconnects

happen between respective nodes. The following scraped telemetry indicates

the same disconnect as above:

zeek_broker_backpressure_disconnects_total{endpoint="worker01",peer="proxy01"} 1

To implement custom handling of a backpressure-induced disconnect, add a

``Broker::peer_removed`` event handler, as follows:

event Broker::peer_removed(endpoint: Broker::EndpointInfo, msg: string)

{

if ( "caf::sec::backpressure_overflow" !in msg )

return;

# The local node has disconnected the given endpoint,

# add your logic here.

}

These new policies fix a problem in which misbehaving nodes could trigger

cascading "lockups" of nodes, each ceasing to transmit any messages.

- The LDAP analyzer now supports handling of non-sealed GSS-API WRAP tokens.

- StartTLS support was added to the LDAP analyzer. The SSL analyzer is enabled

for connections where client and server negotiate to TLS through the extended

request/response mechanism.

- The ``unknown_protocols()`` event now includes the name of all packet

analyzer used for processing the packet when the event is raised. The

``unknown_protocol.log`` file was extended to include this information.

- The MySQL analyzer now generates a ``mysql_user_change()`` event when the user

changes mid-session via the ``COM_USER_CHANGE`` command.

- The DNS analyzer was extended to support TKEY RRs (RFC 2390). A corresponding

``dns_TKEY`` event was added.

- The ``signature_match()`` and custom signature events now receive the end of

match offset within the ``data`` parameter as an optional parameter named

``end_of_match``.

event signature_match(state: signature_state, msg: string, data: string, end_of_match: count);

- A new plugin hook ``InitPreExecution()`` has been added to allow introspection

of Zeek's AST after ZAM optimizations ran. This hook executes right before

the ``zeek_init()`` event is enqueued.

- The SQLite logger now supports setting the value of the SQLite synchronous mode,

as well as of the journal mode. For example, WAL mode can be enabled by setting:

redef LogSQLite::journal_mode=LogSQLite::SQLITE_JOURNAL_MODE_WAL;

- A pseudo protocol analyzer StreamEvent has been added. Attaching this analyzer

to TCP connections allows processing the connection's stream data in the

scripting layer. One example use-case is interactive terminal sessions over

HTTP connections upgraded to TCP.

redef HTTP::upgrade_analyzers += {

["tcp"] = Analyzer::ANALYZER_STREAM_EVENT,

};

event stream_deliver(c: connection, is_orig: bool, data: string);

This comes with performance caveats: For use-cases with high-data rates

a native protocol analyzer with dedicated events will be far more efficient.

- Experimental support for pluggable cluster backends has been added. New plugin

components have been introduced to support switching Zeek's Broker-based

publish-subscribe and remote logging functionality to alternative implementations.

redef Cluster::backend = Cluster::CLUSTER_BACKEND_ZEROMQ;

Besides the backend, the serialization format used for events and log-writes

has become pluggable as well.

- The Zeek distribution now includes an experimental ZeroMQ based cluster backend.

To experiment with it, load the following script on each cluster node.

@load frameworks/cluster/backend/zeromq/connect

Note that Broker-dependent scripts or integrations will become non-functional

when doing so as Zeek nodes will not listen on Broker ports anymore, nor will

they establish a peering to other nodes.

- Zeek now ships with an experimental Spicy-based SSL analyzer, which is

disabled by default. This analyzer can be enabled using the

``--enable-spicy-ssl`` conifgure-time option. The Spicy-based analyzer has

full support for SSL and TLS, just like the current binpac analyzer. It does,

however, not support any version of DTLS. Enabling it will disable DTLS

parsing in Zeek.

The analyzer is currently mostly interesting if you want to experiment with

SSL; we do not yet recommend to enable it in normal Zeek deployments.

- The majority of the metrics reported via stats.log are also now reported via

the Telemetry framework, and are visible in the output passed to Prometheus.

- A new weird ``DNS_unknown_opcode`` was added to the DNS analyzer to report

when it receives opcodes that it cannot process.

Changed Functionality

---------------------

- Heuristics for parsing SASL encrypted and signed LDAP traffic have been

made more strict and predictable. Please provide input if this results in

less visibility in your environment.

- The MySQL analyzer has been improved to better support plugin authentication

mechanisms, like caching_sha2_password, as well as recognizing MySQL query

attributes.

- The ``mysql.log`` for user change commands will contain *just* the username

instead of the remaining parts of the command, including auth plugin data.

- The POP3 parser has been hardened to avoid unbounded state growth in the

face of one-sided traffic capture or when enabled for non-POP3 traffic.

Concretely, the Redis protocol's AUTH mechanism enables the POP3 analyzer

for such connections through DPD.

- Batching and flushing for local log writers can now be controlled via the

options ``Log::flush_interval`` and ``Log::write_buffer_size``. Previously

the ``Threading::heartbeat_interval`` was used for flushing and the buffer

size fixed at 1000.

- Logging of the FTP PASS command in ``ftp.log`` now honors ``FTP::default_capture_password``

and the password is blanked with "<hidden>". Previously, the argument for the PASS

command would be logged in clear.

- The ASCII input reader now suppresses warnings for consecutive invalid lines,

producing a summary of total suppressions once a valid line is encountered.

- The `Telemetry::sync()` hook is now invoked on demand. Either when the metrics

of a node are scraped via the Prometheus HTTP endpoint, or one of the collect

methods is invoked from Zeek script.

- The community-id-logging.zeek policy script was used to set ``c$conn$community_id``

during ``new_connection()`` rather than ``connection_state_remove()``, allowing

other scripts to reuse its value early.

- Calling ``Broker::publish()`` now uses the event time of the currently

executing event as network time metadata attached to the remote event.

Previously, ``network_time()`` was used. This matters if ``Broker::publish()``

is called within scheduled events or called within remote events.

- The SSL analyzer now reports the correct version when an SSLv2 client hello is

used. Zeek previously always reported these as v2, even when the v2 client

hello indicated support for a later version of SSL.

Deprecated Functionality

------------------------

- The ``Broker::auto_publish()`` function has been deprecated and should

be replaced with explicit ``Broker::publish()`` invocations that are

potentially guarded with appropriate ``@if`` or ``@ifdef`` directives.

- The misspelled ``complte_flag`` in the ``dns_binds_rr`` record has been deprecated.

The new ``complete_flag`` uses type ``count`` instead of ``string``.

Zeek 7.0.0

==========

Breaking Changes

----------------

- The Telemetry framework has had a major rework, and includes a number of

breaking changes. The biggest change is a move towards a Prometheus-first

model. This removes the internal aggregation of metrics from nodes onto the

manager node, replacing it with a Prometheus service discovery endpoint. The

usage of this endpoint is described in the updated documentation for the

Telemetry framework. Using this endpoint also requires adding a new

``metrics_port`` field to each node in the cluster configuration, denoting

what port to connect to for each node.

All of the metrics-related script-level options, type, and methods have been

moved to the Telemetry framework:

* Option ``Broker::metrics_port` is now ``Telemetry::metrics_port``

* Option ``Broker::metrics_export_endpoint_name`` is now ``Telemetry::metrics_endpoint_name``

The following options have been removed:

* ``Broker::metrics_export_interval``

* ``Broker::metrics_export_topic``

* ``Broker::metrics_import_topics``

* ``Broker::metrics_export_prefixes``

The ``unit`` field has been removed from the telemetry log.

All of the ``BROKER_METRICS_*`` environment variables have been removed. Use

the equivalent script-level variables for setting configuration settings. For

cluster settings, a new ``metrics_port`` option was added to the ``node.cfg``

options and can be used to set the port number. The cluster layout record

defined in ``cluster-layout.zeek`` has the same option, and it can be

automatically populated by ``zeekctl`` during the ``install`` or ``deploy``

commands.

The instruments that previously supported ``count`` in scripts and ``int64_t``

in C++ were removed in favor of only providing ``double`` versions. Prometheus

only handles ``double`` underneath the Zeek code, so it makes sense to only

support it directly in Zeek as well. This also simplifies the code

significantly.

The ``is_sum`` argument has been removed from the constructors/creation

methods for all of the instruments. This again follows how Prometheus works,

where ``counter`` instruments are always considered sums and ``gauge``

instruments are not. ``Histogram`` instruments don't have the concept of

summing.

- Zeekctl now sets `FileExtract::prefix` to `spool/extract_files/<node>` to avoid

deletion of extracted files when stopping worker nodes. To revert to the

previous behavior, set `FileExtractDir` to an empty string in `zeekctl.cfg`.

If you never enabled Zeek's file extraction functionality, there's no impact.

New Functionality

-----------------

- Support ``delete`` on tables, sets and vectors to clear their contents.

global v = vector(1, 2, 3);

delete v;

assert |v| == 0;

- A new helper function ``can_load()`` backed by a new bif ``find_in_zeekpath()``

was added to determine if a non-relative ``@load`` directive might work. This

can be used to guard ``@load`` directives when script packages may or may not

be installed.

@if ( can_load("my-package") )

@load my-package

@endif

- Zeek packagers can now include a "local" addition into Zeek's version string.

Configure your Zeek build with ``--localversion=XXX`` to add the provided

string, dash-separated, at the end of Zeek's regular version string. The build

setup will refuse strings that contain dashes, to avoid confusing the version

components. For debug builds, the ``-debug`` suffix remains at the end. For

example, the version string on a debug-enabled build with local addition "XXX"

might be "7.0.0-dev.114-XXX-debug". For Docker builds, the ``LOCALVERSION``

environment variable configures the addition.

- SMB2 packets containing multiple PDUs now correctly parse all of the headers,

instead of just the first one and ignoring the rest. This may cause increased

CPU load on SMB2-heavy networks.

- The new built-in function ``lookup_connection_analyzer_id()`` retrieves the

numeric identifier of an analyzer associated with a connection. This enables

the use of the ``disable_analyzer()`` BiF outside of the analyzer

confirmation/violation events that have so far been the only providers of

those identifiers. For example, this allows the suppression of an analyzer

from the outset for specific connections:

event connection_established(c: connection):

{

if ( no_http_for_this_conn_wanted(c) )

{

local aid = lookup_connection_analyzer_id(c$id, Analyzer::ANALYZER_HTTP);

if ( aid > 0 )

disable_analyzer(c$id, aid, T, T);

}

}

Use ``Analyzer::get_tag()`` if you need to obtain an analyzer's tag from its

name (such as "HTTP").

- The ``from_json()`` function now supports ingesting JSON representations of

tables as produced by the ``to_json()`` function. It now also supports reading

the object-based representation of ports that ``to_json()`` generates for that

Zeek type.

- The ``analyzer.log`` now optionally supports logging of disabled analyzers

through the new option ``Analyzer::logging::include_disabling``.

Changed Functionality

---------------------

- The ``ftp.log`` fuid field is now cleared after handling a command with a fuid

associated with it. Previously, fuid was sticky and any subsequent FTP command

would reproduce the same fuid, even if the command itself did not result in

a file transfer over a data connection (e.g., CWD, DEL, PASV, SIZE).

- The type_name field populated by ``global_ids()`` now aligns with the value

returned by ``type_name()`` for each identifier. E.g, ``Site::local_nets``

has a type_name of ``set[subnet]`` rather than ``table``.

- The ISO 9660 file signature has been moved into the policy directory. The

signature has previously been non-functional due to implicit anchoring. Further,

this signature requires users to significantly increase their

``default_file_bof_buffer_size``. Users can now enable this signature by loading

``frameworks/signatures/iso-9660`` which also increases the BOF buffer sufficiently.

Note, doing so may increase memory and CPU usage significantly.

- The ``val_footprint()`` BiF now factors in the size of strings when reporting

footprints, roughly equating a string's size with the number of elements

comparable to that length. As before, footprints are not meant to be precise

but mainly for providing comparisons, which is why this is not a breaking

change.

- The tuning/defaults policy has been deprecated and will be removed in

v7.1. This policy was already being loaded by default via local.zeek. The

settings contained within have become the overall defaults for Zeek now,

instead of having to load the policy. The two changes here are that fragments

now timeout after 5 minutes by default instead of no timeout, and extracted

files now have a default size limit of 100MB instead of unlimited.

- If a Spicy protocol analyzers feeds data into file analysis, it now

needs to call Zeek's `Files::register_protocol()` and provide a

callback for computing file handles. If that's missing, Zeek will

issue a warning. While this was not necessary in previous versions,

it aligns with the same requirement for traditional analyzers and

enables customizing file handles for protocol-specific semantics.

- The Supervisor's API now returns NodeConfig records with a cluster table whose

ClusterEndpoints have a port value of 0/unknown, rather than 0/tcp, to

indicate that the node in question has no listening port.

Removed Functionality

---------------------

Deprecated Functionality

------------------------

- The ``--disable-archiver`` configure flag no longer does anything and will be

removed in 7.1. zeek-archiver has moved into the zeek-aux repository.

- The policy/frameworks/telemetry/prometheus.zeek script has been deprecated

and will be removed with Zeek 7.1. Setting the ``metrics_port`` field on a

``Cluster::Node`` implies listening on that port and exposing telemetry

in Prometheus format.

Zeek 6.2.0

==========

We would like to thank Anthony Verez (netantho), Bijoy Das (mute019), Jan

Grashöfer (J-Gras), Matti Bispham (mbispham), Phil Rzewski (philrz), and

xb-anssi for their contributions to this release.

Breaking Changes

----------------

- The methods ``Dispatcher::Lookup()`` and ``Analyzer::Lookup()`` in the packet_analysis

namespace were changed to return a reference to a std::shared_ptr instead of a copy

for performance reasons.

- Zeek's ``OPENSSL_INCLUDE_DIR`` is not automatically added to an external plugin's

include path anymore. A plugin using OpenSSL functionality directly can use the

following explicit entry to re-use Zeek's ``OPENSSL_INCLUDE_DIR``:

zeek_add_plugin(

Namespace Name

INCLUDE_DIRS "${OPENSSL_INCLUDE_DIR}"

SOURCES ...

)

- The "segment_profiling" functionality and ``load_sample`` event have been removed

without deprecation. This functionality was unmaintained and not known to be used.

- Certain ``ldap.log`` and ``ldap_search.log`` fields have been renamed from

plural to singular and their types changed to scalars. This maps better onto

the expected request-response protocol used between client and server. Additionally,

it removes the burden of working with non-scalar columns from downstream systems.

Specifically, for ``ldap.log``:

* ``arguments: vector of string`` is now ``argument: string``

* ``diagnostic_messages: vector of string`` is now ``diagnostic_message: string``

* ``objects: vector of string`` is now ``object: string``

* ``opcodes: set[string]`` is now ``opcode: string``

* ``results: set[string]`` is now ``result: string``

For ``ldap_search.log``, the following fields were changed:

* ``base_objects: vector of string`` is now ``base_object: string``

* ``derefs: set[string]`` is now ``deref_aliases: string``

* ``diagnostic_messages: vector of string`` is now ``diagnostic_message: string``

* ``results: set[string]`` is now ``result: string``

* ``scopes: set[string]`` is now ``scope: string``

In the unlikely scenario that a request-response pair with the same message

identifier is observed, containing different values for certain fields, new

weirds are raised and will appear in ``weird.log``, including the old and new

values as well as the LDAP message identifier. The value within the LDAP logs

will be the most recently observed one.

- BIF methods now return a ``ValPtr`` directly instead of a ``BifReturnVal`` object

which was just a thin wrapper around ``ValPtr``. This may cause compilation errors

in C++ code that was calling BIF methods directly.

New Functionality

-----------------

- The table type was extended to allow parallel regular expression matching

when a table's index is a pattern. Indexing such tables yields a vector

containing all values of matching patterns for keys of type string.

As an example, the following snippet outputs ``[a, a or b], [a or b]``.

global tbl: table[pattern] of string;

tbl[/a/] = "a";

tbl[/a|b/] = "a or b";

tbl[/c/] = "c";

print tbl["a"], tbl["b"];

Depending on the patterns and input used for matching, memory growth may

be observed over time as the underlying DFA is constructed lazily. Users are

advised to test with realistic and adversarial input data with focus on

memory growth. The DFA's state can be reset by removal/addition of a single

pattern. For observability, a new bif ``table_pattern_matcher_stats()``

can be used to gather ``MatcherStats``.

- Support for delaying log writes.

The logging framework offers two new functions ``Log::delay()`` and ``Log::delay_finish()``

to delay a ``Log::write()`` operation. This new functionality allows delaying of

a specific log record within the logging pipeline for a variable but bounded

amount of time. This can be used, for example, to query and wait for additional

information to attach to the pending record, or even change its final verdict.

Conceptually, delaying a log record happens after the execution of the global

``Log::log_stream_policy`` hook for a given ``Log::write()`` and before the

execution of filter policy hooks. Any mutation of the log record within the

delay period will be visible to filter policy hooks. Calling ``Log::delay()``

is currently only allowed within the context of the ``Log::log_stream_policy`` hook

for the active ``Log::write()` operation (or during the execution of post delay callbacks).

While this may appear restrictive, it makes it explicit which ``Log::write()``

operation is subject to the delay.

Interactions, semantics and conflicts of this feature when writing the same

log record multiple times to the same or different log streams need to be taken

into consideration by script writers.

Given this is the first iteration of this feature, feedback around usability and

use-cases that aren't covered are more than welcome.

- A WebSocket analyzer has been added together with a new ``websocket.log``.

The WebSocket analyzer is instantiated when a WebSocket handshake over HTTP is

recognized. By default, the payload of WebSocket messages is fed into Zeek's dynamic

protocol detection framework, possibly discovering and analyzing tunneled protocols.

The format of the log and the event semantics should be considered preliminary until

the arrival of the next long-term-stable release (7.0).

To disable the analyzer in case of fatal errors or unexpected resource usage,

use the ``Analyzer::disabled_analyzers`` pattern:

redef Analyzer::disabled_analyzers += {

Analyzer::ANALYZER_WEBSOCKET,

};

- The SMTP analyzer was extended to recognize and properly handle the BDAT command

from RFC 3030. This improves visibility into the SMTP protocol when mail agents

and servers support and use this extension.

- The event keyword in signatures was extended to support choosing a custom event

to raise instead of ``signature_match()``. This can be more efficient in certain

scenarios compared to funneling every match through a single event.

The new syntax is to put the name of the event before the string used for the

``msg`` argument. As an extension, it is possible to only provide an event name,

skipping ``msg``. In this case, the framework expects the event's parameters to

consist of only state and data as follows:

signature only-event {

payload /.*root/

event found_root

}

event found_root(state: signature_state, data: string) { }

Using the ``msg`` parameter with a custom event looks as follows. The custom

event's parameters need to align with those for ``signature_match()` event:

signature event-with-msg {

payload /.*root/

event found_root_with_msg "the-message"

}

event found_root_with_msg(state: signature_state, msg: string, data: string) { }

Note, the message argument can currently still be specified as a Zeek identifier

referring to a script-level string value. If used, this is disambiguated behind

the scenes for the first variant. Specifying ``msg`` as a Zeek identifier has

been deprecated with the new event support and will be removed in the future.

Note that matches for signatures with custom events will not be recorded in

``signatures.log``. This log is based on the generation of ``signature_match()``

events.

- The QUIC analyzer has been extended to support analyzing QUIC Version 2

INITIAL packets (RFC 9369). Additionally, prior draft and some of

Facebook's mvfst versions are supported. Unknown QUIC versions will now be

reported in ``quic.log`` as an entry with a ``U`` history field.

- Conditional directives (``@if``, ``@ifdef``, ``@ifndef``, ``@else`` and

``@endif``) can now be placed within a record's definition to conditionally

define or extend a record type's fields.

type r: record {

c: count;

@if ( cond )

d: double;

@else

d: count;

@endif

};

Note that generally you should prefer record extension in conditionally loaded

scripts rather than using conditional directives in the original record definition.

- The 'X' code can now appear in a connection's history. It is meant to indicate

situations where Zeek stopped analyzing traffic due to exceeding certain limits or

when encountering unknown/unsupported protocols. Its first use is to indicate

``Tunnel::max_depth`` being exceeded.

- A new ``Intel::seen_policy`` hook has been introduced to allow intercepting

and changing ``Intel::seen` behavior:

hook Intel::seen_policy(s: Intel::Seen, found: bool)

- A new ``NetControl::rule_added_policy`` hook has been introduced to allow modification

of NetControl rules after they have been added.

- The IP geolocation / ASN lookup features in the script layer provide better

configurability. The file names of MaxMind databases are now configurable via

the new ``mmdb_city_db``, ``mmdb_country_db``, and ``mmdb_asn_db`` constants,

and the previously hardwired fallback search path when not using an

``mmdb_dir`` value is now adjustable via the ``mmdb_dir_fallbacks``

vector. Databases opened explicitly via the ``mmdb_open_location_db`` and

``mmdb_open_asn_db`` functions now behave more predictably when updated or

removed. For details, see:

https://docs.zeek.org/en/master/customizations.html#address-geolocation-and-as-lookups

- The ``zeek-config`` script now provides a set of ``--have-XXX`` checks for

features optionally compiled in. Each check reports "yes"/"no" to stdout and

exits with 0/1, respectively.

Changed Functionality

---------------------

- The ``split_string`` family of functions now respect the beginning-of-line ^ and

end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere

in the input string.

- The ``sub()`` and ``gsub()` functions now respect the beginning-of-line ^ and

end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere

in the input string.

- Ed25519 and Ed448 DNSKEY and RRSIG entries do not cause weirds anymore.

- The OpenSSL references in ``digest.h`` and ``OpaqueVal.h`` headers have been

hidden to avoid unneeded dependencies on OpenSSL headers. Plugins using the

detail API from ``digest.h`` to compute hashes likely need to accommodate for

this change.

- The ``Tunnel::max_depth`` default was changed from 2 to 4 allowing for more than

two encapsulation layers. Two layers are already easily reached in AWS GLB

environments.

- Nested MIME message analysis is now capped at a maximum depth of 100 to prevent

unbounded MIME message nesting. This limit is configurable with ``MIME::max_depth``.

A new weird named ``exceeded_mime_max_depth`` is reported when reached.

- The ``netcontrol_catch_release.log`` now contains a plugin column that shows which

plugin took an action. The logs also contain information when errors or existing

rules are encountered.

- The ``Cluster::PoolSpec`` record no longer provides default values for its

``topic`` and ``node_type`` fields, since defaults don't fit their intended

use and looked confusing in generated documentation.

Removed Functionality

---------------------

- Zeek no longer automatically subscribes to topics prefixed with "bro/" whenever

subscribing to topics prefixed with "zeek/". This was a leftover backward-

compatibility step in the Broker code that should have been removed long ago.

Deprecated Functionality

------------------------

- The virtual functions ``DoSerialize``and ``DoUnserialize`` of the ``OpaqueVal``

(and ``BloomFilter``) class will be removed with Zeek 7.1. Unfortunately, code

implementing the deprecated methods does not produce compiler warnings.

Plugin authors implementing an ``OpaqueVal`` subclass need to convert to

``DoSerializeData`` and ``DoUnserializeData``:

* ``std::optional<BrokerData> OpaqueVal::DoSerializeData() const``

* ``bool OpaqueVal::DoUnserializeData(BrokerDataView data)``

When overriding ``DoSerializeData()``, return ``std::nullopt`` (or a

default-constructed ``optional``) for values that cannot be serialized.

Otherwise, the canonical way to create a ``BrokerData`` for serialization is

by using a ``BrokerListBuilder``. For example, creating a ``BrokerData`` that

contains ``true`` and the count ``42`` could be implemented as follows:

BrokerListBuilder builder;

builder.Add(true);

builder.AddCount(42u);

return std::move(builder).Build();

Please refer to the respective class documentation for a full list of member

functions on ``BrokerListBuilder`` and ``BrokerDataView``.

For plugins that are using the macro ``DECLARE_OPAQUE_VALUE`` to generate the

function prototypes for the serialization functions: please use

``DECLARE_OPAQUE_VALUE_DATA`` instead to generate prototypes for the new API.

Plugin authors that need to support multiple Zeek versions can use the

``ZEEK_VERSION_NUMBER`` macro to conditionally implement the new and old

methods. Provide the new versions with Zeek 6.2 (60200) or later, otherwise

keep the old signature. The default implementations for the new functions

as used by Zeek will call the old signatures and convert the results.

- The ``Cluster::Node$interface`` field has been deprecated. It's essentially

unneeded, unused and not a reliable way to gather the actual interface used

by a worker. In Zeekctl deployments the field will be populated until its

removal. The ``packet_source()`` bif should be used on worker processes to

gather information about the interface.

- The ``policy/misc/load-balancing`` script has been deprecated in favor of

AF_PACKET PF_RING, Netmap or other NIC specific load balancing approaches.

- Time machine related enums, options and fields have been marked for removal.

- The ``check_for_unused_event_handlers`` options the related ``UsedHandlers()``,

``UnusedHandlers()`` and their related ``SetUsed()`` and ``Used()`` methods

have been marked for removal. The feature of finding unused event handlers is

provided by default via the ``UsageAnalyzer`` component.

- Using a Zeek identifier for the ``msg`` argument within a signatures's ``event``

keyword has been deprecated.

Zeek 6.1.0

==========

Breaking Changes

----------------

- ``assert`` is now a reserved keyword for the new ``assert`` statement.

- The ``__bro_plugin__`` file that gets generated as part of plugin builds was

renamed to ``__zeek_plugin__``. This will affect the ability for older

versions of ``zkg`` to use the ``zkg unload`` and ``zkg load`` commands. This

should only cause breakage for people using a version of ``zkg` that doesn't

come bundled with Zeek (which we generally don't recommend doing).

- Zeek does not traverse into dot directories to find plugins or hlto files

anymore. Any dot directories found below the directories specified in

ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are now skipped. Dot directories

explicitly listed in ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are not

skipped.

- External plugins will fail to configure if their minimum required CMake

version is below 3.15. This was a warning with Zeek 6.0, but has caused user

confusion due to unhelpful error messages around the IN_LIST operator policy.

- The FindBISON, FindOpenSSL, FindPackageHandleStandardArgs, FindPackageMessage,

and SelectLibraryConfigurations cmake files were removed from our cmake

repository in favor of the versions that come with CMake. This should not

cause any breakage, but it is possible in the case that someone was using

these in a plugin.

New Functionality

-----------------

- Zeek now includes the LDAP protocol analyzer from the zeek/spicy-ldap project

(https://github.com/zeek/spicy-ldap). This analyzer is enabled by default. The

analyzer's events and its ``ldap.log`` and ``ldap_search.log`` should be

considered preliminary and experimental until the arrival of Zeek's next

long-term-stable release (7.0).

If you observe unusually high CPU consumption or other issues due to this

analyzer being enabled by default, the easiest way to disable it is via the

``Analyzer::disabled_analyzers`` const as follows:

redef Analyzer::disabled_analyzers += {

Analyzer::ANALYZER_LDAP_UDP,

Analyzer::ANALYZER_LDAP_TCP,

};

Please do report issues to us including diagnostic information in case this is

necessary in your environment. We're also open to general feedback about the

structure of the new logs.

- Zeek now includes the QUIC protocol analyzer from the zeek/spicy-quic project

(https://github.com/zeek/spicy-quic). This project is a fork of Fox-IT's

initial implementation (https://github.com/fox-ds/spicy-quic).

As for the LDAP analyzer, the analyzer's events and the new ``quic.log``

should be considered preliminary and experimental until the arrival of Zeek's

next long-term-stable release (7.0). As above, any feedback and contributions

to this analyzer and the new log are welcome.

The analyzer's functionality is limited to decryption of the INITIAL packets

of QUIC version 1. If decryption of these packets is successful, the handshake

data is forwarded to Zeek's SSL analyzer. An ``ssl.log`` entry will appear in

``ssl.log`` for QUIC connections. The entry in the ``conn.log`` will contain

``quic`` and ``ssl`` in the service field.

To disable the analyzer in case of issues, use the following snippet:

redef Analyzer::disabled_analyzers += {

Analyzer::ANALYZER_QUIC,

};

- Added a new ``assert`` statement for assertion based testing and asserting

runtime state.

assert <expr: bool>[, <message: string>];

This statement comes with two hooks. First, ``assertion_failure()`` that is

invoked for every failing assert statement. Second, ``assertion_result()``

which is invoked for every assert statement and its outcome. The latter allows

to construct a summary of failing and passing assert statements. Both hooks

receive the location and call stack for the ``assert`` statement via a

``Backtrace`` vector.

A failing assert will abort execution of the current event handler similar to

scripting errors. By default, a reporter error message is logged. Using the

break statement within ``assertion_failure()`` or ``assertion_result()``

allows to suppress the default message.

- Add a new ``&default_insert`` attribute for tables. This behaves as

``&default`` with the addition that the default value is inserted into the

table upon a failed lookup. Particularly for tables with nested container

values, the ``&default`` behavior of not inserting the value can be of little

use.

- The ``from_json()`` function now takes an optional key_func argument to

normalize JSON object key names. This can be useful if the keys in a JSON

object are not valid Zeek identifiers or reserved keywords.

- Module names are now included in ``global_ids()``. Their key in the returned

table is prefixed with "module " and their value will have the ``type_name``

field set to "module".

- Identifiers in the global scope can now be referenced and defined from within

modules by prefixing their names with ``::``. Previously, these required an

explicit ``GLOBAL::`` prefix to be used. Using ``GLOBAL::`` has been

deprecated.

- The ``as`` keyword now supports casting between ``set`` and ``vector`` values

with the same element type. Converting ``set`` values with multiple index

values is not supported. We plan to extend the use of the ``as`` keyword to

support more type conversions in the future.

- Added new packet analyzer to handle PCAP files DLT_PPP link type.

- Fixed appending of ``any`` to ``vector of any``.

- The ModBus analyzer's function support was expanded, with new handling of the

Encapsulation Interface Transport (function 28) And Diagnostics (function 8)

functions. This adds new ``modbus_encap_interface_transport_{request,response}``

and ``modbus_diagnostics_{request,response}`` events.

- The ModBus file record read and write events now provide the full data from

the request and response messages as part of the event data.

- The full PDU length was added to the ``ModBusHeader`` record type passed with