Skip to content

Commit 9d466c7

Browse files
author
Petr Pchelko
authored
access_log, formatter: Implement omit_empty_values (#12801)
This introduces an option to entirely omit null values from the access log. Risk Level: Low Testing: Unit and integration tests Docs Changes: New option documented in proto file Release Notes: Updated Fixes #12735 Signed-off-by: Petr Pchelko <[email protected]>
1 parent 983e9e9 commit 9d466c7

File tree

18 files changed

+507
-328
lines changed

18 files changed

+507
-328
lines changed

‎api/envoy/config/core/v3/substitution_format_string.proto‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,4 +58,11 @@ message SubstitutionFormatString {
5858
//
5959
google.protobuf.Struct json_format = 2 [(validate.rules).message = {required: true}];
6060
}
61+
62+
// If set to true, when command operators are evaluated to null,
63+
//
64+
// * for ``text_format``, the output of the empty operator is changed from ``-`` to an
65+
// empty string, so that empty values are omitted entirely.
66+
// * for ``json_format`` the keys with null values are omitted in the output structure.
67+
bool omit_empty_values = 3;
6168
}

‎api/envoy/config/core/v4alpha/substitution_format_string.proto‎

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/root/configuration/observability/access_log/usage.rst‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,9 @@ are noted.
116116

117117
Note that if a value is not set/empty, the logs will contain a ``-`` character or, for JSON logs,
118118
the string ``"-"``. For typed JSON logs unset values are represented as ``null`` values and empty
119-
strings are rendered as ``""``.
119+
strings are rendered as ``""``. :ref:`omit_empty_values
120+
<envoy_v3_api_field_config.core.v3.SubstitutionFormatString.omit_empty_values>` option could be used
121+
to omit empty values entirely.
120122

121123
Unless otherwise noted, command operators produce string outputs for typed JSON logs.
122124

‎docs/root/version_history/current.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ New Features
6363
* access log: added a :ref:`dynamic metadata filter<envoy_v3_api_msg_config.accesslog.v3.MetadataFilter>` for access logs, which filters whether to log based on matching dynamic metadata.
6464
* access log: added support for :ref:`%DOWNSTREAM_PEER_FINGERPRINT_1% <config_access_log_format_response_flags>` as a response flag.
6565
* access log: added support for nested objects in :ref:`JSON logging mode <config_access_log_format_dictionaries>`.
66+
* access log: added :ref:`omit_empty_values<envoy_v3_api_field_config.core.v3.SubstitutionFormatString.omit_empty_values>` option to omit unset value from formatted log.
6667
* build: enable building envoy :ref:`arm64 images <arm_binaries>` by buildx tool in x86 CI platform.
6768
* cluster: added new :ref:`connection_pool_per_downstream_connection <envoy_v3_api_field_config.cluster.v3.Cluster.connection_pool_per_downstream_connection>` flag, which enable creation of a new connection pool for each downstream connection.
6869
* dns_filter: added support for answering :ref:`service record<envoy_v3_api_msg_data.dns.v3.DnsTable.DnsService>` queries.

‎generated_api_shadow/envoy/config/core/v3/substitution_format_string.proto‎

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎generated_api_shadow/envoy/config/core/v4alpha/substitution_format_string.proto‎

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎include/envoy/formatter/substitution_formatter.h‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -51,13 +51,14 @@ class FormatterProvider {
5151
* @param response_trailers supplies the response trailers.
5252
* @param stream_info supplies the stream info.
5353
* @param local_reply_body supplies the local reply body.
54-
* @return std::string containing a single value extracted from the given headers/trailers/stream.
54+
* @return absl::optional<std::string> optional string containing a single value extracted from
55+
* the given headers/trailers/stream.
5556
*/
56-
virtual std::string format(const Http::RequestHeaderMap& request_headers,
57-
const Http::ResponseHeaderMap& response_headers,
58-
const Http::ResponseTrailerMap& response_trailers,
59-
const StreamInfo::StreamInfo& stream_info,
60-
absl::string_view local_reply_body) const PURE;
57+
virtual absl::optional<std::string> format(const Http::RequestHeaderMap& request_headers,
58+
const Http::ResponseHeaderMap& response_headers,
59+
const Http::ResponseTrailerMap& response_trailers,
60+
const StreamInfo::StreamInfo& stream_info,
61+
absl::string_view local_reply_body) const PURE;
6162
/**
6263
* Extract a value from the provided headers/trailers/stream, preserving the value's type.
6364
* @param request_headers supplies the request headers.

‎source/common/formatter/substitution_format_string.cc‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,17 @@ namespace Formatter {
77

88
FormatterPtr
99
SubstitutionFormatStringUtils::createJsonFormatter(const ProtobufWkt::Struct& struct_format,
10-
bool preserve_types) {
11-
return std::make_unique<JsonFormatterImpl>(struct_format, preserve_types);
10+
bool preserve_types, bool omit_empty_values) {
11+
return std::make_unique<JsonFormatterImpl>(struct_format, preserve_types, omit_empty_values);
1212
}
1313

1414
FormatterPtr SubstitutionFormatStringUtils::fromProtoConfig(
1515
const envoy::config::core::v3::SubstitutionFormatString& config) {
1616
switch (config.format_case()) {
1717
case envoy::config::core::v3::SubstitutionFormatString::FormatCase::kTextFormat:
18-
return std::make_unique<FormatterImpl>(config.text_format());
18+
return std::make_unique<FormatterImpl>(config.text_format(), config.omit_empty_values());
1919
case envoy::config::core::v3::SubstitutionFormatString::FormatCase::kJsonFormat: {
20-
return createJsonFormatter(config.json_format(), true);
20+
return createJsonFormatter(config.json_format(), true, config.omit_empty_values());
2121
}
2222
default:
2323
NOT_REACHED_GCOVR_EXCL_LINE;

‎source/common/formatter/substitution_format_string.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ class SubstitutionFormatStringUtils {
2525
* Generate a Json formatter object from proto::Struct config
2626
*/
2727
static FormatterPtr createJsonFormatter(const ProtobufWkt::Struct& struct_format,
28-
bool preserve_types);
28+
bool preserve_types, bool omit_empty_values);
2929
};
3030

3131
} // namespace Formatter

0 commit comments

Comments
 (0)