Skip to content

Commit c86c679

Browse files
lambdailambdai
authored
listener: in place filter chain update (envoyproxy#10662)
Provide new listener update path which could update the filter chains without draining all the connections of the old listener. The in place filter chain update flow is an optimization of listener update. If the supportUpdateFilterChain() passes and runtime "envoy.reloadable_features.listener_in_place_filterchain_update" is not explicitly disabled, the existing connections may not be drained if the owning filter chains are not updated in the new listener config. Signed-off-by: Yuchen Dai <[email protected]>
1 parent 49f6e88 commit c86c679

File tree

21 files changed

+1467
-275
lines changed

21 files changed

+1467
-275
lines changed

‎docs/root/configuration/listeners/stats.rst‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -64,13 +64,15 @@ statistics. Any ``:`` character in the stats name is replaced with ``_``.
6464
:header: Name, Type, Description
6565
:widths: 1, 1, 2
6666

67-
listener_added, Counter, Total listeners added (either via static config or LDS)
68-
listener_modified, Counter, Total listeners modified (via LDS)
69-
listener_removed, Counter, Total listeners removed (via LDS)
70-
listener_stopped, Counter, Total listeners stopped
71-
listener_create_success, Counter, Total listener objects successfully added to workers
72-
listener_create_failure, Counter, Total failed listener object additions to workers
73-
total_listeners_warming, Gauge, Number of currently warming listeners
74-
total_listeners_active, Gauge, Number of currently active listeners
75-
total_listeners_draining, Gauge, Number of currently draining listeners
67+
listener_added, Counter, Total listeners added (either via static config or LDS).
68+
listener_modified, Counter, Total listeners modified (via LDS).
69+
listener_removed, Counter, Total listeners removed (via LDS).
70+
listener_stopped, Counter, Total listeners stopped.
71+
listener_create_success, Counter, Total listener objects successfully added to workers.
72+
listener_create_failure, Counter, Total failed listener object additions to workers.
73+
listener_in_place_updated, Counter, Total listener objects created to execute filter chain update path.
74+
total_filter_chains_draining, Gauge, Number of currently draining filter chains.
75+
total_listeners_warming, Gauge, Number of currently warming listeners.
76+
total_listeners_active, Gauge, Number of currently active listeners.
77+
total_listeners_draining, Gauge, Number of currently draining listeners.
7678
workers_started, Gauge, A boolean (1 if started and 0 otherwise) that indicates whether listeners have been initialized on workers.

‎docs/root/version_history/current.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,9 @@ Changes
2424
Can be reverted temporarily by setting runtime feature `envoy.reloadable_features.fix_upgrade_response` to false.
2525
* http: remove legacy connection pool code and their runtime features: `envoy.reloadable_features.new_http1_connection_pool_behavior` and
2626
`envoy.reloadable_features.new_http2_connection_pool_behavior`.
27+
* listener: added in place filter chain update flow for tcp listener update which doesn't close connections if the corresponding network filter chain is equivalent during the listener update.
28+
Can be disabled by setting runtime feature `envoy.reloadable_features.listener_in_place_filterchain_update` to false.
29+
Also added additional draining filter chain stat for :ref:`listener manager <config_listener_manager_stats>` to track the number of draining filter chains and the number of in place update attempts.
2730
* logger: added :ref:`--log-format-prefix-with-location <operations_cli>` command line option to prefix '%v' with file path and line number.
2831
* network filters: added a :ref:`postgres proxy filter <config_network_filters_postgres_proxy>`.
2932
* network filters: added a :ref:`rocketmq proxy filter <config_network_filters_rocketmq_proxy>`.

‎include/envoy/network/connection_handler.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -159,4 +159,4 @@ class ActiveUdpListenerFactory {
159159
using ActiveUdpListenerFactoryPtr = std::unique_ptr<ActiveUdpListenerFactory>;
160160

161161
} // namespace Network
162-
} // namespace Envoy
162+
} // namespace Envoy

‎source/common/runtime/runtime_features.cc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ constexpr const char* runtime_features[] = {
6262
"envoy.deprecated_features.allow_deprecated_extension_names",
6363
"envoy.reloadable_features.ext_authz_http_service_enable_case_sensitive_string_matcher",
6464
"envoy.reloadable_features.fix_upgrade_response",
65+
"envoy.reloadable_features.listener_in_place_filterchain_update",
6566
};
6667

6768
// This is a section for officially sanctioned runtime features which are too

‎source/server/connection_handler_impl.cc‎

Lines changed: 5 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -67,27 +67,15 @@ void ConnectionHandlerImpl::removeListeners(uint64_t listener_tag) {
6767
void ConnectionHandlerImpl::removeFilterChains(
6868
uint64_t listener_tag, const std::list<const Network::FilterChain*>& filter_chains,
6969
std::function<void()> completion) {
70-
// TODO(lambdai): Merge the optimistic path and the pessimistic path.
7170
for (auto& listener : listeners_) {
72-
// Optimistic path: The listener tag provided by arg is not stale.
7371
if (listener.second.listener_->listenerTag() == listener_tag) {
7472
listener.second.tcp_listener_->get().deferredRemoveFilterChains(filter_chains);
7573
// Completion is deferred because the above removeFilterChains() may defer delete connection.
7674
Event::DeferredTaskUtil::deferredRun(dispatcher_, std::move(completion));
7775
return;
7876
}
7977
}
80-
// Fallback to iterate over all listeners. The reason is that the target listener might have began
81-
// another update and the previous tag is lost.
82-
// TODO(lambdai): Remove this once we decide to use the same listener tag during intelligent
83-
// update.
84-
for (auto& listener : listeners_) {
85-
if (listener.second.tcp_listener_.has_value()) {
86-
listener.second.tcp_listener_->get().deferredRemoveFilterChains(filter_chains);
87-
}
88-
}
89-
// Completion is deferred because the above removeFilterChains() may defer delete connection.
90-
Event::DeferredTaskUtil::deferredRun(dispatcher_, std::move(completion));
78+
NOT_REACHED_GCOVR_EXCL_LINE;
9179
}
9280

9381
void ConnectionHandlerImpl::stopListeners(uint64_t listener_tag) {
@@ -399,7 +387,7 @@ void ConnectionHandlerImpl::ActiveTcpListener::newConnection(
399387
std::move(socket), std::move(transport_socket), *stream_info);
400388
ActiveTcpConnectionPtr active_connection(
401389
new ActiveTcpConnection(active_connections, std::move(server_conn_ptr),
402-
parent_.dispatcher_.timeSource(), *config_, std::move(stream_info)));
390+
parent_.dispatcher_.timeSource(), std::move(stream_info)));
403391
active_connection->connection_->setBufferLimits(config_->perConnectionBufferLimitBytes());
404392

405393
const bool empty_filter_chain = !config_->filterChainFactory().createNetworkFilterChain(
@@ -498,13 +486,11 @@ ConnectionHandlerImpl::ActiveConnections::~ActiveConnections() {
498486

499487
ConnectionHandlerImpl::ActiveTcpConnection::ActiveTcpConnection(
500488
ActiveConnections& active_connections, Network::ConnectionPtr&& new_connection,
501-
TimeSource& time_source, Network::ListenerConfig& config,
502-
std::unique_ptr<StreamInfo::StreamInfo>&& stream_info)
489+
TimeSource& time_source, std::unique_ptr<StreamInfo::StreamInfo>&& stream_info)
503490
: stream_info_(std::move(stream_info)), active_connections_(active_connections),
504491
connection_(std::move(new_connection)),
505492
conn_length_(new Stats::HistogramCompletableTimespanImpl(
506-
active_connections_.listener_.stats_.downstream_cx_length_ms_, time_source)),
507-
config_(config) {
493+
active_connections_.listener_.stats_.downstream_cx_length_ms_, time_source)) {
508494
// We just universally set no delay on connections. Theoretically we might at some point want
509495
// to make this configurable.
510496
connection_->noDelay(true);
@@ -521,7 +507,7 @@ ConnectionHandlerImpl::ActiveTcpConnection::ActiveTcpConnection(
521507
}
522508

523509
ConnectionHandlerImpl::ActiveTcpConnection::~ActiveTcpConnection() {
524-
emitLogs(config_, *stream_info_);
510+
emitLogs(*active_connections_.listener_.config_, *stream_info_);
525511

526512
active_connections_.listener_.stats_.downstream_cx_active_.dec();
527513
active_connections_.listener_.stats_.downstream_cx_destroy_.inc();

‎source/server/connection_handler_impl.h‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,6 @@ class ConnectionHandlerImpl : public Network::ConnectionHandler,
199199
public Network::ConnectionCallbacks {
200200
ActiveTcpConnection(ActiveConnections& active_connections,
201201
Network::ConnectionPtr&& new_connection, TimeSource& time_system,
202-
Network::ListenerConfig& config,
203202
std::unique_ptr<StreamInfo::StreamInfo>&& stream_info);
204203
~ActiveTcpConnection() override;
205204

@@ -218,7 +217,6 @@ class ConnectionHandlerImpl : public Network::ConnectionHandler,
218217
ActiveConnections& active_connections_;
219218
Network::ConnectionPtr connection_;
220219
Stats::TimespanPtr conn_length_;
221-
Network::ListenerConfig& config_;
222220
};
223221

224222
/**

‎source/server/filter_chain_manager_impl.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -193,6 +193,10 @@ class FilterChainManagerImpl : public Network::FilterChainManager,
193193
FilterChainFactoryBuilder& b, FilterChainFactoryContextCreator& context_creator);
194194
static bool isWildcardServerName(const std::string& name);
195195

196+
// Return the current view of filter chains, keyed by filter chain message. Used by the owning
197+
// listener to calculate the intersection of filter chains with another listener.
198+
const FcContextMap& filterChainsByMessage() const { return fc_contexts_; }
199+
196200
private:
197201
void convertIPsToTries();
198202
using SourcePortsMap = absl::flat_hash_map<uint16_t, Network::FilterChainSharedPtr>;

0 commit comments

Comments
 (0)