Skip to content

RateLimit v3 with 1.9.0 Kills the Ingressgatway #30900

Closed
Closed
RateLimit v3 with 1.9.0 Kills the Ingressgatway#30900
Labels
area/environmentsarea/test and releasearea/upgradeIssues related to upgradeskind/docs
@acumartini

Description

@acumartini
acumartini
opened on Feb 17, 2021

(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities
To ask questions about how to use Istio, please visit https://discuss.istio.io)

Bug description

I had an Envoy ratelimiter working perfectly in 1.8.2. Ingressgateway fails to run after the upgrade to 1.9.0. I pinpointed the failure to the Ratelimiter type.

Failing: "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
Working (most of the time): "@type": type.googleapis.com/envoy.config.filter.http.rate_limit.v2.RateLimit

v2 type working however we get deprecation warning when deployed. Note that the bookinfo exmaple is using v2. I recommend upgrading the bookinfo example and fixing the bug in the latest 1.9.x release.

Thanks!

[x] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[x] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[x] Upgrade

Expected behavior

1.9.0 should support both ratelimter Envoy types:
"@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
"@type": type.googleapis.com/envoy.config.filter.http.rate_limit.v2.RateLimit

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

❯ istioctl version --remote
client version: 1.8.2
control plane version: 1.9.0
data plane version: 1.9.0 (6 proxies)

❯ kubectl version --short
Client Version: v1.17.17-dispatcher
Server Version: v1.17.15-gke.800

❯ helm version --short
v3.5.2+g167aac7

Why can I not attach the bug_report tarball???
bug-report.log

How was Istio installed?

latest helm charts

Environment where the bug was observed (cloud vendor, OS, etc)

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue.
Refer cluster state archive for more details.

Error from gateway logs:

istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.060375Z	critical	envoy main	std::terminate called! (possible uncaught exception, see trace)
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.060411Z	critical	envoy backtrace	Backtrace (use tools/stack_decode.py to get line numbers):
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.060416Z	critical	envoy backtrace	Envoy version: 298ff36b2d43794816f7d8cdc5461bf6eed71bba/1.17.0/Clean/RELEASE/BoringSSL
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.076082Z	critical	envoy main	std::terminate called! (possible uncaught exception, see trace)
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.076116Z	critical	envoy backtrace	Backtrace (use tools/stack_decode.py to get line numbers):
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.076122Z	critical	envoy backtrace	Envoy version: 298ff36b2d43794816f7d8cdc5461bf6eed71bba/1.17.0/Clean/RELEASE/BoringSSL
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.078802Z	critical	envoy backtrace	#0: Envoy::TerminateHandler::logOnTerminate()::$_0::operator()() [0x55e852efd336]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.103651Z	critical	envoy backtrace	#1: [0x55e852efd199]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.103784Z	critical	envoy backtrace	#0: Envoy::TerminateHandler::logOnTerminate()::$_0::operator()() [0x55e852efd336]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.123074Z	critical	envoy backtrace	#2: std::__terminate() [0x55e853594f13]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.137342Z	critical	envoy backtrace	#1: [0x55e852efd199]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.142035Z	critical	envoy backtrace	#3: std::__1::__function::__func<>::operator()() [0x55e85137a256]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.163627Z	critical	envoy backtrace	#2: std::__terminate() [0x55e853594f13]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.169801Z	critical	envoy backtrace	#4: Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerConfig::createFilterChainForFactories() [0x55e852bbdb6d]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.189743Z	critical	envoy backtrace	#5: Envoy::Http::FilterManager::createFilterChain() [0x55e852beb0ce]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.196723Z	critical	envoy backtrace	#3: std::__1::__function::__func<>::operator()() [0x55e85137a256]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.215261Z	critical	envoy backtrace	#6: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders() [0x55e852bd812d]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.222810Z	critical	envoy backtrace	#4: Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerConfig::createFilterChainForFactories() [0x55e852bbdb6d]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.239972Z	critical	envoy backtrace	#7: Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete() [0x55e852bfa511]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.254430Z	critical	envoy backtrace	#5: Envoy::Http::FilterManager::createFilterChain() [0x55e852beb0ce]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.263055Z	critical	envoy backtrace	#8: Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase() [0x55e852bf87bd]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.281682Z	critical	envoy backtrace	#6: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders() [0x55e852bd812d]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.287966Z	critical	envoy backtrace	#9: Envoy::Http::Http1::ConnectionImpl::$_8::__invoke() [0x55e852bf4ff6]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.307287Z	critical	envoy backtrace	#7: Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete() [0x55e852bfa511]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.312368Z	critical	envoy backtrace	#10: http_parser_execute [0x55e852d62cce]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.332145Z	critical	envoy backtrace	#11: Envoy::Http::Http1::ConnectionImpl::dispatchSlice() [0x55e852bf6e11]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.340458Z	critical	envoy backtrace	#8: Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase() [0x55e852bf87bd]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.358652Z	critical	envoy backtrace	#12: Envoy::Http::Http1::ConnectionImpl::innerDispatch() [0x55e852bf669e]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.369313Z	critical	envoy backtrace	#9: Envoy::Http::Http1::ConnectionImpl::$_8::__invoke() [0x55e852bf4ff6]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.383241Z	critical	envoy backtrace	#13: std::__1::__function::__func<>::operator()() [0x55e852bff232]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.396356Z	critical	envoy backtrace	#10: http_parser_execute [0x55e852d62cce]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.409204Z	critical	envoy backtrace	#14: Envoy::Http::Utility::exceptionToStatus() [0x55e852d33ad7]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.425202Z	critical	envoy backtrace	#11: Envoy::Http::Http1::ConnectionImpl::dispatchSlice() [0x55e852bf6e11]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.436811Z	critical	envoy backtrace	#15: Envoy::Http::Http1::ConnectionImpl::dispatch() [0x55e852bf61c1]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.452788Z	critical	envoy backtrace	#12: Envoy::Http::Http1::ConnectionImpl::innerDispatch() [0x55e852bf669e]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.462631Z	critical	envoy backtrace	#16: Envoy::Http::ConnectionManagerImpl::onData() [0x55e852bd42dc]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.480021Z	critical	envoy backtrace	#13: std::__1::__function::__func<>::operator()() [0x55e852bff232]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.485661Z	critical	envoy backtrace	#17: Envoy::Network::FilterManagerImpl::onContinueReading() [0x55e8528de9ff]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.507377Z	critical	envoy backtrace	#14: Envoy::Http::Utility::exceptionToStatus() [0x55e852d33ad7]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.508664Z	critical	envoy backtrace	#18: Envoy::Network::ConnectionImpl::onReadReady() [0x55e8528d8ec9]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.532988Z	critical	envoy backtrace	#19: Envoy::Network::ConnectionImpl::onFileEvent() [0x55e8528d69d9]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.533230Z	critical	envoy backtrace	#15: Envoy::Http::Http1::ConnectionImpl::dispatch() [0x55e852bf61c1]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.558373Z	critical	envoy backtrace	#16: Envoy::Http::ConnectionManagerImpl::onData() [0x55e852bd42dc]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.558576Z	critical	envoy backtrace	#20: std::__1::__function::__func<>::operator()() [0x55e8528cb8c1]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.581839Z	critical	envoy backtrace	#17: Envoy::Network::FilterManagerImpl::onContinueReading() [0x55e8528de9ff]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.589344Z	critical	envoy backtrace	#21: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x55e8528cc6cc]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.606897Z	critical	envoy backtrace	#18: Envoy::Network::ConnectionImpl::onReadReady() [0x55e8528d8ec9]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.614700Z	critical	envoy backtrace	#22: event_process_active_single_queue [0x55e852d5cd0b]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.631143Z	critical	envoy backtrace	#19: Envoy::Network::ConnectionImpl::onFileEvent() [0x55e8528d69d9]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.640686Z	critical	envoy backtrace	#23: event_base_loop [0x55e852d5b66e]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.656417Z	critical	envoy backtrace	#20: std::__1::__function::__func<>::operator()() [0x55e8528cb8c1]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.667421Z	critical	envoy backtrace	#24: Envoy::Server::WorkerImpl::threadRoutine() [0x55e8528bc473]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.681418Z	critical	envoy backtrace	#21: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x55e8528cc6cc]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.694692Z	critical	envoy backtrace	#25: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x55e852f42663]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.694879Z	critical	envoy backtrace	#26: start_thread [0x7fb0861406db]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.694948Z	critical	envoy backtrace	Caught Aborted, suspect faulting address 0x5390000000d
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.694962Z	critical	envoy backtrace	Backtrace (use tools/stack_decode.py to get line numbers):
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.694967Z	critical	envoy backtrace	Envoy version: 298ff36b2d43794816f7d8cdc5461bf6eed71bba/1.17.0/Clean/RELEASE/BoringSSL
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.695954Z	critical	envoy backtrace	#0: __restore_rt [0x7fb08614b980]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.708924Z	critical	envoy backtrace	#22: event_process_active_single_queue [0x55e852d5cd0b]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.728303Z	critical	envoy backtrace	#1: [0x55e852efd199]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.744257Z	critical	envoy backtrace	#23: event_base_loop [0x55e852d5b66e]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.763722Z	critical	envoy backtrace	#2: std::__terminate() [0x55e853594f13]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.775616Z	critical	envoy backtrace	#24: Envoy::Server::WorkerImpl::threadRoutine() [0x55e8528bc473]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.792289Z	critical	envoy backtrace	#3: std::__1::__function::__func<>::operator()() [0x55e85137a256]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.803029Z	critical	envoy backtrace	#25: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x55e852f42663]
istio-ingressgateway-6cc55b7878-vhvz6 istio-proxy 2021-02-17T19:56:11.803296Z	critical	envoy backtrace	#26: start_thread [0x7fb0861406db]

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/environmentsarea/test and releasearea/upgradeIssues related to upgradeskind/docs

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions