More stable kubernetes port forwarding #11538
Conversation
potiuk
force-pushed
the
more-stable-kubernetes-port-forwarding
branch
from
a4e9a9a to
2782072
Compare
potiuk
force-pushed
the
more-stable-kubernetes-port-forwarding
branch
from
2782072 to
740355d
Compare
|
Hey @dimberman -> I hope this one will fix the recent problems with kubernetes port-forward stability. There were a lot of prroblems caused by it recently and I implemented another workaround - increasing port numbers between subsequent tries. I thin ti will solve the problem for good. |
Seems that port forwarding during kubernetes tests started to behave erratically - seems that kubectl port forward sometimes might hang indefinitely rather than connect or fail. We change the strategy a bit to try to allocate increasing port numbers in case something like that happens.
potiuk
force-pushed
the
more-stable-kubernetes-port-forwarding
branch
from
740355d to
3ff8087
Compare
|
@kaxil @ashb @turbaszek -> this one should solve the Kubernetes problems we started to experienced recently. They apparently were not related to the provider split as I originally suspected - but to some changes in the way how port forwarding started to interact with GA runner. So looking forward to reviews 👍 One more thing and maybe you can help me verify my theory. I believe GA is kinda reusing workers without full restarts between them - that might be the reason for 137 errors and resource exhaustion because they do not clean up the machines fully. It could be an accident this is the only explanation for an error I saw yesterday that some other jobs were affected by the kubectl background processes that we started in other jobs. This was an earlier version of the fix, but it did not have the trap that kills (first gently and then forcefully) all kubectl instances running in the background: There were seemingly unrelated errors (in several other jobs). Seems like for other jobs (theoretically in different machines!), the tests were affected by the background-running hanging kubectls, as if the 8080 port numbers continued to be be "taken". I am not 100% sure of that, but that is the only explanation I have for this. The errors went completely away when I added the trap to kill the kubectls (in unrelated jobs !). |
Seems that port forwarding during kubernetes tests started to behave erratically - seems that kubectl port forward sometimes might hang indefinitely rather than connect or fail. We change the strategy a bit to try to allocate increasing port numbers in case something like that happens. (cherry picked from commit 3447b55)
potiuk
added
the
type:misc/internal
Seems that port forwarding during kubernetes tests started to behave erratically - seems that kubectl port forward sometimes might hang indefinitely rather than connect or fail. We change the strategy a bit to try to allocate increasing port numbers in case something like that happens. (cherry picked from commit 3447b55)
Seems that port forwarding during kubernetes tests started to behave erratically - seems that kubectl port forward sometimes might hang indefinitely rather than connect or fail. We change the strategy a bit to try to allocate increasing port numbers in case something like that happens. (cherry picked from commit 3447b55)
Seems that port forwarding during kubernetes tests started to behave erratically - seems that kubectl port forward sometimes might hang indefinitely rather than connect or fail. We change the strategy a bit to try to allocate increasing port numbers in case something like that happens. (cherry picked from commit 3447b55)
Seems that port forwarding during kubernetes tests started to behave erratically - seems that kubectl port forward sometimes might hang indefinitely rather than connect or fail. We change the strategy a bit to try to allocate increasing port numbers in case something like that happens. (cherry picked from commit 3447b55)
2 participants
Seems that port forwarding during kubernetes tests started to behave
erratically - seems that kubectl port forward sometimes might hang
indefinitely rather than connect or fail.
We change the strategy a bit to try to allocate
increasing port numbers in case something like that happens.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.