alts: Add retry loop when making RPC in ALTS's TestFullHandshake.#6183
Merged
easwars merged 5 commits intogrpc:masterfrom Apr 11, 2023
Merged
alts: Add retry loop when making RPC in ALTS's TestFullHandshake.#6183easwars merged 5 commits intogrpc:masterfrom
easwars merged 5 commits intogrpc:masterfrom
Conversation
matthewstevenson88
changed the title
matthewstevenson88
changed the title
Contributor
Author
|
@easwars This should be ready for review when you have a chance. :) The failing test seems to be an unrelated XDS flake. |
easwars
requested changes
Apr 11, 2023
| c := testgrpc.NewTestServiceClient(conn) | ||
| if _, err = c.UnaryCall(ctx, &testpb.SimpleRequest{}, grpc.WaitForReady(true)); err != nil { | ||
| t.Errorf("c.UnaryCall() failed: %v", err) | ||
| for ; ctx.Err() == nil; <-time.After(5 * time.Second) { |
Contributor
There was a problem hiding this comment.
I think this should be more like this:
- You have an overall deadline of 10s
- You retry every 10ms
- You make the RPCs without the
WaitForReadycall option - I'm not convinced that
DeadlineExceededis something where you retry. It should ideally be something else. You probably will see the correct code now if you make the RPC without theWaitForReadycall option.
const defaultTestTimeout = 10 * time.Second
const defaultTestShortTimeout = 10 * time.Millisecond
ctx, cancel := context.WithTimeout(context.Background(), defaultTestTimeout)
defer cancel()
c := testgrpc.NewTestServiceClient(conn)
// TODO: A comment explaining why we need to retry the RPC until it succeeds.
for ; ctx.Err() == nil; <-time.After(defaultTestShortTimeout) {
_, err = c.UnaryCall(ctx, &testpb.SimpleRequest{})
if err == nil {
break
}
if code := status.Code(err); code == <expected code when handshake fails> {
continue
}
t.Fatalf("c.UnaryCall() failed with unexpected error: %v", err)
}
if ctx.Err() != nil {
t.Fatalf("Timeout when waiting for UnaryCall RPC to succeed")
}What do you think?
Contributor
Author
There was a problem hiding this comment.
Good call, the WaitForReady option was masking an Unavailable error code, which makes more sense. And thanks for the tips! Implemented as you suggested. :)
easwars
approved these changes
Apr 11, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes the flaky test introduced in #6177 and flagged in #6182. Before this PR, I could reproduce the flake ~1.7% of time on local runs. With this PR, I cannot reproduce the flake locally.
Fixes #6182
RELEASE NOTES: none