You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
*[Integration with Service Config](#integration-with-service-config)
39
-
*[Open issues](#open-issues)
40
-
*[Security](#security)
41
39
42
40
## Abstract
43
41
@@ -172,33 +170,34 @@ Servers may explicitly pushback by setting metadata in their response to the cli
172
170
173
171
Pushback may also be received to a hedged request. If the pushback says not to retry, no further hedged requests will be sent. If the pushback says to retry after a given delay, the next hedged request (if any) will be issued after the given delay has elapsed.
174
172
175
-
A new metadata key, `"grpc-retry-pushback-ms"`, will be added to support server pushback. The value is to be an integer that represents how many milliseconds to wait before sending a retry. If the value for pushback is set to -1, then it will be seen as the server asking the client not to retry at all.
173
+
A new metadata key, `"grpc-retry-pushback-ms"`, will be added to support server pushback. The value is to be an integer with no unnecessary leading zeros that represents how many milliseconds to wait before sending a retry. If the value for pushback is set to -1, then it will be seen as the server asking the client not to retry at all.
176
174
177
175
#### Limits on Retries and Hedges
178
-
Both retries and hedges will be subject to an overall limit that is not set by the service config, but rather exists in the client already. The limit will default to 5, but can be reset by the client explicitly through the use of channel args. This mitigates concerns over the security of the service config being transfered to the client via DNS.
176
+
177
+
Both `maxRetryAttempts` in `retryPolicy` and `maxRequests` in `hedgingPolicy` have, by default, a client-side maximum value of 5. This client-side maximum value can be changed by the client through the use of channel arguments. Service owners may specify a higher value for these parameters, but higher values will be treated as equal to the maximum value by the client implementation. This mitigates security concerns related to the service config being transferred to the client via DNS.
179
178
180
179
#### Summary of Retry and Hedging Logic
181
180
There are five possible types of server responses. The list below enumerates the behavior of retry and hedging policies for each type of response. In all cases, if the maximum number of retry attempts or the maximum number of hedged requests is reached, no further RPCs are sent. Hedged RPCs are returned to the client application when all outstanding and pending requests have either received a response or been canceled.
182
181
183
182
1. OK
184
-
* Retry policy: Successful response, return success to client application
185
-
* Hedging policy: Successful response, cancel previous and pending hedges
183
+
1. Retry policy: Successful response, return success to client application
184
+
2. Hedging policy: Successful response, cancel previous and pending hedges
186
185
187
186
2. Retryable/Non-Fatal Status Code
188
-
* Retry policy: Retry according to policy
189
-
* Hedging policy: Immediately send next scheduled hedged request, if any. Subsequent hedged requests will resume at `hedgingDelayMs`
187
+
1. Retry policy: Retry according to policy
188
+
2. Hedging policy: Immediately send next scheduled hedged request, if any. Subsequent hedged requests will resume at `hedgingDelayMs`
190
189
191
190
3. Fatal Status Code
192
-
* Retry policy: Don't retry, return failure to client application
193
-
* Hedging policy: Cancel previous and pending hedges
191
+
1. Retry policy: Don't retry, return failure to client application
192
+
2. Hedging policy: Cancel previous and pending hedges
194
193
195
194
4. Pushback: Don't retry
196
-
* Retry policy: Don't retry, return failure to client application
197
-
* Hedging policy: Don’t send any more hedged requests.
195
+
1. Retry policy: Don't retry, return failure to client application
196
+
2. Hedging policy: Don’t send any more hedged requests.
198
197
199
198
5. Pushback: Retry in *n* ms
200
-
* Retry policy: Retry in *n* ms. If this attempt also fails, retry delay will reset to initial backoff for the following retry (if applicable)
201
-
* Hedging policy: Send next hedged request in *n* ms. Subsequent hedged requests will resume at `n + hedgingDelayMs`
199
+
1. Retry policy: Retry in *n* ms. If this attempt also fails, retry delay will reset to initial backoff for the following retry (if applicable)
200
+
2. Hedging policy: Send next hedged request in *n* ms. Subsequent hedged requests will resume at `n + hedgingDelayMs`
202
201
203
202

204
203
@@ -284,13 +283,13 @@ Additionally, to present a clearer picture of retry attempts, we add three addit
284
283
1. Total number of retry attempts made
285
284
2. Total number of retry attempts which failed
286
285
3. A histogram of retry attempts made:
287
-
1. The number of retry attempts will be classified into the following buckets:
288
-
1.>=1, >=2, >=3, >=4, >=5, >=10, >=100, >=1000
289
-
2. Each retry attempt increments the count in exactly bucket. For example:
290
-
1. The 1st retry attempt adds one to the ">=1" count
291
-
2. The 2nd retry attempt adds one to the ">=2" count, leaving the count for ">=1" unchanged.
292
-
3. The 5th through 9th retry attempts each add one to the ">=5" count.
293
-
4. The 10th through 99th retry attempts each add one to the ">=10" count.
286
+
1. The number of retry attempts will be classified into the following buckets:
287
+
1.\>=1, >=2, >=3, >=4, >=5, >=10, >=100, >=1000
288
+
2. Each retry attempt increments the count in exactly bucket. For example:
289
+
1. The 1st retry attempt adds one to the ">=1" count
290
+
2. The 2nd retry attempt adds one to the ">=2" count, leaving the count for ">=1" unchanged.
291
+
3. The 5th through 9th retry attempts each add one to the ">=5" count.
292
+
4. The 10th through 99th retry attempts each add one to the ">=10" count.
294
293
295
294
For hedged requests, we record the same stats as above, treating the first hedged request as the initial RPC and subsequent hedged requests as retry attempts.
296
295
@@ -460,9 +459,3 @@ The retry policy is transmitted to the client through the service config mechani
460
459
}
461
460
}
462
461
```
463
-
464
-
## Open issues
465
-
466
-
### Security
467
-
468
-
There is an outstanding security issue concerning the use of service config, in that it may enable distributing potentially malicious configuration including retry policies. This does not change any of the retry logic, but will have to be resolved before retries are made available to service owners.
0 commit comments