Skip to content

fix(api-breaker): inconsistent circuit breaking due to premature breaker_time increment#12451

Merged
Revolyssup merged 1 commit into
apache:masterfrom
Revolyssup:revolyssup/fix-api-breaker
Jul 21, 2025
Merged

fix(api-breaker): inconsistent circuit breaking due to premature breaker_time increment#12451
Revolyssup merged 1 commit into
apache:masterfrom
Revolyssup:revolyssup/fix-api-breaker

Conversation

@Revolyssup

@Revolyssup Revolyssup commented Jul 21, 2025

Copy link
Copy Markdown
Contributor

Description

The circuit breaker did not operate as configured. In the following logs, the circuit breaker condition is triggered by 5 occurrences of 500/502/503 responses, and lines 18 and 19 should not have tripped

Reason

When set to ceil breaker_time is prematurely increased causing inconsistency. Consider the flow​:
5 requests fail->2 second breaker_time->6th request fail and due to ceil, breaker_time is immediately set to 4 seconds. So if 7th request comes just before 4 seconds, it will be blocked as well.
The behaviour should be 5 requests fail->2 second circuit break->5 more requests fail->4 second circuit break->5 more requests fail->8 second circuit break.... And so on

Reproduction steps

curl http://127.0.0.1:9180/apisix/admin/routes/3 -H "X-API-KEY: rycegJXdCGBWaFwXbadcNvcbgYgwOmyG" -X PUT -d '
{
    "uri": "/status/*",
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "httpbin.org:80": 1
        }
    },
    "plugins": {
        "api-breaker": {
            "break_response_code": 599,
            "break_response_body": "触发熔断",
            "max_breaker_sec": 60,
            "unhealthy": {
                "http_statuses": [
                    500,
                    502,
                    503
                ],
                "failures": 5
            },
            "healthy": {
                "http_statuses": [
                    200,
                    201,
                    202,
                    203,
                    301,
                    302,
                    304,
                    400,
                    401,
                    403,
                    404,
                    413,
                    429
                ],
                "successes": 3
            }
        }
    }
}'

while true; do
  timestamp=$(date +"[%Y-%m-%d %H:%M:%S]")
  status_code=$(curl -s -o /dev/null -w "%{http_code}" "http://127.0.0.1:9080/status/500")
  echo "$timestamp $status_code"
  sleep 1
done
[2025-07-09 17:03:20] 500
[2025-07-09 17:03:21] 500
[2025-07-09 17:03:24] 500
[2025-07-09 17:03:25] 500
[2025-07-09 17:03:27] 500
[2025-07-09 17:03:28] 599
[2025-07-09 17:03:29] 599
[2025-07-09 17:03:30] 500
[2025-07-09 17:03:33] 500
[2025-07-09 17:03:34] 500
[2025-07-09 17:03:36] 500
[2025-07-09 17:03:39] 502
[2025-07-09 17:03:41] 599
[2025-07-09 17:03:42] 599
[2025-07-09 17:03:43] 599
[2025-07-09 17:03:44] 599
[2025-07-09 17:03:45] 500
[2025-07-09 17:03:47] 599
[2025-07-09 17:03:48] 599
[2025-07-09 17:03:49] 500

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. bug Something isn't working labels Jul 21, 2025
@Revolyssup
Revolyssup requested review from membphis and nic-6443 July 21, 2025 10:46
@Revolyssup
Revolyssup merged commit fa1ab6e into apache:master Jul 21, 2025
35 of 36 checks passed
@Revolyssup
Revolyssup deleted the revolyssup/fix-api-breaker branch July 21, 2025 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants