Skip to content

fix(asynqdl): truncate dead-letter errors on a rune boundary - #3964

Merged
lyingbug merged 1 commit into
Tencent:mainfrom
kevin9327:fix/dead-letter-utf8-truncation
Oct 7, 2026
Merged

lyingbug merged 1 commit into
Tencent:mainfrom
kevin9327:fix/dead-letter-utf8-truncation

Conversation

@kevin9327

Copy link
Copy Markdown
Contributor

Description

asynqdl.truncateError limits last_error to 8192 bytes by slicing the string at a byte offset (s[:max-len(suffix)]). If the error text contains multi-byte characters, such as Chinese error messages, file names or model output, the cut can land inside a rune. last_error then ends with a partial UTF-8 sequence.

PostgreSQL rejects such a value (invalid byte sequence for encoding "UTF8"), and so does MySQL with utf8mb4 in strict mode (Incorrect string value). The Insert fails, and the middleware only logs that failure. The dead-letter row for the task is then lost, and it is exactly the long, verbose failures that hit this path.

This change backs the cut off to the previous rune start with utf8.RuneStart before appending ...(truncated). ASCII-only errors are truncated exactly as before, and the result is still at most 8192 bytes.

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 💥 Breaking change
  • 📚 Documentation update
  • 🎨 Refactor
  • ⚡ Performance improvement
  • 🧪 Test
  • 🔧 Configuration / Build / CI

Related Issue

None. I found this while reading the dead-letter middleware. Similar earlier fixes: #3056, #813.

Testing

  • Added TestMiddleware_LongMultiByteErrorStaysValidUTF8. It runs a ~12 KB CJK error through the middleware and checks that last_error is at most 8192 bytes, is valid UTF-8, and keeps the suffix.
    • On main it fails with got trailing bytes "\xa5解\xe6\x9e...(truncated)".
    • It passes with this change.
  • go test ./internal/middleware/asynqdl/: ok
  • go vet ./internal/middleware/asynqdl/: ok
  • golangci-lint run --new-from-rev=origin/main ./internal/middleware/asynqdl/...: 0 issues
  • git diff --check: clean

Checklist

  • git diff --check origin/main...HEAD passes
  • Changed source files are formatted
  • Targeted tests for the changed packages/components pass
  • Diff-scoped lint passes where applicable (for Go: golangci-lint run --new-from-rev=origin/main ./...)
  • Full-repository checks were run, or any unrelated/environment-dependent failures are documented above (only the changed package was run)
  • Self-reviewed the code
  • Added/updated tests covering the change
  • Updated related documentation (README, website-docs/, Swagger annotations, etc.) (not applicable)
  • Breaking changes are clearly called out in the description above (none)

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@lyingbug
lyingbug merged commit 20038cb into Tencent:main Oct 7, 2026
4 checks passed
lyingbug pushed a commit that referenced this pull request Oct 21, 2026
…4554)

The dead-letter callback capped the knowledge error_message at 8192 bytes by
slicing the string, which can split a multi-byte UTF-8 rune. PostgreSQL
rejects the invalid UTF-8, so the update fails and the knowledge stays in its
in-flight status. Back off to a rune boundary, as asynqdl does for
last_error since #3964.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants