Skip to content

fix(shared): normalize box corner order before drawing - #52

Merged
JJJYmmm merged 2 commits into
QwenLM:mainfrom
MaxFreedomPollard:fix/draw-boxes-corner-order
Sep 9, 2026
Merged

JJJYmmm merged 2 commits into
QwenLM:mainfrom
MaxFreedomPollard:fix/draw-boxes-corner-order

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown
Contributor

What changed

draw_boxes (src/shared/image.py) passes each bbox_pixel straight to Pillow's draw.rectangle, which since Pillow 9.5 raises ValueError: x1 must be greater than or equal to x0 when the second corner precedes the first. This repo pins pillow<12 with no lower bound, so every current Pillow raises. Two callers are exposed:

  • grounding — a model reply that names the corners in the other order, e.g. [{"label": "cat", "bbox_2d": [800, 100, 200, 400]}], parses fine, but with return_img=true the draw runs after the parser's try/except, so the exception escapes the tool and the detections that had parsed are lost with the call.
  • draw_bbox (core) — the same line, with the reversed box coming from the caller's arguments.

The fix sorts each pair of corners before drawing, so any corner order draws the rectangle it describes and the label lands at its top-left. Well-ordered boxes are unchanged. Grounding's JSON still returns bbox_normalized/bbox_pixel exactly as the model emitted them; I left that alone because crop already rejects a reversed box with a clear message, but can canonicalize them at parse time as well if you prefer.

Regression tests

  • tests/test_api_tools.py::test_grounding_draws_boxes_with_reversed_corners — stubbed model reply with reversed corners and return_img=True: the tool returns its detections and the annotated image block instead of raising.
  • tests/test_tools.py::test_draw_bbox_accepts_reversed_corners — draw_bbox with [800, 100, 200, 400] on the 96×64 sample image saves a 96×64 file whose top-edge pixel (60, 6) carries the requested outline colour, i.e. the box was drawn where the sorted corners put it.

Both fail on main and pass with the fix:

# on main:
tests/test_api_tools.py::test_grounding_draws_boxes_with_reversed_corners FAILED
    ValueError: x1 must be greater than or equal to x0
tests/test_tools.py::test_draw_bbox_accepts_reversed_corners FAILED
    ValueError: x1 must be greater than or equal to x0

Verification

macOS, Python 3.12.14, Pillow 11.3.0:

$ python -m pytest -q -m "not reachability" tests/
445 passed, 30 skipped, 7 deselected in 26.94s      # main: 443 passed, 30 skipped, 7 deselected

$ python scripts/check_manifests.py
OK — 10 capabilities consistent across marketplace.json, plugin manifests and pyproject.toml

$ ruff check src/shared/image.py tests/test_api_tools.py tests/test_tools.py
All checks passed!

$ ruff format --check src/shared/image.py tests/test_api_tools.py
2 files already formatted

tests/test_tools.py already fails ruff format --check on main (ruff 0.16.6); I added to it without reformatting it (24 insertions, 0 deletions), so its format diff is identical before and after.

Not run: the opt-in reachability tests (credentialed; also deselected in CI) and the Ubuntu leg of the CI matrix. The grounding test stubs call_openai_chat, so nothing touches the network.

#51 also touches src/shared/image.py (it adds a function after draw_boxes); I merged the two locally in both orders — no conflict, combined tests pass.

Compatibility

  • Existing MCP tool names, inputs, and outputs are unchanged, or the change is explained above. — Names, inputs and outputs are unchanged for well-ordered boxes; a reversed box now draws instead of raising.
  • Tests and documentation were updated where behavior changed. — Tests above; no documentation described the exception.
  • No credentials, private media, generated artifacts, or machine-specific configuration are included.

This fix was developed with AI assistance; every result quoted above comes from running the listed commands locally.

MaxFreedomPollard and others added 2 commits September 4, 2026 23:41
`draw_boxes` passed each `bbox_pixel` straight to Pillow's `draw.rectangle`,
which since Pillow 9.5 raises `ValueError` when the second corner precedes
the first. A grounding reply that names the corners in the other order,
`[800, 100, 200, 400]`, therefore raised out of `grounding.handle` -- the
draw sits outside its try/except -- and the detections that had parsed fine
were lost with the tool call. The same one-line exposure existed in core's
`draw_bbox`, where the reversed box comes from the caller's arguments.

Sort each pair of corners before drawing, so any corner order draws the
rectangle it describes and the label lands at its top-left.

Tests: grounding with a reversed model box returns its detections and the
annotated image instead of raising; draw_bbox with a reversed box saves an
image with the outline where the sorted corners put it.
@JJJYmmm
JJJYmmm merged commit 1108692 into QwenLM:main Sep 9, 2026
9 checks passed
@JJJYmmm

JJJYmmm commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

TY!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants