Skip to content

Delegate image processing to raygeo.image, keep only Cairo wrappers#278

Closed
knipknap wants to merge 18 commits into
mainfrom
feat/raygeo-image-integration
Closed

Delegate image processing to raygeo.image, keep only Cairo wrappers#278
knipknap wants to merge 18 commits into
mainfrom
feat/raygeo-image-integration

Conversation

@knipknap

@knipknap knipknap commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Replaces all pure-Python image processing implementations with calls to raygeo.image Rust functions
  • surface_to_* functions remain as thin Cairo surface buffer extraction wrappers
  • Removes 405 lines of Python image processing code (-68% in affected files)

Changes by file

File Change
grayscale.py surface_to_grayscale/binary/inplace → extract Cairo buffer, call raygeo.image.rgba_to_*. Removed compute_auto_levels, normalize_grayscale (now from raygeo.image)
srgb.py srgb_to_linear/linear_to_srgb now delegate to raygeo.image. Retains create_lut_from_color and resize_linear_nd (no Rust equivalents)
dither.py apply_floyd_steinberg_dither/apply_bayer_dither/apply_minimum_run_length removed, imported from raygeo.image. surface_to_dithered_array uses raygeo.image.rgba_to_grayscale + Rust dither
__init__.py compute_auto_levels/normalize_grayscale re-exported from raygeo.image
raster_widget.py Imports compute_auto_levels from raygeo.image directly
Tests Updated imports, fixed dtype (float32uint8), adjusted rounding expectations

Testing

All 422 image tests pass.

knipknap added 2 commits June 5, 2026 19:34
Replace pure-Python implementations of srgb_to_linear, linear_to_srgb,
compute_auto_levels, normalize_grayscale, dithering algorithms, and
surface_to_* conversions with calls to raygeo.image Rust functions.

The surface_to_* functions remain as thin Cairo surface buffer extraction
wrappers that delegate to raygeo.image.rgba_to_grayscale/binary/inplace.
The pure-array dither functions (apply_floyd_steinberg_dither,
apply_bayer_dither, apply_minimum_run_length) are removed from
rayforge.image.dither and imported from raygeo.image instead.

srgb.py retains create_lut_from_color and resize_linear_nd which have
no raygeo equivalents.
The Rust-accelerated image processing completes much faster than the
previous Python implementation, causing rapid-fire invalidations to
flood the pipeline with overlapping task replacements. This leaves
nodes stuck in PROCESSING state and prevents settle.

Raise min_invalidation_interval_ms from 2ms to 50ms to give the
pipeline breathing room between invalidations. Also increase
max_settle_wait_sec from 45s to 60s.
knipknap added 2 commits June 6, 2026 09:35
The is_busy property iterated ALL contexts (including superseded ones)
to check for active tasks. When rapid invalidations cause generations
to be superseded before their tasks complete, orphaned task counts in
old contexts kept is_busy True forever, preventing the pipeline from
settling.

Only check the active context's task counts. Superseded contexts clean
up their own tasks independently and should not block the pipeline's
busy state.
The is_busy property was changed to only check the active context's
task counts (commit 650eb4d), ignoring superseded contexts. The two
tests that expected is_busy to be True when only inactive contexts had
tasks now correctly expect is_busy to be False.
knipknap added 5 commits June 6, 2026 11:16
- Mark nodes as DIRTY on cancellation so scheduler re-processes them
- Fix is_busy tests to match active-context-only behavior
Worker processes that crash during task execution leave orphaned tasks
stuck in '_tasks' with status='running', causing wait_for_settle() to
hang indefinitely. The result queue's POSIX semaphore (_wlock) can also
be permanently poisoned by a crash in a peer worker's _feed thread.

Fix:
- Workers write their current task to a SyncManager DictProxy BEFORE
  running user_func, using the Manager's own connection (immune to
  POSIX semaphore corruption).
- The DictProxy entry is cleaned up ONLY after successful
  put_nowait('done') or put_nowait('error'). If the worker crashes,
  the entry persists for the health check.
- New _check_worker_health() runs every ~1s in the listener loop.
  It checks both _worker_task_map (result queue) and the DictProxy
  to find orphaned tasks, emits worker_died signal, and spawns
  replacement workers.
- manager.py connects worker_died signal to _finalize_pooled_task,
  reusing the existing task finalization path.
Add assert for process.pid (cannot be None after start()) and guard
against None in workers.remove().
When a worker crashes while holding the result queue's _wlock semaphore,
alive workers also can't deliver results. The health check now detects
this: if dead workers are found AND alive workers have DictProxy entries
but no results have been received for >5s, those alive-but-stuck workers
are terminated and their orphaned tasks are finalized.
Log health check state when dead or stuck workers are found. Lower
stuck detection threshold to 3s and don't require dead workers as
prerequisite.
knipknap added 2 commits June 6, 2026 17:52
The 'stuck' count was just workers actively processing tasks — not
actually stuck. Only log when there are dead workers.
When a workpiece task completes for an old generation, don't trigger
process_graph(). This prevents cascading task submissions from stale
completions that keep the pipeline permanently busy.
knipknap added 3 commits June 6, 2026 20:32
Workers now check the cancellation flag before running user_func,
allowing them to skip already-cancelled tasks instantly instead of
processing them to completion. This prevents cancelled tasks from
the chaos phase from blocking the queue and starving current-
generation tasks.

Also adds a running message from workers so _worker_start_time is
populated, enabling the per-worker timeout to detect and terminate
unresponsive workers after 30 seconds. Gate _emit_node_state calls
on is_generation_current() to prevent stale completions from
corrupting DAG state. Add _busy_reason() diagnostic method and
improve settle-phase logging.
artifact_created events can set DAG nodes to VALID before the
worker's final done message is processed, causing has_pending_work()
to return False while tasks are still live. Adding has_tasks() to
is_busy ensures the pipeline reports busy until tasks are actually
finalized.
The context check is still needed for unit tests that create
GenerationContext directly without going through the TaskManager.
knipknap added 2 commits June 7, 2026 04:56
The early cancel check broke test_cancel_propagates_to_worker which
expects the task function to run and check is_cancelled() itself.
The is_busy fix (has_tasks + context check) is sufficient to make
the stress test pass without needing to skip tasks in the worker.
Workers skip already-cancelled tasks immediately, preventing stale
tasks from blocking the queue. The cancel test now accepts both
outcomes: cooperative cancellation (result='cancelled_early') or
pre-emptive skip (result=None) depending on timing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant