Type of Change

• Bug Fix
• Performance Improvement
• New Feature
• Documentation
• Test/CI
• Refactor
• Other:

Related Issues

• Fixes Data was not automatically recovered during the disconnection #829
• Builds upon PR Fix: Trigger self-heal on read when shards missing from rejoined nodes #871

Summary of Changes

This PR addresses issue #829 where data written while a node is offline is not automatically healed to the rejoined node, even after reading the object. While PR #871 added code to trigger heal requests during reads, the heal queue was using simple FIFO processing that ignored priority levels. After initial implementation and user testing feedback, comprehensive diagnostic logging was added to enable precise identification of where the heal flow may fail.

Problem Analysis

Issue #829: After a node rejoins following an outage, data written while the node was offline is not automatically healed to the rejoined node, even after reading the object.

PR #871 Fix: Added code to detect missing shards during reads and trigger heal requests with HealChannelPriority::Normal (changed from Low).

Remaining Issues:

1. Heal queue used simple FIFO (VecDeque::pop_front()), completely ignoring priority fields
2. Even with Normal priority, requests waited behind all previously queued tasks
3. No priority-based scheduling mechanism
4. High-priority heal requests couldn't be processed faster
5. Insufficient diagnostic information to identify where heal flow failures occur

Root Cause

1. Non-priority queue: HealManager's heal_queue used VecDeque<HealRequest> but processed items with pop_front(), completely ignoring the priority field
2. Single queue approach: All heal requests entered one queue regardless of priority
3. No preemption mechanism: No mechanism to prioritize high-priority tasks over low-priority ones
4. Limited observability: Lack of detailed logging made it difficult to diagnose heal flow issues

Changes Made

Core Priority Queue Implementation:

1. New PriorityHealQueue structure:

   • Uses BinaryHeap for priority-based ordering
   • Maintains sequence numbers for FIFO ordering within same priority
   • Uses HashSet for automatic deduplication of duplicate heal requests
   • Supports priority ordering: Urgent > High > Normal > Low

2. Updated HealPriority:

   • Added Copy and Hash traits to avoid ownership issues
   • Maintains backward compatibility

3. Updated HealManager:

   • Changed heal_queue from VecDeque to PriorityHealQueue
   • Updated submit_heal_request with deduplication support
   • Updated process_heal_queue to process by priority
   • Updated auto disk scanner to use new queue API

Enhanced Monitoring & Observability:

4. Queue monitoring features:
   • Added get_priority_stats() to track request counts by priority
   • Logs queue statistics when adding high/urgent priority tasks
   • Warns when queue capacity exceeds 80% full
   • Logs queue status when backlog exceeds 10 items

Adaptive Task Processing:

5. Improved throughput:
   • Process up to 2 tasks per cycle instead of 1
   • Intelligent slot allocation based on available capacity
   • Improves throughput for high-priority heal operations

Robust Error Handling:

6. Better error reporting:
   • Fixed all instances where heal request errors were silently ignored
   • Added contextual warning logs when heal requests fail to enqueue
   • Improved error messages across ecstore operations

Comprehensive Diagnostic Logging:

7. Read path diagnostics:

   • Detailed shard availability logging (total_shards, available_shards, missing_shards)
   • Success confirmation when heal requests are enqueued
   • Improved variable naming for clarity (nil_count → available_shards)

8. Heal execution diagnostics:

   • Log available disk count vs total disks from disks_with_all_parts
   • Per-disk heal decision logging with reason and endpoint information
   • Summary of how many disks need healing

9. Heal writing diagnostics:

   • Detailed logging for rename phase with disk endpoints
   • Success/failure logging for each disk's heal operation
   • Clear indication when healed data is successfully written to rejoined nodes

Comprehensive Testing:

10. Test coverage:
    • 8 unit tests covering all queue functionality
    • Tests for priority ordering, FIFO within priority, deduplication
    • Tests for queue statistics and empty state checking
    • Tests for different heal types and mixed priorities

Checklist

• I have read and followed the CONTRIBUTING.md guidelines
• Passed make pre-commit
• Added/updated necessary tests
• Documentation updated (if needed)
• CI/CD passed (if applicable)

Impact

• Breaking change (compatibility)
• Requires doc/config/deployment update
• Other impact: Internal implementation change with improved observability and performance

Additional Notes

Test Results

• ✅ All 8 new unit tests pass
• ✅ All existing tests pass
• ✅ Clippy checks pass (no warnings)
• ✅ Code formatting complete

Performance Impact

• Minimal overhead from BinaryHeap operations (O(log n))
• Improved throughput from processing multiple tasks per cycle
• Better resource utilization through priority-based scheduling

Behavior Changes

• Heal requests now processed by priority instead of FIFO order
• Duplicate heal requests automatically deduplicated
• Queue capacity warnings help prevent queue overflow
• Failed heal request submissions now logged instead of silently ignored
• Comprehensive logging at all heal flow stages for diagnostics

Diagnostic Guidance

The enhanced logging enables precise diagnosis of heal flow issues. After testing, check logs for:

1. Read detection: "Detected missing shards during read" - confirms heal is triggered
2. Request acceptance: "Successfully enqueued heal request" - confirms queue accepts request
3. Task creation: "Starting heal task" - confirms HealManager processes queue
4. Disk marking: "Disk X heal check: should_heal=true" - identifies which disks need healing
5. Heal completion: "Successfully renamed healed data" - confirms data written to disk

This diagnostic information helps identify exactly where in the heal flow any issues occur, enabling targeted fixes.

Thank you for your contribution! Please ensure your PR follows the community standards (CODE_OF_CONDUCT.md) and sign the CLA if this is your first contribution.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.