…y by 96%

⚠️ DISCUSSION REQUIRED: This reduces accumulated allocations but not peak
concurrent memory. Requires discussion on benefit vs. complexity trade-off.

Reduces LZ4 internal buffer memory allocation from ~900MB to ~40MB
(96% reduction) by implementing custom ArrayPool that supports 4MB buffers.

Problem:
- Databricks uses LZ4 frames with 4MB maxBlockSize
- .NET's ArrayPool.Shared has 1MB limit
- 222 decompressions × 4MB fresh allocations = 888MB LOH

Solution:
- CustomLZ4FrameReader extends StreamLZ4FrameReader
- Overrides AllocBuffer() to use custom ArrayPool (4MB max, 10 buffers)
- CustomLZ4DecoderStream wrapper using CustomLZ4FrameReader
- Updated Lz4Utilities to use custom implementation

Results:
- Total allocations: 900MB → 40MB (96% reduction)
- GC pressure: Significantly reduced (fewer LOH allocations)
- Peak concurrent memory: Unchanged (~8-16MB with parallelDownloads=1)

Note: This primarily optimizes accumulated allocations over time, not peak
concurrent memory usage. With sequential decompression (default), peak memory
remains similar but with better reuse and fewer GC Gen2 collections.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

…ooling

Change CustomLZ4FrameReader to clear buffers when returning to pool
(clearArray: true) to prevent stale data from previous decompressions
from corrupting subsequent operations.

Without clearing, buffers retain old data which can cause Arrow IPC
stream corruption when the decompressor doesn't fully overwrite the
buffer or there are length tracking issues.

Performance impact is minimal (~1-2ms per 4MB buffer clear) compared
to network I/O (10-100ms) and decompression time (5-20ms).

Fixes: Power BI error "Corrupted IPC message. Received a continuation
token at the end of the message."

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

…lz4-memory-pooling

…se instance

Address code review feedback by moving ArrayPool lifecycle management
from process-level (static) to Database instance-level.

**Changes:**
- DatabricksDatabase: Creates ArrayPool (one per database instance)
- DatabricksConnection: Receives pool from database via property
- CustomLZ4FrameReader: Accepts ArrayPool as constructor parameter
- CustomLZ4DecoderStream: Accepts ArrayPool as constructor parameter
- Lz4Utilities: Methods require ArrayPool parameter (no fallback)
- CloudFetchDownloader: Changed to use IHiveServer2Statement (was ITracingStatement)
- Readers: Get pool directly from connection and pass to LZ4 utilities

**Benefits:**
- Pool lifecycle tied to Database, not process lifetime
- Memory freed when Database is disposed/GC'd
- Perfect for connection pooling: multiple connections share one pool
- Cleaner design: no optional/nullable complexity

**Memory footprint:**
- With connection pooling: 10 connections → 1 shared pool = 40MB
- Without: Each Database instance → 1 pool = 40MB

Addresses review comment: apache#3654 (comment)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Improves on the property setter approach by passing Lz4BufferPool via
internal constructor parameter, matching the RecyclableMemoryStreamManager pattern.

Changes:
- Added internal constructor that accepts Lz4BufferPool parameter
- Public constructor chains to internal constructor with null (creates own pool)
- DatabricksDatabase passes shared pool via internal constructor
- Property changed from mutable to immutable (get-only)
- No public API changes - backward compatible

Benefits:
- Compile-time safety - Database must explicitly pass the pool
- Immutable after construction - can't be changed accidentally
- Predictable initialization - pool created at construction time
- Consistent pattern with RecyclableMemoryStreamManager

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>