This is cherry pick of this PR: apache/arrow-adbc#3756

Fixes memory starvation issue in CloudFetch download pipeline where newer downloads could win memory allocation over older waiting downloads, causing indefinite delays.

Problem

The CloudFetch downloader was acquiring memory inside parallel download tasks with a polling-based wait (10ms intervals). This non-FIFO behavior caused memory starvation:

1. Multiple download tasks acquire semaphore slots and start polling for memory
2. All tasks wake up simultaneously every 10ms to check memory availability
3. When memory becomes available, any waiting task could acquire it (non-deterministic)
4. Newer downloads (e.g., file 5, 7, 8) could repeatedly win the race over older downloads (e.g., file 4)
5. File 4 never gets memory despite waiting the longest → indefinite starvation

Example scenario with 200MB memory, 50MB files, 3 parallel downloads:

• Files 1, 2, 3 download (150MB used)
• File 1 completes and releases 50MB
• Files 4, 5, 6 are all waiting for memory
• File 5 wins the race and acquires the 50MB
• File 2 completes, file 7 wins the race
• File 3 completes, file 8 wins the race
• File 4 never gets memory because files 5, 7, 8 keep winning

Solution

Move memory acquisition from inside parallel download tasks to the main sequential loop. This ensures FIFO ordering:

1. Main loop acquires memory sequentially for each download
2. Only after memory is acquired does the download task start
3. Downloads are guaranteed to get memory in the order they were queued
4. No starvation possible

Additionally, increased default memory buffer from 100MB to 200MB to allow more parallel downloads without hitting memory limits.

Changes

• CloudFetchDownloader.cs:
  • Moved AcquireMemoryAsync from inside DownloadFileAsync to main loop (line 278-280)
  • Ensures FIFO ordering before spawning download tasks
• CloudFetchDownloadManager.cs:
  • Increased DefaultMemoryBufferSizeMB from 100 to 200
  • Better performance for large result sets

Testing

• Manually verified on fast VM with Power BI Desktop
• Existing CloudFetchDownloader tests still pass
• FIFO ordering prevents the starvation scenario

TODO

The memory buffer design have a flaw that it only captures the compressed size, but does not cover decompressed size. There are a couple of options we can take but I wanna land this first since this one actually can get blocked for customer.