Skip to content

Make Azure file verification optional reverify by download#559

Merged
irees merged 2 commits intomainfrom
az-fix-md5check
Jan 14, 2026
Merged

Make Azure file verification optional reverify by download#559
irees merged 2 commits intomainfrom
az-fix-md5check

Conversation

@irees
Copy link
Copy Markdown
Contributor

@irees irees commented Jan 14, 2026

This PR fixes Azure blob upload verification by replacing the broken TransactionalValidation approach with a download-and-compare method.

Problem

The Azure SDK's TransactionalValidation with blob.TransferValidationTypeMD5() does not work correctly for UploadStream because it uploads files in chunks and validates MD5 against each chunk rather than the whole file.

Solution

  • New verification approach: Calculate MD5 before upload, then download the uploaded file and recalculate MD5 to verify integrity
  • Opt-in behavior: Verification is disabled by default to avoid performance overhead. Enable via VerifyUpload field or ?verify_upload=true query parameter in the storage URL
  • Refactored MD5 helpers: md5FromStream is now the core function, with md5FromReader wrapping it for seekable readers

Changes

  • az.go:

    • Added VerifyUpload field to Az struct
    • Parse ?verify_upload=true query parameter from URL
    • Implement download-and-verify logic with detailed error messages including hex digests
    • Remove unused blob import, add bytes for bytes.Equal
  • request.go:

    • Refactored md5FromReader to use md5FromStream as core
    • Removed custom md5Equal in favor of bytes.Equal
  • az_test.go:

    • Added TestNewAzFromUrl with comprehensive test cases for URL parsing including verify_upload parameter

Usage

// Via URL parameter
az, _ := NewAzFromUrl("az://myaccount.blob.core.windows.net/container?verify_upload=true")

// Or set directly
az := &Az{
    Account:      "myaccount.blob.core.windows.net",
    Container:    "container",
    VerifyUpload: true,
}

@irees irees marked this pull request as ready for review January 14, 2026 12:13
Copilot AI review requested due to automatic review settings January 14, 2026 12:13
@irees irees changed the title Make Azure file verification optional and use redownload method Make Azure file verification optional reverify by download Jan 14, 2026
@irees irees merged commit c9bfe0c into main Jan 14, 2026
12 checks passed
@irees irees deleted the az-fix-md5check branch January 14, 2026 12:34
@irees irees review requested due to automatic review settings March 23, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant