Skip to content

Pull registries from HTTP(S) URLs, with private-source auth#1356

Merged
jerbly merged 23 commits into
open-telemetry:mainfrom
jerbly:http-pull
Apr 30, 2026
Merged

Pull registries from HTTP(S) URLs, with private-source auth#1356
jerbly merged 23 commits into
open-telemetry:mainfrom
jerbly:http-pull

Conversation

@jerbly

@jerbly jerbly commented Apr 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Adds HTTP Bearer token authentication for downloading remote registries from private HTTP(S) sources (closes #1344). Primary use case: pulling published registry artifacts (manifest.yaml + resolved.yaml) from private GitHub release assets.

  • New RemoteFile source type for VirtualDirectoryPath - supports non-archive, non-git HTTP(S) URLs.
  • HTTP auth is configured per-URL via [[auth]] entries in .weaver.toml. Each entry pairs a url_prefix (longest match wins) with a token source: a literal token, a token_env variable name, or a token_command helper whose first stdout line is the token (e.g. ["gh", "auth", "token"]). Matched requests get Authorization: Bearer <token> and a User-Agent: weaver header.
  • GitHub URL normalization - browser-style URLs like https://github.com/org/repo/releases/download/v1.0.0/manifest.yaml are transparently resolved to GitHub API asset URLs (which accept Bearer auth). Release metadata is cached per-release to avoid redundant API calls.
  • Shared ureq Agent with RedirectAuthHeaders::SameHost so auth headers survive GitHub's same-host redirects but are stripped on cross-origin redirects.

Breaking change: A bare HTTP(S) URL with no .git/.zip/.tar.gz suffix and no refspec or [sub_folder] now parses as RemoteFile instead of GitRepo. Add .git to such URLs to keep the old behaviour.

@jerbly
jerbly requested a review from a team as a code owner April 14, 2026 00:35
@codecov

codecov Bot commented Apr 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.78882% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.4%. Comparing base (7061f2d) to head (7b270cd).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/weaver_common/src/http_auth.rs 90.9% 4 Missing ⚠️
crates/weaver_common/src/vdir.rs 96.8% 3 Missing ⚠️
crates/weaver_resolver/src/lib.rs 50.0% 2 Missing ⚠️
crates/weaver_semconv/src/registry_repo.rs 85.7% 1 Missing ⚠️
Additional details and impacted files 
@@           Coverage Diff           @@
##            main   #1356     +/-   ##
=======================================
+ Coverage   82.1%   82.4%   +0.2%     
=======================================
  Files        118     120      +2     
  Lines       9960   10092    +132     
=======================================
+ Hits        8183    8320    +137     
+ Misses      1777    1772      -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jsuereth
jsuereth reviewed Apr 15, 2026
View reviewed changes
Comment thread crates/weaver_common/src/vdir.rs Outdated
@jsuereth jsuereth moved this to To consider for the next release in OTel Weaver Project Apr 15, 2026
jsuereth
jsuereth approved these changes Apr 16, 2026
View reviewed changes

@jsuereth jsuereth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe there is a non-breaking way to support multiple credentials per-domain you resolve from and this flag at the same time.

Specifically I think we can have a url - token config, similar to cargo, in the future that would "win" over this broad credential config.

Approve-ing but want to confirm you agree with me first, @jerbly. WDYT?

github-advanced-security[bot]
github-advanced-security AI found potential problems Apr 23, 2026
View reviewed changes
Comment thread src/registry/mod.rs Fixed
Comment thread src/registry/mod.rs Fixed
@jerbly jerbly changed the title Bearer token authentication for downloading remote registries Pull registries from HTTP(S) URLs, with private-source auth Apr 29, 2026
@jerbly
jerbly requested a review from jsuereth April 30, 2026 00:39
@jerbly

jerbly commented Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

@jsuereth - I think this is good now. Requested re-review since it's changed a lot since you looked last. This is what I demoed to you a couple of weeks ago.

@jerbly
jerbly merged commit dbbabda into open-telemetry:main Apr 30, 2026
27 checks passed
@github-project-automation github-project-automation Bot moved this from To consider for the next release to Done in OTel Weaver Project Apr 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jsuereth jsuereth jsuereth approved these changes
+1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

OTel Weaver Project
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support securely pulling artifacts over http

3 participants

@jerbly @jsuereth @github-advanced-security