Skip to content

Add automatic retry for notarization failures in SignTool#16508

Merged
mmitche merged 7 commits intomainfrom
copilot/add-notarization-retry-loop
Feb 6, 2026
Merged

Add automatic retry for notarization failures in SignTool#16508
mmitche merged 7 commits intomainfrom
copilot/add-notarization-retry-loop

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Feb 5, 2026

Notarization experiences transient failures in bursts, causing all MacOS builds to fail for extended periods.

Changes

SignTool.cs - AuthenticodeSignAndNotarize()

  • Retry notarization up to 5 times on failure (no delay between attempts)
  • Unique log files per attempt: NotarizationRound{X}-Attempt{Y}.binlog
  • Logs each attempt and final error if exhausted

SignToolTests.cs

  • Test verifying retry behavior with simulated failures
  • FakeBuildEngineWithFailures helper for failure injection

The retry loop only applies to the notarization pass, not authenticode signing.

To double check:

Original prompt

This section details on the original issue you should resolve

<issue_title>Notarization passes in signtool should have an automatic retry</issue_title>
<issue_description>Notarization is quick flaky in bursts. We see things work well for quite a while, then all of a sudden, all of MacOS starts failing for a day or two. SignTool should have a retry loop for notarization. This should only occur for notarization. No wait is necessary between retries. The number of retries should be 5.</issue_description>

Comments on the Issue (you are @copilot in this section)


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI changed the title [WIP] Add automatic retry for notarization process in signtool Add automatic retry for notarization failures in SignTool Feb 5, 2026
Copilot AI requested a review from mmitche February 5, 2026 16:43
Comment thread .gitignore
Copilot AI requested a review from mmitche February 5, 2026 17:11
@mmitche mmitche marked this pull request as ready for review February 5, 2026 18:14
Copy link
Copy Markdown
Member

@mmitche mmitche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mmitche mmitche merged commit 1954e5b into main Feb 6, 2026
9 checks passed
@mmitche mmitche deleted the copilot/add-notarization-retry-loop branch February 6, 2026 19:01
@steveisok
Copy link
Copy Markdown
Member

/backport to release/10.0

@github-actions
Copy link
Copy Markdown
Contributor

Started backporting to release/10.0 (link to workflow run)

steveisok added a commit to steveisok/arcade that referenced this pull request Apr 2, 2026
The notarization retry logic (added in dotnet#16508) correctly retries up to 5 times
when Apple notarization fails transiently. However, when the out-of-process
MSBuild invocation fails on attempt N, RealSignTool.RunMSBuild logs via
_log.LogError(), which permanently records the error in MSBuild's error list.
Even when a subsequent retry succeeds, the build fails with '1 Error(s)' from
the earlier attempt.

This was observed in 4+ internal unified-build failures where:
- Notarization attempt 1 failed (transient Apple service issue)
- Attempt 2 succeeded (package verified as 'trusted by Apple notary service')
- Build still reported FAILED due to the error from attempt 1

The fix adds a SuppressRunMSBuildErrors flag that RealSignTool checks during
retryable attempts. Failed attempts are logged as high-importance messages
(still visible in logs for diagnostics) instead of errors. The last attempt
uses normal error logging so exhausted retries are correctly reported.

Fixes: macOS notarization builds failing despite successful retry
Related: dotnet#16507, dotnet#16508, dotnet#15756

Co-authored-by: Copilot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Notarization passes in signtool should have an automatic retry

4 participants