[build] Add escalating backoff between download retry attempts - #11817
Merged
jonathanpeppers merged 1 commit intoJun 30, 2026
Merged
Conversation
The shared download helper in DownloadFileWithRetry.targets wraps MSBuild's `<DownloadFile>` in outer retry attempts because the `ResponseEnded` failure flaky CDNs produce is a top-level `HttpIOException` that MSBuild's inner `Retries` does not cover. However the outer attempts fired back-to-back with no delay, so a short `dl.google.com` outage blew through every attempt within a few seconds and failed the build (observed: 3 attempts in ~8s during the 'install OpenJDK and accept Android SDK licenses' step). Add a cross-platform inline `Sleep` task and an escalating backoff between attempts (30s, 60s, 90s, 120s), and bump the outer attempts from 3 to 5, so a transient CDN hiccup has time to clear before the next retry instead of failing the build instantly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves resiliency of the build-tool download pipeline by adding an outer retry backoff to DownloadFileWithRetry.targets, addressing intermittent dl.google.com mid-stream disconnects (ResponseEnded) that aren’t covered by MSBuild’s built-in <DownloadFile Retries=...> logic.
Changes:
- Adds an inline (RoslynCodeTaskFactory)
SleepMSBuild task for cross-platform delays. - Introduces escalating backoff sleeps between outer
<DownloadFile>attempts (30s → 60s → 90s → 120s). - Increases the number of outer attempts from 3 to 5 and updates the header comment to explain the behavior.
Show a summary per file
| File | Description |
|---|---|
| build-tools/scripts/DownloadFileWithRetry.targets | Adds a cross-platform sleep task and escalating backoff between outer download retry attempts to better tolerate transient CDN/network hiccups. |
Copilot's findings
- Files reviewed: 1/1 changed files
- Comments generated: 1
jonathanpeppers
approved these changes
Jun 30, 2026
simonrozsival
added a commit
that referenced
this pull request
Jul 7, 2026
## Summary This is another incremental hardening PR for the recurring `install OpenJDK and accept Android SDK licenses` CI failure where large Android SDK archives from `dl.google.com` fail with: ```text MSB3923: Failed to download file "https://dl.google.com/android/repository/...". The response ended prematurely. (ResponseEnded) ``` The specific recent failure already had the newer retry/backoff behavior: the log shows retries waiting 30s, 60s, 90s, and 120s before ultimately exhausting `x86_64-29_r08-darwin.zip`. So this PR does **not** replace the existing download retry work; it builds on it and covers the next failure mode. ## Prior work this builds on - [#11348](#11348) moved OpenJDK installation into an MSBuild NoTargets project. - [#11440](#11440) moved Android SDK/NDK component downloads into `src/androidsdk/androidsdk.targets`, which is what this setup step builds. - [#11618](#11618) added the shared `android-archives` pipeline cache for build jobs and bumped the built-in `DownloadFile` retries. - [#11647](#11647) added the shared `DownloadFileWithRetry` wrapper because `ResponseEnded` is a top-level `HttpIOException` that MSBuild's built-in `DownloadFile` retry logic does not catch. - [#11693](#11693) made `DownloadOneFileWithRetry` skip cleanly on no-op builds using per-file stamps, which makes whole-step retries less wasteful once some archives already succeeded. - [#11817](#11817) added escalating outer backoff between retry attempts, so transient CDN hiccups get several minutes to recover instead of burning through attempts immediately. ## Remaining gap Those PRs made individual downloads much more resilient, but the failing path is the **test environment setup** template, not the main build template: - `build-{linux,macos,windows}-steps.yaml` already uses `cache-android-archives.yaml`. - `setup-test-environment-steps.yaml` did **not** use that cache before running `src/androidsdk/androidsdk.csproj`. - The `install OpenJDK and accept Android SDK licenses` `run-dotnet-preview.yaml` invocation also had `retryCountOnTaskFailure: 0` through the template default. That means a macOS test setup job could still cold-download several large archives concurrently and fail the entire job if one archive exhausted the per-file retry/backoff loop. ## Change This PR hardens that remaining test-setup path by: - adding a `condition` parameter to `cache-android-archives.yaml`, preserving the existing default behavior for build jobs - invoking the Android archive cache from `setup-test-environment-steps.yaml` for macOS test setup before SDK downloads run - setting `retryCountOnTaskFailure: 2` on the `install OpenJDK and accept Android SDK licenses` step The cache is intentionally limited to macOS test setup for now. Linux archive caching already has a separate disk-pressure concern tracked by [#11837](#11837), so this PR avoids expanding Linux cache usage while still targeting the observed macOS `ResponseEnded` failure. ## Why step retry helps The per-file retry wrapper is still the first line of defense. The task-level retry is a second line of defense for the case where a specific archive exhausts all per-file attempts. On a second task attempt, successfully downloaded archives should be reused/skipped by the existing cache/stamp logic, so the retry mostly focuses on the archive that failed rather than restarting all work from scratch. ## Validation - `git diff --check`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI builds intermittently fail in the "install OpenJDK and accept Android SDK licenses" step when
dl.google.comhas a brief network hiccup, e.g. build 1486699:build-tools/scripts/DownloadFileWithRetry.targetsalready wraps MSBuild's<DownloadFile>in outer retry attempts (because theResponseEndedfailure is a top-levelHttpIOExceptionthat MSBuild's innerRetriesdoes not cover). But the outer attempts fired back-to-back with no delay — the build log shows all 3 attempts for a single file blowing through in ~8 seconds (22:47:09 -> 22:47:17). The existingRetryDelayMilliseconds=5000never applies, since the inner retry never engages forResponseEnded. So even a ~10s outage fails the build instantly.Change
Sleeptask (viaRoslynCodeTaskFactory, matching existing repo usage — MSBuild has no built-in sleep).!Exists()so it only waits when a retry is actually needed.A transient CDN outage now gets up to ~5 minutes of spaced retries to recover instead of failing in 8 seconds.
Test
Sleeptask and$([MSBuild]::Multiply(...))produce 30000…120000 correctly.