Skip to content

[build] Add escalating backoff between download retry attempts - #11817

Merged
jonathanpeppers merged 1 commit into
dotnet:mainfrom
simonrozsival:dev/simonrozsival/download-retry-backoff
Jun 30, 2026
Merged

[build] Add escalating backoff between download retry attempts#11817
jonathanpeppers merged 1 commit into
dotnet:mainfrom
simonrozsival:dev/simonrozsival/download-retry-backoff

Conversation

@simonrozsival

@simonrozsival simonrozsival commented Jun 30, 2026

Copy link
Copy Markdown
Member

Summary

CI builds intermittently fail in the "install OpenJDK and accept Android SDK licenses" step when dl.google.com has a brief network hiccup, e.g. build 1486699:

error MSB3923: Failed to download file
  "https://dl.google.com/android/repository/platform-36.1_r01.zip".
  The response ended prematurely. (ResponseEnded)

build-tools/scripts/DownloadFileWithRetry.targets already wraps MSBuild's <DownloadFile> in outer retry attempts (because the ResponseEnded failure is a top-level HttpIOException that MSBuild's inner Retries does not cover). But the outer attempts fired back-to-back with no delay — the build log shows all 3 attempts for a single file blowing through in ~8 seconds (22:47:09 -> 22:47:17). The existing RetryDelayMilliseconds=5000 never applies, since the inner retry never engages for ResponseEnded. So even a ~10s outage fails the build instantly.

Change

  • Add a cross-platform inline Sleep task (via RoslynCodeTaskFactory, matching existing repo usage — MSBuild has no built-in sleep).
  • Insert an escalating backoff between outer attempts: 30s -> 60s -> 90s -> 120s, each gated by !Exists() so it only waits when a retry is actually needed.
  • Bump outer attempts from 3 -> 5.
  • Update the header comment to document the new behavior.

A transient CDN outage now gets up to ~5 minutes of spaced retries to recover instead of failing in 8 seconds.

Test

  • Targets file parses; the Sleep task and $([MSBuild]::Multiply(...)) produce 30000…120000 correctly.
  • Real end-to-end download succeeds on first attempt with no sleeps firing (backoff correctly no-ops on success) and the stamp file is written as before.

The shared download helper in DownloadFileWithRetry.targets wraps
MSBuild's `<DownloadFile>` in outer retry attempts because the
`ResponseEnded` failure flaky CDNs produce is a top-level
`HttpIOException` that MSBuild's inner `Retries` does not cover.

However the outer attempts fired back-to-back with no delay, so a
short `dl.google.com` outage blew through every attempt within a few
seconds and failed the build (observed: 3 attempts in ~8s during the
'install OpenJDK and accept Android SDK licenses' step).

Add a cross-platform inline `Sleep` task and an escalating backoff
between attempts (30s, 60s, 90s, 120s), and bump the outer attempts
from 3 to 5, so a transient CDN hiccup has time to clear before the
next retry instead of failing the build instantly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 30, 2026 07:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves resiliency of the build-tool download pipeline by adding an outer retry backoff to DownloadFileWithRetry.targets, addressing intermittent dl.google.com mid-stream disconnects (ResponseEnded) that aren’t covered by MSBuild’s built-in <DownloadFile Retries=...> logic.

Changes:

  • Adds an inline (RoslynCodeTaskFactory) Sleep MSBuild task for cross-platform delays.
  • Introduces escalating backoff sleeps between outer <DownloadFile> attempts (30s → 60s → 90s → 120s).
  • Increases the number of outer attempts from 3 to 5 and updates the header comment to explain the behavior.
Show a summary per file
File Description
build-tools/scripts/DownloadFileWithRetry.targets Adds a cross-platform sleep task and escalating backoff between outer download retry attempts to better tolerate transient CDN/network hiccups.

Copilot's findings

  • Files reviewed: 1/1 changed files
  • Comments generated: 1

Comment thread build-tools/scripts/DownloadFileWithRetry.targets
@simonrozsival simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Jun 30, 2026
@jonathanpeppers
jonathanpeppers merged commit a1acf82 into dotnet:main Jun 30, 2026
40 checks passed
simonrozsival added a commit that referenced this pull request Jul 7, 2026
## Summary

This is another incremental hardening PR for the recurring `install OpenJDK and accept Android SDK licenses` CI failure where large Android SDK archives from `dl.google.com` fail with:

```text
MSB3923: Failed to download file "https://dl.google.com/android/repository/...".
The response ended prematurely. (ResponseEnded)
```

The specific recent failure already had the newer retry/backoff behavior: the log shows retries waiting 30s, 60s, 90s, and 120s before ultimately exhausting `x86_64-29_r08-darwin.zip`. So this PR does **not** replace the existing download retry work; it builds on it and covers the next failure mode.

## Prior work this builds on

- [#11348](#11348) moved OpenJDK installation into an MSBuild NoTargets project.
- [#11440](#11440) moved Android SDK/NDK component downloads into `src/androidsdk/androidsdk.targets`, which is what this setup step builds.
- [#11618](#11618) added the shared `android-archives` pipeline cache for build jobs and bumped the built-in `DownloadFile` retries.
- [#11647](#11647) added the shared `DownloadFileWithRetry` wrapper because `ResponseEnded` is a top-level `HttpIOException` that MSBuild's built-in `DownloadFile` retry logic does not catch.
- [#11693](#11693) made `DownloadOneFileWithRetry` skip cleanly on no-op builds using per-file stamps, which makes whole-step retries less wasteful once some archives already succeeded.
- [#11817](#11817) added escalating outer backoff between retry attempts, so transient CDN hiccups get several minutes to recover instead of burning through attempts immediately.

## Remaining gap

Those PRs made individual downloads much more resilient, but the failing path is the **test environment setup** template, not the main build template:

- `build-{linux,macos,windows}-steps.yaml` already uses `cache-android-archives.yaml`.
- `setup-test-environment-steps.yaml` did **not** use that cache before running `src/androidsdk/androidsdk.csproj`.
- The `install OpenJDK and accept Android SDK licenses` `run-dotnet-preview.yaml` invocation also had `retryCountOnTaskFailure: 0` through the template default.

That means a macOS test setup job could still cold-download several large archives concurrently and fail the entire job if one archive exhausted the per-file retry/backoff loop.

## Change

This PR hardens that remaining test-setup path by:

- adding a `condition` parameter to `cache-android-archives.yaml`, preserving the existing default behavior for build jobs
- invoking the Android archive cache from `setup-test-environment-steps.yaml` for macOS test setup before SDK downloads run
- setting `retryCountOnTaskFailure: 2` on the `install OpenJDK and accept Android SDK licenses` step

The cache is intentionally limited to macOS test setup for now. Linux archive caching already has a separate disk-pressure concern tracked by [#11837](#11837), so this PR avoids expanding Linux cache usage while still targeting the observed macOS `ResponseEnded` failure.

## Why step retry helps

The per-file retry wrapper is still the first line of defense. The task-level retry is a second line of defense for the case where a specific archive exhausts all per-file attempts.

On a second task attempt, successfully downloaded archives should be reused/skipped by the existing cache/stamp logic, so the retry mostly focuses on the archive that failed rather than restarting all work from scratch.

## Validation

- `git diff --check`
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 31, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

flaky-tests ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants