[ci] Retry on-device install, fail fast, diagnose failures, always capture logcat - #11831
Merged
Merged
Conversation
The -t:Install step in apk-instrumentation.yaml ran with continueOnError: true, so an install failure (e.g. XAGCPU7000 "device offline" when the emulator drops off ADB) was swallowed and the pipeline still ran 'dotnet test' against a device where the app was never installed. That wastes CI time and produces a misleading INSTRUMENTATION_FAILED, with the lane only going red at the final 'fail if any issues occurred' gate. Set continueOnError: false on the install step so a failed install fails the lane immediately and the test run is skipped (its condition defaults to succeeded()). Context: #11830 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the on-device CI template to fail the job immediately when the -t:Install step fails (e.g., when the emulator/device goes offline), avoiding a subsequent dotnet test run against a device where the test app was never installed.
Changes:
- Switch the install/build step from
continueOnError: truetocontinueOnError: falseto stop the lane as soon as install fails. - Add inline rationale in the YAML explaining why fail-fast is preferred (reduces wasted CI time and misleading instrumentation failures).
Show a summary per file
| File | Description |
|---|---|
| build-tools/automation/yaml-templates/apk-instrumentation.yaml | Makes the install step fail-fast so the job doesn’t proceed to test execution after an install failure. |
Copilot's findings
- Files reviewed: 1/1 changed files
- Comments generated: 1
Wrap the -t:Install step in a retry loop (3 attempts, 15s delay between them) so a transient emulator/ADB drop (XAGCPU7000 "device offline") during install can recover instead of failing the lane outright. Only if all 3 attempts fail does the step exit non-zero; combined with continueOnError: false, the lane then fails fast and the test run is skipped. Context: #11830 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
simonrozsival
changed the title
Address review feedback on PR #11831: with continueOnError: false on the -t:Install step, a failed install skips all subsequent succeeded()-gated steps, including the capture-logcat step - losing device logs for exactly the 'device offline' scenario this change targets. Change the capture-logcat step's condition to succeededOrFailed() so the best-effort 'adb logcat -d' still runs after a failed install (it is already continueOnError: true and tolerates an offline device). This matches the step's stated 'Always capture full device logcat' intent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the inline PowerShell retry loop with the run-dotnet-preview template's existing retryCountOnTaskFailure parameter (set to 3). This keeps the shared template invocation instead of duplicating the dotnet PATH/invocation logic, while still retrying the -t:Install step on transient emulator/ADB 'device offline' failures before failing fast. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use always() instead of succeededOrFailed() for the capture-logcat step so device logs are also captured when the job is canceled - e.g. a hung/timed-out test run - not just on a failed step. This matches the step's 'Always capture full device logcat' intent; the capture is best-effort (continueOnError + '|| echo'). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
simonrozsival
changed the title
Add a best-effort, time-bounded device-state snapshot right after the -t:Install step, gated on failed(), so when an install fails we capture the data needed to classify it next time instead of guessing: connectivity (adb devices/get-state), disk pressure (df, dumpsys diskstats), storage-service readiness (dumpsys storaged - the StorageStatsManager NPE seen during install-create), boot completion, and how many test apps have accumulated. Context: #11830 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
simonrozsival
changed the title
simonrozsival
added
the
copilot
jonathanpeppers
approved these changes
Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The on-device package-test template
apk-instrumentation.yamlbuilds+installs the app with-t:Installand then runsdotnet test. When the emulator drops off ADB mid-install (error XAGCPU7000: Mono.AndroidTools.AdbException: device offline) the previous behavior was doubly bad:continueOnError: true, so the failure was swallowed and the pipeline still randotnet testagainst a device where the app was never installed, producing a misleadingINSTRUMENTATION_FAILEDand only going red at the finalfail if any issues occurredgate — after wasting time.Changes
run-dotnet-previewtemplate's existingretryCountOnTaskFailure: 3, so a transientdevice offlineblip can recover on a retry instead of failing the lane.continueOnError: false) once retries are exhausted: the lane fails immediately and therunstep (whoseconditiondefaults tosucceeded()) is skipped — no point running tests against a device with no app installed.failed()-gated, time-bounded step right after install snapshots device state so we can classify the failure next time instead of guessing — connectivity (adb devices -l/get-state), disk pressure (df,dumpsys diskstats), storage-service readiness (dumpsys storaged— theStorageStatsManagerNPE seen duringinstall-create), boot completion, and accumulated test apps (pm list packages -3).capture logcatstep's condition is changed from the defaultsucceeded()toalways(), so the best-effortadb logcat -druns on success, on a failed step (e.g. fail-fast install), and on job cancellation/timeout (e.g. a hung test). The capture is best-effort (continueOnError: true+|| echo) and tolerates an offline device. This matches the step's "Always capture full device logcat" intent and addresses review feedback.Why
always()(logcat) and not just removing the conditionOmitting
condition:is not neutral — an Azure Pipelines step with no condition defaults tosucceeded(), i.e. it is skipped as soon as any prior step fails. That is exactly the old, buggy behavior.succeededOrFailed()fixes the failed-step case but still skips on cancellation (a job-level timeout counts as cancellation), which is precisely when a hung test's logcat is most valuable — soalways()is used.Behavior change
succeeded(), a hard install failure in one flavor will now skip the remaining flavors in that job. This is acceptable: a device offline across all retries is very unlikely to install the next flavor either, and fail-fast gives a clear signal.Context
Tracking issue for the underlying emulator/ADB install flakiness (
device offlineand theStorageStatsManagerNPE): #11830Observed on build 1488505 (
Package Tests > macOS > Tests > APKs 1).A companion PR adds an equivalent device-state snapshot to the
DeviceTeston-failure teardown for the MSBuildDeviceIntegrationDeployToDevice/InstallAndRuntests.