Add content-based build caching with ADO Cache@2#3826
Merged
Conversation
Replace the old get-build-type.ps1 / DOWNLOAD_EXTERNALS mechanism with Azure DevOps Cache@2 task using content-based cache keys for all native build jobs. Cache key composition: - Job name (unique per matrix combination: arch, variant, features) - Skia submodule commit SHA (all C++ source state) - Platform-specific build.cake hash (per-platform granularity) - Shared Cake script hashes (native-shared.cake, shared.cake) - Dockerfile hash (for Linux/WASM Docker-based builds) - VERSIONS.txt hash (version numbers, sonames) Cache policy: - main/release/develop branches: ALWAYS build, seed cache for PRs - PRs and feature branches: skip build on cache hit, restore from main - ADO scope isolation prevents cross-PR contamination automatically Per-platform granularity: - Changing native/windows/build.cake only invalidates Windows cache keys - Changing externals/skia (submodule SHA) invalidates ALL cache keys - C#-only or test-only changes: all 50+ native jobs hit cache What was removed: - get-build-type.ps1 invocation from bootstrapper template - DOWNLOAD_EXTERNALS variable and all ~30 condition checks - buildExternals parameter from all pipeline templates (~12 files) - Download-from-previous-build artifact step What was added: - scripts/compute-native-cache-key.ps1 — deterministic cache key script - Cache@2 task in bootstrapper for native_ jobs - NATIVE_CACHE_SKIP variable for conditional build skipping - Cache hit/miss logging for visibility Expected impact: - C#-only PR: ~4 hours → ~30 minutes (all native jobs cache hit) - Platform script change: only affected platform rebuilds - Main branch: unchanged (always builds, seeds cache) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
📦 Try the packages from this PRWarning Do not run these scripts without first reviewing the code in this PR. Step 1 — Download the packages bash / macOS / Linux: curl -fsSL https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.sh | bash -s -- 3826PowerShell / Windows: iex "& { $(irm https://raw.githubusercontent.com/mono/SkiaSharp/main/scripts/get-skiasharp-pr.ps1) } 3826"Step 2 — Add the local NuGet source dotnet nuget add source ~/.skiasharp/hives/pr-3826/packages --name skiasharp-pr-3826More options
Or download manually from Azure Pipelines — look for the Remove the source when you're done: dotnet nuget remove source skiasharp-pr-3826 |
Contributor
|
📖 Documentation Preview The documentation for this PR has been deployed and is available at: 🔗 View Staging Site This preview will be updated automatically when you push new commits to this PR. This comment is automatically updated by the documentation staging workflow. |
The sed-based removal of buildExternals lines left orphaned 'type: string' lines in 5 template files where buildExternals was the first parameter declaration. This broke YAML parsing and caused the pipeline to fail at validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The buildExternals parameter in this file left behind orphaned 'type: string' and 'default: latest' lines when removed, causing ADO validation error: 'type' is already defined. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add scripts/generate-native-dep-graph.py that scans the repository to build a complete dependency graph for native builds: - Traces Cake #load chains (e.g., windows/build.cake → msbuild.cake → native-shared.cake → shared.cake) - Discovers all files in native platform directories (vcxproj, sln, map files, Android.mk, etc.) - Catalogs Docker context files (Dockerfiles, startup.sh, _clang-cross-common.sh) - Maps install script conditions from the bootstrapper template - Outputs scripts/native-build-deps.json The cache key script (compute-native-cache-key.ps1) now reads this graph to hash ALL files that affect a given target, not just a hardcoded list. This fixes gaps where msbuild.cake, ndk.cake, Docker context scripts, and platform project files were not included in the cache key. The graph generator can be run in CI on every push to main to keep the dep graph up to date: python3 scripts/generate-native-dep-graph.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The generator now always produces both files side by side: - native-build-deps.json (machine-readable, used by cache key script) - native-build-deps.md (human-readable, renders on GitHub) The Markdown doc includes: - Impact table: shows exactly which targets rebuild for each file change - Per-target file lists - Mermaid flowchart of the full dependency graph Key insights from the impact table: - shared.cake / VERSIONS.txt / externals/skia → ALL targets - msbuild.cake → windows, winui, winui-angle only - ndk.cake → android only - xcode.cake → ios, macos, tvos only - Any native/<platform>/ file → that platform only Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Eliminated all hardcoded file names, platform lists, and path patterns. The script now discovers everything by scanning: - target_roots: directories containing build entry points (build.cake) - script_roots: directories with shared scripts to scan for #load - load_patterns: configurable regex patterns for dependency directives - docker_roots: directories to search for Dockerfiles - global_deps: glob patterns for files that affect all targets - submodules: git submodules tracked by commit SHA Configuration is built-in via DEFAULT_CONFIG but can be overridden with --config for use in other repositories. The script is now generic enough for any Cake-based build system — not SkiaSharp-specific. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move configuration out of the script into a sibling JSON file so you can adjust target roots, script paths, Docker roots, global deps, and submodules without touching the generator script. The script auto-discovers the config file next to the output or next to itself. Can also be passed explicitly with --config. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the category-based config (target_roots, script_roots, docker_roots) with a unified model: scan: globs for files to index targets: globs for build entry points links: rules for connecting files (regex patterns + sibling rules) global: globs for files that affect all targets exclude: globs to skip The script has zero domain knowledge — it just scans files, follows link rules, and computes transitive deps. All repo-specific knowledge lives in native-build-deps.config.json. Link rules support two modes: - pattern: regex extracts a file path from content (e.g., #load) - siblings: all files in the same directory tree are linked (e.g., Docker contexts) To adapt for another repo, just change the config file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dep graph now discovers YAML pipeline dependencies: - template: references (stages -> jobs -> bootstrapper) - target: references (YAML -> native/*/build.cake, bidirectional) - docker: context path references (YAML -> scripts/Docker/*) - pwsh/bash: script invocations in pipeline steps Link resolution supports prefix + suffix modes for YAML patterns like 'target: externals-wasm' -> 'native/wasm/build.cake'. Reverse-link walking ensures that YAML templates which reference a target's build.cake are included in that target's deps. For example, azure-templates-stages-native-wasm.yml references externals-wasm, so it now appears in the wasm target's dependency set. Impact table now correctly shows: - stages-native-wasm.yml -> wasm only - stages-native-macos.yml -> android, ios, maccatalyst, macos, tizen, tvos - stages-native-windows.yml -> android, nanoserver, tizen, windows, winui, winui-angle - jobs-bootstrapper.yml -> ALL targets Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix unbalanced double-paren on target nodes: (( )) not (( ) - Remove impact table and per-target lists from .md output — just the header and Mermaid diagram (JSON has the full data) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix the YAML script reference pattern to catch both ./scripts/ and .\scripts\ path formats used in the bootstrapper template. This adds all install scripts (install-android-ndk.ps1, install-llvm.ps1, install-tizen.ps1, etc.) to the dependency graph. Also adds azure-templates-variables.yml and the bootstrapper template as global deps since they affect all native builds. 119 links discovered (up from 41 in initial version). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the dense per-file edge diagram with a layered overview:
externals/skia (C++ source)
↓
Global (shared.cake, VERSIONS.txt, bootstrapper, variables)
↓
Platform families (Apple, Windows, Linux, Mobile, Web)
Each target shows its UNIQUE deps (cake, yaml, docker)
The diagram is now readable at a glance — global deps flow down
into platform families, and each target card shows what's unique
to it (xcode.cake for Apple, msbuild.cake for Windows, etc).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract all 65 native CI job names from YAML templates and map them to their target + host + docker context. The Mermaid diagram now shows the full path: files → targets → CI jobs. Jobs with 3 or fewer per target show individually (e.g., win32_x86, win32_x64, win32_arm64). Larger groups show as compact nodes (e.g., '24 jobs debian/11' for linux-clang-cross). This lets you trace: 'if I touch msbuild.cake, which CI jobs run?' → msbuild.cake → windows/winui/winui-angle → 9 specific CI jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Global deps like the bootstrapper YAML link to install scripts (install-ninja.ps1, install-llvm.ps1, etc.) which weren't being included in per-target dependency sets. Now the generator walks forward from each global dep to include their transitive deps. Validated against the fully expanded ADO pipeline YAML (27K lines, build 157227) — zero gaps. Every script referenced in any native CI job is now tracked in the dep graph. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Walk forward from ALL discovered deps (not just entry point and globals) to pick up transitive deps of reverse-linked files. This catches Docker files linked from YAML templates found via the backward walk. - Resolve directory paths to Dockerfiles: when a YAML template references 'scripts/Docker/wasm' (a directory), also link to 'scripts/Docker/wasm/Dockerfile' which triggers the sibling rule to include startup.sh, build-local.sh, etc. All Docker individual files and all YAML templates are now tracked. Validated against expanded ADO pipeline YAML — zero gaps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The transitive expansion was too aggressive — walking forward from reverse-discovered YAML templates pulled in other platforms' deps (e.g., android got ios/build.cake and all Docker files through stages-native.yml → stages-native-linux.yml → Docker). Fix: only expand forward from the entry point's forward deps and global deps, NOT from reverse-walked YAML templates. The reverse walk adds the YAML files themselves (they affect the pipeline) but doesn't follow their forward edges into other platforms' deps. Result: - android: 40 files (was 92) — no Docker, no ios/tvos cross-contamination - wasm: 35 files (was 88) — clean - Per-target file counts now match actual dependencies Docker contexts are handled at runtime via the --Docker parameter to compute-native-cache-key.ps1, not from the static dep graph. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move native build infrastructure scripts into a platform-mirrored structure under scripts/infra/: scripts/infra/shared/ — affects ALL targets (shared.cake, VERSIONS.txt, ninja, etc.) scripts/infra/android/ — ndk.cake, install-android-ndk.ps1 scripts/infra/apple/ — xcode.cake, select-xcode.sh scripts/infra/windows/ — msbuild.cake, install-llvm.ps1, select-vs.ps1, etc. scripts/infra/linux/ — Docker contexts (alpine, bionic, debian) scripts/infra/tizen/ — install-tizen.ps1 scripts/infra/wasm/ — Docker context, install-emsdk.sh This makes the cache key trivially correct: hash(native/<platform>/**) + hash(scripts/infra/shared/**) + skia_sha Each target now has zero cross-platform contamination: android: scripts/infra/android/ only ios: scripts/infra/apple/ only windows: scripts/infra/windows/ only wasm: (only shared) Updated: all #load paths in Cake files, all script paths in YAML templates, dep graph config, and dep graph output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The cache key script is native build infrastructure — belongs with the other shared infra scripts. The remaining scripts in scripts/ root are either: - Managed-only (install-android-sdk.ps1, install-dotnet-workloads.ps1, etc.) - Developer utilities (build-debug-app.ps1, download-*.ps1) - Obsolete (get-build-type.ps1) None affect native build output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restructure scripts/infra/ with a stage prefix for future extensibility: scripts/infra/native/shared/ ← affects all native builds scripts/infra/native/android/ ← android-specific native infra scripts/infra/native/apple/ ← ios, macos, tvos, maccatalyst scripts/infra/native/windows/ ← windows, winui, winui-angle, nanoserver scripts/infra/native/linux/ ← linux, linux-clang-cross (Docker) scripts/infra/native/tizen/ ← tizen scripts/infra/native/wasm/ ← wasm (Docker + emsdk) scripts/infra/managed/ ← managed build infra (SDK, workloads, JDK) scripts/infra/tests/ ← test infra (xharness) This pattern extends to future stages: scripts/infra/package/ ← NuGet packaging infra scripts/infra/samples/ ← sample build infra scripts/infra/tests/android/ ← platform-specific test infra The cache key for native builds is now: hash(native/<platform>/**) + hash(scripts/infra/native/shared/**) + skia_sha Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
scripts/ root now contains ONLY Azure YAML pipeline templates. Everything else is organized under subdirectories: scripts/ ├── azure-*.yml ← pipeline definitions (root-level) ├── infra/ │ ├── native/ ← native build infrastructure │ │ ├── shared/ ← ALL native targets │ │ ├── android/ ← android-specific │ │ ├── apple/ ← ios, macos, tvos, maccatalyst │ │ ├── windows/ ← windows, winui, winui-angle, nanoserver │ │ ├── linux/docker/ ← Docker contexts for Linux cross-compilation │ │ ├── tizen/ ← tizen-specific │ │ ├── wasm/ ← wasm-specific (Docker + emsdk) │ │ └── tools/ ← dep graph generator and output │ ├── managed/ ← managed build infra (SDK, workloads, JDK) │ ├── package/ ← NuGet packaging (SignList, nuspec, snk) │ ├── tests/ ← test infrastructure (xharness) │ └── security/ ← security scanning config (guardian) ├── utils/ ← developer utilities └── legacy/ ← unreferenced scripts (candidates for deletion) 21 scripts moved to scripts/legacy/ (zero references found): benchmark-pipeline.sh, build-debug-app.*, build-linux.sh, checkout-skia.ps1, download-externals.ps1, get-build-type.ps1, install-mono.sh, install-nuget.ps1, install-python.ps1, install-vs.ps1, merge-files.ps1, provisionator.csx, retry-command.*, select-xamarin.sh, split-file.ps1, and more. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These scripts had zero references anywhere in the repository: benchmark-pipeline.sh, build-debug-app.ps1, build-debug-app.sh, build-linux.sh, checkout-skia.ps1, download-externals.ps1, get-artifact-names.ps1, get-build-type.ps1, get-dotnet-framework-version.ps1, get-largest-folders.ps1, install-mono.sh, install-nuget.ps1, install-python.ps1, install-requirements.ps1, install-vs.ps1, merge-files.ps1, provisionator.csx, retry-command.ps1, retry-command.sh, select-xamarin.sh, split-file.ps1 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Delete download-artifact.ps1 (zero references) - Restore get-skiasharp-pr.ps1/.sh to scripts/ root (GitHub raw URLs in workflow and docs hardcode this path) - Restore download-file.ps1 to scripts/ root (called by infra scripts with .\scripts\download-file.ps1 path) - Keep persist-to-cache.ps1 in utils/ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This should result in: - native/*: all cache hits - managed/libs: cache hit - managed/package: cache hit - managed/tests: cache MISS (test file changed) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+18
to
+22
| try { | ||
| var wasmProj = MakeAbsolute (File ($"{ROOT_PATH}/tests/SkiaSharp.Tests.Wasm/SkiaSharp.Tests.Wasm.csproj")).FullPath; | ||
| serverProc = RunAndReturnProcess ("dotnet", $"run --project {wasmProj} --no-build -c {CONFIGURATION}"); | ||
| DotNetRun ($"{ROOT_PATH}/utils/WasmTestRunner/WasmTestRunner.csproj", | ||
| $"--output=\"{ROOT_PATH}/output/logs/testlogs/SkiaSharp.Tests.Wasm/{DATE_TIME_STR}/\" " + |
Comment on lines
+63
to
+65
| var DATE_TIME_NOW = DateTime.Now; | ||
| var DATE_TIME_STR = DATE_TIME_NOW.ToString ("yyyyMMdd_hhmmss"); | ||
|
|
Comment on lines
+1
to
+2
| // Cache hit test - 1777791390 | ||
| // cache test - test file change |
| dotnet_naming_rule.camel_case_for_private_internal_fields.severity = suggestion | ||
| dotnet_naming_rule.camel_case_for_private_internal_fields.symbols = private_internal_fields | ||
| dotnet_naming_rule.camel_case_for_private_internal_fields.style = camel_case | ||
| # Cache test Wed May 6 20:46:01 SAST 2026 |
| to a PR/issue or uploading as a gist. | ||
|
|
||
| To push to the data-cache branch separately: `pwsh scripts/persist-to-cache.ps1` | ||
| To push to the data-cache branch separately: `pwsh scripts/infra/persist-to-cache.ps1` |
Expected: ALL jobs cache hit — no file in any job's include changed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Expected: - native/*: ALL cache hits (binding doesn't affect native) - managed/libs: cache MISS (binding file changed) - managed/package: cache MISS (inherits managed) - managed/tests: cache MISS (inherits managed) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cs)" This reverts commit 8a8ecf9.
Expected: - native/android: cache MISS (file changed) - native/ios, native/windows, etc: cache HIT (siblings unaffected) - managed/libs: cache MISS (dependsOn native) - managed/package: cache MISS (dependsOn native) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1. Remove cache test debug lines from .editorconfig, android/build.cake, and delete tests/SkiaSharp.Tests/SkiaSharpTest.cs (test-only file) 2. Remove duplicate tree-renderer code block in repo-deps.py (lines executed outside the for loop using leaked loop variable) 3. Update stale scripts/cake/ and scripts/Docker/ paths in docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix 12h→24h clock format in DATE_TIME_STR (hh→HH) to avoid AM/PM collisions - Quote wasmProj path in dotnet run to handle spaces - Update stale persist-to-cache.ps1 reference in review-skia-update SKILL.md - Merge 3 commits from origin/main (auto-skia-sync workflow, persist-aw-data, 4.147.0 release notes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… cache keys Move from exclude to root-level include: - .config/** (dotnet-tools.json controls Cake/xharness/docfx versions) - .gitmodules (submodule URL/branch changes affect checkout) - scripts/azure-*.yml (pipeline logic changes need full rebuild) Move from exclude to managed/package include: - scripts/infra/security/** (security scan config affects package scanning) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
.editorconfig has 34 dotnet_diagnostic/csharp_style rules that the SDK evaluates during build. Changing severity from suggestion to error can break builds, so it must invalidate caches. build.ps1/build.sh confirmed not used in pipeline YAML — kept excluded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…build wrappers - scripts/infra/security/** now in root include (invalidates ALL jobs) — security config changes are critical and should force full rebuild - Delete build.ps1 and build.sh — not used in pipeline YAML, only local dev convenience wrappers that duplicate 'dotnet tool restore && dotnet cake' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mattleibow
added a commit
that referenced
this pull request
May 19, 2026
Update Tizen SDK installer from 6.1 to 10.0 (#3743) The Tizen SDK 6.1 `Certificate-Manager` package hosted on download.tizen.org is permanently failing to install, causing every Tizen native build that bypasses the output cache to crash with: java.lang.NoClassDefFoundError: org/tizen/sbilib/for_cli/sbiplugin/SBIPluginManager This was masked on main since May 8 by the output caching PR (#3826) which sets CACHE_SKIP=true on cache hit and gates all build steps — including SDK install. Every PR that triggers a Tizen cache miss fails deterministically. Samsung rebranded "Tizen Studio" to "Tizen SDK" starting with version 10.0 (released Nov 2025), which changed the download URL structure. * Default version: 6.1 → 10.0 * Download URL pattern: `tizen-studio_` → `tizen-sdk_`, `Tizen_Studio_` → `Tizen_SDK_` * Removed unused `$UpgradeLLVM` parameter and dead `Swap-Tool` function (LLVM 10 ships natively with SDK 10.0; the old LLVM 4→10 swap is unnecessary) Installed packages unchanged: `MOBILE-6.0-NativeAppDevelopment` and `TIZEN-8.0-NativeAppDevelopment` — both rootstraps still ship in SDK 10.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Matthew Leibowitz <mattleibow@live.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds content-based build caching via ADO
Cache@2and restructures the Cake build system into isolated, standalone files. Native builds skip in 1-2 minutes on cache hit. Managed builds and NuGet packaging also cache. Full dependency tracking ensures native code changes cascade to managed/package stages.Cache Performance
Main averages based on last 10 builds.
Cache Invalidation (verified on CI)
How It Works
Content-based cache keys:
{job}_{name}_{submodule_shas}_{file_hash}Cache@2restoresoutput/on hit → skips full checkout + builddependsOnin config propagates: native change → managed/package/test invalidateWhat invalidates ALL caches (root-level tracked)
These files are in the root-level
include— changing any of them busts every cache key:scripts/VERSIONS.txtscripts/infra/shared/**scripts/azure-*.ymlnuget.configglobal.json.config/**.gitmodules.editorconfigdotnet buildscripts/infra/security/**Cache key tool (
scripts/infra/caching/repo-deps.py)Controls
enableCachingparameter: UI checkbox in ADO "Run pipeline" (default: on)*markers for changed filesCake Build Isolation
build.cakeis now a pure dispatcher — zero#addin, zero#tool, single#load:All constants in
scripts/infra/shared/shared.cake. Each domain usesROOT_PATHfor repo-relative paths. Editing one domain cannot affect another.Scripts Restructure
32 unused legacy scripts deleted (including
build.ps1/build.sh— not used in CI).Dependency Config (
repo-deps.json){ "native": { "submodules": ["externals/skia", "externals/depot_tools"], "children": { "android": {...}, "ios": {...}, ... } }, "managed": { "dependsOn": ["native"], "children": { "libs": {}, "package": { "children": { "api_diff": {...}, "samples": {...} } }, "tests": {} } } }Children inherit parent paths.
dependsOnaccumulates all files from the referenced job tree. Tests and samples are isolated siblings — editing test files doesn't trigger sample rebuilds and vice versa.Other Changes
build-local.shpaths after restructure#addin(Cake.Xamarin, Cake.XCode, Mono.Cecil) and#tool vswhereDATE_TIME_STRto avoid AM/PM collisionsPre-existing Failures
Linux/Windows sample builds fail with
-mretpoline(same on main).