Page MenuHomePhabricator
Create Task
Maniphest T419049

Upgrade the MediaWiki servers to ICU 72 ☂️
Closed, DeclinedPublic
Actions

Description

Goal

In order to upgrade all our MediaWiki clusters to Debian Bookworm, we need to upgrade the current Debian Bullseye installation to use ICU 72. We will test-drive the new process created in T263437 on smaller wikis, and use the old (more disruptive) process for the big ones this time. The aim is to test the new process with a limited blast radius, so that we can confidently use it on all wikis next time, and thus improve user experience.

Based on T419980#11734256, we will use the new process on s3 wikis minus ruwikinews and the old process everywhere else.

Roadmap

These need to happen in sequence.

Prep

  1. Prepare packages and production images for ICU 72 upgrade — T419058
  2. [new process] Copy the categorylinks tables — T419980
  3. [new process] enable remote ICU collation writes — T419274
  4. [new process] Migrate collation data to ICU 72 — T419242
  5. [new process] Confirm migration date, sync with DBA and MW Engineering, put "no deployments" into deployment calendar

Day of migration

  1. upgrade production systems/images to the build with ICU 72
  2. scap lock
  3. [new process] swap the tables — T419980
  4. scap unlock
  5. [old process] start collation data migration maintenance script for old process wikis

SRE on point: @Raine
SRE backup: @Scott_French
DBA on point: TODO
MW on point: TODO

TODO:

  • monitoring and rollback steps

Cleanup

  • [new process] disable remote writes
  • [old process] monitor the maintenance script
  • after things have been running on the new version for at least a few days:
    • clean up shellbox deployment
    • drop old table

In parallel:

  • Build and test production images for MW
  • CommRel support

Still needs clarification:

  • Discuss this process with DBA: T419980
  • Discuss this process with MW engineers
    • get sign-off on this procedure
    • discuss risks & risk mitigations
    • ask them to look at whether the code for this functionality is still current
    • double-check order of operations, in particular:
      • how to minimize the disruption/funky sorting window
  • Find which wikis on which DB sections actually use a non-standard collation: find_collations.py
    • and put the checklist into the relevant tasks’ runbooks
  • Come up with a way to check the newly-written collation data — swap the table and deploy the images somewhere ahead of swapping production, and/or run a one-off sanity checks script
  • When should we upgrade deployment hosts?

Out of scope

Upgrading ICU on Beta.

References

T345561 - upgrade using the old process
T329491 - preparation for the upgrade (before it was determined that the old process would be used)
T263437 - implementation of the new process

Details

Other Assignee
Scott_French
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
maintenance/updateCollation: add --tablemediawiki/coremaster+20 -12
Temporarily add shellbox-icu ClusterIP endpointoperations/mediawiki-configmaster+2 -1
Temporarily add shellbox-icu to $wgShellboxUrlsoperations/mediawiki-configmaster+9 -0
Enable $wgTempCategoryCollations for testwiki.operations/mediawiki-configmaster+12 -0
shellbox-icu72: Add ClusterIP to TLS cert SANsoperations/deployment-chartsmaster+4 -0
Enable $wgTempCategoryCollations for s3 wikis.operations/mediawiki-configmaster+20 -12
Enable $wgTempCategoryCollations for s3 wikis.operations/mediawiki-configmaster+18 -12
DB schema: Create temporary table for ICU upgrademediawiki/coremaster+163 -0
k8s: create shellbox-icu72operations/puppetproduction+4 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Change base image to use ICU 72repos/releng/release!239kamilaicu72main
Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
MLechvien-WMF added a subtask: T419212: Upgrade ServiceOps roles from Bullseye to Debian Trixie.Mar 6 2026, 11:02 AM
Raine created subtask T419242: Migrate collation data to ICU 72.Mar 6 2026, 1:56 PM
Raine changed the status of subtask T419242: Migrate collation data to ICU 72 from Open to Stalled.Mar 6 2026, 1:58 PM
Scott_French mentioned this in T418262: deploy2003 implementation tracking.Mar 6 2026, 3:48 PM
Raine created subtask T419274: ICU 72 upgrade: enable remote ICU collation writes.Mar 6 2026, 6:41 PM
Raine closed subtask T418752: ICU 72 upgrade: double-check feasibility of new approach; come up with a runbook as Resolved.Mar 6 2026, 7:30 PM
Raine reopened subtask T418752: ICU 72 upgrade: double-check feasibility of new approach; come up with a runbook as In Progress.
Raine updated the task description. (Show Details)Mar 6 2026, 7:58 PM
Scott_French updated the task description. (Show Details)Mar 6 2026, 10:08 PM
Raine updated the task description. (Show Details)Mar 9 2026, 12:12 PM
Raine moved this task from "Should I do it?" triage to In Progress on the User-Raine board.Mar 10 2026, 1:52 PM
Raine attached a referenced file: F32455073: find_collations.py. (Show Details)Mar 10 2026, 2:21 PM
Raine updated the task description. (Show Details)Mar 10 2026, 2:28 PM
Raine updated the task description. (Show Details)
Raine closed subtask T418752: ICU 72 upgrade: double-check feasibility of new approach; come up with a runbook as Resolved.Mar 10 2026, 3:03 PM
Raine updated the task description. (Show Details)Mar 10 2026, 3:52 PM
Raine updated the task description. (Show Details)Mar 10 2026, 7:36 PM
Scott_French changed the status of subtask T419058: Prepare packages and production images for ICU 72 upgrade from Open to In Progress.Mar 12 2026, 5:12 PM
JMeybohm subscribed.Mar 12 2026, 5:29 PM
Scott_French closed subtask T419058: Prepare packages and production images for ICU 72 upgrade as Resolved.Mar 12 2026, 9:05 PM
Raine mentioned this in T419980: ICU 72 upgrade: `categorylinks` table swap.Mar 13 2026, 12:59 PM
Raine updated the task description. (Show Details)
ItsNyoty subscribed.EditedMar 14 2026, 1:47 PM

Please keep in mind that there are currently 2 vulnerabilities inside ICU 67.1 (the version currently used).

https://nvd.nist.gov/vuln/detail/CVE-2025-5222
https://nvd.nist.gov/vuln/detail/cve-2020-21913

CVE-2025-5222 is a vulnerability up to (excluding) 77.1

taavi subscribed.Mar 14 2026, 1:51 PM
In T419049#11709963, @ItsNyoty wrote:

Please keep a mind that there are currently 2 vulnerabilities inside ICU 67.1 (the version currently used).

Both of these have been fixed in the Debian packaging we use:

gerritbot added a comment.Mar 17 2026, 5:25 PM

Change #1254266 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/puppet@production] k8s: create shellbox-icu72

https://gerrit.wikimedia.org/r/1254266

gerritbot added a project: Patch-For-Review.Mar 17 2026, 5:25 PM
gerritbot added a comment.Mar 17 2026, 7:33 PM

Change #1254266 abandoned by Kamila Součková:

[operations/puppet@production] k8s: create shellbox-icu72

Reason:

Will go with just another release in one of the existing namespaces.

https://gerrit.wikimedia.org/r/1254266

Maintenance_bot removed a project: Patch-For-Review.Mar 17 2026, 8:31 PM
gerritbot added a comment.Mar 17 2026, 9:52 PM

Change #1254348 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[mediawiki/core@master] DB schema: Create temporary table for ICU upgrade

https://gerrit.wikimedia.org/r/1254348

gerritbot added a project: Patch-For-Review.Mar 17 2026, 9:52 PM
ItsNyoty unsubscribed.Mar 18 2026, 6:59 AM
gerritbot added a comment.Mar 18 2026, 6:58 PM

Change #1254348 abandoned by Kamila Součková:

[mediawiki/core@master] DB schema: Create temporary table for ICU upgrade

Reason:

Not needed for temporary tables.

https://gerrit.wikimedia.org/r/1254348

Maintenance_bot removed a project: Patch-For-Review.Mar 18 2026, 7:37 PM
Stashbot added a comment.Mar 19 2026, 10:36 AM

Mentioned in SAL (#wikimedia-operations) [2026-03-19T10:36:44Z] <Raine> created temporary categorylinks_icu72 tables -- T419980, T419049

Raine changed the status of subtask T419242: Migrate collation data to ICU 72 from Stalled to In Progress.Mar 20 2026, 12:49 PM
gerritbot added a comment.Mar 20 2026, 1:32 PM

Change #1256384 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/mediawiki-config@master] Temporarily add shellbox-icu to $wgShellboxUrls

https://gerrit.wikimedia.org/r/1256384

gerritbot added a project: Patch-For-Review.Mar 20 2026, 1:32 PM
Raine created subtask T420748: CommRel support for ICU 72 upgrade.Mar 20 2026, 4:04 PM
Raine mentioned this in T420748: CommRel support for ICU 72 upgrade.
gerritbot added a comment.Mar 26 2026, 1:16 PM

Change #1256384 merged by jenkins-bot:

[operations/mediawiki-config@master] Temporarily add shellbox-icu to $wgShellboxUrls

https://gerrit.wikimedia.org/r/1256384

Stashbot mentioned this in T419274: ICU 72 upgrade: enable remote ICU collation writes.Mar 26 2026, 1:17 PM
Stashbot mentioned this in T419242: Migrate collation data to ICU 72.

Mentioned in SAL (#wikimedia-operations) [2026-03-26T13:17:11Z] <kamila@deploy2002> Started scap sync-world: Backport for [[gerrit:1256384|Temporarily add shellbox-icu to $wgShellboxUrls (T419049 T419242 T419274)]]

Stashbot added a comment.Mar 26 2026, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-26T13:19:12Z] <kamila@deploy2002> kamila: Backport for [[gerrit:1256384|Temporarily add shellbox-icu to $wgShellboxUrls (T419049 T419242 T419274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mar 26 2026, 1:24 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-26T13:24:27Z] <kamila@deploy2002> Finished scap sync-world: Backport for [[gerrit:1256384|Temporarily add shellbox-icu to $wgShellboxUrls (T419049 T419242 T419274)]] (duration: 07m 16s)

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2026, 1:30 PM
gerritbot added a comment.Mar 26 2026, 2:39 PM

Change #1261470 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/mediawiki-config@master] Enable $wgTempCategoryCollations for testwiki.

https://gerrit.wikimedia.org/r/1261470

gerritbot added a project: Patch-For-Review.Mar 26 2026, 2:39 PM
gerritbot added a comment.Mar 26 2026, 8:05 PM

Change #1261470 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable $wgTempCategoryCollations for testwiki.

https://gerrit.wikimedia.org/r/1261470

Stashbot added a comment.Mar 26 2026, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-26T20:06:29Z] <kamila@deploy1003> Started scap sync-world: Backport for [[gerrit:1261545|Wrap 'centralauthtoken' in a JWT (T420280)]], [[gerrit:1261470|Enable $wgTempCategoryCollations for testwiki. (T419274 T419049)]]

Stashbot mentioned this in T420280: Authenticated cross-origin requests are being throttled as if unauthenticated (centralauth).Mar 26 2026, 8:06 PM
Stashbot added a comment.Mar 26 2026, 8:25 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-26T20:25:05Z] <kamila@deploy1003> matmarex, kamila: Backport for [[gerrit:1261545|Wrap 'centralauthtoken' in a JWT (T420280)]], [[gerrit:1261470|Enable $wgTempCategoryCollations for testwiki. (T419274 T419049)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2026, 8:31 PM
Stashbot added a comment.Mar 26 2026, 8:44 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-26T20:44:01Z] <kamila@deploy1003> Finished scap sync-world: Backport for [[gerrit:1261545|Wrap 'centralauthtoken' in a JWT (T420280)]], [[gerrit:1261470|Enable $wgTempCategoryCollations for testwiki. (T419274 T419049)]] (duration: 37m 32s)

gerritbot added a comment.Mar 27 2026, 12:55 PM

Change #1262091 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/mediawiki-config@master] Enable $wgTempCategoryCollations for s3 wikis.

https://gerrit.wikimedia.org/r/1262091

gerritbot added a project: Patch-For-Review.Mar 27 2026, 12:55 PM
Raine updated the task description. (Show Details)Mar 27 2026, 1:05 PM
gerritbot added a comment.Mar 30 2026, 2:20 PM

Change #1262091 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable $wgTempCategoryCollations for s3 wikis.

https://gerrit.wikimedia.org/r/1262091

Stashbot added a comment.Mar 30 2026, 2:20 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T14:20:56Z] <kamila@deploy1003> Started scap sync-world: Backport for [[gerrit:1262091|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]]

Stashbot added a comment.Mar 30 2026, 2:22 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T14:22:41Z] <kamila@deploy1003> kamila: Backport for [[gerrit:1262091|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mar 30 2026, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T14:30:55Z] <kamila@deploy1003> Finished scap sync-world: Backport for [[gerrit:1262091|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] (duration: 09m 59s)

Maintenance_bot removed a project: Patch-For-Review.Mar 30 2026, 2:31 PM
gerritbot added a comment.Mar 30 2026, 3:54 PM

Change #1264670 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/mediawiki-config@master] Enable $wgTempCategoryCollations for s3 wikis.

https://gerrit.wikimedia.org/r/1264670

gerritbot added a project: Patch-For-Review.Mar 30 2026, 3:54 PM
gerritbot added a comment.Mar 30 2026, 5:05 PM

Change #1264670 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable $wgTempCategoryCollations for s3 wikis.

https://gerrit.wikimedia.org/r/1264670

Stashbot added a comment.Mar 30 2026, 5:05 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T17:05:53Z] <kamila@deploy1003> Started scap sync-world: Backport for [[gerrit:1264670|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]]

Stashbot added a comment.Mar 30 2026, 5:07 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T17:07:40Z] <kamila@deploy1003> kamila: Backport for [[gerrit:1264670|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mar 30 2026, 5:18 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T17:18:30Z] <kamila@deploy1003> Finished scap sync-world: Backport for [[gerrit:1264670|Enable $wgTempCategoryCollations for s3 wikis. (T419274 T419049)]] (duration: 12m 36s)

Maintenance_bot removed a project: Patch-For-Review.Mar 30 2026, 5:31 PM
Raine created subtask T421913: Create MW images with ICU 72 libs.Mar 31 2026, 2:48 PM
CodeReviewBot added a project: Patch-For-Review.Mar 31 2026, 3:26 PM

kamila opened https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/239

Draft: Change base image to use ICU 72

gerritbot added a comment.Apr 1 2026, 1:23 PM

Change #1266250 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/deployment-charts@master] shellbox-icu72: Add ClusterIP to TLS cert SANs

https://gerrit.wikimedia.org/r/1266250

gerritbot added a comment.Apr 1 2026, 1:42 PM

Change #1266264 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/mediawiki-config@master] Temporarily add shellbox-icu ClusterIP endpoint

https://gerrit.wikimedia.org/r/1266264

gerritbot added a comment.Apr 1 2026, 2:33 PM

Change #1266278 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[mediawiki/core@master] maintenance/updateCollation: add --source-table

https://gerrit.wikimedia.org/r/1266278

Raine closed this task as Declined.Apr 7 2026, 7:31 PM

Upon further discussion with DBA, we will fall back to the old process (i.e. just running the maintenance script in place) this time around.

This is because the table swap is, and always will be, a risky DB operation on very large wikis (e.g. enwiki). While this does not prevent this smaller wikis test, we will have to change the table swap procedure to be safe for larger wikis. At this point we have tested almost everything other than the table swap, so there is no added value in running a test of a procedure we won't be able to reuse.

Therefore:

  1. We will fall back to the old process for this ICU upgrade, tracked in T422544.
    • We can reuse a lot of the work done on this task.
  2. We will discuss how to make the new process safe. A possible suggestion is to move the table swapping logic to MW, and thus not put the DBs at risk at all.
    • By not trying to do this right now, we will avoid further stalling the current upgrade with unknown unknowns.
  3. We will capture all the work, patches, processes and learnings we figured out on this task. Thereby, once the above is done, we can directly reuse all the work done here.
    • This might be best done by (1) being careful to not delete any information here, and (2) following the subtasks here and creating a runbook on wikitech, either now or during the next ICU upgrade.
Raine created subtask T422546: Clean up after the ICU 72 upgrade.Apr 7 2026, 7:34 PM
Raine closed subtask T419274: ICU 72 upgrade: enable remote ICU collation writes as Declined.Apr 7 2026, 7:51 PM
Raine mentioned this in T422544: Upgrade the MediaWiki servers to ICU 72 (no table swap).Apr 7 2026, 9:01 PM
jijiki removed a subtask: T419212: Upgrade ServiceOps roles from Bullseye to Debian Trixie.Apr 8 2026, 9:42 AM
Raine closed subtask T420748: CommRel support for ICU 72 upgrade as Declined.Apr 8 2026, 12:39 PM
gerritbot added a comment.Apr 9 2026, 1:49 PM

Change #1266250 abandoned by Kamila Součková:

[operations/deployment-charts@master] shellbox-icu72: Add ClusterIP to TLS cert SANs

Reason:

T419049 declined

https://gerrit.wikimedia.org/r/1266250

Raine created subtask T423138: Make the new ICU upgrade process production-ready.Apr 13 2026, 2:31 PM
Raine mentioned this in T423138: Make the new ICU upgrade process production-ready.
gerritbot added a comment.Apr 13 2026, 2:54 PM

Change #1266264 abandoned by Kamila Součková:

[operations/mediawiki-config@master] Temporarily add shellbox-icu ClusterIP endpoint

Reason:

T419049 declined

https://gerrit.wikimedia.org/r/1266264

Raine changed the status of subtask T422546: Clean up after the ICU 72 upgrade from Open to In Progress.Apr 17 2026, 1:50 PM
gerritbot added a comment.Apr 20 2026, 1:33 AM

Change #1266278 merged by jenkins-bot:

[mediawiki/core@master] maintenance/updateCollation: add --table

https://gerrit.wikimedia.org/r/1266278

Marostegui closed subtask T422546: Clean up after the ICU 72 upgrade as Resolved.Apr 20 2026, 9:16 AM
CodeReviewBot added a comment.Apr 20 2026, 9:30 AM

kamila merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/239

Change base image to use ICU 72

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2026, 9:34 AM
Raine closed subtask T419242: Migrate collation data to ICU 72 as Declined.Apr 20 2026, 9:48 AM
Raine closed subtask T421913: Create MW images with ICU 72 libs as Resolved.Apr 21 2026, 12:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits