Page MenuHomePhabricator

Sunsetting mirrors.wikimedia.org
Open, MediumPublic

Description

We've been operating a Debian mirror since approx. ten years (it also hosts a mirror of Ubuntu, Tails and Debian Openstack Backports repository). This has historically been a great way to contribute back to our favourite OS. But Debian defaults to the deb.debian.org CDN since many years and that shrank the use of the traditional mirrors significantly.

Despite being part of the ftp.us.debian.org mirror rotation our mirror is hardly used these days, even despite idiotic AI scrapers hitting the server randomly and excessively, our bandwidth usage practically never exceeds 10% of the available bandwidth (and usually even less than that).

The current server is ending it's hardware lifespan, so in the current state it currently does not really make sense to refresh it, so we'll likely decommision it entirely.

Event Timeline

Change #1256371 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch our servers to use deb.debian.org

https://gerrit.wikimedia.org/r/1256371

Hi folks! Any idea when this is likely to happen? I will need to coordinate for openstack nodes which use a bespoke openstack repo hosted on the mirror.

The alert MirrorHighLag has started firing 1 month ago. Would it make sense to disable it at this point?

Labels
alertname=MirrorHighLag
cluster=misc
instance=mirror1001:9100
job=node
path=/srv/mirrors/ubuntu
prometheus=ops
severity=warning
site=eqiad
source=prometheus
team=infrastructure-foundations
Annotations
NameContent
dashboardhttps://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1
descriptionMirrors - /srv/mirrors/ubuntu synchronization lag is behind 38d 16h 31m 9s
runbookhttps://wikitech.wikimedia.org/wiki/Mirrors
summaryMirrors - /srv/mirrors/ubuntu synchronization lag
Links

Triage metadata. Do not delete.
fingerprint=60b74e4f03d5302a

Change #1256371 merged by Muehlenhoff:

[operations/puppet@production] Switch our servers to use deb.debian.org

https://gerrit.wikimedia.org/r/1256371

Change #1268515 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch three test systems to deb.debian.org

https://gerrit.wikimedia.org/r/1268515

Change #1268515 merged by Muehlenhoff:

[operations/puppet@production] Switch three test systems to deb.debian.org

https://gerrit.wikimedia.org/r/1268515

Change #1268522 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch our servers to use deb.debian.org

https://gerrit.wikimedia.org/r/1268522

Change #1268522 merged by Muehlenhoff:

[operations/puppet@production] Switch our servers to use deb.debian.org

https://gerrit.wikimedia.org/r/1268522

thcipriani added subscribers: dduvall, thcipriani.

Checking my understanding of "sunsetting" here:

  • We're no longer hosting a mirror? vs.
  • the mirrors.wikimedia.org url will cease to function?

Currently all the base docker images have mirrors in their apt source:

(◍•ᴗ•◍)❤ docker run --rm -it docker-registry.wikimedia.org/trixie:20260412 cat /etc/apt/sources.list | grep mirrors
deb http://mirrors.wikimedia.org/debian/ trixie main
deb http://mirrors.wikimedia.org/debian/ trixie-updates main
deb http://mirrors.wikimedia.org/debian/ trixie-backports main contrib
(◍•ᴗ•◍)❤ docker run --rm -it docker-registry.wikimedia.org/bookworm:20260412 cat /etc/apt/sources.list | grep mirrors
deb http://mirrors.wikimedia.org/debian/ bookworm main
deb http://mirrors.wikimedia.org/debian/ bookworm-updates main
deb http://mirrors.wikimedia.org/debian/ bookworm-backports main contrib
(◍•ᴗ•◍)❤ docker run --rm -it docker-registry.wikimedia.org/bullseye:20260412 cat /etc/apt/sources.list | grep mirrors
deb http://mirrors.wikimedia.org/debian/ bullseye main
deb http://mirrors.wikimedia.org/debian/ bullseye-updates main

Trying to think through the consequences of this for CI images and pipeline building (CC @dduvall )

Checking my understanding of "sunsetting" here:

  • We're no longer hosting a mirror? vs.
  • the mirrors.wikimedia.org url will cease to function?

These are both true. We will no longer operate a mirror (which is running under mirrors.wikimedia.org), so these image should be built using deb.debian.org as the mirror going forward.

These are both true. We will no longer operate a mirror (which is running under mirrors.wikimedia.org), so these image should be built using deb.debian.org as the mirror going forward.

And going backward? Is there a way we can help "cool URIs don't change" by redirecting mirrors.wikimedia.org elsewhere?

And going backward? Is there a way we can help "cool URIs don't change" by redirecting mirrors.wikimedia.org elsewhere?

I'm obviously not WMF staff; but from e.g. T243056#6095651 it feels like a .wikimedia.org URL maybe shouldn't be redirected to an alternative mirror-site that isn't controlled by the WMF. But maybe e.g. collaboration-services could host a microsite at the mirrors.wikimedia.org domain, that just serves to point to alternative mirrors/sites for the downloads that it's currently providing?

But maybe e.g. collaboration-services could host a microsite at the mirrors.wikimedia.org domain, that just serves to point to alternative mirrors/sites for the downloads that it's currently providing?

That would fix nothing of value for me. The problem is needing to update the apt sources for anything that is currently pointed to mirrors.wikimedia.org. A redirect would keep that working, at least for a bit. A static page telling me that mirrors.wikimedia.org has been taken offline as an apt repo would still leave everything broken.

I assume that @Krinkle and @Jdforrester-WMF were concerned about things like HTTP cookie leaks in that exchange, but maybe there is something even more insidious that I'm not considering. Anyway, I'd like to hear from a WMF SRE making system changing decisions on the general topic.

@bd808 In addition to the good point raised by @A_smart_kitten above the general intent here is to reduce complexity. Leaving a dependency that is no longer tied to a service and has potential to break silently in the future would go against that. The preference is for end users to switch to the canonical (no pun intended) way of accessing repositories.

Change #1273441 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Switch Cloud VPS to deb.debian.org

https://gerrit.wikimedia.org/r/1273441

Change #1273441 merged by Muehlenhoff:

[operations/puppet@production] Switch Cloud VPS to deb.debian.org

https://gerrit.wikimedia.org/r/1273441

Change #1294284 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Retire the Ubuntu mirror

https://gerrit.wikimedia.org/r/1294284

Change #1294285 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] autoinstall: Stop using mirrors.wikimedia.org

https://gerrit.wikimedia.org/r/1294285

Change #1294306 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Disable tails mirror

https://gerrit.wikimedia.org/r/1294306

I opened a ticket at https://anonticket.torproject.org/ and mirrors.wikimedia.org has been removed from the download mirror rotation for Tails.

Change #1294285 merged by Muehlenhoff:

[operations/puppet@production] autoinstall: Stop using mirrors.wikimedia.org

https://gerrit.wikimedia.org/r/1294285

Change #1294306 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Disable tails mirror

https://gerrit.wikimedia.org/r/1294306

Change #1294938 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove remaining bits of Tails mirror

https://gerrit.wikimedia.org/r/1294938

Change #1294938 merged by Muehlenhoff:

[operations/puppet@production] Remove remaining bits of Tails mirror

https://gerrit.wikimedia.org/r/1294938

Change #1294980 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Disable osbpo sync

https://gerrit.wikimedia.org/r/1294980

Change #1295362 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] package_builder: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295362

Change #1295362 merged by Muehlenhoff:

[operations/puppet@production] package_builder: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295362

Change #1295945 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] autoinstall: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295945

Change #1295945 merged by Muehlenhoff:

[operations/puppet@production] autoinstall: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295945

Change #1295956 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] autoinstall: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295956

Change #1295956 merged by Muehlenhoff:

[operations/puppet@production] autoinstall: Switch to deb.debian.org

https://gerrit.wikimedia.org/r/1295956

Change #1294980 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Disable osbpo sync

https://gerrit.wikimedia.org/r/1294980

Change #1301232 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Removing remaining bits of osbpo mirror

https://gerrit.wikimedia.org/r/1301232

Change #1301232 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Removing remaining bits of osbpo mirror

https://gerrit.wikimedia.org/r/1301232

Change #1301237 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/alerts@master] Stop monitoring the Ubuntu part of our mirror

https://gerrit.wikimedia.org/r/1301237

Change #1301238 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] mirrors: Disable the NRPE check for the Ubuntu mirror

https://gerrit.wikimedia.org/r/1301238

Change #1301237 merged by Muehlenhoff:

[operations/alerts@master] Stop monitoring the Ubuntu part of our mirror

https://gerrit.wikimedia.org/r/1301237

Change #1301238 merged by Muehlenhoff:

[operations/puppet@production] mirrors: Disable the NRPE check for the Ubuntu mirror

https://gerrit.wikimedia.org/r/1301238