Page MenuHomePhabricator
Create Task
Maniphest T421366

Test Kitchen Experiment setup to measure the impact of the banner
Closed, ResolvedPublic
Actions

Description

Experiment: Nudging users with unconfirmed emails to confirm them
Team: Product Safety and Integrity team
Experiment launch date: April 16, 2026

Motivation

We want to A/B test the banner presence before a complete rollout to ensure there is no unintended negative impact on the user behavior.

Hypothesis
  • We see an increase in confirmed emails for users who see the banner
Experiment setup
  • Users: Logged-in users with unconfirmed emails
  • Wikis: English, French, German, Polish, Japanese, Hebrew, Wikidata, Commons, Mediawiki, Meta
  • Experiment setup:
    • A/B test
    • Arm A: 50% of users in the experiment see banner version 1 as implemented in T418336
    • Arm B: 50% users in control group
  • Length of experiment: 2 weeks
Metrics
  • Primary metric: Number of accounts with confirmed emails
  • Secondary metric: Banner impressions and Banner clicks (tracked in T420007)
  • Guardrail metric: Number of accounts with emails

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Niharika created this task.Mar 26 2026, 12:40 PM
Niharika triaged this task as Medium priority.
Restricted Application added subscribers: Dragoniez, Aklapper. · View Herald TranscriptMar 26 2026, 12:40 PM
cchen subscribed.Mar 26 2026, 3:09 PM
gerritbot added a comment.Mar 26 2026, 6:09 PM

Change #1261516 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on selected wikis

https://gerrit.wikimedia.org/r/1261516

gerritbot added a project: Patch-For-Review.Mar 26 2026, 6:09 PM
gerritbot added a comment.Mar 26 2026, 6:50 PM

Change #1261525 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on mediawikiwiki

https://gerrit.wikimedia.org/r/1261525

gerritbot added a comment.Mar 26 2026, 6:51 PM

Change #1261525 abandoned by Mmartorana:

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on mediawikiwiki

https://gerrit.wikimedia.org/r/1261525

gerritbot added a comment.Mar 26 2026, 6:53 PM

Change #1261526 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on mediawikiwiki

https://gerrit.wikimedia.org/r/1261526

gerritbot added a comment.Mar 26 2026, 9:29 PM

Change #1261608 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[mediawiki/core@master] Email confirmation banner: Add Test Kitchen A/B gating

https://gerrit.wikimedia.org/r/1261608

Niharika updated the task description. (Show Details)Mar 27 2026, 11:58 AM
gerritbot added a comment.Mar 30 2026, 3:19 PM

Change #1261516 abandoned by Mmartorana:

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on selected wikis

Reason:

cancelling scheduled deploy

https://gerrit.wikimedia.org/r/1261516

gerritbot added a comment.Mar 30 2026, 3:20 PM

Change #1261526 abandoned by Mmartorana:

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on mediawikiwiki

Reason:

cancelling scheduled deploy

https://gerrit.wikimedia.org/r/1261526

gerritbot added a comment.Mar 30 2026, 3:28 PM

Change #1264662 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on testwiki

https://gerrit.wikimedia.org/r/1264662

gerritbot added a comment.Mar 30 2026, 8:10 PM

Change #1264662 merged by jenkins-bot:

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on testwiki

https://gerrit.wikimedia.org/r/1264662

Stashbot added a comment.Mar 30 2026, 8:10 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T20:10:19Z] <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1264662|config: Enable EmailConfirmationBanner on testwiki (T421366)]]

Stashbot added a comment.Mar 30 2026, 8:12 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T20:12:09Z] <cjming@deploy1003> cjming, mmartorana: Backport for [[gerrit:1264662|config: Enable EmailConfirmationBanner on testwiki (T421366)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Mar 30 2026, 8:29 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T20:29:36Z] <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1264662|config: Enable EmailConfirmationBanner on testwiki (T421366)]] (duration: 19m 16s)

gerritbot added a comment.Mar 31 2026, 3:04 AM

Change #1261608 merged by jenkins-bot:

[mediawiki/core@master] Email confirmation banner: Add Test Kitchen A/B gating

https://gerrit.wikimedia.org/r/1261608

gerritbot added a comment.Mar 31 2026, 3:19 AM

Change #1264921 had a related patch set uploaded (by Catrope; author: Mmartorana):

[mediawiki/core@wmf/1.46.0-wmf.22] Email confirmation banner: Add Test Kitchen A/B gating

https://gerrit.wikimedia.org/r/1264921

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.23; 2026-04-07).Mar 31 2026, 4:00 AM
gerritbot added a comment.Mar 31 2026, 8:44 PM

Change #1264921 merged by jenkins-bot:

[mediawiki/core@wmf/1.46.0-wmf.22] Email confirmation banner: Add Test Kitchen A/B gating

https://gerrit.wikimedia.org/r/1264921

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.22; 2026-03-31); removed: MW-1.46-notes (1.46.0-wmf.23; 2026-04-07).Mar 31 2026, 9:00 PM
Stashbot added a comment.Mar 31 2026, 9:25 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-31T21:25:15Z] <urbanecm@deploy1003> Started scap sync-world: Email confirmation banner: Add Test Kitchen A/B gating (T421366)

Maintenance_bot removed a project: Patch-For-Review.Mar 31 2026, 9:30 PM
Stashbot added a comment.Mar 31 2026, 9:56 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-31T21:56:48Z] <urbanecm@deploy1003> Finished scap sync-world: Email confirmation banner: Add Test Kitchen A/B gating (T421366) (duration: 31m 33s)

gerritbot added a comment.Apr 1 2026, 4:11 PM

Change #1266314 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on mediawikiwiki

https://gerrit.wikimedia.org/r/1266314

gerritbot added a project: Patch-For-Review.Apr 1 2026, 4:11 PM
gerritbot added a comment.Apr 1 2026, 4:22 PM

Change #1266315 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[mediawiki/core@master] i18n: Update arm_b email confirmation banner copy

https://gerrit.wikimedia.org/r/1266315

gerritbot added a comment.Apr 1 2026, 5:47 PM

Change #1266315 merged by jenkins-bot:

[mediawiki/core@master] i18n: Update arm_b email confirmation banner copy

https://gerrit.wikimedia.org/r/1266315

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.23; 2026-04-07); removed: MW-1.46-notes (1.46.0-wmf.22; 2026-03-31).Apr 1 2026, 6:00 PM
gerritbot added a comment.Apr 1 2026, 8:04 PM

Change #1266314 merged by jenkins-bot:

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on mediawikiwiki

https://gerrit.wikimedia.org/r/1266314

Stashbot added a comment.Apr 1 2026, 8:04 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T20:04:52Z] <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1266314|config: Enable EmailConfirmationBanner on mediawikiwiki (T421366)]]

Stashbot added a comment.Apr 1 2026, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T20:06:53Z] <cjming@deploy1003> mmartorana, cjming: Backport for [[gerrit:1266314|config: Enable EmailConfirmationBanner on mediawikiwiki (T421366)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Apr 1 2026, 8:13 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T20:13:40Z] <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1266314|config: Enable EmailConfirmationBanner on mediawikiwiki (T421366)]] (duration: 08m 47s)

Maintenance_bot removed a project: Patch-For-Review.Apr 1 2026, 8:31 PM
gerritbot added a comment.Apr 2 2026, 4:23 PM

Change #1261516 restored by Mmartorana:

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on selected wikis

https://gerrit.wikimedia.org/r/1261516

gerritbot added a project: Patch-For-Review.Apr 2 2026, 4:23 PM
gerritbot added a comment.Apr 2 2026, 4:26 PM

Change #1261516 abandoned by Mmartorana:

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on selected wikis

https://gerrit.wikimedia.org/r/1261516

gerritbot added a comment.Apr 2 2026, 4:32 PM

Change #1267116 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on selected wikis

https://gerrit.wikimedia.org/r/1267116

Dragoniez unsubscribed.Apr 2 2026, 4:33 PM
Niharika mentioned this in T422487: Update banner text to remove reference to edit reverts.Apr 7 2026, 1:03 PM
Niharika updated the task description. (Show Details)Apr 15 2026, 2:50 PM
Niharika updated the task description. (Show Details)
gerritbot added a comment.Apr 16 2026, 8:03 PM

Change #1267116 merged by jenkins-bot:

[operations/mediawiki-config@master] config: Enable EmailConfirmationBanner on selected wikis

https://gerrit.wikimedia.org/r/1267116

Stashbot added a comment.Apr 16 2026, 8:03 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-16T20:03:30Z] <mstyles@deploy1003> Started scap sync-world: Backport for [[gerrit:1267116|config: Enable EmailConfirmationBanner on selected wikis (T421366)]]

Stashbot added a comment.Apr 16 2026, 8:05 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-16T20:05:12Z] <mstyles@deploy1003> mmartorana, mstyles: Backport for [[gerrit:1267116|config: Enable EmailConfirmationBanner on selected wikis (T421366)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Apr 16 2026, 8:09 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-16T20:09:37Z] <mstyles@deploy1003> Finished scap sync-world: Backport for [[gerrit:1267116|config: Enable EmailConfirmationBanner on selected wikis (T421366)]] (duration: 06m 06s)

gerritbot added a comment.Apr 24 2026, 4:34 PM

Change #1277129 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[mediawiki/core@master] Email confirmation banner: Remove obsolete arm_b variant

https://gerrit.wikimedia.org/r/1277129

gerritbot added a comment.Apr 30 2026, 3:10 PM

Change #1277129 merged by jenkins-bot:

[mediawiki/core@master] Email confirmation banner: Remove obsolete arm_b variant

https://gerrit.wikimedia.org/r/1277129

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.1; 2026-05-05).Apr 30 2026, 4:01 PM
gerritbot added a comment.May 1 2026, 1:51 PM

Change #1281501 had a related patch set uploaded (by Mmartorana; author: Mmartorana):

[mediawiki/core@wmf/1.46.0-wmf.26] Email confirmation banner: Remove obsolete arm_b variant

https://gerrit.wikimedia.org/r/1281501

gerritbot added a comment.May 5 2026, 9:15 PM

Change #1281501 merged by jenkins-bot:

[mediawiki/core@wmf/1.46.0-wmf.26] Email confirmation banner: Remove obsolete arm_b variant

https://gerrit.wikimedia.org/r/1281501

Stashbot mentioned this in T417513: Switch to CSS/JS solution for thumbnail size in legacy parser.May 5 2026, 9:16 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-05T21:16:13Z] <arlolra@deploy1003> Started scap sync-world: Backport for [[gerrit:1281501|Email confirmation banner: Remove obsolete arm_b variant (T421366)]], [[gerrit:1283056|Legacy parser no longer varies by user thumbnail size. (T417513)]]

Stashbot added a comment.May 5 2026, 9:33 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-05T21:33:52Z] <arlolra@deploy1003> jdlrobson, mmartorana, arlolra: Backport for [[gerrit:1281501|Email confirmation banner: Remove obsolete arm_b variant (T421366)]], [[gerrit:1283056|Legacy parser no longer varies by user thumbnail size. (T417513)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.May 5 2026, 9:49 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-05T21:49:08Z] <arlolra@deploy1003> Finished scap sync-world: Backport for [[gerrit:1281501|Email confirmation banner: Remove obsolete arm_b variant (T421366)]], [[gerrit:1283056|Legacy parser no longer varies by user thumbnail size. (T417513)]] (duration: 32m 55s)

ReleaseTaggerBot edited projects, added: MW-1.46-notes (1.46.0-wmf.26; 2026-04-28); removed: MW-1.46-notes (1.46.0-wmf.23; 2026-04-07).May 5 2026, 10:00 PM
mmartorana created subtask T425677: SRM for email confirmation banner experiment.May 7 2026, 2:12 PM
mmartorana changed the status of subtask T425677: SRM for email confirmation banner experiment from Open to In Progress.
mmartorana closed subtask T425677: SRM for email confirmation banner experiment as Resolved.May 13 2026, 1:46 PM
mmartorana closed this task as Resolved.Fri, Jun 5, 4:23 PM
mmartorana claimed this task.
mmartorana mentioned this in T428291: Disable Email Confirmation Banner on all wikis.Fri, Jun 5, 4:57 PM
gerritbot added a comment.Thu, Jun 11, 6:45 PM

Change #1300887 had a related patch set uploaded (by SomeRandomDeveloper; author: Mmartorana):

[mediawiki/core@REL1_46] Email confirmation banner: Remove obsolete arm_b variant

https://gerrit.wikimedia.org/r/1300887

gerritbot added a comment.Thu, Jun 11, 7:12 PM

Change #1300887 merged by jenkins-bot:

[mediawiki/core@REL1_46] Email confirmation banner: Remove obsolete arm_b variant

https://gerrit.wikimedia.org/r/1300887

ReleaseTaggerBot edited projects, added: MW-1.46-notes; removed: MW-1.46-notes (1.46.0-wmf.26; 2026-04-28).Thu, Jun 11, 8:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits