One hurdle in investigating T421152: Could not hoist data into experiment.subject_id for event was that validation errors don't have user agent information associated with them, which makes it difficult to understand whether the bug is being triggered by a particular version of a particular browser.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | phuedx | T427092 Run and synthetic A/A test that captures UA to investigate hoisting errors | |||
| Resolved | PRODUCTION ERROR | phuedx | T421152 Could not hoist data into experiment.subject_id for event | ||
| Resolved | phuedx | T425096 ConfigsFetcher cache is not multi-DC | |||
| Resolved | phuedx | T419513 JS SDK: Read everyone experiment enrollment from the WMF-Uniq server timing header | |||
| Resolved | BBlack | T420586 Varnish sends invalid Server-Timing header when device is enrolled in an experiment |
Change #1293674 had a related patch set uploaded (by Phuedx; author: Phuedx):
[mediawiki/extensions/WikimediaEvents@master] ext.wikimediaEvents: Add hoisting error detection test
Change #1293674 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] ext.wikimediaEvents: Add hoisting error detection test
Change #1294217 had a related patch set uploaded (by Phuedx; author: Phuedx):
[mediawiki/extensions/WikimediaEvents@wmf/1.47.0-wmf.4] ext.wikimediaEvents: Add hoisting error detection test
Change #1294217 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.47.0-wmf.4] ext.wikimediaEvents: Add hoisting error detection test
Mentioned in SAL (#wikimedia-operations) [2026-05-27T13:28:50Z] <phuedx@deploy1003> Started scap sync-world: Backport for [[gerrit:1294217|ext.wikimediaEvents: Add hoisting error detection test (T427092)]]
Mentioned in SAL (#wikimedia-operations) [2026-05-27T13:30:45Z] <phuedx@deploy1003> phuedx: Backport for [[gerrit:1294217|ext.wikimediaEvents: Add hoisting error detection test (T427092)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2026-05-27T13:40:25Z] <phuedx@deploy1003> Finished scap sync-world: Backport for [[gerrit:1294217|ext.wikimediaEvents: Add hoisting error detection test (T427092)]] (duration: 11m 35s)
SELECT
month,
day,
SUM(1) AS n,
SUM(IF(CAST(STRPOS(json_extract_scalar(raw_event, '$.agent.ua_string'), 'compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm') AS BOOLEAN), 1, 0)) AS n_bingbot,
ROUND(
SUM(IF(CAST(STRPOS(json_extract_scalar(raw_event, '$.agent.ua_string'), 'compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm') AS BOOLEAN), 1, 0)) / CAST(SUM(0.01) AS DOUBLE),
2
) AS pct_bingbot
FROM
event.eventgate_analytics_external_error_validation
WHERE
year = 2026
AND month >= 5
AND starts_with(message, 'Could not hoist data')
AND json_extract_scalar(raw_event, '$.experiment.enrolled') = 'synth-aa-detect-hoisting-errors-1'
GROUP BY
1, 2
ORDER BY
1, 2 ASC
;+-------+-----+------+-----------+-------------+ | month | day | n | n_bingbot | pct_bingbot | +-------+-----+------+-----------+-------------+ | 5 | 27 | 28 | 12 | 42.86 | | 5 | 28 | 1089 | 513 | 47.11 | | 5 | 29 | 2906 | 2155 | 74.16 | | 5 | 30 | 1819 | 890 | 48.93 | | 5 | 31 | 1664 | 721 | 43.33 | | 6 | 1 | 3489 | 2679 | 76.78 | | 6 | 2 | 3012 | 2145 | 71.22 | | 6 | 3 | 2086 | 1506 | 72.2 | +-------+-----+------+-----------+-------------+
As discussed with the Experiment Platform team earlier this week, Bingbot is overrepresented in the experiment results. To put it in perspective, Bingbot accounts for ~0.2% of requests on 2026/05/28.
We suggest implementing a UA denylist in EventGate. Bingbot (and other bots like it) will continue to crawl the site but shouldn't affect our SLOs and shouldn't be represented in experiment results.