Motivation
To evaluate the effectiveness of the Incident Reporting System (IRS) during the 5% rollout on enwiki, we need to update our instrumentation. Specifically, we need to understand whether users can successfully navigate each step of the flow, where they drop off, and whether they proceed from support pages to the intended reporting destinations. These insights will inform whether the system is functioning as intended and support decisions on scaling beyond the initial 5% rollout.
Background
The Incident Reporting System (IRS) guides users through a structured flow:
- Entry point (report button from comment or sidebar)
- Flow selection (emergency vs non-emergency)
- Category selection
- Support page with guidance
- Outbound action (link click)
As part of this rollout, we are introducing an updated categories list. Resulting category set for enwiki trial:
- Bullying, intimidation, threats, or insults
- Sexual harassment
- Exposing private personal information (doxxing)
- Trolling or Hounding (Stalking)
- Hateful or discriminatory content
- Illegitimate use of multiple accounts (sockpuppetry)
- Vandalism
- Disruptive editing
- Disputes with another user
- Other
Instrumentation
Update instrumentation as per this spec:
https://docs.google.com/spreadsheets/d/1067j3sZiH8mOMJmGx7Djwu2bOnaV21EUY5Y4E4zS6YM/edit?gid=0#gid=0
How we will use the information
This instrumentation will allow us to measure:
- Whether users can successfully navigate each step of the flow by tracking progression from one step to the next (form -> flow selection -> category -> support page -> outbound / link clicked)
- Where users drop off
- Whether users proceed from support pages to reporting destinations
- Which categories perform well or poorly
- A category that performs well: low drop-off rate after selection, high % click a support link
- A category that performs poorly: users frequently abandon after selecting it, users reconsider their choice (backtrack)
- Whether users understand the categories
- e.g. Users rarely use "Other", low drop-off/backtracking after selection
- Whether users get stuck or abandon the flow
- How emergency vs non-emergency flows perform
- Whether users complete the reporting flow (up to link clicked)
This will help validate whether the IRS flow works end-to-end and support rollout decisions (beyond 5%).