Page MenuHomePhabricator
Create Task
Maniphest T417067

[Share Highlights] Support CJK text better for fragment for zh pilot
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

The initial implementation of long text fragmentation in textFragment.js tries to take the first 5 words and the last 5 words and remove the middle; this falls down in a couple places:

  • if there are in fact less than 10 words, you get duplicates
  • languages like Chinese and Japanese that don't use spacing between words will treat long strings as individual words, making that situation more likely

For instance sharing a link to an entire paragraph of Chinese text from the zh "Paris" article gave me a URL over 5000 characters long (tweaked to point at zhwiki instead of my localhost):

https://zh.wikipedia.org/wiki/%E5%B7%B4%E9%BB%8E#:~:text=%E5%85%AC%E5%85%83508%E5%B9%B4%EF%BC%8C%E6%B3%95%E8%98%AD%E5%85%8B%E4%BA%BA%E5%9C%8B%E7%8E%8B%E5%85%8B%E6%B4%9B%E7%B6%AD%E4%B8%80%E4%B8%96%E5%B0%87%E5%B7%B4%E9%BB%8E%E5%AE%9A%E7%82%BA%E5%A2%A8%E6%B4%9B%E6%BA%AB%E7%8E%8B%E6%9C%9D%E7%9A%84%E9%A6%96%E9%83%BD%EF%BC%8C%E6%B3%95%E8%98%AD%E5%85%8B%E4%BA%BA%E7%94%A8%E6%9C%A8%E6%9D%BF%E5%9C%A8%E9%80%99%E8%A3%A1%E5%BC%80%E5%A7%8B%E5%BB%BA%E9%80%A0%E6%95%99%E5%A0%82%E5%92%8C%E5%AE%AE%E6%AE%BF%E3%80%82%E4%BD%86%E6%98%AF%E6%AD%A4%E6%99%82%E7%9A%84%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E5%8F%AA%E6%98%AF%E4%B8%80%E5%80%8B%E9%83%A8%E8%90%BD%E7%9A%84%E8%81%9A%E5%90%88%E9%AB%94%EF%BC%8C%E6%94%BF%E6%B2%BB%E4%BD%93%E7%B3%BB%E4%BB%8D%E4%B8%8D%E7%A8%B3%E5%AE%9A%E3%80%82%E5%85%8B%E6%B4%9B%E7%B6%AD%E4%B8%80%E4%B8%96%E6%AD%BB%E5%BE%8C%EF%BC%8C%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E8%A2%AB%E5%85%B6%E5%90%8E%E4%BB%A3%E7%93%9C%E5%88%86%EF%BC%8C%E5%B7%B4%E9%BB%8E%E5%9B%A0%E6%AD%A4%E5%86%8D%E5%BA%A6%E6%B7%AA%E7%82%BA%E5%9C%B0%E6%96%B9%E6%80%A7%E5%9F%8E%E5%B8%82%E3%80%82%E5%8A%A0%E6%B4%9B%E6%9E%97%E7%8E%8B%E6%9C%9D%E6%99%82%E6%9C%9F%EF%BC%8C%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E7%9A%84%E9%A6%96%E9%83%BD%E4%B8%80%E7%9B%B4%E5%9C%A8%E4%BA%9E%E7%90%9B%E7%AD%89%E5%9F%8E%E5%B8%82%E4%B9%8B%E9%96%93%E5%8F%98%E6%8D%A2%EF%BC%8C%E5%B7%B4%E9%BB%8E%E7%95%B6%E6%99%82%E7%94%B1%E3%80%8C%E5%BC%B7%E8%80%85%E3%80%8D%E7%BE%85%E8%B2%9D%E7%88%BE%E4%BE%86%E7%B5%B1%E6%B2%BB%E3%80%82%E5%85%AC%E5%85%839%E4%B8%96%E7%B4%80%EF%BC%8C%E7%B6%AD%E4%BA%AC%E4%BA%BA%E5%85%A5%E4%BE%B5%E6%B3%95%E5%9C%8B%EF%BC%8C%E4%B8%A6%E6%96%BC%E5%85%AC%E5%85%83845%E5%B9%B4%E9%80%B2%E6%94%BB%E5%B7%B4%E9%BB%8E%EF%BC%8C%E6%96%BC%E6%98%AF%E5%B7%B4%E9%BB%8E%E4%BA%BA%E5%9C%A8%E5%9F%8E%E5%B8%82%E5%91%A8%E5%9C%8D%E5%BB%BA%E8%B5%B7%E4%BA%86%E5%9F%8E%E7%89%86%E6%8A%B5%E7%A6%A6%E7%B6%AD%E4%BA%AC%E4%BA%BA%E4%BE%B5%E7%95%A5%E3%80%82%E5%9B%A0%E7%82%BA%E5%8A%A0%E6%B4%9B%E6%9E%97%E7%8E%8B%E6%9C%9D%E7%9A%84%E6%9C%80%E5%BE%8C%E4%B8%80%E5%80%8B%E5%9C%8B%E7%8E%8B%E8%83%96%E5%AD%90%E6%9F%A5%E7%90%86%E8%BB%9F%E5%BC%B1%E7%84%A1%E8%83%BD%EF%BC%8C%E6%96%BC%E6%98%AF%E5%9C%A8%E6%8A%B5%E6%8A%97%E7%B6%AD%E4%BA%AC%E4%BA%BA%E9%80%B2%E6%94%BB%E4%B8%AD%E4%BA%AB%E6%9C%89%E7%9B%9B%E8%AD%BD%E7%9A%84%E5%B7%B4%E9%BB%8E%E4%BC%AF%E7%88%B5%E3%80%81%E7%BE%85%E8%B2%9D%E7%88%BE%E4%B9%8B%E5%AD%90%E5%8E%84%E5%BE%B7%EF%BC%8C%E5%9C%A8%E5%85%AC%E5%85%83888%E5%B9%B4%E7%94%B1%E5%A4%A7%E9%A0%98%E4%B8%BB%E5%80%91%E6%8E%A8%E9%81%B8%E7%82%BA%E8%A5%BF%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E7%9A%84%E5%9C%8B%E7%8E%8B%E3%80%82%E5%A5%A7%E5%A4%9A%E7%9A%84%E6%9B%BE%E5%AD%AB%E9%9B%A8%E6%9E%9C%C2%B7%E5%8D%A1%E4%BD%A9%E6%96%BC987%E5%B9%B4%E5%8A%A0%E5%86%95%E7%82%BA%E6%B3%95%E8%98%AD%E8%A5%BF%E5%9C%8B%E7%8E%8B%EF%BC%8C%E5%88%9B%E7%AB%8B%E4%BA%86%E5%8D%A1%E4%BD%A9%E7%8E%8B%E6%9C%9D%EF%BC%8C%E5%90%8C%E6%99%82%E5%B7%B4%E9%BB%8E%E4%B9%9F%E9%A6%96%E6%AC%A1%E6%88%90%E7%82%BA%E8%A5%BF%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E7%9A%84%E9%A6%96%E9%83%BD%5B36%5D%E3%80%82,%E5%85%AC%E5%85%83508%E5%B9%B4%EF%BC%8C%E6%B3%95%E8%98%AD%E5%85%8B%E4%BA%BA%E5%9C%8B%E7%8E%8B%E5%85%8B%E6%B4%9B%E7%B6%AD%E4%B8%80%E4%B8%96%E5%B0%87%E5%B7%B4%E9%BB%8E%E5%AE%9A%E7%82%BA%E5%A2%A8%E6%B4%9B%E6%BA%AB%E7%8E%8B%E6%9C%9D%E7%9A%84%E9%A6%96%E9%83%BD%EF%BC%8C%E6%B3%95%E8%98%AD%E5%85%8B%E4%BA%BA%E7%94%A8%E6%9C%A8%E6%9D%BF%E5%9C%A8%E9%80%99%E8%A3%A1%E5%BC%80%E5%A7%8B%E5%BB%BA%E9%80%A0%E6%95%99%E5%A0%82%E5%92%8C%E5%AE%AE%E6%AE%BF%E3%80%82%E4%BD%86%E6%98%AF%E6%AD%A4%E6%99%82%E7%9A%84%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E5%8F%AA%E6%98%AF%E4%B8%80%E5%80%8B%E9%83%A8%E8%90%BD%E7%9A%84%E8%81%9A%E5%90%88%E9%AB%94%EF%BC%8C%E6%94%BF%E6%B2%BB%E4%BD%93%E7%B3%BB%E4%BB%8D%E4%B8%8D%E7%A8%B3%E5%AE%9A%E3%80%82%E5%85%8B%E6%B4%9B%E7%B6%AD%E4%B8%80%E4%B8%96%E6%AD%BB%E5%BE%8C%EF%BC%8C%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E8%A2%AB%E5%85%B6%E5%90%8E%E4%BB%A3%E7%93%9C%E5%88%86%EF%BC%8C%E5%B7%B4%E9%BB%8E%E5%9B%A0%E6%AD%A4%E5%86%8D%E5%BA%A6%E6%B7%AA%E7%82%BA%E5%9C%B0%E6%96%B9%E6%80%A7%E5%9F%8E%E5%B8%82%E3%80%82%E5%8A%A0%E6%B4%9B%E6%9E%97%E7%8E%8B%E6%9C%9D%E6%99%82%E6%9C%9F%EF%BC%8C%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E7%9A%84%E9%A6%96%E9%83%BD%E4%B8%80%E7%9B%B4%E5%9C%A8%E4%BA%9E%E7%90%9B%E7%AD%89%E5%9F%8E%E5%B8%82%E4%B9%8B%E9%96%93%E5%8F%98%E6%8D%A2%EF%BC%8C%E5%B7%B4%E9%BB%8E%E7%95%B6%E6%99%82%E7%94%B1%E3%80%8C%E5%BC%B7%E8%80%85%E3%80%8D%E7%BE%85%E8%B2%9D%E7%88%BE%E4%BE%86%E7%B5%B1%E6%B2%BB%E3%80%82%E5%85%AC%E5%85%839%E4%B8%96%E7%B4%80%EF%BC%8C%E7%B6%AD%E4%BA%AC%E4%BA%BA%E5%85%A5%E4%BE%B5%E6%B3%95%E5%9C%8B%EF%BC%8C%E4%B8%A6%E6%96%BC%E5%85%AC%E5%85%83845%E5%B9%B4%E9%80%B2%E6%94%BB%E5%B7%B4%E9%BB%8E%EF%BC%8C%E6%96%BC%E6%98%AF%E5%B7%B4%E9%BB%8E%E4%BA%BA%E5%9C%A8%E5%9F%8E%E5%B8%82%E5%91%A8%E5%9C%8D%E5%BB%BA%E8%B5%B7%E4%BA%86%E5%9F%8E%E7%89%86%E6%8A%B5%E7%A6%A6%E7%B6%AD%E4%BA%AC%E4%BA%BA%E4%BE%B5%E7%95%A5%E3%80%82%E5%9B%A0%E7%82%BA%E5%8A%A0%E6%B4%9B%E6%9E%97%E7%8E%8B%E6%9C%9D%E7%9A%84%E6%9C%80%E5%BE%8C%E4%B8%80%E5%80%8B%E5%9C%8B%E7%8E%8B%E8%83%96%E5%AD%90%E6%9F%A5%E7%90%86%E8%BB%9F%E5%BC%B1%E7%84%A1%E8%83%BD%EF%BC%8C%E6%96%BC%E6%98%AF%E5%9C%A8%E6%8A%B5%E6%8A%97%E7%B6%AD%E4%BA%AC%E4%BA%BA%E9%80%B2%E6%94%BB%E4%B8%AD%E4%BA%AB%E6%9C%89%E7%9B%9B%E8%AD%BD%E7%9A%84%E5%B7%B4%E9%BB%8E%E4%BC%AF%E7%88%B5%E3%80%81%E7%BE%85%E8%B2%9D%E7%88%BE%E4%B9%8B%E5%AD%90%E5%8E%84%E5%BE%B7%EF%BC%8C%E5%9C%A8%E5%85%AC%E5%85%83888%E5%B9%B4%E7%94%B1%E5%A4%A7%E9%A0%98%E4%B8%BB%E5%80%91%E6%8E%A8%E9%81%B8%E7%82%BA%E8%A5%BF%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E7%9A%84%E5%9C%8B%E7%8E%8B%E3%80%82%E5%A5%A7%E5%A4%9A%E7%9A%84%E6%9B%BE%E5%AD%AB%E9%9B%A8%E6%9E%9C%C2%B7%E5%8D%A1%E4%BD%A9%E6%96%BC987%E5%B9%B4%E5%8A%A0%E5%86%95%E7%82%BA%E6%B3%95%E8%98%AD%E8%A5%BF%E5%9C%8B%E7%8E%8B%EF%BC%8C%E5%88%9B%E7%AB%8B%E4%BA%86%E5%8D%A1%E4%BD%A9%E7%8E%8B%E6%9C%9D%EF%BC%8C%E5%90%8C%E6%99%82%E5%B7%B4%E9%BB%8E%E4%B9%9F%E9%A6%96%E6%AC%A1%E6%88%90%E7%82%BA%E8%A5%BF%E6%B3%95%E8%98%AD%E5%85%8B%E7%8E%8B%E5%9C%8B%E7%9A%84%E9%A6%96%E9%83%BD%5B36%5D%E3%80%82

Consider treating each Han character as an entire word (which may help with Chinese and Japanese specifically), or cropping the total length of each side of the pair.

Acceptance criteria:

  • Never repeat words on both sides of the "long string" start/end pair in a Chinese string
  • If cropping within long strings, do not break Unicode surrogate pairs

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
[ShareHighlight] Text fragment generation tests and fixesmediawiki/extensions/ReaderExperimentsmaster+152 -8
Customize query in gerrit

Related Objects
Search...

Event Timeline

bvibber created this task.Feb 10 2026, 8:08 PM
Restricted Application added subscribers: Dragoniez, Stang, Aklapper. · View Herald TranscriptFeb 10 2026, 8:08 PM
bvibber renamed this task from [Share Highlights] Support CJK text better for ellipsis for zh pilot to [Share Highlights] Support CJK text better for fragment for zh pilot.Feb 10 2026, 8:08 PM
bvibber added a parent task: T416432: [Epic] Share Card experiment.
egardner mentioned this in T416434: [Share Highlights] Set up codebase for Share Highlights.Feb 10 2026, 9:18 PM
Stang unsubscribed.Feb 10 2026, 11:43 PM
Shizhao added a project: Chinese-Sites.Feb 11 2026, 2:59 AM
egardner mentioned this in T417105: [Share Highlights] Text fragment URL generation.Feb 11 2026, 4:42 AM
egardner moved this task from Incoming/Inbox to Needs Refinement on the Reader Growth Team board.Feb 11 2026, 4:49 AM
SherryYang-WMF triaged this task as Low priority.Feb 11 2026, 5:30 PM
SherryYang-WMF raised the priority of this task from Low to Medium.
Shizhao moved this task from Backlog to Research on the Chinese-Sites board.Feb 24 2026, 4:01 AM
egardner mentioned this in T416432: [Epic] Share Card experiment.Mar 3 2026, 7:53 PM
egardner moved this task from Needs Refinement to Backlog on the Reader Growth Team board.Mar 3 2026, 10:01 PM
SherryYang-WMF moved this task from Backlog to Needs Refinement on the Reader Growth Team board.Mar 10 2026, 10:42 PM
HSwan-WMF set the point value for this task to 3.Mar 12 2026, 4:36 PM
SherryYang-WMF moved this task from Needs Refinement to Ready on the Reader Growth Team board.Mar 17 2026, 12:00 AM
gerritbot added a comment.Mar 24 2026, 7:44 PM

Change #1259248 had a related patch set uploaded (by Bvibber; author: Bvibber):

[mediawiki/extensions/ReaderExperiments@master] [ShareHighlight] Text fragment generation tests and fixes

https://gerrit.wikimedia.org/r/1259248

gerritbot added a project: Patch-For-Review.Mar 24 2026, 7:44 PM
gerritbot added a comment.Mar 25 2026, 7:44 PM

Change #1259248 merged by jenkins-bot:

[mediawiki/extensions/ReaderExperiments@master] [ShareHighlight] Text fragment generation tests and fixes

https://gerrit.wikimedia.org/r/1259248

mfossati edited projects, added: Reader Growth Team (Sprint 5 (Mar 18 - Mar 31) Q3 25/26); removed: Reader Growth Team.Mar 25 2026, 7:46 PM
mfossati moved this task from Committed to QA on the Reader Growth Team (Sprint 5 (Mar 18 - Mar 31) Q3 25/26) board.
ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.22; 2026-03-31).Mar 25 2026, 8:00 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 25 2026, 8:31 PM
Etonkovidova moved this task from QA to Signoff on the Reader Growth Team (Sprint 5 (Mar 18 - Mar 31) Q3 25/26) board.Mar 27 2026, 6:55 PM
Etonkovidova subscribed.

Checked on enwiki beta on 第85屆奧斯卡金像獎 (the article was imported from zhwiki).

The resulting url from the shared highlighted text (see the screenshot below) - https://en.wikipedia.beta.wmcloud.org/wiki/%E7%AC%AC85%E5%B1%86%E5%A5%A7%E6%96%AF%E5%8D%A1%E9%87%91%E5%83%8F%E7%8D%8E#:~:text=2012%E5%B9%B412%E6%9C%881%E6%97%A5%EF%BC%8C%E5%AD%A6%E9%99%A2%E5%9C%A8%E5%A5%BD%E8%8E%B1%E5%9D%9E,%E9%A2%81%E5%8F%91%E4%BA%86%E5%A5%A5%E6%96%AF%E5%8D%A1%E7%A7%91%E6%8A%80%E6%88%90%E6%9E%9C%E5%A5%96%5B7%5D%E3%80%82

Screenshot 2026-03-27 at 2.36.17 PM.png (2,378×1,274 px, 563 KB)

Compare with the link for the same highlighted text on zhwiki wmf.21

https://zh.wikipedia.org/wiki/%E7%AC%AC85%E5%B1%86%E5%A5%A7%E6%96%AF%E5%8D%A1%E9%87%91%E5%83%8F%E7%8D%8E#:~:text=2012%E5%B9%B412%E6%9C%881%E6%97%A5%EF%BC%8C%E5%AD%A6%E9%99%A2%E5%9C%A8%E5%A5%BD%E8%8E%B1%E5%9D%9E%E9%AB%98%E5%9C%B0%E4%B8%AD%E5%BF%83%E7%9A%84%E5%AE%B4%E4%BC%9A%E5%A4%A7%E7%A4%BC%E5%A0%82%E4%B8%BE%E8%A1%8C%E4%BA%86%E7%AC%AC%E5%9B%9B%E5%B1%8A%E5%B9%B4%E5%BA%A6%E5%AD%A6%E9%99%A2%E4%B8%BB%E5%B8%AD%E5%A5%96%E9%A2%81%E5%A5%96%E6%99%9A%E4%BC%9A%5B8%5D%E3%80%822013%E5%B9%B42%E6%9C%889%E6%97%A5%EF%BC%8C%E5%85%8B%E9%87%8C%E6%96%AF%C2%B7%E6%BD%98%E6%81%A9%E5%92%8C%E4%BD%90%E4%BC%8A%C2%B7%E7%B4%A2%E5%B0%94%E8%BE%BE%E5%A8%9C%E4%B8%80%E8%B5%B7%E5%9C%A8%E6%AF%94%E4%BD%9B%E5%88%A9%E5%B1%B1%E7%9A%84%E6%AF%94%E4%BD%9B%E5%88%A9%E5%B1%B1%E9%85%92%E5%BA%97%E4%B8%BB%E6%8C%81%E9%A2%81%E5%8F%91%E4%BA%86%E5%A5%A5%E6%96%AF%E5%8D%A1%E7%A7%91%E6%8A%80%E6%88%90%E6%9E%9C%E5%A5%96%5B9%5D%E3%80%82,2012%E5%B9%B412%E6%9C%881%E6%97%A5%EF%BC%8C%E5%AD%A6%E9%99%A2%E5%9C%A8%E5%A5%BD%E8%8E%B1%E5%9D%9E%E9%AB%98%E5%9C%B0%E4%B8%AD%E5%BF%83%E7%9A%84%E5%AE%B4%E4%BC%9A%E5%A4%A7%E7%A4%BC%E5%A0%82%E4%B8%BE%E8%A1%8C%E4%BA%86%E7%AC%AC%E5%9B%9B%E5%B1%8A%E5%B9%B4%E5%BA%A6%E5%AD%A6%E9%99%A2%E4%B8%BB%E5%B8%AD%E5%A5%96%E9%A2%81%E5%A5%96%E6%99%9A%E4%BC%9A%5B8%5D%E3%80%822013%E5%B9%B42%E6%9C%889%E6%97%A5%EF%BC%8C%E5%85%8B%E9%87%8C%E6%96%AF%C2%B7%E6%BD%98%E6%81%A9%E5%92%8C%E4%BD%90%E4%BC%8A%C2%B7%E7%B4%A2%E5%B0%94%E8%BE%BE%E5%A8%9C%E4%B8%80%E8%B5%B7%E5%9C%A8%E6%AF%94%E4%BD%9B%E5%88%A9%E5%B1%B1%E7%9A%84%E6%AF%94%E4%BD%9B%E5%88%A9%E5%B1%B1%E9%85%92%E5%BA%97%E4%B8%BB%E6%8C%81%E9%A2%81%E5%8F%91%E4%BA%86%E5%A5%A5%E6%96%AF%E5%8D%A1%E7%A7%91%E6%8A%80%E6%88%90%E6%9E%9C%E5%A5%96%5B9%5D%E3%80%82
lwatson added a project: ReaderExperiments-ShareHighlight.Mar 31 2026, 6:19 PM
KSarabia-WMF closed this task as Resolved.Mar 31 2026, 8:29 PM
KSarabia-WMF claimed this task.
KSarabia-WMF subscribed.

LGTM

Stang moved this task from Research to Closed on the Chinese-Sites board.Mar 31 2026, 8:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits