Page MenuHomePhabricator
Create Task
Maniphest T98035

[Task] Drop support for php-serialized output from Special:EntityData
Closed, ResolvedPublic
Actions

Description

Context:
Special:EntityData is intended to be a LinkedData interface.
It should support various RDF flavours, and plain JSON.

Problem:
Special:EntityData currently inherits some formats from the MediaWiki API result formats, specifically php-serialized. This will be dropped.

This would allow us to use the JSON serialization of the entity directly, and drop the clunky dependency on the API result formatters.

Acceptance criteria:

  • The “php” format of Special:EntityData is completely gone.
    • https://www.wikidata.org/wiki/Special:EntityData/Q42.php returns an error (almost certainly 415 Unsupported Media Type, like Q42.blah)
    • curl -I -H 'Accept: application/vnd.php.serialized' https://www.wikidata.org/wiki/Special:EntityData/Q42 returns an error (almost certainly 406 Not Acceptable, like curl -I -H 'Accept: application/vnd.microsoft.portable-executable' https://www.wikidata.org/wiki/Special:EntityData/Q42)
  • The code and tests have been cleaned up as described above (“drop the clunky dependency on the API result formatters”), or we determine that this no longer applies.

Notes:

Details

Other Assignee
Lucas_Werkmeister_WMDE
Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Lucas_Werkmeister_WMDE mentioned this in T166470: Include links in Wikidata HTTP responses to different entity representations as Link headers.Aug 18 2020, 3:48 PM
Ladsgroup added a subscriber: Lydia_Pintscher.Aug 18 2020, 3:57 PM
In T98035#6092192, @Addshore wrote:

This is a product decision, so putting in the product column.
From the tech side removing this will leave us with less things to maintain, I'm not really sure if we "support" the php format here really.
I would guess that most of the PHP calls are perhaps scrapers and things requesting this format by accident.
Migration path would be to use JSON instead.

I can look it up in hadoop if that helps PM decision (@Lydia_Pintscher) on this.

Ladsgroup added a comment.Aug 18 2020, 5:53 PM

Yesterday we had 3400 hits on php endpoints, 2089 were spiders and 1300 were from users (at least they faked user UA which is possible and happens quite often). 1000 of the hits belong to only four countries (which are not usual suspects) but I can't disclose more in a public ticket.

Addshore added a project: [DEPRECATED] wdwb-tech.Sep 21 2021, 8:40 AM
Addshore moved this task from Inbox to Product Realm on the [DEPRECATED] wdwb-tech board.Sep 21 2021, 8:45 AM

I'll raise this with Lydia in my next 1:1 with her

WMDE-leszek subscribed.Sep 21 2021, 12:45 PM
Addshore added a comment.Sep 21 2021, 12:46 PM
In T98035#6394020, @Ladsgroup wrote:

Yesterday we had 3400 hits on php endpoints, 2089 were spiders and 1300 were from users (at least they faked user UA which is possible and happens quite often). 1000 of the hits belong to only four countries (which are not usual suspects) but I can't disclose more in a public ticket.

@Ladsgroup can we get up to date numbers and also put this in relation to the other formats we expose?

Then @Lydia_Pintscher can make a current and informed decision on the future of the output.

Addshore updated the task description. (Show Details)Sep 21 2021, 12:48 PM
Ladsgroup added a comment.Sep 22 2021, 7:51 PM

Reasons to drop:

  • It adds a non-negligible amount of code that needs maintaining
  • It adds a stable interface to our stable interfaces that we need to communicate and follow the procedure for each and every change.
  • It adds two urls for cache busting (as explained above)
  • It doesn't give much benefit, the views of it are small (I put numbers below), its only unserializable in php and not any other language (unless with gymnastics).
  • (Might not be a big deal): Serialization and deserialization are security sensitive, we might expose something we shouldn't or receive something which would lead to arbitrary code execution.
    • This is not true here AFAIK but avoiding seriliazation and deserilazation in language the server is running is highly encouraged to reduce attack vectors.
Ladsgroup added a comment.Sep 22 2021, 8:15 PM
TypeNumber of hits in September 21
json7,598,854
rdf89,861
ttl11,388,708
php2,116
Lydia_Pintscher added a comment.Sep 23 2021, 8:13 AM

Thank you!
Alright. Then let's do this.

@Ladsgroup can you say if the hits for the php-serialozed output are coming from one/very few individuals making a lot of requests or a lot of individuals making a few requests? Is there any discernable pattern in the requests or the tools they are made with? (I'm asking as this might change the communication a bit.)

Lucas_Werkmeister_WMDE added a comment.Sep 23 2021, 10:08 AM

It adds two urls for cache busting (as explained above)

No, we only cache a limited set of URLs and the RDF format is not included in those. (The earlier comment was from before the caching story was resolved, we changed plans at some point in there.)

(But to be clear, I also support getting rid of this.)

Ladsgroup added a comment.Sep 23 2021, 3:25 PM
In T98035#7373635, @Lydia_Pintscher wrote:

@Ladsgroup can you say if the hits for the php-serialozed output are coming from one/very few individuals making a lot of requests or a lot of individuals making a few requests? Is there any discernable pattern in the requests or the tools they are made with? (I'm asking as this might change the communication a bit.)

It has four major consumers, Three seems to be bots and one is either a bot with fake UA or a gadget. There are maybe in total thirty usages outside these four but negligible.

Lydia_Pintscher added a comment.Sep 23 2021, 6:02 PM

Thanks! :)

Lydia_Pintscher updated the task description. (Show Details)Sep 23 2021, 6:03 PM
Lydia_Pintscher added a subscriber: Mohammed_Sadat_WMDE.
• Manuel moved this task from Needs Wikidata PM Work to Miscellaneous on the Wikidata-Campsite board.Jul 12 2022, 11:57 AM
• Manuel moved this task from Miscellaneous to Tech realm on the Wikidata-Campsite board.Aug 2 2022, 11:57 AM
• Manuel removed a project: Wikidata-Campsite.Nov 8 2022, 11:11 AM
Addshore unsubscribed.Jun 27 2023, 12:42 PM
Lucas_Werkmeister_WMDE added subscribers: Ifrahkhanyaree_WMDE, Arian_Bozorg.May 1 2026, 10:09 PM

This task has gained some new urgency in the context of T118538: Reduce the usage of API format=php. Given T98035#7373635 and T98035#7375023, I assume we still have a valid product decision to go ahead with this removal.

I do have one question: Is this a breaking (or significant) change according to our Stable Interface Policy? The Linked Data Interface is a listed under Stable Public APIs, but Stable Data Formats only includes RDF and JSON.

(If we decide this isn’t a breaking change, IMHO it would be nice to clarify this in the policy. Example: “The Linked Data Interface accessible at …, when used with one of the Stable Data Formats, is considered a stable interface.”)

CC @Arian_Bozorg and @Ifrahkhanyaree_WMDE, I’m not sure if this falls in the scope of Omega or Reuse. Cain we just go ahead with this removal, or do we want to announce it first?

Lucas_Werkmeister_WMDE added a parent task: T118538: Reduce the usage of API format=php.May 1 2026, 10:09 PM
Novem_Linguae subscribed.May 2 2026, 12:37 AM
Arian_Bozorg added a comment.EditedMay 2 2026, 9:53 AM

I 'm not sure which team it falls under, but I think although this is only used by a handful of people and is depricated we should announce it as a significant change

thiemowmde added a comment.May 4 2026, 5:54 AM

Is this a breaking (or significant) change […]

I would argue that the PHP serialized format never was and never can be stable and thus can never fall under https://www.wikidata.org/wiki/Wikidata:Stable_Interface_Policy. It's still nice to inform people ahead of time when we remove an old feature like this. But that happened, didn't it?

Rosalie_WMDE added a project: Wikidata-Omega.May 5 2026, 6:59 PM
karapayneWMDE moved this task from Incoming to Triaging on the Wikidata-Omega board.May 8 2026, 9:53 AM
karapayneWMDE edited projects, added: Wikidata-Omega (Triaging); removed: Wikidata-Omega.
karapayneWMDE moved this task from Backlog to Misc. on the Wikidata-Omega (Triaging) board.
Lucas_Werkmeister_WMDE updated the task description. (Show Details)May 13 2026, 2:24 PM
ARamirez_WMF subscribed.Fri, May 15, 2:54 PM

@Lucas_Werkmeister_WMDE confirmed that this will get moved into Sprint planning on Tuesday.

Arian_Bozorg added a comment.Tue, May 19, 8:52 AM

The significant change announcement will be made on 22 May for a config change on 28 May

Arian_Bozorg updated the task description. (Show Details)Tue, May 19, 8:53 AM
Arian_Bozorg moved this task from Triaging to The Backlog on the Wikidata-Omega board.
Arian_Bozorg edited projects, added: Wikidata-Omega; removed: Wikidata-Omega (Triaging).
ArthurTaylor claimed this task.Tue, May 19, 12:03 PM
ArthurTaylor moved this task from The Backlog to The Board on the Wikidata-Omega board.
ArthurTaylor edited projects, added: Wikidata-Omega (The Board); removed: Wikidata-Omega.
ArthurTaylor moved this task from new to the board to In Development on the Wikidata-Omega (The Board) board.
gerritbot added a comment.Tue, May 19, 12:29 PM

Change #1289331 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/Wikibase@master] [WIP] Remove support for PHP EntityData format

https://gerrit.wikimedia.org/r/1289331

gerritbot added a project: Patch-For-Review.Tue, May 19, 12:29 PM
Lucas_Werkmeister_WMDE added a comment.Tue, May 19, 1:31 PM

(FTR, my assumption is that we want a (very temporary) config flag for this, so we can turn this off at a predictable point in time next Thursday rather than whenever the train rolls forward.)

ArthurTaylor added a comment.Tue, May 19, 2:28 PM

@Lucas_Werkmeister_WMDE We already have the wmgWikibaseEntityDataFormats config option. Removing 'php' from that list has the desired effect (at least locally) of making the two requests in the acceptance criteria fail with 415 and 406 as required. We can simply make the config change a suitable amount of time after the interface change is announced, and merge the deprecation of the feature in a subsequent version / train.

Lucas_Werkmeister_WMDE added a comment.Tue, May 19, 3:12 PM

Alright, sounds good to me. Thanks for investigating!

gerritbot added a comment.Wed, May 20, 6:00 AM

Change #1289736 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[operations/mediawiki-config@master] Disable support for PHP-serialized EntityData on Wikidata

https://gerrit.wikimedia.org/r/1289736

gerritbot added a comment.Wed, May 20, 7:13 AM

Change #1289751 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/Wikibase@master] [WIP] Remove ApiFormatter dependency

https://gerrit.wikimedia.org/r/1289751

Lucas_Werkmeister_WMDE added a comment.Wed, May 20, 9:22 AM

Something I didn’t think about during task time but realized while reviewing the config change: we should deploy this to Test Wikidata and Beta Wikidata before the announcement, right? Then the announcement can say that people can already test the behavior there, as usual.

gerritbot added a comment.Wed, May 20, 9:24 AM

Change #1289898 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[operations/mediawiki-config@master] Disable support for PHP-serialized EntityData on Wikidata

https://gerrit.wikimedia.org/r/1289898

ArthurTaylor added a comment.Wed, May 20, 9:36 AM

@Lucas_Werkmeister_WMDE sure - patch is now attached

gerritbot added a comment.Wed, May 20, 1:18 PM

Change #1289736 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable support for PHP-serialized EntityData on Beta / Test Wikidata

https://gerrit.wikimedia.org/r/1289736

Stashbot added a comment.Wed, May 20, 1:18 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-20T13:18:47Z] <arthurtaylor@deploy1003> Started scap sync-world: Backport for [[gerrit:1289736|Disable support for PHP-serialized EntityData on Beta / Test Wikidata (T98035)]]

Stashbot added a comment.Wed, May 20, 1:20 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-20T13:20:43Z] <arthurtaylor@deploy1003> arthurtaylor: Backport for [[gerrit:1289736|Disable support for PHP-serialized EntityData on Beta / Test Wikidata (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Wed, May 20, 1:26 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-20T13:26:14Z] <arthurtaylor@deploy1003> Finished scap sync-world: Backport for [[gerrit:1289736|Disable support for PHP-serialized EntityData on Beta / Test Wikidata (T98035)]] (duration: 07m 26s)

gerritbot added a comment.Thu, May 21, 9:36 AM

Change #1290698 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/Wikibase@master] [WIP] Serialization fixups v2

https://gerrit.wikimedia.org/r/1290698

gerritbot added a comment.Thu, May 21, 11:15 AM

Change #1290698 abandoned by Arthur taylor:

[mediawiki/extensions/Wikibase@master] [WIP] Serialization fixups v2

Reason:

merged into parent change

https://gerrit.wikimedia.org/r/1290698

ArthurTaylor removed ArthurTaylor as the assignee of this task.Fri, May 22, 7:39 AM
ArthurTaylor moved this task from In Development to Ready for Peer Review on the Wikidata-Omega (The Board) board.
ArthurTaylor subscribed.
karapayneWMDE updated Other Assignee, added: ArthurTaylor.Tue, May 26, 8:10 AM
Lucas_Werkmeister_WMDE claimed this task.Tue, May 26, 2:59 PM
Lucas_Werkmeister_WMDE moved this task from Ready for Peer Review to In Peer Review on the Wikidata-Omega (The Board) board.
gerritbot added a comment.Thu, May 28, 7:03 AM

Change #1289898 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable support for PHP-serialized EntityData on Wikidata production

https://gerrit.wikimedia.org/r/1289898

Stashbot added a comment.Thu, May 28, 7:04 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-28T07:04:03Z] <wmde-fisch@deploy1003> Started scap sync-world: Backport for [[gerrit:1289898|Disable support for PHP-serialized EntityData on Wikidata production (T98035)]]

Stashbot added a comment.Thu, May 28, 7:06 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-28T07:06:08Z] <wmde-fisch@deploy1003> wmde-fisch, arthurtaylor: Backport for [[gerrit:1289898|Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Thu, May 28, 7:11 AM

Mentioned in SAL (#wikimedia-operations) [2026-05-28T07:11:18Z] <wmde-fisch@deploy1003> Finished scap sync-world: Backport for [[gerrit:1289898|Disable support for PHP-serialized EntityData on Wikidata production (T98035)]] (duration: 07m 15s)

gerritbot added a comment.Thu, May 28, 7:14 AM

Change #1289331 merged by Arthur taylor:

[mediawiki/extensions/Wikibase@master] Remove support for PHP EntityData format

https://gerrit.wikimedia.org/r/1289331

gerritbot added a comment.Thu, May 28, 11:45 AM

Change #1294961 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/Wikibase@master] Tests for JSON EntityData formatting

https://gerrit.wikimedia.org/r/1294961

gerritbot added a comment.Thu, May 28, 11:46 AM

Change #1294962 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/Wikibase@master] Refactor AddPageInfo to make item order stable

https://gerrit.wikimedia.org/r/1294962

gerritbot added a comment.Fri, May 29, 11:29 AM

Change #1295438 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/WikibaseLexeme@master] Introduce test of JSON serialization

https://gerrit.wikimedia.org/r/1295438

gerritbot added a comment.Fri, May 29, 11:54 AM

Change #1295444 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/WikibaseLexeme@master] Remove test temporarily for change of EntityDataSerializationService

https://gerrit.wikimedia.org/r/1295444

gerritbot added a comment.Fri, May 29, 11:54 AM

Change #1295445 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/WikibaseLexeme@master] Restore test of EntityDataSerializationService after refactor

https://gerrit.wikimedia.org/r/1295445

gerritbot added a comment.Fri, May 29, 3:29 PM

Change #1294961 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Tests for JSON EntityData formatting

https://gerrit.wikimedia.org/r/1294961

ArthurTaylor removed Lucas_Werkmeister_WMDE as the assignee of this task.Mon, Jun 1, 10:13 AM
ArthurTaylor moved this task from In Peer Review to Ready for Development on the Wikidata-Omega (The Board) board.
ArthurTaylor updated Other Assignee, removed: ArthurTaylor.
Lucas_Werkmeister_WMDE claimed this task.Wed, Jun 3, 2:24 PM
Lucas_Werkmeister_WMDE moved this task from Ready for Development to In Development on the Wikidata-Omega (The Board) board.
gerritbot added a comment.Wed, Jun 3, 2:34 PM

Change #1295438 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] Introduce test of JSON serialization

https://gerrit.wikimedia.org/r/1295438

Lucas_Werkmeister_WMDE removed Lucas_Werkmeister_WMDE as the assignee of this task.Wed, Jun 3, 2:58 PM
Lucas_Werkmeister_WMDE moved this task from In Development to Ready for Peer Review on the Wikidata-Omega (The Board) board.
Lucas_Werkmeister_WMDE updated Other Assignee, added: Lucas_Werkmeister_WMDE.
gerritbot added a comment.Thu, Jun 4, 12:22 PM

Change #1295444 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] Disable test temporarily for change of EntityDataSerializationService

https://gerrit.wikimedia.org/r/1295444

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.6; 2026-06-09).Thu, Jun 4, 1:00 PM
gerritbot added a comment.Thu, Jun 4, 3:16 PM

Change #1289751 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Remove ApiFormatter dependency

https://gerrit.wikimedia.org/r/1289751

gerritbot added a comment.Thu, Jun 4, 3:16 PM

Change #1295445 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] Restore test of EntityDataSerializationService after refactor

https://gerrit.wikimedia.org/r/1295445

gerritbot added a comment.Thu, Jun 4, 3:57 PM

Change #1294962 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Wikibase@master] Refactor AddPageInfo to make item order stable

https://gerrit.wikimedia.org/r/1294962

Lucas_Werkmeister_WMDE moved this task from Ready for Peer Review to Ready for Tech Verification on the Wikidata-Omega (The Board) board.Thu, Jun 4, 3:57 PM
Maintenance_bot removed a project: Patch-For-Review.Thu, Jun 4, 4:31 PM
mahmoud.abdelsattar.wmde subscribed.Fri, Jun 5, 6:59 AM

Looks like it is done and no more effort needed here.
Thank you!

mahmoud.abdelsattar.wmde closed this task as Resolved.Fri, Jun 5, 6:59 AM
mahmoud.abdelsattar.wmde claimed this task.
mahmoud.abdelsattar.wmde moved this task from Ready for Tech Verification to Done - Moving a task here will move it to the completed tasks workboard, resolve the ticket, and assign it to whomever made the move on the Wikidata-Omega (The Board) board.
mahmoud.abdelsattar.wmde edited projects, added: Wikidata-Omega (Completed Tasks); removed: Wikidata-Omega (The Board).
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits