Add the final "Qid to BabyGlobe state" mapping
Open, HighPublic
Actions

Description

Context:
Baby Globe is configured to show different animations on different pages through the use of WikiData Qids. We have a list of Qids matching different wiki pages, mapped to BG state names. Compiling all the proposed mappings - there are 667170 Qids. That mapping was planned to be stored in a json file.

Problem:
As a json file this is ~15M, which IIUC does not fit into WANObjectCache

Current implementation:

Storing of the json file in WANObjectCache https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WP25EasterEggs/+/refs/heads/master/src/PageCompanionService.php#102
Look up of Qids in the mapping https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WP25EasterEggs/+/refs/heads/master/src/PageCompanionConfigResolver.php#165

Proposed solutions:
s1. Shard the json file and create a WANObjectCache for each shard. This would get a bit messy, as we would need to have at least 15 json files with sorted Qids
s2. Remove entries from the configuration mapping to the json file under 1M. Probably the last resort solution
s3. Use Cdb https://www.mediawiki.org/wiki/CDB, compile the binary file locally, commit it in the extension repo, utilize \Cdb\Reader in extension to check the Qids mapping.

Further questions:
q1. Is using Cdb in extension permitted?
q2. Would using Cdb in extension require some tedious setup, or is it "plug and play"?
q3. Will there be security review complications because it is a binary file?
q4. Are the other better suited solutions than the ones mentioned here?

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Remove WANObjectCache from PageCompanionService, add soft-launch Qids	mediawiki/extensions/WP25EasterEggs	master	+47 -81

Customize query in gerrit

Event Timeline

ATitkov created this task.Wed, Jan 28, 10:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Jan 28, 10:46 PM

ATitkov updated the task description. (Show Details)Wed, Jan 28, 10:48 PM

ATitkov updated the task description. (Show Details)Wed, Jan 28, 10:55 PM

ATitkov added subscribers: Jdlrobson, Catrope, aude and 3 others.

Size issues aside, I do not think WANObjectCache caching is helpful for a json config file, as memcached involves a network request vs loading a static file cached by the file system. (though it needs to be loaded for each request)

I think Cdb would be okay for this.

There could be a maintenance script that generates the cdb file from the json, and we could run the script locally and commit the file to the extension. If easter eggs was a longer-term feature, then maybe this could be made part of the CI or deployment process.

Change #1236375 had a related patch set uploaded (by ATitkov; author: ATitkov):

[mediawiki/extensions/WP25EasterEggs@master] Refactor PageCompanionService to use CDB for companion configs

https://gerrit.wikimedia.org/r/1236375

gerritbot added a project: Patch-For-Review.Tue, Feb 3, 10:15 PM

We decided to use a json file containing a limited dataset instead of CDB. Also we decided to remove WANObjectCache

Jdrewniak added a project: Reader Experience Team.Fri, Feb 6, 9:11 PM

Jdrewniak moved this task from Incoming to Needs refinement on the Reader Experience Team board.

Change #1236375 merged by jenkins-bot:

[mediawiki/extensions/WP25EasterEggs@master] Remove WANObjectCache from PageCompanionService, add soft-launch Qids

https://gerrit.wikimedia.org/r/1236375

Maintenance_bot removed a project: Patch-For-Review.Mon, Feb 9, 9:32 PM

CDekock-WMF subscribed.Tue, Feb 10, 8:40 AM