Page MenuHomePhabricator

Sunset ShortUrl extension in favour of UrlShortener extension
Open, MediumPublic

Description

NOTE: This is a proposed outcome for T187045: Code Stewardship Review: ShortUrl Extension.

Before a wiki can switch from ShortUrl to UrlShortener, the ShortUrl extension first needs to gain a feature to keep the exsiting URLs working. There are also a number of user interface improvements and transitional optoins needed during the migration.

Before and after

From T107188#11181721:

ShortUrl UI (old)UrlShortener UI (new)
Screenshot 2025-09-15 at 16.37.30.png (1×1 px, 356 KB)
Screenshot 2025-09-15 at 16.38.28.png (1×2 px, 322 KB)

List of wikis from wmf-config:

30 wikis with ShortUrl installed
'wmgUseShortUrl' => [
	'default' => false,
	'bdwikimedia' => true, // T146014
	'bhwiki' => true, // T113348
	'bnwiki' => true, // T62956
	'bnwikisource' => true, // T127968
	'eswikibooks' => true, // T96668
	'gomwiki' => true, // T206741
	'hiwiki' => true,
	'hiwikiversity' => true, // T177187
	'knwiki' => true, // T97218
	'knwikisource' => true, // T189287
	'maiwiki' => true, // T125802
	'mrwiki' => true, // T103646
	'newiki' => true, // T92820
	'orwiki' => true,
	'orwikisource' => true, // T124429
	'orwiktionary' => true, // T103644
	'pawiki' => true, // T178919
	'sawiki' => true, // T94660
	'sawikibooks' => true, // T94660
	'sawikiquote' => true, // T94660
	'sawikisource' => true, // T94660
	'sawiktionary' => true, // T94660
	'tawiki' => true,
	'tawikibooks' => true,
	'tawikinews' => true,
	'tawikiquote' => true,
	'tawikisource' => true,
	'tawiktionary' => true,
	'tcywiki' => true, // T150166
	'urwiki' => true, // T138507
],

Discovery for new short URLs.

T288653: Make URLShortener service discoverable for users

Discover short URL for current article when reading

The "ShortUrl" extension promoted the URL under the page title. This was cheap to do as the URLs were deterministic. We may not be able to do this from a performance perspective (T107188#6275214) and it may not be desirable design-wise, but we can instead expose an affordance in the sidebar toolbox.

Migration plan

Revised plan from T107188#11181721 by @Krinkle and @JTweed-WMF, based on earlier plan by @Ladsgroup and @Legoktm:

  • Develop a read-only flag for the legacy ShortUrl extension.
  • Develop a sidebar UI in the new UrlShortener extension. T267921
  • Reach out to the ~30 communities where the legacy ShortUrl extension is enabled.
  • Develop simplified read-only URL resolver for ShortUrl database in UrlShortener extension repo behind a feature flag: UrlShortenerEnableLegacyShortUrl.
  • Enable UrlShortenerEnableLegacyShortUrl on test2wiki (which already has ShortUrl disabled, so the old dataset can be tested there).
  • (One week after outreach) Enable the read-only flag of the legacy ShortUrl extension. T256993
  • Enable UrlShortenerEnableLegacyShortUrl on these ~30 wikis and disable the ShortUrl extension. T256993
  • Empty and archive the ShortUrl repository.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Is there a plan for this? We now have UrlShorterner enabled everywhere but no plan for getting rid of ShortUrl.

Krinkle changed the task status from Open to Stalled.Jul 2 2020, 4:19 PM
Krinkle triaged this task as Medium priority.

The main thing is that we no longer expose (old) ShortUrls anywhere, and we don't create any *new* ones.

we could keep ShortUrl as is, put it in a readonly mode.

So it looks like the outcome willl be that the current UI deployed on some wikis where short URLs are eagerly generated on page view (violates DBPerf, T122708) and shows them atop the page for sharing, is something that will no longer be supported.

Instead, people would need to use https://w.wiki, possibly via a gadget that produces a link like https://w.wiki/?url=URL so that users can still get there easily and then only have to submit the form to find or make the short url accordingly.

Perhaps we can already remove the current UI and eager generation today, ahead of T187045 and this task (T107188). If not, what is blocking that?

This comment was removed by Krinkle.
Ammarpad changed the task status from Stalled to Open.EditedJul 12 2020, 1:57 PM
Ammarpad subscribed.

I think keeping the extension in 'readonly' mode, while easier, it has cost and we will be back here to undeploy again. (cf. CodeReview T116948).

I will be happy to move this forward to make UrlShortener understands these legacy urls but need some clarification. Since ShortUrl has database table per each wiki, so where will they eventually live so as to undeploy the extension? Once UrlShortener can have access to the static table it will be able to read and process them.

The plan would be to add a configuration flag that will only be enabled on these projects where the ShortUrls is currently deployed.

Instead, people would need to use https://w.wiki, possibly via a gadget that produces a link like https://w.wiki/?url=URL so that users can still get there easily and then only have to submit the form to find or make the short url accordingly.

UrlShortener already provides a sidebar link that does the same thing. On click, it sends api POST request to create the link and display it. It is disabled on production

I have been thinking about this. I think merging two extension is not a good idea, it makes the code less coherent and turn it into some sort of spagethhi . My counter proposal is:

  • Introduce a readonly mode for ShortUrl extension
  • Enable it on production at the same time of enabling sidebar link (with UrlShortner) on those wikis. i.e. Turn ShortUrl readonly, Turn off read only mode of UrlShortener in those wikis (the wikis should be small, so not much work)
  • Remove the write parts of ShortUrl extension
  • Keep and maintain the shell left of ShortUrl in production

What do you think?

I am fine with that. My comment was largely based on what the task proposed: Add support to UrlShortener to support ShortUrl redirects, may be we should rename it now.

Indeed making UrlShortener to support ShortUrl codes would make the code some kind of garbled, because you have got to explain that this extension is supporting another different routine of abandoned extension. But maintaining ShortUrl extension, even skinned 'shell' of it is not free either and, as I said we may probably be back here talking of 'undeploy' again when it starts causing more harm than good, due to deprecations and other inevitable refactoring of the parts it depend on.

So that makes it worthwhile to consider both options and their short term and long-term repercussions. In the short term, making ShortUrl read-only and leaving UrlShortener out of this is both easy and arguably clean solution. but I am not sure if that's also the case from the long-term perspective.

But do we care about that? I am not sure who will finally decide here (to merge, go for readonly, do something else, etc), but if I may give my opinion, both options are OK. Although I will favor dropping ShortUrl completely and redirect the special page: T240188 (But this may not necessarily means merging it with UrlShortener, there might be other options out there).

Turn off read only mode of UrlShortener in those wikis (the wikis should be small, so not much work)
[Enable] sidebar link (with UrlShortner) on those wikis.

So can you help me understand why this was disabled for other wikis? For instance on English Wiki a user will see this. Is it just we want force them to create it on meta-wiki? I think this is relevant now we want turn it on on subset of projects and what if another project request it to be turned on for them too? possibly after if they see it enabled on project near them. Or the sidebar link that will become visibily present on subset of the projects and not the others.

I am fine with that. My comment was largely based on what the task proposed: Add support to UrlShortener to support ShortUrl redirects, may be we should rename it now.

Indeed making UrlShortener to support ShortUrl codes would make the code some kind of garbled, because you have got to explain that this extension is supporting another different routine of abandoned extension. But maintaining ShortUrl extension, even skinned 'shell' of it is not free either and, as I said we may probably be back here talking of 'undeploy' again when it starts causing more harm than good, due to deprecations and other inevitable refactoring of the parts it depend on.

True but if we remove most of the code and leave only the shell of it, I assume these kind of issues stop and at least reduced to minimum.

But do we care about that? I am not sure who will finally decide here (to merge, go for readonly, do something else, etc), but if I may give my opinion, both options are OK. Although I will favor dropping ShortUrl completely and redirect the special page: T240188 (But this may not necessarily means merging it with UrlShortener, there might be other options out there).

Turn off read only mode of UrlShortener in those wikis (the wikis should be small, so not much work)
[Enable] sidebar link (with UrlShortner) on those wikis.

So can you help me understand why this was disabled for other wikis? For instance on English Wiki a user will see this. Is it just we want force them to create it on meta-wiki? I think this is relevant now we want turn it on on subset of projects and what if another project request it to be turned on for them too? possibly after if they see it enabled on project near them. Or the sidebar link that will become visibily present on subset of the projects and not the others.

There is two parts: Making it non-readonly and enabling the link on sidebar.

  • Making it non-readonly is mostly blocked on checking if the ratelimit works (it might be you can bypass the ratelimit by requesting it in another wiki, we want to make sure it doesn't happen). Once we are sure the ratelimit works, it can go live everywhere.
  • Enabling the link on the sidebar on the other hand is a complicated mess, it needs a UX review first (probably a rewrite) then a performance review, then slowly rolling it out (to make sure pressure on the database stays within healthy boundaries) but OTOH, this can go live on the wikis that have shorturl enabled (as an alternative to shorturl) because a) they are pretty small and b) this thing at any shape is better than shorturl and c) if the ratelimit doesn't work, the number of wikis we roll this out to is still small.

@Ammarpad mostly summed up my points about maintenance overhead with having a separate extension. I think subsuming ShortUrl into UrlShortener makes sense long term as it:

  • removes the overhead for maintaining a separate git repo think of all the CI/npm/composer/phan/PHPCS/etc. config
  • clearly indicates that UrlShortener is the future of ShortUrl
  • unifies all short URL logic, making it straightforward to look in just one place when you're trying to find or debug something (there's already enough confusion just based on the names alone)

I don't think it'll be as messy as feared, the ShortUrl stuff will be relatively self-contained and marked as legacy, guarded by feature flags.

We could do it in stages too, first turn ShortUrl into the shell, deploy all that, and then merge it into UrlShortener as cleanup.

Sure, if that's preferred for you too. I'm fine as well, let's first make the shell of it to make the merge easier. Does that sound good?

Krinkle renamed this task from Add support to UrlShortener to support ShortUrl redirects to Sunset ShortUrl extension in favour of UrlShortener extension.Jul 8 2023, 6:28 PM
Krinkle updated the task description. (Show Details)
Krinkle moved this task from Limbo to Perf recommendation on the Performance-Team (Radar) board.
pmiazga subscribed.

FWIW, shorturl tables are the only tables in production that have not been cataloged yet.

Would the tables be kept and transferred to UrlShortener? Or would there be some kind of migration script that can convert ShortUrl entries into UrlShortener entries? I imagine the link format is too different for that...

Would the tables be kept and transferred to UrlShortener? Or would there be some kind of migration script that can convert ShortUrl entries into UrlShortener entries? I imagine the link format is too different for that...

Yeah, that wouldn't be really possible. We could set up a redirect service with the list as files being loaded, they shouldn't be too massive but they have to be somewhere.

Change #1184153 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/mediawiki-config@master] [WIP] Disable ShortURL everywhere, without migration

https://gerrit.wikimedia.org/r/1184153

Honestly at this point I'm in favor of simply shutting it down without no replacement. We can let the community know beforehand but if/when goo.gl can do it, I wonder why we can't. We can definitely provide a public dump of the tables so people could fix links retroactively but the benefit of the redirect mechanism IMHO is not worth the effort as this is a very small set of requests coming to our infra.

Revised plan

Based on chatting with @JTweed-WMF:

  1. (Already done.) Develop a read-only flag for the legacy ShortUrl extension.
  2. (Already done.) Develop a sidebar UI in the new UrlShortener extension. T267921
  3. Reach out to the ~30 communities where the legacy ShortUrl extension is enabled.
  4. Develop simplified read-only URL resolver for ShortUrl database to UrlShortener extension repo behind a feature flag: UrlShortenerEnableLegacyShortUrl.
  5. Enable UrlShortenerEnableLegacyShortUrl on test2wiki (which already has ShortUrl disables, so the old dataset can be tested there).
  6. (One week after outreach) Enable the read-only flag of the legacy ShortUrl extension. T256993
  7. Enable UrlShortenerEnableLegacyShortUrl on these ~30 wikis and disable the ShortUrl extension.
  8. Archive the ShortUrl repository.

Context

The read-only flag hides the legacy UI (title caption and sidebar link) and prevents creation of new URLs in the backend. All existing short URLs, such as https://hi.wikipedia.org/s/177k, will keep working.

ShortUrl UI (legacy)UrlShortener UI (new)
Screenshot 2025-09-15 at 16.37.30.png (1×1 px, 356 KB)
Screenshot 2025-09-15 at 16.38.28.png (1×2 px, 322 KB)

The outreach will inform these ~30 communities that the ShortUrl extension, with its "title caption" link and "Short URL" sidebar link, will be replaced by the new "Get shortened URL" sidebar link. This new link is already available on their wiki today and can be tested.

Some of the benefits:

The main reason for the proactive outreach directly to local commnities is to give time to migrate any custom integrations they might have. For example, Hindi Wikipedia has a gadget that replaces the "title caption" link via HTML class="title-shortlink", with a general share menu.

Default interface (legacy ShortUrl extension)Hindi Wikipedia gadget
Screenshot 2025-09-15 at 16.59.05.png (560×970 px, 105 KB)
Screenshot 2025-09-15 at 16.59.09.png (560×1 px, 138 KB)

Anyone we did not reach through our communication will notice the change at step 6 (one week after outreach) when we hide the legacy UI, and may find our Village Pump or Tech News message retroactively at this time, or otherwise contact us or local Wikitech ambassadors who did receive the message. Either way we're ready to help update any gadgets and CSS mods.

Develop simplified read-only URL resolver for ShortUrl database to UrlShortener extension repo behind a feature flag: wgUrlShortenerResolveLegacyShortUrl.

By 'to UrlShortener extension repo' do you mean this code will live in the UrlShortener extension's repo? If so, that sounds reasonable.

Proposed text for Tech-News W45 edition of Monday 3 Nov:

Since 2019 the Wikimedia URL Shortener is available on all Wikimedia wikis to create short links to articles, permalinks, diffs, searches, and more. There are 30 wikis that also install the older "ShortUrl" extension. This older extension will soon be removed. This means old /s/ URLs will not be advertised under article titles via HTML class="title-shortlink". The /s/ URLs will keep working. Task

One small correction on tech news, The w.wiki url shortener got deployed to production in 2019: T108557#5103986

Oops, I misread that change as enabling the local sidebar UI, because the domain appears to be live and routing to Meta-Wiki since 2016. Fixed.

Krinkle updated the task description. (Show Details)

Oops, I misread that change as enabling the local sidebar UI, because the domain appears to be live and routing to Meta-Wiki since 2016. Fixed.

Yeah, it was technically deployed but noone could shorten any url and only w.wiki/2 worked as POC.

Change #1199717 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/UrlShortener@master] Allow wgServer by default even when it includes a port

https://gerrit.wikimedia.org/r/1199717

Change #1199717 merged by jenkins-bot:

[mediawiki/extensions/UrlShortener@master] Allow wgServer by default even when it includes a port

https://gerrit.wikimedia.org/r/1199717