@timofei-iatsenko compiling is now so fast, I think I'll just go ahead and close that other PR because fetching from the cache is slower than compilation! Awesome work 🔥

I could not find the action for the extraction in passed checks, could you give a link or just share the numbers, before and after?

I think you can also close this one #7138 as superseded

It's all done in one step, so intl:extract:all takes ~18 seconds, and intl:compile takes ~1 second 🤯

Benchmarks on my laptop

Extraction: 4.81s -> 6.30s

Before	After

Compilation: 18.78s -> 0.66s 🔥

Before	After

What were the previous metrics for intl:extract:all?

I did some benchmarking yesterday and found that on my local machine, the difference between running the extract command with --workers 6 vs. --workers 1 was only 6s vs.7s. This is unexpected, as extraction is a highly parallelizable process; theoretically, we should see closer to a 6x speed increase (especially since I have 16 cores available).

This suggests there is a bottleneck elsewhere in the extraction process. I’ll investigate further to identify what’s stalling the parallel execution.

I did some benchmarking yesterday and found that on my local machine, the difference between running the extract command with --workers 6 vs. --workers 1 was only 6s vs.7s. This is unexpected, as extraction is a highly parallelizable process; theoretically, we should see closer to a 6x speed increase (especially since I have 16 cores available).

Yep I see the same result with --workers 13 vs. --workers 1

The execution of extraction on a single core might slightly rise, because macro plugin now doing a bit more job then before. It now not only change the AST but also re-build scopes and bindings so lingui macro doesn't mess a react-compiler which is relying on the bindings information.

Since the same macro babel plugin used for app build and for extraction it might affect extraction time.

@mozzius i also suggest you to switch lingui config to ts or at least use defineConfig function for the better validation/developer experience:

import {defineConfig} from '@lingui/cli'

export default defineConfig({
  locales: [
    'en',
    'an',
    'ast',
    'ca',
    'cy',
    'da',
    'de',
    'el',
    'en-GB',
    'eo',
    'es',
    'eu',
    'fi',
    'fr',
    'fy',
    'ga',
    'gd',
    'gl',
    'hi',
    'hu',
    'ia',
    'id',
    'it',
    'ja',
    'km',
    'ko',
    'ne',
    'nl',
    'pl',
    'pt-BR',
    'pt-PT',
    'ro',
    'ru',
    'sv',
    'th',
    'tr',
    'uk',
    'vi',
    'zh-CN',
    'zh-HK',
    'zh-TW',
  ],
  catalogs: [
    {
      path: '<rootDir>/src/locale/locales/{locale}/messages',
      include: ['src'],
    },
  ]
})

I dropped a format: po, field, since it's a default one, and string formatters are no longer supported in upcoming v6.

Also i recommend to add a sourceLocale: 'en', this will slightly change the cli behaviour when it comes to extracting and compiling.

@timofei-iatsenko thanks for the tip! I also added compileNamespace: 'ts'.

What exactly does sourceLocale: 'en' do? I don't quite understand from the docs. How will it change the CLI output?

I found a culprit of a slower extraction. It was a sorting, the sort function creates a

const collator = new Intl.Collator("en-US");

on every invocation. So without a patch, the numbers look like:

$ lingui extract --clean --verbose --workers 4
Extracting messages from source files…
Use worker pool of size 4
collecting: 1.524s (tasks inside goes in parallel)
  - readAll: 415.598ms
  - extracting from files: 1.501s
merging: 412.112ms
sorting: 3.981s
  order en: 93.221ms
  order an: 94.806ms
  order ast: 95.919ms
  [...]
write: 596.106ms
✔ Done in 7s

With a patch it looks like

sorting: 138.494ms
  order en: 5.282ms
  order an: 5.81ms
  order ast: 3.125ms
    [...]

3.981s vs 138.494ms

@timofei-iatsenko thanks for the tip! I also added compileNamespace: 'ts'.

What exactly does sourceLocale: 'en' do? I don't quite understand from the docs. How will it change the CLI output?

It basically tells the Lingui CLI that you are using en as your source language. When sourceLocale: 'en' is set, the CLI handles statistics differently. Instead of saying all translations are "missing" for English, it marks it as the source:

┌─────────────┬─────────────┬─────────┐
│ Language    │ Total count │ Missing │
├─────────────┼─────────────┼─────────┤
│ en (source) │    2595     │    -    │
│ an          │    2595     │   1022  │
│ ast         │    2595     │   1595  │

Also, when Lingui writes the catalog for the source locale, it automatically pre-populates translations for new entries:

# en.po

#. Accessibility label for a category
#: src/components/InterestTabs.tsx:330
msgid "(contains embedded content)"
msgstr "(contains embedded content)" # <-- Automatically filled for source locale

When you compile catalogs, the fallback system uses the sourceLocale catalog as a backup. For example, if you add a new message "Hello!" but only extract it for en and not for pl, that message won't exist in the pl catalog. Without a source locale, the app would show the ID instead of the message at runtime.

If sourceLocale is set, the compiler merges the pl keys with the en keys. So even if you haven't extracted it intopl catalog yet, the app won't break - it will just fall back to the English version.

This helps you avoid updating every single catalog file every time you run an extract. You only need to touch the other catalogs when you actually have new translations to add.

A further step would be using lingui extract-template, which creates a idiomatic .pot file with the actual messages from your app.

I know this is a bit complicated - let me know if it’s still not clear and I can try explaining it differently!

1. Awesome, nice find! This is going to speed up our CI significantly :)

2. Interesting, thank you! I think I will enable it then - cc @surfdude29 for visibility

@mozzius here is a patch for extraction performance @lingui+cli+5.9.1.patch

You can use it temporarly, while I preparing a fix in the lingui repo.

Nice, CI for extract+compile has gone from 19.81s to 13.21s 🎉

All sounds great to me! :)

Sorry if this is a silly question, but as long as a full extract and compile is run before a release is cut, that's the main thing that matters to ensure that all the updated translations make it into a release, right? 🤔

@surfdude29 yes, all our CI does this and will continue to do so just to be safe

@mozzius Also worth noting that the compile step could be skipped via: https://lingui.dev/ref/metro-transformer

Not sure if it has performance implications or perhaps compatibility issues for web, just wanted to bring attention to it.

@mozzius since compiling time is no longer an issue - you can use bundler loaders (metro-transformer or webpack/vite) This will eliminate a manual step of compiling and made a local development a bit more pleasant.

But it's better to test performance, because as i mentioned bundler loaders could not benefit from multithreading of compile cli command.

Thanks @fobos531 and @timofei-iatsenko - I'll look into it!