Skip to content

librarian: allow degrees of parallelization to be configured #3933

@jskeet

Description

@jskeet

Currently, we run prepare and clean sequentially, generate in parallel, and format sequentially.

I assume this suits Rust - but generation going to be a pain (at least temporarily) for Python, where libraries are generated into a fixed staging directory first. Whereas prepare/clean could be run in parallel for Python, and if we extract the formatting from generation I'd expect that to be parallelizable too.

I'd also note that for debug purposes it can be useful to run everything sequentially too (to get an easy-to-follow log file, for example).

Finally, it's possible that we may want to parallelize but with a limited degree of parallelization. (@coryan discussed this in our recent shadowing for release.)

I think we could reasonably have something in librarian.yaml stating the degree of parallelization (possibly in line with group.SetLimit) for each of clean, generate, format - and probably build/publish later on.

@zhumin8 @coryan @julieqiu thoughts? Is this worth doing at all do you think, and if so, do you have any thoughts on the configuration design?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions