Skip to content

RUM-14050: Reduce CI pipeline time via parallelism and Gradle consolidation#3559

Merged
hamorillo merged 5 commits into
developfrom
hector.morilloprieto/RUM-14050-analysis-optimization
Jun 19, 2026
Merged

RUM-14050: Reduce CI pipeline time via parallelism and Gradle consolidation#3559
hamorillo merged 5 commits into
developfrom
hector.morilloprieto/RUM-14050-analysis-optimization

Conversation

@hamorillo

Copy link
Copy Markdown
Contributor

What does this PR do?

Optimises the analysis:detekt-custom and test stage jobs by introducing Gradle parallel execution and consolidating redundant sequential ./gradlew --no-daemon invocations. No changes to what is compiled or tested.

Motivation

analysis:detekt-custom ran its assembly and custom rules sequentially. test:debug and test:kover each paid full Gradle JVM startup and project-configuration overhead across 5 and 4 separate invocations respectively, adding unnecessary wall-clock time to every PR pipeline.

Changes

.gitlab-ci.ymlKUBERNETES_MEMORY_LIMIT: 13Gi → 32Gi globally (matching datadog-android on the same runner pool; no infrastructure request needed).

DetektCustomConfig.ktmaxHeapSize = "2g" on customDetektRules to bound per-worker JVM heap for safe parallel execution.

ci/pipelines/default-pipeline.yml:

Job Change
analysis:detekt-custom assembleLibrariesRelease --parallel (in-process workers, shared daemon heap)
analysis:detekt-custom customDetektRules --parallel --max-workers=4 (31 JVMs sequential → 4 concurrent, heap capped at 2g each)
test:debug 5 sequential ./gradlew --no-daemon calls → 1 consolidated with --parallel --max-workers=4
test:kover 4 sequential ./gradlew --no-daemon calls → 1 consolidated with --parallel --max-workers=4

Note: >- YAML folded scalar is used for multiline commands — shell \ continuation breaks in GitLab CI YAML because the backslash is preserved literally and becomes an escaped space, causing Gradle to reject task paths.

Measured results

Baseline: 27 recent PRs on develop (excluding this branch).

analysis stage

Job Baseline avg (27 PRs) This PR Saved
analysis:detekt-custom 15m 48s 9m 32s −6m 16s (−40%)

test stage

Job Baseline avg (27 PRs) This PR Saved
test:debug 18m 23s 9m 23s −9m 00s (−49%)
test:kover 18m 35s 12m 13s −6m 22s (−34%)
Test stage gate 19m 31s 12m 13s −7m 18s (−37%)

Total pipeline wall-clock

Baseline avg (27 PRs) This PR Saved
Total 54m 04s 39m 14s −14m 50s (−27%)

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Make sure you discussed the feature or bugfix with the maintaining team in an Issue
  • Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

The analysis:detekt-custom job runs custom Detekt rules by invoking
./gradlew customDetektRules, which sequentially executes 31 independent
JavaExec tasks — one JVM per module. Because the tasks share no state and
declare no inter-dependencies, they are safe to run concurrently.

Adding --parallel --max-workers=4 reduces step 6 of the job from ~155s to
~69s (2.2× speedup) on a local benchmark of all 31 modules.

Each worker JVM is capped at maxHeapSize=2g to bound heap usage. The job's
Kubernetes memory limit is increased to 32Gi (matching datadog-android) to
accommodate 4 concurrent JVMs: 4 × ~3.25Gi (heap + metaspace) + 4Gi
Gradle daemon ≈ 18Gi, well within the new limit.

# Conflicts:
#	ci/pipelines/default-pipeline.yml
assembleLibrariesRelease is the dominant step in analysis:detekt-custom,
taking ~8m 17s (56% of the job) in CI. Adding --parallel enables Gradle
to compile independent library modules concurrently within the same daemon
process (in-process workers sharing the 4Gi heap) rather than sequentially.

Unlike customDetektRules --parallel which spawns separate JVMs per task,
--parallel on assembleLibrariesRelease uses Gradle's built-in worker
infrastructure — no additional memory per concurrent task. The module
dependency graph has a natural fan-out: dd-sdk-android-internal and
dd-sdk-android-core must compile first, then 25+ feature and integration
modules can compile in parallel.
Move KUBERNETES_MEMORY_LIMIT from 13Gi to 32Gi in the root .gitlab-ci.yml,
matching datadog-android which uses the same limit on the same runner pool.

The higher limit is needed for jobs that run parallel JVM workers
(analysis:detekt-custom, test:debug). Setting it globally is simpler than
per-job overrides and has no scheduling cost since KUBERNETES_MEMORY_REQUEST
remains at 8Gi — the scheduler still only needs 8Gi free on the node.
Actual memory consumed per job is unchanged for jobs that don't use the
extra headroom.

Removes the now-redundant KUBERNETES_MEMORY_LIMIT: 32Gi override from
analysis:detekt-custom.
…--parallel

test:debug ran 5 separate ./gradlew --no-daemon invocations sequentially.
Each call pays the full Gradle startup and project configuration overhead
(~10-15s in CI × 4 extra calls = ~40-60s wasted). Consolidating into a
single invocation eliminates that redundancy.

--parallel --max-workers=4 lets Gradle run up to 4 modules' test tasks
concurrently. Each module forks its own test JVM (~768MB); 4 concurrent
test JVMs + 3Gi Gradle JVM ≈ 6Gi total — well within the 32Gi limit now
set globally.

Note: unlike assembleLibrariesRelease --parallel (in-process workers,
shared heap), test tasks fork separate JVMs. --max-workers=4 acts as the
memory safety valve, the same role it plays for customDetektRules.
…--parallel

test:kover ran 4 separate ./gradlew --no-daemon invocations for coverage
generation, paying ~33s of Gradle startup overhead across 3 extra calls.
Consolidating into a single invocation using the >- YAML folded scalar
eliminates that overhead (same fix applied to test:debug).

--parallel --max-workers=4 lets Gradle run up to 4 modules' koverXmlReport
tasks concurrently. koverXmlReportRelease forks a test JVM per module for
instrumented coverage tracking (~1.5-2GB each); 4 workers stay well within
the 32Gi global limit set in the previous commit.

test:kover is the current critical path of the test stage (~17m 55s avg)
now that test:debug has been reduced to ~7m 57s.
@hamorillo

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 28868f2b6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@hamorillo
hamorillo marked this pull request as ready for review June 19, 2026 08:32
@hamorillo
hamorillo requested review from a team as code owners June 19, 2026 08:32
Comment thread ci/pipelines/default-pipeline.yml

@kikoveiga kikoveiga left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! Is it also possible to add some of these --parallel optimizations to local_ci.sh?

Comment thread ci/pipelines/default-pipeline.yml
@aleksandr-gringauz

Copy link
Copy Markdown
Contributor

Very nice! Is it also possible to add some of these --parallel optimizations to local_ci.sh?

Ideally this should live in gradle.properties that is commited to Git. However on CI we get gradle.properties from secrets that complicates things a bit.

@hamorillo hamorillo left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! Is it also possible to add some of these --parallel optimizations to local_ci.sh?

Ideally this should live in gradle.properties that is commited to Git. However on CI we get gradle.properties from secrets that complicates things a bit.

Not sure if I understand what you mean with the gradle.properties. I would say we can leverage a similar approach in the local-ci.sh for some gradlew tasks. (In fact, that was something I wanted to add in following PRs).

Comment thread ci/pipelines/default-pipeline.yml
Comment thread ci/pipelines/default-pipeline.yml
@aleksandr-gringauz

Copy link
Copy Markdown
Contributor

Not sure if I understand what you mean with the gradle.properties. I would say we can leverage a similar approach in the local-ci.sh for some gradlew tasks. (In fact, that was something I wanted to add in following PRs).

I mean doing essentially the same thing as --parallel but by setting it from gradle.properties so that the execution for all gradle tasks in this project is parallel by default (locally, on CI, everywhere). https://docs.gradle.org/current/userguide/performance.html#sec:enable_parallel_execution

@hamorillo

Copy link
Copy Markdown
Contributor Author

I mean doing essentially the same thing as --parallel but by setting it from gradle.properties

I see what you mean 👍

@aleksandr-gringauz aleksandr-gringauz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@aleksandr-gringauz

Copy link
Copy Markdown
Contributor

I mean doing essentially the same thing as --parallel but by setting it from gradle.properties

I see what you mean 👍

we can think about it in future PRs

current PR LGTM

@hamorillo
hamorillo merged commit d4b6be3 into develop Jun 19, 2026
27 checks passed
@hamorillo
hamorillo deleted the hector.morilloprieto/RUM-14050-analysis-optimization branch June 19, 2026 11:59
kikoveiga pushed a commit that referenced this pull request Jun 19, 2026
…analysis-optimization

RUM-14050: Reduce CI pipeline time via parallelism and Gradle consolidation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants