Move Jenkins job configuration into the repository #3893

rcurtin · 2025-02-13T16:26:44Z

This is a "warmup" to defining the CI jobs on Jenkins for embedded systems. It will take me some attempts to transfer our existing Jenkins job configurations to a .jenkins/ directory full of Jenkinsfiles. The Jenkinsfile syntax is truly awful (it is Groovy), but, better to have the configurations here instead of hidden inside of the Jenkins configuration.

Update: this is ready!

This PR moves the Jenkins jobs into the mlpack repository, just like the Github Actions jobs. This makes them a lot more visible to users and developers, and easier to modify. (We should hopefully no longer need to give out permissions to Jenkins, but we'll see!)

There are five jobs I've defined:

.jenkins/cross-compilation/Jenkinsfile: cross-compile mlpack for an RPi5 and run mlpack_test on the RPi5; in another PR, I will add more embedded targets.
.jenkins/doc-link-check/Jenkinsfile: check all the links in the documentation, maintaining a cache of links that are known-good to accelerate the job.
.jenkins/doc-snippet-build/Jenkinsfile: build all the C++ documentation snippets and make sure they all run.
.jenkins/memory-checks/Jenkinsfile: run any tests that have changed through valgrind to make sure there are no memory problems. I don't think this job was working correctly before this PR, but hopefully it will be useful in the future.
.jenkins/style-checks/Jenkinsfile: run the style linter to make sure there are no problems.

The style check and link check job take less than 10 minutes each, so they are pretty fast. The other jobs use ccache, but the compile times seem to be long. I am not totally sure why that is. I also think the C++ documentation snippets may need some amount of auditing because the slowdown there is runtime, not compile time... but I will leave that for another time.

I've disabled all the other Jenkins jobs and updated the documentation. The static code analysis job used Facebook's infer tool, which was so computationally intensive that I made the call to just remove the job.

There were a couple other small changes and bugfixes, especially make the junit test reporter work correctly for Catch2. They should hopefully be pretty self-explanatory in the diff.

rcurtin · 2025-02-13T16:27:13Z

@mlpack-jenkins test this please

See catchorg/Catch2@c29e198.

rcurtin · 2025-03-05T16:54:41Z

Ok, I made everything work here and it is ready for review! I updated the description of the PR with the details.

eddelbuettel · 2025-03-05T19:36:59Z

Very amazing patience and persistence! Congratulations for seeing it through.

shrit

Looks good to me, thanks for doing all of migration, I know how tedious this looks like

shrit · 2025-03-10T11:45:56Z

src/mlpack/tests/ann/feedforward_network_test.cpp

+  // Make sure the data loaded okay.
+  REQUIRE(!dataset.is_empty());


Thanks for the fix

shrit · 2025-03-10T11:47:56Z

.jenkins/cross-compilation/Jenkinsfile

+
+                  ssh jenkins@${hostname} -t mkdir -p test_${BRANCH_NAME}_${BUILD_ID}/
+                  scp build/bin/mlpack_test jenkins@${hostname}:test_${BRANCH_NAME}_${BUILD_ID}/
+                  scp -r src/mlpack/tests/data/* jenkins@${hostname}:test_${BRANCH_NAME}_${BUILD_ID}/
+                  # Unpack all compressed test data.
+                  ssh jenkins@${hostname} -t "
+                      cd test_${BRANCH_NAME}_${BUILD_ID};
+                      find ./ -iname '*.bz2' -exec tar xvf \\{\\} \\;"
+
+                  mkdir -p reports;
+                  ssh jenkins@${hostname} -t "
+                      cd test_${BRANCH_NAME}_${BUILD_ID};
+                      mkdir -p reports;
+                      ./mlpack_test -r junit -o reports/mlpack_test.junit.xml"
+
+                  # Clean up afterwards.
+                  scp jenkins@${hostname}:test_${BRANCH_NAME}_${BUILD_ID}/reports/mlpack_test.junit.xml reports/mlpack_test.${hostname}.junit.xml;
+                  ssh jenkins@${hostname} -t rm -rf test_${BRANCH_NAME}_${BUILD_ID}/;


I suppose this part took much of the time

Yes, most of the time was actually figuring out that I was compiling with 4KB page sizes but couscous (that RPi5) was running with 16KB page sizes and so it would just segfault...

shrit · 2025-03-10T11:49:22Z

.jenkins/utils.groovy

+def setBuildStatus(Map paramsMap)
+{
+  // Extract arguments from the map.
+  def result = paramsMap.result;
+  def context = paramsMap.context;
+  def successMessage = paramsMap.successMessage;


Groovy looks interesting

I guess it's not bad, but I am tired of having so many different languages... I would have rather just had a shell script.

shrit · 2025-03-10T11:50:05Z

doc/embedded/supported_boards.md

+-DTOOLCHAIN_PREFIX=/path/to/bootlin/toolchain/aarch64--glibc--stable-2024.02-1/bin/aarch64-buildroot-linux-gnu-
+-DCMAKE_SYSROOT=/path/to/bootlin/toolchain/aarch64--glibc--stable-2024.02-1/aarch64-buildroot-linux-gnu/sysroot


Any reason for this change ?

It seems like it was a typo in the original; the gnueabihf prefix doesn't exist:

$ ls /opt/bootlin/aarch64--glibc--stable-2024.02-1/bin/aarch64-buildroot-linux-*-g++ /opt/bootlin/aarch64--glibc--stable-2024.02-1/bin/aarch64-buildroot-linux-gnu-g++

This may be an issue with some of the other boards too; I didn't check fully.

github-actions

Second approval provided automatically after 24 hours. 👍

rcurtin added 2 commits February 13, 2025 11:03

Add a first attempt at explicitly using Jenkinsfiles.

baaaf0a

A first attempt...

ed0927d

rcurtin added c: testing t: added feature labels Feb 13, 2025

rcurtin added 25 commits February 13, 2025 13:02

Put code in a script block.

d85bf74

Add first attempt at link check job pipeline.

db60dc9

Hopefully correct shell block.

8abb11e

Install git.

a0bded5

Install packages as root.

d88d9a7

Use custom image that already has dependencies installed.

97fd88d

Try processing the JUnit results.

e86b2bf

Fix syntax (hopefully).

644c3b1

Try to get the build to set its status on Github.

c5727eb

Try and see if I can get the snippet build to run too.

d4eac2b

A first attempt at reviving the static code analysis build.

8cc0d17

Refactor style check job a bit.

9ef2118

Try to clean up other files and have them set statuses.

aaa7cd4

Set status in script blocks.

b6cf9a4

Fix directory (this may not fix my problem).

ea61882

Should I load in the script step?

f7c6d58

Maybe I can just load it without a name.

6e792f3

Maybe I have my path wrong.

bb1cd45

Will this work? Just a test...

c84b600

Try using a plugin instead.

90f4575

And if I define the function manually at the top?

200c11d

Maybe this will fix the load.

978b0a4

Try to turn unstable into failed.

68dd436

Hopefully fix documentation builds.

6620a47

Fix script blocks.

03008d2

rcurtin added 21 commits February 27, 2025 13:26

Fail tests when the data isn't there.

df6ff7f

Maybe I can use find instead.

5136018

Double escape for backslash?

4cefd10

What if we just run the test?

4631c5c

Port Catch2 improvement for junit runner.

862c21b

See catchorg/Catch2@c29e198.

Re-enable junit testing (hopefully it will work now).

bef0330

Output directly to the xml file.

f89d6c3

Try to clean up regex.

a642b36

Try to set IN PROGRESS status.

7d5e8e6

Could it be called RUNNING?

55ab7a7

I guess I don't get access to set jobs in progress through this API.

f3cfd1c

Fix regex for test name extraction.

a3e7dec

Try to clean up Jenkinsfiles.

88b6f20

Fix parameter name.

e0bb8c5

Maybe fix syntax?

cb215f6

Does it work without keyword arguments?

7e87ae2

Correctly accept named parameters.

d8cc820

Abort previous builds to reduce load on Jenkins.

24d2708

Use optimization when compiling.

04d3dd5

Fix syntax for abortPrevious.

e8db9f8

Fix missing closing brace...

fe1b1dc

rcurtin mentioned this pull request Mar 5, 2025

Bootstrap strategy for RandomForest #3829

Merged

rcurtin marked this pull request as ready for review March 5, 2025 16:48

Update links in CI documentation and try to fix memory check job.

d0d530b

rcurtin mentioned this pull request Mar 5, 2025

Add more cross-compilation hosts for CI job #3906

Merged

shrit approved these changes Mar 10, 2025

View reviewed changes

github-actions bot approved these changes Mar 11, 2025

View reviewed changes

rcurtin merged commit a44fbe4 into mlpack:master Mar 12, 2025
16 checks passed

		// Make sure the data loaded okay.
		REQUIRE(!dataset.is_empty());

		-DTOOLCHAIN_PREFIX=/path/to/bootlin/toolchain/aarch64--glibc--stable-2024.02-1/bin/aarch64-buildroot-linux-gnu-
		-DCMAKE_SYSROOT=/path/to/bootlin/toolchain/aarch64--glibc--stable-2024.02-1/aarch64-buildroot-linux-gnu/sysroot

Uh oh!

Move Jenkins job configuration into the repository #3893

Move Jenkins job configuration into the repository #3893

Uh oh!

Conversation

rcurtin commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rcurtin commented Feb 13, 2025

Uh oh!

rcurtin commented Mar 5, 2025

Uh oh!

eddelbuettel commented Mar 5, 2025

Uh oh!

shrit left a comment

Choose a reason for hiding this comment

Uh oh!

shrit Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

shrit Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

rcurtin Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

shrit Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

rcurtin Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

shrit Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

rcurtin Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rcurtin commented Feb 13, 2025 •

edited

Loading