Skip to content

External storage prototype#110

Closed
jglick wants to merge 25 commits intojenkinsci:masterfrom
jglick:external-storage
Closed

External storage prototype#110
jglick wants to merge 25 commits intojenkinsci:masterfrom
jglick:external-storage

Conversation

@jglick
Copy link
Copy Markdown
Member

@jglick jglick commented Aug 28, 2018

Prototype of an SPI for keeping test results anywhere other than junitResults.xml.

Subsumes #109.

@jglick jglick requested a review from davidcurrie September 25, 2018 17:56
@timja
Copy link
Copy Markdown
Member

timja commented Mar 28, 2019

Hey @jglick

What would it take to get some movement on this?
Did you implement a prototype consumer of this api?

I'm happy to spend a bit of time on this if you think it could be moved forward

@jglick
Copy link
Copy Markdown
Member Author

jglick commented Mar 29, 2019

@timja indeed this was put on hold due to lack of interest from my employer in spending resources on this general area. You should be able to get a rough idea of how far I got by looking at outstanding TODO comments and the coverage in TestResultStorageTest. If I get permission to spend some more time here, I would want to get it far enough along that all the basic functions of the plugin work (so far as automated tests are concerned) against the dummy provider using local H2 SQL, including at least one complex feature such as the trend graph implemented using a proper query. That would be enough for someone else with more practical database / cloud storage experience to prototype a realistic provider plugin (not necessarily SQL-based), which could be the reference implementation for a JEP, and help fill in some of the gaps in functionality needed for production use.

What is your interest in this? Is there a particular issue that is hurting you in the field—disk space consumption after many builds, difficulty creating custom reports, compliance requirement for archival storage, Remoting channel congestion transferring massive test results, slow build load time parsing junitResult.xml, etc.?

@timja
Copy link
Copy Markdown
Member

timja commented Mar 29, 2019

Specific interest in this currently is:
We're trying to decouple our jenkins from our infrastructure and reduce the access that it has, it's basically a god everywhere right now.

In this case we're running our tests in Kubernetes and wanting to report them back to jenkins, not have jenkins collect them

We were easily able to get artifacts back to jenkins using: https://github.com/jenkinsci/azure-artifact-manager-plugin and storing them in blob storage. It would be nice to be able to do something similar with the test results

General interest:
Jenkins is our most critical piece of infrastructure, anything that moves its storage off of the master and into blob storage is great for us.
Makes it more resilient to failure and easier to upgrade.

We also have disk spaces issues,
image

I suspect the test reports don't contribute much to our disk space problems, but its hard to tell as there's soo many files that du doesn't respond in a reasonable time, I can run it against one teams jobs folder, come back 10 minutes later and not have a result.

@jglick jglick marked this pull request as draft April 21, 2020 19:33
This was referenced Aug 15, 2020
@timja
Copy link
Copy Markdown
Member

timja commented Aug 19, 2020

Replaced by #141

@timja timja closed this Aug 19, 2020
@jglick jglick deleted the external-storage branch August 19, 2020 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants