Skip to content

buildcache push: improve printing#36141

Merged
haampie merged 11 commits intospack:developfrom
scottwittenburg:print-msg-for-each-pushed-buildcache
May 3, 2023
Merged

buildcache push: improve printing#36141
haampie merged 11 commits intospack:developfrom
scottwittenburg:print-msg-for-each-pushed-buildcache

Conversation

@scottwittenburg
Copy link
Copy Markdown
Contributor

@scottwittenburg scottwittenburg commented Mar 15, 2023

If for any reason dependent jobs begin (or continue) pushing buildcaches for all of their dependencies, it will be easy to spot in job traces (like this one from a previous test PR).

@spackbot-app spackbot-app bot added binary-packages core PR affects Spack core functionality labels Mar 15, 2023
@haampie
Copy link
Copy Markdown
Member

haampie commented Mar 16, 2023

nitpick: I'm generally not a fan of putting tty.info(...) in source files other than cmd/*.py, since non-cmd/*.py files have other entrypoints where no terminal is involved at all (e.g. user scripts that call binary_distribution.push(...)).

Alternatively, can you make:

  1. def push(spec: spack.spec.Spec) -> Result accept just an individual spec and let it return whether it succeeded and if so where it pushed the spec?
  2. def push_all(*spec: spack.spec.Spec) -> List[Result] do basically return [push(s) for s in specs]
  3. Make cmd/buildcache.py do the logic of listing the specs that can be pushed to the cache.
  4. Make CI just call push(...) so it's very clear it will only ever push one spec.

Then the tty.info business can go into cmd/buildcache.py and cmd/ci.py.

This also ties in with #35601 since error handling can be improved too. Maybe push_all could instead return a List[Either[Result, Error]] by catching errors in push.

@scottwittenburg
Copy link
Copy Markdown
Contributor Author

Make CI just call push(...) so it's very clear it will only ever push one spec.

This is the bit I thought was really key from your review, thanks. Let me know if I missed the mark in your opinion, I didn't really see the benefit of a push_all, but could still be convinced.

@scottwittenburg scottwittenburg force-pushed the print-msg-for-each-pushed-buildcache branch from 603f5cb to 2389977 Compare March 17, 2023 00:39
@haampie haampie force-pushed the print-msg-for-each-pushed-buildcache branch from 2389977 to 99a3469 Compare March 17, 2023 12:42
@haampie haampie changed the title bindist: Print spec/hash and dest url when pushing buildcace push: improve UI. Mar 17, 2023
@haampie
Copy link
Copy Markdown
Member

haampie commented Mar 17, 2023

I've hijacked this PR to improve the UI more generally, cause it was still extremely verbose in some cases. Now it looks like this:

Screenshot 2023-03-17 at 13 26 03

@haampie haampie changed the title buildcace push: improve UI. buildcace push: improve printing Mar 17, 2023
@haampie haampie changed the title buildcace push: improve printing buildcache push: improve printing Mar 18, 2023
haampie
haampie previously approved these changes Mar 20, 2023
Copy link
Copy Markdown
Member

@alalazo alalazo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have test-driven this PR trying to create buildcaches on my local filesystem. This LGTM from a user perspective.

Concerning the code, is it possible to update type-hints and docstring of all the functions that have been modified?

Comment on lines +1217 to +1232

This method raises NoOverwriteException when force=False and the tarball or
spec.json file already exist in the buildcache.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a fault of this PR, but as we are at it, can we add a description of each argument in the docstring (and type-hints)?

scottwittenburg and others added 4 commits April 28, 2023 16:19
Also refactor so that it is clear that the ci module is only ever going
to push a single packaged spec to the mirror.
@haampie haampie force-pushed the print-msg-for-each-pushed-buildcache branch from 5a4b1ea to f297ac0 Compare April 28, 2023 14:21
@alalazo
Copy link
Copy Markdown
Member

alalazo commented May 2, 2023

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented May 2, 2023

I encountered an error attempting to run the pipeline.

Details
Error: <class 'aiohttp.client_exceptions.ContentTypeError'>, 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://gitlab.spack.io/api/v4/projects/2/repository/commits/pr36141_print-msg-for-each-pushed-buildcache')
Stack trace:
  File "/usr/local/lib/python3.7/site-packages/rq/worker.py", line 1359, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 1178, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 1218, in _execute
    coro_result = loop.run_until_complete(result)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "./spackbot/workers.py", line 197, in run_pipeline_task
    if not await check_gitlab_has_latest(branch, head_sha, gh, comments_url):
  File "./spackbot/workers.py", line 77, in check_gitlab_has_latest
    gitlab_commit = await helpers.get(commit_url, headers)
  File "./spackbot/helpers.py", line 202, in get
    return await response.json()
  File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 1110, in json
    headers=self.headers,

@alalazo
Copy link
Copy Markdown
Member

alalazo commented May 2, 2023

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented May 2, 2023

I've started that pipeline for you!

@alalazo
Copy link
Copy Markdown
Member

alalazo commented May 2, 2023

@haampie Feel free to merge, if this seems ready. All the points in my previous review seem addressed, so ✔️

@haampie haampie merged commit c7e60f4 into spack:develop May 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

binary-packages commands core PR affects Spack core functionality new-variant tests General test capability(ies) update-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants