CD: Package Index by ax3l · Pull Request #11665 · spack/spack

ax3l · 2019-06-07T22:11:42Z

Build the docker image for the new package index REST API on each merge/push to develop and master.

Follow-up to: #11652

Manual Build Instructions

Here is how to build the image for the service manually/locally.

spack list --format version_json > packages.json
docker build . -t spack/packages.spack.io:latest

docker run -p 8080:80 spack/packages.spack.io:latest

Now the server is started at http://localhost:8080

tgamblin

minor nitpick and one major request: can we not check packages.json into the Spack repo? The CI should just run it.

ax3l · 2019-06-07T22:19:31Z

removed it already, accidentally pushed

tgamblin · 2019-06-07T22:20:07Z

ok can we remove initial spaces from shebangs?

ax3l · 2019-06-07T22:21:46Z

huh! clumsy f1ngers

opadron

I'm guessing this docker image provides a listing of packages... available for installation from the built-in repo? Is that right?

In any case, this all looks OK to me. I might make a pass on some of this after further simplifying #11621, just to make sure all of our docker stuff is consistent.

ax3l · 2019-06-07T22:53:56Z

Wonderful! Yes, this provides a nginx image with static files for all packages as shown in #11652

ax3l · 2019-06-07T23:06:54Z

@opadron I already pushed a first image manually to spack/packages.spack.io. If you spin up packages.spack.io with it we can provide version badges to our users! :)
badges/shields#3536

opadron · 2019-06-08T00:35:08Z

@ax3l Done!

But, it looks like something isn't working quite right. Here's some log output that I see after hitting packages.spack.io a few times:

2019/06/08 00:30:44 [error] 7#7: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.118.41, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "packages.spack.io"
192.168.118.41 - - [08/Jun/2019:00:30:44 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0" "75.183.5.177"
192.168.118.41 - - [08/Jun/2019:00:31:16 +0000] "GET /.git/config HTTP/1.1" 404 153 "http://packages.spack.io/.git/config" "Go-http-client/1.1" "195.176.3.20"
2019/06/08 00:31:16 [error] 7#7: *2 open() "/usr/share/nginx/html/.git/config" failed (2: No such file or directory), client: 192.168.118.41, server: localhost, request: "GET /.git/config HTTP/1.1", host: "packages.spack.io", referrer: "http://packages.spack.io/.git/config"
192.168.118.41 - - [08/Jun/2019:00:31:40 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0" "75.183.5.177"
2019/06/08 00:31:40 [error] 7#7: *3 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.118.41, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "packages.spack.io"
2019/06/08 00:32:44 [error] 7#7: *4 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.118.41, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "packages.spack.io"
192.168.118.41 - - [08/Jun/2019:00:32:44 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0" "75.183.5.177"

opadron · 2019-06-08T00:41:57Z

Taking a closer look, it seems like there isn't supposed to be a landing page, and the default page was just left unmodified? I'm not familiar with the API this server is supposed to serve, but e.g.: packages.spack.io/z/zlib.json seems to work just fine.

ax3l · 2019-06-08T08:00:10Z

Yes, we didn't add one yet and consider adding a little search frontend later on, such as https://github.com/Homebrew/formulae.brew.sh

For the API itself a landing page is not needed, but will look more professional once we add one.
Note: Update to add an api/ prefix, so we can add a static page later on without potential collisions.

The package index API is as simple as:
https://packages.spack.io/api/:firstLetter/:packageName.json
which will 200 a JSON file for all valid packages (content from spack list in #11652) and 404 for all other package names.
e.g. https://packages.spack.io/api/a/adios2.json and https://packages.spack.io/api/p/py-pandas.json
There is also the full index available under https://packages.spack.io/api/packages.json

The corresponding shields.io API (badges/shields#3536) for badges/shields will then be as simple as:
https://img.shields.io/spack/v/:packageName.svg,
e.g. adios2: and py-pandas:

ax3l · 2019-06-08T09:16:48Z

@opadron I pushed the endpoint change to include api/ now but the service does not seam to automatically re-deploy with the updated image on dockerhub.

Build the docker image for the new package index REST API on each merge/push to develop and master.

ax3l · 2019-06-10T15:12:00Z

@opadron @tgamblin shall we merge this already?

tgamblin · 2019-06-10T15:36:59Z

@ax3l: it’s approved and I think you have merge rights :)

ax3l · 2019-06-10T16:21:51Z

Okay, I just wasn't sure if self-merge is fine :)

Build the docker image for the new package index REST API on each merge/push to develop and master.

ax3l added documentation Improvements or additions to documentation docker labels Jun 7, 2019

ax3l requested review from opadron and tgamblin June 7, 2019 22:11

ax3l force-pushed the topic-CDPackageIndex branch 2 times, most recently from 86fe733 to 7ad5cb3 Compare June 7, 2019 22:15

tgamblin requested changes Jun 7, 2019

View reviewed changes

ax3l force-pushed the topic-CDPackageIndex branch from 7ad5cb3 to 51c1e7a Compare June 7, 2019 22:21

ax3l requested a review from tgamblin June 7, 2019 22:31

ax3l mentioned this pull request Jun 7, 2019

[Spack] Package Manager: Latest Version badges/shields#3536

Merged

2 tasks

opadron approved these changes Jun 7, 2019

View reviewed changes

ax3l force-pushed the topic-CDPackageIndex branch from 51c1e7a to d65c778 Compare June 8, 2019 09:05

CD: Package Index

2a139fe

Build the docker image for the new package index REST API on each merge/push to develop and master.

ax3l force-pushed the topic-CDPackageIndex branch from d65c778 to 2a139fe Compare June 8, 2019 15:12

tgamblin approved these changes Jun 8, 2019

View reviewed changes

ax3l merged commit d4008db into spack:develop Jun 10, 2019

ax3l deleted the topic-CDPackageIndex branch June 10, 2019 16:21

carsonwoods pushed a commit to carsonwoods/spack that referenced this pull request Jun 27, 2019

CD: Package Index (spack#11665)

3c39389

Build the docker image for the new package index REST API on each merge/push to develop and master.

dev-zero pushed a commit to dev-zero/spack that referenced this pull request Aug 13, 2019

CD: Package Index (spack#11665)

ef54bc5

Build the docker image for the new package index REST API on each merge/push to develop and master.

ax3l mentioned this pull request Nov 20, 2019

Package Index: Build in Dockerhub #13810

Merged

2 tasks

Conversation

ax3l commented Jun 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Manual Build Instructions

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

ax3l commented Jun 7, 2019

Uh oh!

tgamblin commented Jun 7, 2019

Uh oh!

ax3l commented Jun 7, 2019

Uh oh!

opadron left a comment

Choose a reason for hiding this comment

Uh oh!

ax3l commented Jun 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ax3l commented Jun 7, 2019

Uh oh!

opadron commented Jun 8, 2019

Uh oh!

opadron commented Jun 8, 2019

Uh oh!

ax3l commented Jun 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ax3l commented Jun 8, 2019

Uh oh!

ax3l commented Jun 10, 2019

Uh oh!

tgamblin commented Jun 10, 2019

Uh oh!

ax3l commented Jun 10, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ax3l commented Jun 7, 2019 •

edited

Loading

ax3l commented Jun 7, 2019 •

edited

Loading

ax3l commented Jun 8, 2019 •

edited

Loading