Adding script to build and publish docker image #4200

xiangfu0 · 2019-05-10T18:33:41Z

Adding docker support to build image.

Jackie-Jiang

You need to add license header for these files

docker/Dockerfile

Jackie-Jiang · 2019-05-10T20:34:15Z

docker/Dockerfile

+ARG PINOT_BRANCH=master
+ARG PINOT_GIT_URL="https://github.com/apache/incubator-pinot.git"
+RUN echo "Trying to build Pinot from [ ${PINOT_GIT_URL} ] on branch [ ${PINOT_BRANCH} ]"
+ENV PINOT_HOME=/opt/pinot


Just try to understand, can we pass these through environment variables?

This is image path, it could be passed from outside, but not really necessary.

docker/Dockerfile

Jackie-Jiang · 2019-05-10T20:36:54Z

docker/Dockerfile

+
+VOLUME ["${PINOT_HOME}/configs", "${PINOT_HOME}/data"]
+
+EXPOSE 9000 8099 8098 8097 8096 9514


What does these numbers stand for? Maybe add some comments?

Those are default ports for controller/broker/server/admin.

Jackie-Jiang · 2019-05-10T20:38:07Z

docker/README.md

+docker inspect pinot-zookeeper|grep IPAddress
+            "SecondaryIPAddresses": null,
+            "IPAddress": "172.17.0.2",
+                    "IPAddress": "172.17.0.2",


Duplicate line? Also, extra ,?

Not really, this field appears also inside the nested field:
Below is a sample example of NetworkSettings

"NetworkSettings": { "Bridge": "", "SandboxID": "23406af2ceaa015751faab22e383216e173c111073510c0134aa8d8189acce47", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": { "2181/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "2191" } ], "2888/tcp": null, "3888/tcp": null }, "SandboxKey": "/var/run/docker/netns/23406af2ceaa", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "72e7e3628415fcf12c972ac5e8ceaa5536cfdb6d90df6f3b13f5f05bcdd36628", "Gateway": "172.17.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "172.17.0.3", "IPPrefixLen": 16, "IPv6Gateway": "", "MacAddress": "02:42:ac:11:00:03", "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "b6b2f7c3d4c82fd85370d998f0de3d6a517f0350c0495169a1d7751968ee24bb", "EndpointID": "72e7e3628415fcf12c972ac5e8ceaa5536cfdb6d90df6f3b13f5f05bcdd36628", "Gateway": "172.17.0.1", "IPAddress": "172.17.0.3", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "02:42:ac:11:00:03", "DriverOpts": null } } }

docker/docker-build-and-push.sh

docker/docker-build.sh

snleee · 2019-05-11T03:07:32Z

Where do we publish the docker image? Apache Incubator had the restriction on releasing non-apache approved releases to public. The policy seems that as long as we publish the docker image created from the official apache releases (e.g. 0.1.0), it's fine. But, it's NOT ok to publish any arbitrary version that did not go through the voting process.

Some guideline from the general@incubator for docker is the following.

Docker

Artefacts need to be placed in https://hub.docker.com/r/apache/<project> or https://hub.docker.com/u/apache<project>/<project>

To comply with ASF release and distributions please ensue the following:
- The overview should include the incubator disclaimer.
- The docker file should include an ASF header.
- The docker file should include the incubator disclaimer.
- "docker pull apache/<project>" should not install an artefact containing unapproved code.
- Release candidates, nightly or snapshots need to be clearly tagged.
- The latest tag should not point to an artefact containing unapproved code e.g. to master or dev branches or to a RC or snapshot.

Some related discussions
https://lists.apache.org/thread.html/9cdc55e46a6fe23e1c0fcbdee6032f4949572311b0ded10cf6353be0@%3Cgeneral.incubator.apache.org%3E

https://issues.apache.org/jira/browse/LEGAL-270
https://issues.apache.org/jira/browse/LEGAL-427

kishoreg · 2019-05-16T03:29:06Z

@snleee, thanks for the pointers. I agree that we should all published releases under https://hub.docker.com/r/apache/ must be based on the released code. But individuals are free to take this and publish it under a different repo right?

I am assuming we will have an instruction page such as https://zeppelin.apache.org/docs/0.7.0/install/docker.html

codecov-io · 2019-05-27T06:36:01Z

Codecov Report

Merging #4200 into master will increase coverage by 1.79%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #4200      +/-   ##
============================================
+ Coverage     65.42%   67.21%   +1.79%     
  Complexity       20       20              
============================================
  Files          1052     1041      -11     
  Lines         54590    51518    -3072     
  Branches       7771     7217     -554     
============================================
- Hits          35714    34628    -1086     
+ Misses        16319    14521    -1798     
+ Partials       2557     2369     -188

Impacted Files	Coverage Δ	Complexity Δ
...apache/pinot/common/metrics/ValidationMetrics.java	`20.28% <0%> (-55.06%)`	`0% <0%> (ø)`
...ller/validation/OfflineSegmentIntervalChecker.java	`32.46% <0%> (-40.61%)`	`0% <0%> (ø)`
...egation/function/customobject/MinMaxRangePair.java	`75.86% <0%> (-24.14%)`	`0% <0%> (ø)`
...ot/controller/helix/core/util/HelixSetupUtils.java	`62.31% <0%> (-20.59%)`	`0% <0%> (ø)`
...ot/pql/parsers/pql2/ast/IntegerLiteralAstNode.java	`66.66% <0%> (-16.67%)`	`0% <0%> (ø)`
.../parsers/pql2/ast/FloatingPointLiteralAstNode.java	`66.66% <0%> (-16.67%)`	`0% <0%> (ø)`
...impl/dictionary/DoubleOnHeapMutableDictionary.java	`69.56% <0%> (-13.05%)`	`0% <0%> (ø)`
...e/impl/dictionary/LongOnHeapMutableDictionary.java	`82.6% <0%> (-13.05%)`	`0% <0%> (ø)`
.../pinot/controller/ControllerLeadershipManager.java	`74.41% <0%> (-9.31%)`	`0% <0%> (ø)`
.../impl/dictionary/FloatOnHeapMutableDictionary.java	`82.6% <0%> (-8.7%)`	`0% <0%> (ø)`
... and 155 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f3fe4e...ba8dd0d. Read the comment docs.

xiangfu0 · 2019-05-29T04:39:30Z

I think we should maintain an official Pinot dockerhub account and make docker image publish as part of the release process.

This script could also be a reference for pinot users to publish their own images.

s-sahay · 2019-06-12T20:01:20Z

I think we should maintain an official Pinot dockerhub account and make docker image publish as part of the release process.

This script could also be a reference for pinot users to publish their own images.

+1 on making it part of our release process. I'll create an issue for this to be tackled in our 0.3 release.

snleee

LGTM otherwise

snleee · 2019-06-12T20:01:16Z

docker/docker-build-and-push.sh

+  DOCKER_TAG=test
+fi
+echo "Trying to push docker image to ${DOCKER_TAG}"
+docker push ${DOCKER_TAG}


Which repository this will push? As long as this won't push to the official apache/pinot for docker hub, I'm fine with this pr.

snleee · 2019-06-12T20:43:10Z

Synced with @fx19880617 offline. Running the script will be no-op (permission denied). Users will need to set their custom tag for publishing to their own repository.

For the official apache release, we will need to set up apache/pinot repository and publish the image to the repository as part of the release process.

The push refers to repository [docker.io/library/pinot]
b7c6cccc78d8: Preparing
6f2fe2187e57: Preparing
604345bc37d0: Preparing
e8f1623f0c2c: Preparing
e3b08ec2022d: Preparing
4230ff7f2288: Waiting
2c719774c1e1: Waiting
ec62f19bb3aa: Waiting
f94641f1fe1f: Waiting
denied: requested access to the resource is denied

Adding script to build and publish docker image (#2)

b1dd9e3

xiangfu0 requested review from Jackie-Jiang and kishoreg May 10, 2019 18:33

Jackie-Jiang reviewed May 10, 2019

View reviewed changes

xiangfu0 added 2 commits May 26, 2019 22:53

Address comments

071d7d1

Adding license header

0b5d164

xiangfu0 force-pushed the docker_image branch from fe82cbf to 0b5d164 Compare May 27, 2019 06:01

xiangfu0 requested a review from Jackie-Jiang May 27, 2019 06:36

Adding thrift support in build image

6ed117b

snleee approved these changes Jun 12, 2019

View reviewed changes

s-sahay mentioned this pull request Jun 12, 2019

Publish Docker image as part of Apache Release #4310

Open

update readme

ba8dd0d

xiangfu0 force-pushed the docker_image branch from 6b069fb to ba8dd0d Compare June 12, 2019 21:42

xiangfu0 merged commit 27ad26c into apache:master Jun 13, 2019

xiangfu0 deleted the docker_image branch June 13, 2019 17:09


		VOLUME ["${PINOT_HOME}/configs", "${PINOT_HOME}/data"]

		EXPOSE 9000 8099 8098 8097 8096 9514

Adding script to build and publish docker image #4200

Adding script to build and publish docker image #4200

Uh oh!

Conversation

xiangfu0 commented May 10, 2019

Uh oh!

Jackie-Jiang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Jackie-Jiang May 10, 2019

Choose a reason for hiding this comment

Uh oh!

xiangfu0 May 27, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Jackie-Jiang May 10, 2019

Choose a reason for hiding this comment

Uh oh!

xiangfu0 May 27, 2019

Choose a reason for hiding this comment

Uh oh!

Jackie-Jiang May 10, 2019

Choose a reason for hiding this comment

Uh oh!

xiangfu0 May 27, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

snleee commented May 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kishoreg commented May 16, 2019

Uh oh!

codecov-io commented May 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

xiangfu0 commented May 29, 2019

Uh oh!

s-sahay commented Jun 12, 2019

Uh oh!

snleee left a comment

Choose a reason for hiding this comment

Uh oh!

snleee Jun 12, 2019

Choose a reason for hiding this comment

Uh oh!

snleee commented Jun 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

snleee commented May 11, 2019 •

edited

Loading

codecov-io commented May 27, 2019 •

edited

Loading

snleee commented Jun 12, 2019 •

edited

Loading