Added support for maximum replicas per node by olljanat · Pull Request #37940 · moby/moby

olljanat · 2018-09-30T15:29:38Z

- What I did
Added support for maximum replicas per node

Second step to be able solve #26259

- How I did it
Addes support for Placement parameter MaxReplicas

TODO:

Get Added support for maximum replicas per node swarmkit#2758 merged.
Wait that mount: add BindOptions.NonRecursive (API v1.40) #38003 is merged.

kolyshkin · 2018-10-01T21:59:46Z

build fails:

00:07:31.935 Building: bundles/binary-daemon/dockerd-dev
00:08:09.664 # github.com/docker/docker/daemon/cluster/convert
00:08:09.664 daemon/cluster/convert/service.go:249:4: unknown field 'Maxreplicas' in struct literal of type api.Placement

olljanat · 2018-10-02T03:53:05Z

Sure it will until swarmkit PR have been merged that why this is marked as WIP now.

Test automation set on Moby is quite complex and I'm not able run all these tests on my dev env so that why I prefer to have PR open here which can run those tests for me.

Next steps are:

Do requested changes swarmkit PR
Make this PR match with that one.
Temporarily add commit which takes swarmkit from my repo

After these build and all these tests (should) pass.

codecov · 2018-10-02T19:00:07Z

Codecov Report

❗ No coverage uploaded for pull request base (master@d147fe0). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master   #37940   +/-   ##
=========================================
  Coverage          ?    36.6%           
=========================================
  Files             ?      608           
  Lines             ?    44990           
  Branches          ?        0           
=========================================
  Hits              ?    16467           
  Misses            ?    26245           
  Partials          ?     2278

olljanat · 2018-10-17T20:38:56Z

Swarmkit PR was merged so I included swarmkit bump to this.

Builds should pass as they was earlier when I tested to include swarmkit changes from my repo.

@cpuguy83 can you review ?

EDIT: Swarmkit bump looks to be also on #38056 so I will drop it from this when it is merged.

EDIT 2: #38056 was merged. Failing test on experimental looks to be flaky ( #32673 ).

thaJeztah

just some minor comments, but I want to give this a spin (haven't had time for that yet).

One thing I'd like to test is the behavior if the --max-replicas-per-node value is updated for an existing service (e.g. change from max 3 replicas to max 2; are the existing tasks properly killed and/or re-scheduled in that case?)

daemon/cluster/convert/service.go

api/swagger.yaml

olljanat · 2018-10-18T21:01:31Z

One thing I'd like to test is the behavior if the --max-replicas-per-node value is updated for an existing service (e.g. change from max 3 replicas to max 2; are the existing tasks properly killed and/or re-scheduled in that case?)

Currently it looks working on way that if user only updates --max-replicas-per-node (or actually it is --replicas-max-per-node so it shorts nicely on cli help) then only service spec will be updated and extra tasks will be killed on next service update. If user also updates --image value or uses --force switch then extra one will be killed immediately (as all tasks are re-scheduled).

If we need change that logic then it would mean one more change to swarmkit.

Anyway, I will wait that #38003 is merged and implement these API versions checks on meanwhile.

olljanat · 2018-10-27T08:03:17Z

Temporarily included commit from #38003 will cleanup after it is merged.

olljanat · 2018-11-07T15:03:12Z

@thaJeztah rebased on master and included changes you requested. Can you review again?

integration/service/create_test.go

olljanat · 2018-12-21T11:38:49Z

@cpuguy83 @thaJeztah PTAL, now this contains only simple API test without logic test as it is on swarmkit side.

cpuguy83

LGTM

thaJeztah

Left some minor nits inline (no critical issues, but would be nice to have them in the PR)

I was finally able to give this a spin (see below for some playing with the new option); one thing I noticed (also, see below) that reducing the max-replicas value does not terminate existing tasks. I wondered if this was by design, or an oversight? Not a show-stopper; just curious 🤗

Here's some things I tried:

Create a service with 2 replicas, but maxreplicas 1

curl -s \
  --unix-socket /var/run/docker.sock \
  -X POST \
  "http://localhost/v1.40/services/create" \
  -H "Content-Type: application/json" \
  -d '{"EndpointSpec":{"Mode":"vip"},"Labels":{},"Mode":{"Replicated":{"Replicas":2}},"Name":"test","TaskTemplate":{"ContainerSpec":{"DNSConfig":{},"Image":"nginx:alpine@sha256:2e497c294e3ba84aaeab7a0fbb1027819cd1f5f5892ed3c4a82b8b05010090da"},"ForceUpdate":0,"Placement":{"MaxReplicas":1,"Platforms":[{"Architecture":"amd64","OS":"linux"},{"Architecture":"arm","OS":"linux"},{"Architecture":"arm64","OS":"linux"},{"Architecture":"386","OS":"linux"},{"Architecture":"ppc64le","OS":"linux"},{"Architecture":"s390x","OS":"linux"}]},"Resources":{"Limits":{},"Reservations":{}}}}'

{"ID":"eln7vce863tedwd5q72kjaxkg"}

Verify the MaxReplicas configuration is stored:

curl -s \
  --unix-socket /var/run/docker.sock \
  "http://localhost/v1.40/services/test?insertDefaults=true" | jq .Spec.TaskTemplate.Placement.MaxReplicas

1

And check that no more than 1 replica is running (on a single-node swarm);

docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
qy2qc9p1yder        test                replicated          1/2                 nginx:alpine

Check the task, and confirm it's unable to start due to the MaxReplicas restriction:

docker service ps test --no-trunc --format 'table {{.Name}}\t{{.Error}}'
NAME                ERROR
test.1              
test.2              "no suitable node (max replicas per node limit exceed)"

Updated MaxReplicas to 2;

Get the service definition

curl -s \
  --unix-socket /var/run/docker.sock \
  "http://localhost/v1.40/services/test"

{"ID":"cqnbsmr9foblwucehzu9ekrpb","Version":{"Index":87},"CreatedAt":"2018-12-23T22:23:22.415002133Z","UpdatedAt":"2018-12-23T22:23:22.415002133Z","Spec":{"Name":"test","Labels":{},"TaskTemplate":{"ContainerSpec":{"Image":"nginx:alpine@sha256:2e497c294e3ba84aaeab7a0fbb1027819cd1f5f5892ed3c4a82b8b05010090da","DNSConfig":{},"Isolation":"default"},"Resources":{"Limits":{},"Reservations":{}},"Placement":{"MaxReplicas":1,"Platforms":[{"Architecture":"amd64","OS":"linux"},{"Architecture":"arm","OS":"linux"},{"Architecture":"arm64","OS":"linux"},{"Architecture":"386","OS":"linux"},{"Architecture":"ppc64le","OS":"linux"},{"Architecture":"s390x","OS":"linux"}]},"ForceUpdate":0,"Runtime":"container"},"Mode":{"Replicated":{"Replicas":2}},"EndpointSpec":{"Mode":"vip"}},"Endpoint":{"Spec":{}}}

Update the service to set MaxReplicas to 2

curl -s \
  --unix-socket /var/run/docker.sock \
  -X POST \
  "http://localhost/v1.40/services/test/update?version=87" \
  -H "Content-Type: application/json" \
  -d '{"Name":"test","Labels":{},"TaskTemplate":{"ContainerSpec":{"Image":"nginx:alpine@sha256:2e497c294e3ba84aaeab7a0fbb1027819cd1f5f5892ed3c4a82b8b05010090da","DNSConfig":{},"Isolation":"default"},"Resources":{"Limits":{},"Reservations":{}},"Placement":{"MaxReplicas":2,"Platforms":[{"Architecture":"amd64","OS":"linux"},{"Architecture":"arm","OS":"linux"},{"Architecture":"arm64","OS":"linux"},{"Architecture":"386","OS":"linux"},{"Architecture":"ppc64le","OS":"linux"},{"Architecture":"s390x","OS":"linux"}]},"ForceUpdate":0,"Runtime":"container"},"Mode":{"Replicated":{"Replicas":2}},"EndpointSpec":{"Mode":"vip"}}'

Verify the service was updated with the new value (2);

curl -s \
  --unix-socket /var/run/docker.sock \
  "http://localhost/v1.40/services/test" | jq .Spec.TaskTemplate.Placement.MaxReplicas

2

And confirm that the service was now able to deploy 2 replicas:

docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
cqnbsmr9fobl        test                replicated          2/2                 nginx:alpine

Now update the service again, and set MaxReplicas back to 1

curl -s \
  --unix-socket /var/run/docker.sock \
  -X POST \
  "http://localhost/v1.40/services/test/update?version=172" \
  -H "Content-Type: application/json" \
  -d '{"Name":"test","Labels":{},"TaskTemplate":{"ContainerSpec":{"Image":"nginx:alpine@sha256:2e497c294e3ba84aaeab7a0fbb1027819cd1f5f5892ed3c4a82b8b05010090da","DNSConfig":{},"Isolation":"default"},"Resources":{"Limits":{},"Reservations":{}},"Placement":{"MaxReplicas":1,"Platforms":[{"Architecture":"amd64","OS":"linux"},{"Architecture":"arm","OS":"linux"},{"Architecture":"arm64","OS":"linux"},{"Architecture":"386","OS":"linux"},{"Architecture":"ppc64le","OS":"linux"},{"Architecture":"s390x","OS":"linux"}]},"ForceUpdate":0,"Runtime":"container"},"Mode":{"Replicated":{"Replicas":2}},"EndpointSpec":{"Mode":"vip"}}'

Verify the service was updated with the new value (1)

curl -s \
  --unix-socket /var/run/docker.sock \
  "http://localhost/v1.40/services/test" | jq .Spec.TaskTemplate.Placement.MaxReplicas

1

Note that when reducing the maximum number of replicas, existing tasks are not terminated;

docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
cqnbsmr9fobl        test                replicated          2/2                 nginx:alpine

Another thing that needs to be taken into account is that scaling/updating a service with an older version of the CLI will strip the MaxReplicas. That's the expected behavior, but perhaps something we should mention somewhere in the documentation :thinking_face:

docker service scale test=3
test scaled to 3

After scaling, the MaxReplicas property is reset (because the CLI does not know the new property);

curl -s \
  --unix-socket /var/run/docker.sock \
  "http://localhost/v1.40/services/test?insertDefaults=true" | jq .Spec.TaskTemplate.Placement.MaxReplicas

null

api/server/router/swarm/cluster_routes.go

docs/api/version-history.md

api/swagger.yaml

olljanat · 2018-12-23T23:26:49Z

one thing I noticed (also, see below) that reducing the max-replicas value does not terminate existing tasks. I wondered if this was by design, or an oversight? Not a show-stopper; just curious 🤗

By design as we don't want kill running containers when it is not absolutely necessary and user have option to use --force (not sure what it is on API side) and then all tasks will be re-scheduled and situation fixed.

Signed-off-by: Olli Janatuinen <[email protected]>

olljanat · 2018-12-24T00:06:03Z

@thaJeztah updated based on review. Now hopefully ready to be merged?

thaJeztah · 2018-12-24T09:45:26Z

By design as we don't want kill running containers when it is not absolutely necessary and user have option to use --force (not sure what it is on API side) and then all tasks will be re-scheduled and situation fixed.

Gotcha; makes sense. We'll have to document that, and explain the behavior (i.e., if they want to force a reschedule, use the --force)

Trying that scenario:

Create a service with MaxReplicas: 2, and Replicas: 2

curl -s \
  --unix-socket /var/run/docker.sock \
  -X POST \
  "http://localhost/v1.40/services/create" \
  -H "Content-Type: application/json" \
  -d '{"EndpointSpec":{"Mode":"vip"},"Labels":{},"Mode":{"Replicated":{"Replicas":2}},"Name":"test","TaskTemplate":{"ContainerSpec":{"DNSConfig":{},"Image":"nginx:alpine@sha256:2e497c294e3ba84aaeab7a0fbb1027819cd1f5f5892ed3c4a82b8b05010090da"},"ForceUpdate":0,"Placement":{"MaxReplicas":2,"Platforms":[{"Architecture":"amd64","OS":"linux"},{"Architecture":"arm","OS":"linux"},{"Architecture":"arm64","OS":"linux"},{"Architecture":"386","OS":"linux"},{"Architecture":"ppc64le","OS":"linux"},{"Architecture":"s390x","OS":"linux"}]},"Resources":{"Limits":{},"Reservations":{}}}}'

{"ID":"kdu0l3uxhk6p52xa719581h0h"}

Verify the MaxReplicas configuration is stored:

curl -s \
  --unix-socket /var/run/docker.sock \
  "http://localhost/v1.40/services/test" | jq .Spec.TaskTemplate.Placement

2

Confirm the service has two tasks running:

docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
rmg5iwvxl158        test                replicated          2/2                 nginx:alpine

Update MaxReplicas to 2, and set ForceUpdate to a new value (to force the update);

Get the service definition

curl -s \
  --unix-socket /var/run/docker.sock \
  "http://localhost/v1.40/services/test"

{"ID":"kdu0l3uxhk6p52xa719581h0h","Version":{"Index":27},"CreatedAt":"2018-12-24T09:37:12.633108733Z","UpdatedAt":"2018-12-24T09:37:12.633108733Z","Spec":{"Name":"test","Labels":{},"TaskTemplate":{"ContainerSpec":{"Image":"nginx:alpine@sha256:2e497c294e3ba84aaeab7a0fbb1027819cd1f5f5892ed3c4a82b8b05010090da","DNSConfig":{},"Isolation":"default"},"Resources":{"Limits":{},"Reservations":{}},"Placement":{"MaxReplicas":2,"Platforms":[{"Architecture":"amd64","OS":"linux"},{"Architecture":"arm","OS":"linux"},{"Architecture":"arm64","OS":"linux"},{"Architecture":"386","OS":"linux"},{"Architecture":"ppc64le","OS":"linux"},{"Architecture":"s390x","OS":"linux"}]},"ForceUpdate":0,"Runtime":"container"},"Mode":{"Replicated":{"Replicas":2}},"EndpointSpec":{"Mode":"vip"}},"Endpoint":{"Spec":{}}}

Update the service, set MaxReplicas to 1, and set ForceUpdate to a new value (to force the update);

curl -s \
  --unix-socket /var/run/docker.sock \
  -X POST \
  "http://localhost/v1.40/services/test/update?version=27" \
  -H "Content-Type: application/json" \
  -d '{"Name":"test","Labels":{},"TaskTemplate":{"ContainerSpec":{"Image":"nginx:alpine@sha256:2e497c294e3ba84aaeab7a0fbb1027819cd1f5f5892ed3c4a82b8b05010090da","DNSConfig":{},"Isolation":"default"},"Resources":{"Limits":{},"Reservations":{}},"Placement":{"MaxReplicas":1,"Platforms":[{"Architecture":"amd64","OS":"linux"},{"Architecture":"arm","OS":"linux"},{"Architecture":"arm64","OS":"linux"},{"Architecture":"386","OS":"linux"},{"Architecture":"ppc64le","OS":"linux"},{"Architecture":"s390x","OS":"linux"}]},"ForceUpdate":1,"Runtime":"container"},"Mode":{"Replicated":{"Replicas":2}},"EndpointSpec":{"Mode":"vip"}}'

{"Warnings":null}

Confirm that the service now has 1 task running:

docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
kdu0l3uxhk6p        test                replicated          1/2                 nginx:alpine

And that this is due to the new MaxReplicas limit:

docker service ps test --no-trunc --format 'table {{.Name}}\t{{.Error}}'
NAME                ERROR
test.1              
test.2              "no suitable node (max replicas per node limit exceed)"
 \_ test.2

Looking good 👍

thaJeztah

LGTM, thanks!!!

thaJeztah · 2018-12-24T17:54:20Z

@olljanat was the failure only a flaky test? If so, I'm happy to ignore PowerPC and merge

olljanat · 2018-12-24T18:17:40Z

@thaJeztah yes. Those which I disabled on #38423 looks causing issues.

thaJeztah · 2018-12-24T18:29:49Z

Alright let's get this one merged 👍

vpotseluyko · 2019-02-13T17:02:26Z

Hi! I greatly love this commit. With what docker & swarm-kit version could it be actually used at production

thaJeztah · 2019-02-13T17:13:04Z

It's not released yet; will be in the next docker release (19.03) (but you should be able to give it a spin with the nightly builds https://download.docker.com/linux/ubuntu/dists/bionic/pool/nightly/amd64/

GordonTheTurtle added the status/0-triage label Sep 30, 2018

olljanat force-pushed the replicas-max-per-node branch 2 times, most recently from 68f5387 to 8b32d49 Compare October 2, 2018 19:00

olljanat mentioned this pull request Oct 2, 2018

Add maximum replicas per node support to stack version 3.8 docker/cli#1410

Merged

2 tasks

olljanat force-pushed the replicas-max-per-node branch 2 times, most recently from 22d5ad2 to 4c6a698 Compare October 4, 2018 20:58

GordonTheTurtle assigned vieux Oct 15, 2018

olljanat force-pushed the replicas-max-per-node branch from 4c6a698 to 7584627 Compare October 17, 2018 18:45

olljanat changed the title ~~WIP: Added support for maximum replicas per node~~ Added support for maximum replicas per node Oct 17, 2018

thaJeztah added impact/api impact/changelog area/swarm labels Oct 18, 2018

olljanat force-pushed the replicas-max-per-node branch 2 times, most recently from de44ba4 to f422e25 Compare October 18, 2018 14:37

thaJeztah requested changes Oct 18, 2018

View reviewed changes

daemon/cluster/convert/service.go Outdated Show resolved Hide resolved

api/swagger.yaml Outdated Show resolved Hide resolved

olljanat mentioned this pull request Oct 22, 2018

mount: add BindOptions.NonRecursive (API v1.40) #38003

Merged

olljanat force-pushed the replicas-max-per-node branch from f422e25 to 261b3d6 Compare October 27, 2018 07:55

olljanat requested a review from vdemeester as a code owner October 27, 2018 07:55

olljanat force-pushed the replicas-max-per-node branch from 261b3d6 to 169e690 Compare October 27, 2018 07:57

olljanat mentioned this pull request Oct 29, 2018

Add --max-replicas-per-node for docker service create #26259

Closed

olljanat force-pushed the replicas-max-per-node branch from 169e690 to 8ac1832 Compare November 7, 2018 14:34

olljanat force-pushed the replicas-max-per-node branch 2 times, most recently from f24d569 to 298228b Compare November 9, 2018 18:45

cpuguy83 reviewed Dec 20, 2018

View reviewed changes

integration/service/create_test.go Outdated Show resolved Hide resolved

olljanat force-pushed the replicas-max-per-node branch from 7bca461 to 92a4d55 Compare December 20, 2018 19:08

cpuguy83 approved these changes Dec 21, 2018

View reviewed changes

thaJeztah requested changes Dec 23, 2018

View reviewed changes

api/server/router/swarm/cluster_routes.go Outdated Show resolved Hide resolved

docs/api/version-history.md Outdated Show resolved Hide resolved

api/swagger.yaml Outdated Show resolved Hide resolved

api/swagger.yaml Outdated Show resolved Hide resolved

olljanat force-pushed the replicas-max-per-node branch from 92a4d55 to 6e86627 Compare December 24, 2018 00:00

Added support for maximum replicas per node to services

153171e

Signed-off-by: Olli Janatuinen <[email protected]>

olljanat force-pushed the replicas-max-per-node branch from 6e86627 to 153171e Compare December 24, 2018 00:04

derek bot added the rebuild/* label Dec 24, 2018

GordonTheTurtle removed the rebuild/* label Dec 24, 2018

thaJeztah approved these changes Dec 24, 2018

View reviewed changes

thaJeztah added status/4-merge and removed status/2-code-review labels Dec 24, 2018

derek bot added the rebuild/powerpc label Dec 24, 2018

GordonTheTurtle removed the rebuild/powerpc label Dec 24, 2018

derek bot added the rebuild/powerpc label Dec 24, 2018

GordonTheTurtle removed the rebuild/powerpc label Dec 24, 2018

derek bot added the rebuild/powerpc label Dec 24, 2018

GordonTheTurtle removed the rebuild/powerpc label Dec 24, 2018

thaJeztah merged commit 8fbf259 into moby:master Dec 24, 2018

olljanat deleted the replicas-max-per-node branch December 25, 2018 04:00

Drewster727 mentioned this pull request Aug 5, 2019

add npipe support to docker_swarm_service ansible/ansible#60073

Merged

Conversation

olljanat commented Sep 30, 2018 • edited by thaJeztah Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kolyshkin commented Oct 1, 2018

Uh oh!

olljanat commented Oct 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

olljanat commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

olljanat commented Oct 18, 2018

Uh oh!

olljanat commented Oct 27, 2018

Uh oh!

olljanat commented Nov 7, 2018

Uh oh!

Uh oh!

olljanat commented Dec 21, 2018

Uh oh!

cpuguy83 left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

olljanat commented Dec 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

olljanat commented Dec 24, 2018

Uh oh!

thaJeztah commented Dec 24, 2018

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Dec 24, 2018

Uh oh!

olljanat commented Dec 24, 2018

Uh oh!

thaJeztah commented Dec 24, 2018

Uh oh!

vpotseluyko commented Feb 13, 2019

Uh oh!

thaJeztah commented Feb 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

olljanat commented Sep 30, 2018 •

edited by thaJeztah

Loading

olljanat commented Oct 2, 2018 •

edited

Loading

codecov bot commented Oct 2, 2018 •

edited

Loading

olljanat commented Oct 17, 2018 •

edited

Loading

olljanat commented Dec 23, 2018 •

edited

Loading

thaJeztah commented Feb 13, 2019 •

edited

Loading