Skip to content

Bump SwarmKit to 1a0ebd43b2d156983a695f90e56f4ecba6ced902#38520

Merged
thaJeztah merged 2 commits intomoby:masterfrom
thaJeztah:fix_update_status_check
Feb 1, 2019
Merged

Bump SwarmKit to 1a0ebd43b2d156983a695f90e56f4ecba6ced902#38520
thaJeztah merged 2 commits intomoby:masterfrom
thaJeztah:fix_update_status_check

Conversation

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jan 9, 2019

moby/swarmkit@8af8c42...1a0ebd4

relevant changes:

See description below for the original PR description (before swarmkit bump)

In situations where only annotations of a service were updated, SwarmKit never marks a service as UpdateStateUpdating; the service's UpdateStatus is set to nil, and never updated after.

Because of this, it's not possible to verify if the update actually happened (unless comparing the services Version before and after).

This patch modifies the SwarmKit vendored code to try to make this situation non-ambigious, and mark the services as "updated" in all cases.

Relates to #38499 (comment) :

Looking at that code; #38499 (comment)

As part of the Service update, controlapi.UpdateService() sets Updatestatus to nil to indicate that an update is in progress; after that the updated service is written to the store (store.UpdateService()), which updates the UpdatedAt and Version.

Updater.Run() checks if any tasks are "dirty" and need updating; https://github.com/docker/swarmkit/blob/486ad69951868792367a503042ef3f55b399cc40/manager/orchestrator/update/updater.go#L137-L141

	var dirtySlots []orchestrator.Slot
	for _, slot := range slots {
		if u.isSlotDirty(slot) {
			dirtySlots = append(dirtySlots, slot)
		}
	}

Some changes on a service don't require updates to the task (container); to check if updates are needed, these functions are called:

If changes are needed, len(dirtySlots) > 0, in which case and update is started (if no update is in progress yet). From that part of the code, it's clear that service.UpdateStatus == nil is required to even consider starting an update. Updater.startUpdate() is called, which sets the service's UpdateStatus to UpdateStatus_UPDATING, and sets the UpdateStatus.StartedAt to the current time.

However, in case of the service-label update, no updates to the tasks are needed, so len(dirtySlots) == 0; in that case, no action is needed, given that no update (or "rollback") was ever started for this. In that case, the Service.UpdateStatus is not updated, so remains nil

	// Abort immediately if all tasks are clean.
	if len(dirtySlots) == 0 {
		if service.UpdateStatus != nil &&
			(service.UpdateStatus.State == api.UpdateStatus_UPDATING ||
				service.UpdateStatus.State == api.UpdateStatus_ROLLBACK_STARTED) {
			u.completeUpdate(ctx, service.ID)
		}
		return
	}

Even if we would skip the check (if an updated or rollback was started), update.completeUpdate() checks service.UpdateStatus. If service.UpdateStatus == nil, it assumes that another update was started, so the service.UpdateStatus is not updated;

	if service.UpdateStatus == nil {
		// The service was changed since we started this update
		return nil
	}

Based on the above, it looks like service.UpdateStatus == nil is ambiguous; as it can either be;

  • An update was triggered, but no action was required (in which case it's equal to UpdateStatus_COMPLETED or UpdateStatus_ROLLBACK_COMPLETED)
  • An update was triggered, but the actual update was not yet started

I think we can make it non-ambiguous if we set the status to UpdateStatus_COMPLETED if there are no tasks to update (len(dirtySlots) == 0); this requires a change to Updater.Run() (change the check), and to update.completeUpdate()

@thaJeztah
Copy link
Member Author

Let's see if this works; updated the vendored code in this PR just to give it a quick try 😅

@thaJeztah thaJeztah force-pushed the fix_update_status_check branch from ad96b5a to b3b739a Compare January 9, 2019 13:58
@codecov
Copy link

codecov bot commented Jan 9, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@87903f2). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #38520   +/-   ##
=========================================
  Coverage          ?    36.6%           
=========================================
  Files             ?      610           
  Lines             ?    45233           
  Branches          ?        0           
=========================================
  Hits              ?    16558           
  Misses            ?    26392           
  Partials          ?     2283

@thaJeztah thaJeztah force-pushed the fix_update_status_check branch from b3b739a to cf9f75e Compare January 9, 2019 14:00
@thaJeztah thaJeztah changed the title [WIP] try make Service.UpdateStatus non-ambigious [WIP] try make Service.UpdateStatus non-ambiguous Jan 9, 2019
@olljanat
Copy link
Contributor

olljanat commented Jan 9, 2019

Tested with my new tool #38523 works (renamed test to trigger test). Passed nicely all 25 tries. Log from that test: 38520_test-integration-flaky.log

Looks good 👍

@thaJeztah
Copy link
Member Author

Opened an upstream PR in SwarmKit; moby/swarmkit#2804

Full diff: moby/swarmkit@8af8c42...1a0ebd4

relevant changes:

- swarmkit#2771 Allow using Configs as CredentialSpecs
- swarmkit#2804 Make Service.UpdateStatus non-ambiguous
- swarmkit#2805 Refactor condition in restart supervisor
- swarmkit#2780 api: add BindOptions.NonRecursive
  - related to moby#38003
- swarmkit#2790 Fix possible panic if NetworkConfig is nil
- swarmkit#2797 Include old error-message for backward compatibility
  - related to swarmkit#2779 / moby#38140 / moby#38142

Signed-off-by: Sebastiaan van Stijn <[email protected]>
It's no longer needed with the latest swarmkit changes

Signed-off-by: Sebastiaan van Stijn <[email protected]>
@thaJeztah thaJeztah force-pushed the fix_update_status_check branch from cf9f75e to 94429d4 Compare February 1, 2019 00:36
@thaJeztah thaJeztah changed the title [WIP] try make Service.UpdateStatus non-ambiguous Bump SwarmKit to 1a0ebd43b2d156983a695f90e56f4ecba6ced902 Feb 1, 2019
@thaJeztah
Copy link
Member Author

Ping @dperny - guess we need an updated description in the API reference for
moby/swarmkit#2771 ? Do we need a test for that?

ping @AkihiroSuda this brings in moby/swarmkit#2780 (did you prepare changes already for docker service create? Are changes needed?

Copy link
Member

@vdemeester vdemeester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐯

@AkihiroSuda
Copy link
Member

ping @AkihiroSuda this brings in moby/swarmkit#2780 (did you prepare changes already for docker service create? Are changes needed?

I can open follow-up PR

@thaJeztah
Copy link
Member Author

All green

@thaJeztah thaJeztah merged commit acf0853 into moby:master Feb 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants