Merged
Conversation
thaJeztah
added
area/daemon
thaJeztah
requested changes
May 19, 2021
Member
thaJeztah
left a comment
thaJeztah
left a comment
There was a problem hiding this comment.
thanks! left some comments / thoughts
cmd/dockerd/daemon.go
Outdated
Member
There was a problem hiding this comment.
mkdir /foo
echo '{' > /foo/malformed-config.json
echo '' > /foo/empty-config-1.json
echo '{}' > /foo/empty-config-2.json
echo '{"unknown-option": true}' > /foo/invalid-config-1.json
echo '{"debug": true}' > /foo/valid-config-1.jsonThen trying them;
for file in /foo/*.json; do sh -xc "dockerd --validate --config-file=${file}"; done
+ dockerd --validate --config-file=/foo/empty-config-1.json
unable to configure the Docker daemon with file /foo/empty-config-1.json: EOF
+ dockerd --validate --config-file=/foo/empty-config-2.json
INFO[2021-05-19T10:31:30.747037342Z] Config OK
+ dockerd --validate --config-file=/foo/invalid-config-1.json
unable to configure the Docker daemon with file /foo/invalid-config-1.json: the following directives don't match any configuration option: unknown-option
+ dockerd --validate --config-file=/foo/malformed-config.json
unable to configure the Docker daemon with file /foo/malformed-config.json: unexpected EOF
+ dockerd --validate --config-file=/foo/valid-config-1.json
INFO[2021-05-19T10:31:30.883917472Z] Config OKLooking at the output:
- For a follow-up: I think we should ignore the "empty" file. It's probably ok to ignore an empty
daemon.json(same as how we ignore empty json ({}) - Looking at it now; using
logrus.Infof()adds a lot of noise; I think we should just printConfig OK. This also allows the output to be more easily used in scripts (check if the output is "Config OK"
I think we should also have a simple integration test for this. I see there's a TestConfigDaemonLibtrustID test in
moby/integration/config/config_test.go
Lines 382 to 400 in 7b9275c
Some notes on that test though;
- it looks like it's "actually" in the wrong place (that test suite is testing
docker config(swarm configs), so perhaps should be moved to a new suite, but we can look at that later - the "test utils" we have to start the daemon may be a bit complicated for this (they're mostly to setup a daemon and then interact with the running daemon, but not so much to test the daemon CLI); it would need a bit looking into (but happy to help if needed). This particular would probably have been easier to implement in the old
integration-clitests, but those were marked "deprecated", so we can't add new tests there. So instead, perhaps just usingos/execto start the binary and check stdout/stderr would work.
Member
|
Failure is unrelated, but .. interesting? edit: opened #42416 for tracking Details |
aiordache
force-pushed
the
daemon_config
branch
2 times, most recently
from
e54bcb8 to
c905a87
Compare
thaJeztah
requested changes
May 20, 2021
integration/daemon/daemon_test.go
Outdated
Comment on lines
58
to
65
Member
There was a problem hiding this comment.
nit: could make these cons to be explicit they're "immutable";
Suggested change
| validOut := "Config OK" | |
| failedOut := "unable to configure the Docker daemon with file" | |
| const ( | |
| validOut = "Config OK" | |
| failedOut = "unable to configure the Docker daemon with file" | |
| ) |
integration/daemon/daemon_test.go
Outdated
Comment on lines
71
to
78
Member
There was a problem hiding this comment.
I think this could use CombinedOutput(), which both executes the command, and collects the output;
cmd := exec.Command(dockerdBinary, "--validate", "--config-file", config)
out, err := cmd.CombinedOutput()However, it would be useful to verify that in the "fail" case, it exists with an error code, and in the "non-fail" case exits with a "zero" exit status; perhaps something like;
if expectedOut == failedOut {
assert.ErrorContains(t, err, "", "expected an error, but got none")
} else {
assert.NilError(t, err)
}
Member
There was a problem hiding this comment.
For better debugging which of the tests failed, you could use sub-tests here; basically, that's wrapping everything inside the loop in a t.Run(name, func(){});
for filename, expectedOut := range tests {
t.Run(filename, func(t *testing.T) {
config := filepath.Join("testdata", filename)
cmd := exec.Command(dockerdBinary, "--validate", "--config-file", config)
out, err := cmd.CombinedOutput()
assert.Check(t, is.Contains(string(out), expectedOut))
if expectedOut == failedOut {
assert.ErrorContains(t, err, "", "expected an error, but got none")
} else {
assert.NilError(t, err)
}
})
}I just checked out the branch to test that (with my other suggestions), and; with that;
make DOCKER_GRAPHDRIVER=vfs BIND_DIR=. TESTDIRS='github.com/docker/docker/integration/daemon' TESTFLAGS='-test.run ^TestDaemonConfigValidation$' test-integration
Running /go/src/github.com/docker/docker/integration/daemon (amd64.integration.daemon) flags=-test.v -test.timeout=5m -test.run ^TestDaemonConfigValidation
INFO: Testing against a local daemon
=== RUN TestDaemonConfigValidation
=== RUN TestDaemonConfigValidation/valid-config-1.json
=== RUN TestDaemonConfigValidation/empty-config-1.json
=== RUN TestDaemonConfigValidation/empty-config-2.json
=== RUN TestDaemonConfigValidation/invalid-config-1.json
=== RUN TestDaemonConfigValidation/malformed-config.json
--- PASS: TestDaemonConfigValidation (0.40s)
--- PASS: TestDaemonConfigValidation/valid-config-1.json (0.08s)
--- PASS: TestDaemonConfigValidation/empty-config-1.json (0.07s)
--- PASS: TestDaemonConfigValidation/empty-config-2.json (0.08s)
--- PASS: TestDaemonConfigValidation/invalid-config-1.json (0.07s)
--- PASS: TestDaemonConfigValidation/malformed-config.json (0.07s)
PASS
be26d52 to
19976a0
Compare
thaJeztah
reviewed
May 20, 2021
Member
|
Rootless is failing (which we somewhat expected could happen); I think it's ok to skip the test on rootless, because it's not testing anything that's specific to rootless / non-rootless, so we'd have enough coverage for the other situations. |
thaJeztah
reviewed
May 20, 2021
thaJeztah
requested changes
May 21, 2021
thaJeztah
requested changes
May 25, 2021
Member
thaJeztah
left a comment
thaJeztah
left a comment
There was a problem hiding this comment.
changes LGTM, but can you squash the commits?
Member
Member
|
RS5 failure is unrelated, but posting here in case it becomes an issue; |
ebriney
approved these changes
May 27, 2021
ebriney
left a comment
ebriney
left a comment
There was a problem hiding this comment.
Code is clear and it do the job for me.
LGTM
thaJeztah
reviewed
Jun 3, 2021
cmd/dockerd/daemon.go
Outdated
Member
There was a problem hiding this comment.
Discussing with @tonistiigi @tianon and (for comparison), NGINX prints this on stderr
nginx -t > /dev/null
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
samuelkarp
reviewed
Jun 4, 2021
integration/daemon/daemon_test.go
Outdated
Member
There was a problem hiding this comment.
It looks like this is now out of date:
[2021-06-04T18:45:32.940Z] --- FAIL: TestDaemonConfigValidation (0.01s)
[2021-06-04T18:45:32.940Z] --- PASS: TestDaemonConfigValidation/malformed_config (0.11s)
[2021-06-04T18:45:32.940Z] --- FAIL: TestDaemonConfigValidation/config_with_no_content (0.11s)
[2021-06-04T18:45:32.940Z] daemon_test.go:96: assertion failed: string "the configuration file /go/src/github.com/docker/docker/integration/daemon/testdata/empty-config-1.json is valid\n" does not contain "Config OK"
[2021-06-04T18:45:32.940Z] --- PASS: TestDaemonConfigValidation/invalid_config (0.08s)
[2021-06-04T18:45:32.940Z] --- FAIL: TestDaemonConfigValidation/config_with_{} (0.10s)
[2021-06-04T18:45:32.940Z] daemon_test.go:96: assertion failed: string "the configuration file /go/src/github.com/docker/docker/integration/daemon/testdata/empty-config-2.json is valid\n" does not contain "Config OK"
[2021-06-04T18:45:32.940Z] --- FAIL: TestDaemonConfigValidation/valid_config (0.06s)
[2021-06-04T18:45:32.940Z] daemon_test.go:96: assertion failed: string "the configuration file /go/src/github.com/docker/docker/integration/daemon/testdata/valid-config-1.json is valid\n" does not contain "Config OK"
[2021-06-04T18:45:32.940Z] FAIL
[2021-06-04T18:45:32.940Z]
[2021-06-04T18:45:32.940Z] === Failed
[2021-06-04T18:45:32.940Z] === FAIL: amd64.integration.daemon TestDaemonConfigValidation/config_with_no_content (0.11s)
[2021-06-04T18:45:32.940Z] --- FAIL: TestDaemonConfigValidation/config_with_no_content (0.11s)
[2021-06-04T18:45:32.940Z] daemon_test.go:96: assertion failed: string "the configuration file /go/src/github.com/docker/docker/integration/daemon/testdata/empty-config-1.json is valid\n" does not contain "Config OK"
[2021-06-04T18:45:32.940Z]
[2021-06-04T18:45:32.940Z] === FAIL: amd64.integration.daemon TestDaemonConfigValidation/config_with_{} (0.10s)
[2021-06-04T18:45:32.940Z] --- FAIL: TestDaemonConfigValidation/config_with_{} (0.10s)
[2021-06-04T18:45:32.940Z] daemon_test.go:96: assertion failed: string "the configuration file /go/src/github.com/docker/docker/integration/daemon/testdata/empty-config-2.json is valid\n" does not contain "Config OK"
[2021-06-04T18:45:32.940Z]
[2021-06-04T18:45:32.940Z] === FAIL: amd64.integration.daemon TestDaemonConfigValidation/valid_config (0.06s)
[2021-06-04T18:45:32.940Z] --- FAIL: TestDaemonConfigValidation/valid_config (0.06s)
[2021-06-04T18:45:32.940Z] daemon_test.go:96: assertion failed: string "the configuration file /go/src/github.com/docker/docker/integration/daemon/testdata/valid-config-1.json is valid\n" does not contain "Config OK"
[2021-06-04T18:45:32.940Z]
[2021-06-04T18:45:32.940Z] === FAIL: amd64.integration.daemon TestDaemonConfigValidation (0.01s)
Member
|
@samuelkarp or @tonistiigi ptal 🤗 |
tonistiigi
reviewed
Jun 7, 2021
daemon/config/config.go
Outdated
Member
There was a problem hiding this comment.
what change caused this? Also, this ignores the errors after first json block. As the data is already in byte slice then Unmarshal is cleaner.
Member
There was a problem hiding this comment.
I think this was added to allow for an empty daemon.json file (empty file, not empty {})
Member
There was a problem hiding this comment.
Not sure if this is a good idea. But if added should be an explicit empty check then. Tbh I'm surprised this returns io.EOF and not ErrUnexpectedEOF.
Member
There was a problem hiding this comment.
I think it would return ErrUnexpectedEOF if the JSON is incomplete ({), and EOF if the file is empty (or only whitespace);
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
)
func main() {
var reader io.Reader
var jsonConfig map[string]interface{}
for _, config := range []string{"{", "[", `""`, "", " ", `[]`, `{}`} {
reader = bytes.NewReader([]byte(config))
if err := json.NewDecoder(reader).Decode(&jsonConfig); err != nil {
fmt.Println(err)
continue
}
fmt.Println(config, "is ok")
}
}Prints:
unexpected EOF
unexpected EOF
json: cannot unmarshal string into Go value of type map[string]interface {}
EOF
EOF
json: cannot unmarshal array into Go value of type map[string]interface {}
{} is ok
Not sure if this is a good idea
I thought slightly relaxing would be ok; we already did allow an empty {}; considering an empty file to be the equivalent (config with nothing set).
Do you have specific concerns?
Member
There was a problem hiding this comment.
Yeah, idk why empty and spaces are considered as valid json for Decode. JSON.parse in js in unexpectedeof for both cases.
I'm not strictly against allowing empty file, if detection for it is clear in code. It might lead to error going unnoticed though. If Desktop fills this file from a field I do think correct behavior would be to not create a file if the field is empty, not to create an empty file.
aiordache
force-pushed
the
daemon_config
branch
3 times, most recently
from
367683c to
136dbed
Compare
thaJeztah
reviewed
Jun 17, 2021
daemon/config/config.go
Outdated
Comment on lines
394
to
401
Member
There was a problem hiding this comment.
I'm inclined to do the reverse here; not convert "empty" to {}, but check if it's empty, and if so, return early:
Suggested change
| // trim leading and trailing spaces | |
| b = bytes.TrimSpace(b) | |
| // accept empty config files, set content to {} | |
| if len(b) == 0 { | |
| b = []byte("{}") | |
| } | |
| var config Config | |
| var config Config | |
| b = bytes.TrimSpace(b) | |
| if len(b) == 0 { | |
| // empty config file | |
| return &config, nil | |
| } | |
Looks like all the code below should only be for loading the JSON file, and for validating if there's conflicts, so we wouldn't have to run that code afaics (yeah, this code and logic needs a rewrite, as it's quite messy)
aiordache
force-pushed
the
daemon_config
branch
2 times, most recently
from
8c7c9bd to
d5d3231
Compare
Member
|
@tonistiigi PTAL |
tonistiigi
reviewed
Jun 17, 2021
integration/daemon/main_test.go
Outdated
Member
There was a problem hiding this comment.
How do these tests use frozen images? Or any of this environment/protect calls actually. The added tests do not seem to use shared daemon at all.
Member
There was a problem hiding this comment.
Yes, looks like we should be able to remove for now as the current tests don't use the shared daemon.
w.r.t. the testEnv, perhaps we can just skip on ! linux instead.
@aiordache could you have a look at removing the unneeded setup code?
aiordache
force-pushed
the
daemon_config
branch
4 times, most recently
from
02bbfa5 to
e39f972
Compare
Member
|
Just realised a minor inconsistency: If no config-file is configured, it will use the default location, but the default is "optional" (as in: Perhaps we should change the message back to the original |
aiordache
force-pushed
the
daemon_config
branch
2 times, most recently
from
d608765 to
8f80e55
Compare
Fixes moby#36911 If config file is invalid we'll exit anyhow, so this just prevents the daemon from starting if the configuration is fine. Mainly useful for making config changes and restarting the daemon iff the config is valid. Signed-off-by: Rich Horwood <[email protected]> Signed-off-by: Sebastiaan van Stijn <[email protected]> Signed-off-by: Anca Iordache <[email protected]>
thaJeztah
approved these changes
Jun 23, 2021
Member
thaJeztah
left a comment
thaJeztah
left a comment
There was a problem hiding this comment.
LGTM
failure is unrelated (TestBuildWCOWSandboxSize) and a known flaky
Member
|
@tonistiigi PTAL: the test setup was updated ( |
Member
|
CI failure is unrelated (a windows machine running out of disk space) |
tonistiigi
approved these changes
Jun 23, 2021
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On
dockerd --validate, return after the daemon config has been loaded and validated.Continuation of #38138, avoids the
os.Exit.closes #38138
fixes #36911