Skip to content

Comments

increase the maxJSONDecodeRetry in json log reader#17606

Merged
cpuguy83 merged 1 commit intomoby:masterfrom
mountkin:increase-max-retries
Nov 3, 2015
Merged

increase the maxJSONDecodeRetry in json log reader#17606
cpuguy83 merged 1 commit intomoby:masterfrom
mountkin:increase-max-retries

Conversation

@mountkin
Copy link
Contributor

@mountkin mountkin commented Nov 2, 2015

This is another try to fix the flaky tests TestLogsFollowSlowStdoutConsumer and TestRunSlowStdoutConsumer
The previous one is #17537

screenshot from 2015-11-02 17 53 13

I noticed another failure in one of the PRs after #17537 was merged but it seems that the test was re-triggered and I couldn't find the exact failed URL in jenkins.

If I add the following debug lines after https://github.com/docker/docker/blob/master/daemon/logger/jsonfilelog/jsonfilelog.go#L370

        buf, _ := ioutil.ReadAll(dec.Buffered())
        logrus.Errorf("failed to decode log entry after %d retries. buffered: |||%s|||, Error: %v", retries, string(buf), err)

when the test fails, I can see the following error log:

time="2015-11-02T08:23:17.008895080Z" level=error msg="failed to decode log entry after 11 retries. buffered: |||\n{\"log\":\"X\\n\",\"stream\":\"stdout\",\"time\":\"2015-11|||, Error: unexpected EOF"
time="2015-11-02T08:23:17.008955856Z" level=error msg="Error streaming logs: unexpected EOF"

After changed maxJSONDecodeRetry to 20000 I run the test for 300 times locally and all succeed.
Before the change it usually fails within 100 times.

/cc @cpuguy83 @LK4D4

@mountkin
Copy link
Contributor Author

mountkin commented Nov 3, 2015

@LK4D4
Copy link
Contributor

LK4D4 commented Nov 3, 2015

LGTM

1 similar comment
@cpuguy83
Copy link
Member

cpuguy83 commented Nov 3, 2015

LGTM

cpuguy83 added a commit that referenced this pull request Nov 3, 2015
increase the maxJSONDecodeRetry in json log reader
@cpuguy83 cpuguy83 merged commit c30b9e5 into moby:master Nov 3, 2015
@mountkin mountkin deleted the increase-max-retries branch February 10, 2017 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants