Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker -d crash w/ port mapping (stacktrace included) #97

Closed
niallo opened this issue Mar 15, 2013 · 16 comments
Closed

Docker -d crash w/ port mapping (stacktrace included) #97

niallo opened this issue Mar 15, 2013 · 16 comments

Comments

@niallo
Copy link
Contributor

niallo commented Mar 15, 2013

I was trying to run sshd inside one of my images with port forwarding when dockerd crashed:

ubuntu@hosted:$ docker ps
ID IMAGE COMMAND CREATED STATUS COMMENT
ubuntu@hosted:$ docker run -a -p=22 strider-worker-ssh /usr/sbin/sshd -Dd
ubuntu@hosted:$ docker ps
2013/03/15 21:26:32 unable to open database file
ubuntu@hosted:$ ps auxw | grep docker
ubuntu 23998 0.0 0.0 8104 920 pts/3 S+ 21:27 0:00 grep --color=auto docker

Here is the trace:

2013/03/15 21:26:28 docker run -a -p=22 strider-worker-ssh /usr/sbin/sshd -Dd
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x423183]

goroutine 17 [running]:
github.com/dotcloud/docker.(*PortMapper).Map(0x0, 0xf80000c001, 0xf840085f7c, 0x400000004, 0x16, ...)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/network.go:138 +0xa0
----- stack segment boundary -----
github.com/dotcloud/docker.(*NetworkInterface).AllocatePort(0xf8400bf5a0, 0x16, 0xf84014a240, 0x0, 0x0, ...)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/network.go:282 +0xda
github.com/dotcloud/docker.(*Container).allocateNetwork(0xf8400bcf00, 0x0, 0x0, 0x40f66a)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/container.go:421 +0x121
github.com/dotcloud/docker.(*Container).Start(0xf8400bcf00, 0xf8400b6930, 0xf840000b60, 0x0)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/container.go:315 +0x9a
github.com/dotcloud/docker/commands.(*Server).CmdRun(0xf8401235a0, 0xf8400d5690, 0xf8400ec250, 0xf8400d56c0, 0xf840088968, ...)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/commands/commands.go:920 +0xeb2
----- stack segment boundary -----
reflect.Value.call(0x61e408, 0x4314f1, 0x130, 0x67fea4, 0x6c6c614300000009, ...)
        /usr/lib/go/src/pkg/reflect/value.go:521 +0x135e
reflect.Value.CallSlice(0x61e408, 0x4314f1, 0x130, 0x7f9e3d993c58, 0x400000004, ...)
        /usr/lib/go/src/pkg/reflect/value.go:347 +0x85
github.com/dotcloud/docker/rcli._func_005(0xf8400f1d70, 0xf8400ec230, 0x43582b, 0xf8400d5690, 0xf8400ec1c0, ...)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/rcli/types.go:81 +0x21e
github.com/dotcloud/docker/rcli.LocalCall(0xf8400c2c30, 0xf8401235a0, 0xf8400d5690, 0xf8400ec1c0, 0xf8400d56c0, ...)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/rcli/types.go:50 +0x61e
github.com/dotcloud/docker/rcli.call(0xf8400c2c30, 0xf8401235a0, 0xf8400d5690, 0xf8400ec1c0, 0xf8400d56c0, ...)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/rcli/types.go:30 +0x89
github.com/dotcloud/docker/rcli.Serve(0xf8400d5600, 0xf840088968, 0xf8400c2c30, 0xf8401235a0, 0xf840088968, ...)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/rcli/tcp.go:66 +0x2d3
github.com/dotcloud/docker/rcli._func_002(0xf8400ccf90, 0xf8401235b0, 0x0, 0x0)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/rcli/tcp.go:45 +0x60
created by github.com/dotcloud/docker/rcli.ListenAndServe
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/rcli/tcp.go:50 +0x2b7

goroutine 1 [chan receive]:
net.(*pollServer).WaitRead(0xf8400c45c0, 0xf8400f23f0, 0xf84007b660, 0xb, 0x1, ...)
        /usr/lib/go/src/pkg/net/fd.go:268 +0x73
net.(*netFD).accept(0xf8400f23f0, 0x4bc616, 0x0, 0xf8400787b0, 0xf840088040, ...)
        /usr/lib/go/src/pkg/net/fd.go:622 +0x20d
net.(*TCPListener).AcceptTCP(0xf8400be6c8, 0x1, 0x0, 0x0, 0x10, ...)
        /usr/lib/go/src/pkg/net/tcpsock_posix.go:320 +0x71
net.(*TCPListener).Accept(0xf8400be6c8, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/lib/go/src/pkg/net/tcpsock_posix.go:330 +0x49
github.com/dotcloud/docker/rcli.ListenAndServe(0x67dbf4, 0x70637400000003, 0x691a74, 0x2e3732310000000e, 0xf8400c2c30, ...)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/rcli/tcp.go:41 +0x23d
main.daemon(0x67d204, 0x6400000001)
        /go/src/github.com/dotcloud/docker/docker/docker.go:43 +0xb9
main.main()
        /go/src/github.com/dotcloud/docker/docker/docker.go:28 +0xf0

goroutine 2 [syscall]:
created by runtime.main
        /build/buildd/golang-1.0.2/src/pkg/runtime/proc.c:221

goroutine 3 [finalizer wait]:
created by runtime.gc
        /build/buildd/golang-1.0.2/src/pkg/runtime/mgc0.c:882

goroutine 4 [syscall]:
syscall.Syscall6()
        /build/buildd/golang-1.0.2/src/pkg/syscall/asm_linux_amd64.s:40 +0x5
syscall.EpollWait(0xf800000049, 0xf8400baf30, 0xa0000000a, 0xffffffff, 0xc, ...)
        /usr/lib/go/src/pkg/syscall/zerrors_linux_amd64.go:1781 +0xa1
net.(*pollster).WaitFD(0xf8400baf20, 0xf8400c45c0, 0x0, 0x0, 0x0, ...)
        /usr/lib/go/src/pkg/net/fd_linux.go:146 +0x110
net.(*pollServer).Run(0xf8400c45c0, 0x0)
        /usr/lib/go/src/pkg/net/fd.go:236 +0xe4
created by net.newPollServer
        /usr/lib/go/src/pkg/net/newpollserver.go:35 +0x382

goroutine 18 [semacquire]:
sync.runtime_Semacquire(0xf8400c3fb0, 0xf8400c3fb0)
        /build/buildd/golang-1.0.2/src/pkg/runtime/zsema_amd64.c:146 +0x25
sync.(*Cond).Wait(0xf840099268, 0x100000001)
        /usr/lib/go/src/pkg/sync/cond.go:67 +0xaa
io.(*pipe).read(0xf840099240, 0xf840141000, 0x40000000400, 0x0, 0x0, ...)
        /usr/lib/go/src/pkg/io/pipe.go:52 +0x1da
io.(*PipeReader).Read(0xf8400c3ec8, 0xf840141000, 0x40000000400, 0xf840141000, 0x0, ...)
        /usr/lib/go/src/pkg/io/pipe.go:130 +0x57
github.com/dotcloud/docker.(*bufReader).drain(0xf840000bd0, 0x0)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/utils.go:64 +0x9b
created by github.com/dotcloud/docker.newBufReader
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/utils.go:57 +0xcb

goroutine 19 [semacquire]:
sync.runtime_Semacquire(0xf8400c3fb8, 0xf8400c3fb8)
        /build/buildd/golang-1.0.2/src/pkg/runtime/zsema_amd64.c:146 +0x25
sync.(*Cond).Wait(0xf8400991a8, 0x100000001)
        /usr/lib/go/src/pkg/sync/cond.go:67 +0xaa
io.(*pipe).read(0xf840099180, 0xf840141400, 0x40000000400, 0x7f9e00000000, 0x0, ...)
        /usr/lib/go/src/pkg/io/pipe.go:52 +0x1da
io.(*PipeReader).Read(0xf8400c3ed8, 0xf840141400, 0x40000000400, 0xf840141400, 0x0, ...)
        /usr/lib/go/src/pkg/io/pipe.go:130 +0x57
github.com/dotcloud/docker.(*bufReader).drain(0xf840000b60, 0x0)
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/utils.go:64 +0x9b
created by github.com/dotcloud/docker.newBufReader
        /usr/lib/go/src/pkg/github.com/dotcloud/docker/utils.go:57 +0xcb
@niallo
Copy link
Contributor Author

niallo commented Mar 15, 2013

docker run -a -p=22 strider-worker-ssh <cmd> is now consistently crashing docker -d on my system no matter what command I use.

Other images (e.g. base) run fine.

@niallo
Copy link
Contributor Author

niallo commented Mar 15, 2013

Additional info:

  • This only occurs with -p port mapping option
  • If I reboot the system, the issue goes away.

My guess is kernel IP tables rules got into some state which upset Docker, and the reboot cleared this.

@chooper
Copy link
Contributor

chooper commented Mar 19, 2013

Without the strider-worker-ssh image, I've been unable to reproduce.

Since we don't have push functionality yet, is there anyway you can share the image (or a sanitized version that still reproduces the bug, if that's what you're comfortable with)?

@niallo
Copy link
Contributor Author

niallo commented Mar 19, 2013

I will see if I can reproduce. I haven't had the issue since I rebooted the system.

@ghost ghost assigned shin- Mar 20, 2013
@shawnsi
Copy link

shawnsi commented Mar 22, 2013

I'm seeing this issue on the vagrant local VM after building from master. Only occurs with port forwarding enabled.

@shawnsi
Copy link

shawnsi commented Mar 22, 2013

The crash is occurring on the base and custom images for me.

@jpetazzo
Copy link
Contributor

The bug also happens if I run docker run -p 22 base /usr/sbin/sshd -D on a fresh docker install on an EC2 instance.

Edit: forgot to mention that I have exactly the same traceback.

@shykes
Copy link
Contributor

shykes commented Mar 23, 2013

I can't reproduce it on my virtualbox+official binaries install.

@shawnsi
Copy link

shawnsi commented Mar 23, 2013

The bug only manifests itself periodically on virtualbox for me. Once it does occur only a reboot seems to fix it.

@shykes
Copy link
Contributor

shykes commented Mar 23, 2013

Looking into it. Can someone give me a combination of (a) full traceback
and (b) exact revision which generated the traceback?

On Fri, Mar 22, 2013 at 6:47 PM, Shawn Siefkas [email protected]:

The bug only manifests itself periodically on virtualbox for me. Once it
does occur only a reboot seems to fix it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/97#issuecomment-15329155
.

@niallo
Copy link
Contributor Author

niallo commented Mar 23, 2013

So the binaries have a git sha compiled in anywhere ?

Sent from my iPhone

On Mar 22, 2013, at 6:52 PM, Solomon Hykes [email protected] wrote:

Looking into it. Can someone give me a combination of (a) full traceback
and (b) exact revision which generated the traceback?

On Fri, Mar 22, 2013 at 6:47 PM, Shawn Siefkas [email protected]:

The bug only manifests itself periodically on virtualbox for me. Once it
does occur only a reboot seems to fix it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/97#issuecomment-15329155
.

—
Reply to this email directly or view it on GitHub.

@shykes
Copy link
Contributor

shykes commented Mar 23, 2013

Unfortunately not... It's on the todo-list! (if anybody finds the time go
for it!)

On Fri, Mar 22, 2013 at 7:13 PM, niallo [email protected] wrote:

So the binaries have a git sha compiled in anywhere ?

Sent from my iPhone

On Mar 22, 2013, at 6:52 PM, Solomon Hykes [email protected]
wrote:

Looking into it. Can someone give me a combination of (a) full traceback
and (b) exact revision which generated the traceback?

On Fri, Mar 22, 2013 at 6:47 PM, Shawn Siefkas [email protected]:

The bug only manifests itself periodically on virtualbox for me. Once
it
does occur only a reboot seems to fix it.

—
Reply to this email directly or view it on GitHub<
https://github.com/dotcloud/docker/issues/97#issuecomment-15329155>
.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/97#issuecomment-15330181
.

@jpetazzo
Copy link
Contributor

With version 690cae0

2013/03/23 02:17:09 Listening for RCLI/tcp on 127.0.0.1:4242
2013/03/23 02:17:11 docker run -p 22 toto /usr/sbin/sshd -D
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x43040a]

goroutine 5 [running]:
github.com/dotcloud/docker.(*PortMapper).Map(0x0, 0xc001, 0xf84008bfbc, 0x400000004, 0x16, ...)
    /root/go/src/github.com/dotcloud/docker/network.go:142 +0xa0
----- stack segment boundary -----
github.com/dotcloud/docker.(*NetworkInterface).AllocatePort(0xf8401f2280, 0x16, 0xf8401eb5a0, 0x0, 0x0, ...)
    /root/go/src/github.com/dotcloud/docker/network.go:286 +0xda
github.com/dotcloud/docker.(*Container).allocateNetwork(0xf8401ac000, 0x0, 0x0, 0xf8401a3600)
    /root/go/src/github.com/dotcloud/docker/container.go:421 +0x121
github.com/dotcloud/docker.(*Container).Start(0xf8401ac000, 0xf840189f00, 0xf84018ce98, 0x100000001)
    /root/go/src/github.com/dotcloud/docker/container.go:315 +0x9a
github.com/dotcloud/docker.(*Server).CmdRun(0xf84018a200, 0xf84018bd20, 0xf84018a4a0, 0xf84018bd50, 0xf84018cd40, ...)
    /root/go/src/github.com/dotcloud/docker/commands.go:988 +0x1130
----- stack segment boundary -----
reflect.Value.call(0x640e48, 0x427a23, 0x130, 0x6a804c, 0x6c6c614300000009, ...)
    /usr/lib/go/src/pkg/reflect/value.go:521 +0x1329
reflect.Value.CallSlice(0x640e48, 0x427a23, 0x130, 0x7fa52a9afc58, 0x400000004, ...)
    /usr/lib/go/src/pkg/reflect/value.go:347 +0x85
github.com/dotcloud/docker/rcli._func_005(0xf84016ea00, 0xf84018a480, 0x436448, 0xf84018bd20, 0xf84018a440, ...)
    /root/go/src/github.com/dotcloud/docker/rcli/types.go:81 +0x21e
github.com/dotcloud/docker/rcli.LocalCall(0xf84018b420, 0xf84018a200, 0xf84018bd20, 0xf84018a440, 0xf84018bd50, ...)
    /root/go/src/github.com/dotcloud/docker/rcli/types.go:50 +0x61e
github.com/dotcloud/docker/rcli.call(0xf84018b420, 0xf84018a200, 0xf84018bd20, 0xf84018a440, 0xf84018bd50, ...)
    /root/go/src/github.com/dotcloud/docker/rcli/types.go:30 +0x89
github.com/dotcloud/docker/rcli.Serve(0xf84018bc00, 0xf84018cd40, 0xf84018b420, 0xf84018a200, 0xf84018cd40, ...)
    /root/go/src/github.com/dotcloud/docker/rcli/tcp.go:66 +0x2d3
github.com/dotcloud/docker/rcli._func_002(0xf84018a290, 0xf84018a210, 0x0, 0x0)
    /root/go/src/github.com/dotcloud/docker/rcli/tcp.go:45 +0x60
created by github.com/dotcloud/docker/rcli.ListenAndServe
    /root/go/src/github.com/dotcloud/docker/rcli/tcp.go:50 +0x2b7

goroutine 1 [chan receive]:
net.(*pollServer).WaitRead(0xf840189480, 0xf84017f5a0, 0xf8400806c0, 0xb, 0x1, ...)
    /usr/lib/go/src/pkg/net/fd.go:268 +0x73
net.(*netFD).accept(0xf84017f5a0, 0x4e6775, 0x0, 0xf84007ef00, 0xf84008f040, ...)
    /usr/lib/go/src/pkg/net/fd.go:622 +0x20d
net.(*TCPListener).AcceptTCP(0xf84018ccc8, 0x1, 0x0, 0x0, 0x10, ...)
    /usr/lib/go/src/pkg/net/tcpsock_posix.go:322 +0x71
net.(*TCPListener).Accept(0xf84018ccc8, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/lib/go/src/pkg/net/tcpsock_posix.go:332 +0x49
github.com/dotcloud/docker/rcli.ListenAndServe(0x6a5c1c, 0x70637400000003, 0x6bacb4, 0x2e3732310000000e, 0xf84018b420, ...)
    /root/go/src/github.com/dotcloud/docker/rcli/tcp.go:41 +0x23d
main.daemon(0x6a51cc, 0x6400000001)
    /root/go/src/github.com/dotcloud/docker/docker/docker.go:42 +0xb9
main.main()
    /root/go/src/github.com/dotcloud/docker/docker/docker.go:27 +0xf0

goroutine 2 [syscall]:
created by runtime.main
    /build/buildd/golang-1/src/pkg/runtime/proc.c:221

goroutine 3 [finalizer wait]:
created by runtime.gc
    /build/buildd/golang-1/src/pkg/runtime/mgc0.c:882

goroutine 4 [syscall]:
syscall.Syscall6()
    /build/buildd/golang-1/src/pkg/syscall/asm_linux_amd64.s:40 +0x5
syscall.EpollWait(0xf8000000a3, 0xf8400c1f30, 0xa0000000a, 0xffffffff, 0xc, ...)
    /usr/lib/go/src/pkg/syscall/zerrors_linux_amd64.go:1781 +0xa1
net.(*pollster).WaitFD(0xf8400c1f20, 0xf840189480, 0x0, 0x0, 0x0, ...)
    /usr/lib/go/src/pkg/net/fd_linux.go:146 +0x110
net.(*pollServer).Run(0xf840189480, 0x0)
    /usr/lib/go/src/pkg/net/fd.go:236 +0xe4
created by net.newPollServer
    /usr/lib/go/src/pkg/net/newpollserver.go:35 +0x382

@srid srid mentioned this issue Mar 23, 2013
Merged
@srid
Copy link
Contributor

srid commented Mar 23, 2013

@shykes this is what i reported yesterday, and you fixed it in commit 301a8af

@titanous
Copy link
Contributor

Looks like this is all fixed up?

@unclejack
Copy link
Contributor

This issue can probably be closed.

@shykes shykes closed this as completed Apr 4, 2013
icecrime pushed a commit to icecrime/docker that referenced this issue Nov 22, 2014
Merge pull request moby#97 from erikh/extensions
f1b5081 
Initial VXLAN documentation with (broken) instructions.
@jtwile2 jtwile2 mentioned this issue Nov 10, 2015
Closed
runcom pushed a commit to runcom/docker that referenced this issue Apr 12, 2016
@rhatdan
Merge pull request moby#97 from runcom/stopsignal-bp-fedora
84238d5 
BACKPORT: Inherit StopSignal from Dockerfile.
@gggeek gggeek mentioned this issue Apr 9, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@srid @titanous @shykes @shawnsi @chooper @jpetazzo @niallo @shin- @unclejack