benchmarks: pulled in code from github.com/dgonyeo/rkt-monitor by cgonyeo · Pull Request #2324 · rkt/rkt

cgonyeo · 2016-03-24T19:04:30Z

Added the tests/rkt-monitor directory, which includes the code for a
golang binary that can start rkt and watch its resource usage. This
directory also includes bash scripts for generating a handful of test
scenarios.

cgonyeo · 2016-03-24T19:05:58Z

Questions I have about this:

Would there be a better way to display the information? If we're only concerned about systemd usage, perhaps I just print that?
Should I look into adding a make benchmark to our autotools setup?

cgonyeo · 2016-03-24T19:09:51Z

Also I've already found something interesting:

derek@rokot ~/go/src/github.com/coreos/rkt/tests/rkt-monitor> sudo ./rkt-monitor too-many-apps.podmanifest -d 30s | grep systemd
systemd(27448): seconds alive: 16  avg CPU: 92.531073%  avg Mem: 5 mB  peak Mem: 5 mB
systemd-journal(27449): seconds alive: 16  avg CPU: 0.000000%  avg Mem: 6 mB  peak Mem: 6 mB

cgonyeo · 2016-03-24T21:31:42Z

With the too-many-apps pod that's included in this PR it looks like systemd is hunting for some files as hard as it can for a while, and after a while finishes what it was doing and drops back to pretty much no CPU usage.

yifan-gu · 2016-03-28T20:23:19Z

With the too-many-apps pod that's included in this PR it looks like systemd is hunting for some files as hard as it can for a while, and after a while finishes what it was doing and drops back to pretty much no CPU usage.

@dgonyeo Do you made any notes on what are the files systemd is looking for?

cc @alban @iaguis

alban · 2016-03-31T12:15:26Z

systemd notices the cgroup filesystems being mounted for each app and tries to read the configuration for each of them: https://gist.github.com/alban/0bc92a9dab9a722b05d343753fbc51b1

iaguis · 2016-03-31T12:52:12Z

Not mounting /sys (or not mounting it recursively) inside each app's chroot seems to make the problem go away.

jonboulle · 2016-03-31T13:03:46Z

Gah. Can we disable systemd's "smartness" somehow?

alban · 2016-03-31T13:10:22Z

Issue filed in #2351

iaguis · 2016-04-11T13:48:30Z

We shouldn't check in ACIs or binaries.

jonboulle · 2016-04-11T13:51:24Z

Great catch - presumably an accident since the build scripts are there..

iaguis · 2016-04-11T15:23:47Z

tests/rkt-monitor/main.go

+	if err != nil {
+		return nil, err
+	}
+	c, err := p.CPUPercent(0)


It seems this doesn't exist anymore in https://github.com/shirou/gopsutil/blob/master/process/process.go

Also, we should vendor it.

cgonyeo · 2016-04-11T15:26:50Z

Whoops, removed the binary and ACI.

iaguis · 2016-04-11T15:32:29Z

😱

$ sudo ./rkt-monitor cpu-stresser.aci -v -d 30s
panic: process does not have children

goroutine 1 [running]:
panic(0x69aa40, 0xc8200c6300)
    /usr/lib/go/src/runtime/panic.go:464 +0x3e6
main.runRktMonitor(0x893260, 0xc8200d60c0, 0x1, 0x4)
    /home/iaguis/work/coreos/go/src/github.com/coreos/rkt/tests/rkt-monitor/main.go:133 +0xa50
github.com/spf13/cobra.(*Command).execute(0x893260, 0xc82000a060, 0x4, 0x4, 0x0, 0x0)
    /home/iaguis/work/coreos/go/src/github.com/spf13/cobra/command.go:565 +0x85a
github.com/spf13/cobra.(*Command).ExecuteC(0x893260, 0x893260, 0x0, 0x0)
    /home/iaguis/work/coreos/go/src/github.com/spf13/cobra/command.go:651 +0x55c
github.com/spf13/cobra.(*Command).Execute(0x893260, 0x0, 0x0)
    /home/iaguis/work/coreos/go/src/github.com/spf13/cobra/command.go:610 +0x2d
main.main()
    /home/iaguis/work/coreos/go/src/github.com/coreos/rkt/tests/rkt-monitor/main.go:65 +0x23
$ ps -o cmd axf | grep -A3 "[s]ystemd-nspawn"
 \_ stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --register=true --link-journal=try-guest --quiet --uuid=9d7e3c2c-0f90-4a65-98d7-35e4805a7709 --machine=rkt-9d7e3c2c-0f90-4a65-98d7-35e4805a7709 --directory=stage1/rootfs -- --default-standard-output=tty --log-target=null --show-status=0
     \_ /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0
         \_ /usr/lib/systemd/systemd-journald
         \_ /worker

iaguis · 2016-04-14T13:36:18Z

Hopefully @dgonyeo has some time to fix this for the next release ;)

jonboulle · 2016-04-19T18:14:05Z

@dgonyeo do you have time to get this? we need it for rktnetes.

cgonyeo · 2016-04-19T18:18:13Z

I'll have time to fix this up tomorrow. After I fix the panic, is it fine in its current form? Any changes you'd like to see?

jonboulle · 2016-04-19T18:31:41Z

For an initial cut yes I think so

cgonyeo · 2016-04-20T14:53:01Z

The panic was due to gopsutil updating, and me not vendoring the dependencies for this. I pulled in the dependencies and added a little build script to point the GOPATH to Godeps/_workspace. Work for you now, @iaguis?

iaguis · 2016-04-20T15:24:47Z

Yep! But it seems you're still including binary files.

cgonyeo · 2016-04-20T15:30:12Z

Wow I'm bad at not doing that. Fixed.

jonboulle · 2016-04-20T16:22:28Z

this is unreviewable, mind splitting out the godeps?

Added the tests/benchmarks directory, which includes the code for a golang binary that can start rkt and watch its resource usage. This directory also includes bash scripts for generating a handful of test scenarios.

Vendored the dependencies necessary for rkt-monitor.

cgonyeo · 2016-04-20T18:31:09Z

Done.

alban · 2016-04-21T10:23:21Z

The Godeps changes add a dependency on https://github.com/StackExchange/wmi and it does not have releases yet. We would like to only have godeps dependencies on tagged versions. Can you ask the authors of wmi to tag a release? As an example of such request, please see the rationale on peterbourgon/diskv#27

Same thing for https://github.com/shirou/w32 and maybe others.

jonboulle · 2016-04-21T10:25:38Z

I think this might not be such an issue for packaging because the deps in
question aren't used in rkt itself, just in ancillary code... but I'm not
sure - @onlyjob can probably clarify.

On Thu, Apr 21, 2016 at 12:23 PM, Alban Crequy [email protected]
wrote:

The Godeps changes add a dependency on
https://github.com/StackExchange/wmi and it does not have releases yet.
We would like to only have godeps dependencies on tagged versions. Can you
ask the authors of wmi to tag a release? As an example of such request,
please see the rationale on peterbourgon/diskv#27
peterbourgon/diskv#27

Same thing for https://github.com/shirou/w32 and maybe others.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#2324 (comment)

cgonyeo · 2016-04-25T13:51:00Z

Pinging @onlyjob

onlyjob · 2016-04-25T22:10:46Z

Pinging @onlyjob

Why @dgonyeo? What you expect me to do?? Probably I missed the question...

We can mock-up date-based version of dependency library when semantic releases are not available...

cgonyeo · 2016-04-25T22:33:39Z

Oh sorry, should have clarified. The question is if it's acceptable in your eyes to use/vendor untagged commits of dependencies, when the dependency is only used in ancillary code that won't make it into the packaged release in a distro like Debian.

cgonyeo mentioned this pull request Mar 24, 2016

docs: benchmark stage1 systemd resource usage #1788

Closed

jonboulle added area/developer tooling area/performance labels Mar 25, 2016

jonboulle added this to the v1.3.0 milestone Mar 29, 2016

jonboulle added the needs/some-changes label Mar 29, 2016

alban mentioned this pull request Mar 31, 2016

prepare-app: perf issue with mounting /sys (benchmarks) #2351

Closed

alban modified the milestones: v1.4.0, v1.3.0 Mar 31, 2016

iaguis reviewed Apr 11, 2016
View reviewed changes

cgonyeo force-pushed the benchmarks branch from 5a34011 to ccbeacc Compare April 11, 2016 15:26

iaguis added the priority/P2 label Apr 12, 2016

iaguis modified the milestones: v1.5.0, v1.4.0 Apr 14, 2016

jonboulle added priority/P0 technology/kubernetes and removed priority/P2 labels Apr 19, 2016

alban mentioned this pull request Apr 20, 2016

stage1/prepare-app: avoid recursive bind-mounts on /sys #2386

Merged

cgonyeo force-pushed the benchmarks branch 2 times, most recently from 54ce659 to a6eb383 Compare April 20, 2016 14:50

cgonyeo force-pushed the benchmarks branch from a6eb383 to 9b819ae Compare April 20, 2016 15:29

cgonyeo force-pushed the benchmarks branch from 9b819ae to 300069d Compare April 20, 2016 18:29

Derek Gonyeo added 2 commits April 20, 2016 14:30

benchmarks: pulled in code from github.com/dgonyeo/rkt-monitor

ddd59b8

Added the tests/benchmarks directory, which includes the code for a golang binary that can start rkt and watch its resource usage. This directory also includes bash scripts for generating a handful of test scenarios.

benchmarks: godeps for rkt-monitor

6ecb355

Vendored the dependencies necessary for rkt-monitor.

cgonyeo force-pushed the benchmarks branch from 300069d to 6ecb355 Compare April 20, 2016 18:30

alban added needs/review and removed needs/some-changes labels Apr 21, 2016

cgonyeo merged commit 8a6e511 into rkt:master Apr 26, 2016

cgonyeo deleted the benchmarks branch April 26, 2016 13:26

yifan-gu mentioned this pull request May 20, 2016

rkt: Measure the performance for rktnetes kubernetes/kubernetes#25581

Closed

3 tasks

Conversation

cgonyeo commented Mar 24, 2016

Uh oh!

cgonyeo commented Mar 24, 2016

Uh oh!

cgonyeo commented Mar 24, 2016

Uh oh!

cgonyeo commented Mar 24, 2016

Uh oh!

yifan-gu commented Mar 28, 2016

Uh oh!

alban commented Mar 31, 2016

Uh oh!

iaguis commented Mar 31, 2016

Uh oh!

jonboulle commented Mar 31, 2016

Uh oh!

alban commented Mar 31, 2016

Uh oh!

iaguis commented Apr 11, 2016

Uh oh!

jonboulle commented Apr 11, 2016

Uh oh!

iaguis Apr 11, 2016

Choose a reason for hiding this comment

Uh oh!

cgonyeo commented Apr 11, 2016

Uh oh!

iaguis commented Apr 11, 2016

Uh oh!

iaguis commented Apr 14, 2016

Uh oh!

jonboulle commented Apr 19, 2016

Uh oh!

cgonyeo commented Apr 19, 2016

Uh oh!

jonboulle commented Apr 19, 2016

Uh oh!

cgonyeo commented Apr 20, 2016

Uh oh!

iaguis commented Apr 20, 2016

Uh oh!

cgonyeo commented Apr 20, 2016

Uh oh!

jonboulle commented Apr 20, 2016

Uh oh!

cgonyeo commented Apr 20, 2016

Uh oh!

alban commented Apr 21, 2016

Uh oh!

jonboulle commented Apr 21, 2016

Uh oh!

cgonyeo commented Apr 25, 2016

Uh oh!

onlyjob commented Apr 25, 2016

Uh oh!

cgonyeo commented Apr 25, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants