Skip to content

Measure USS on OSX#745

Merged
giampaolo merged 2 commits intogiampaolo:masterfrom
EricRahm:uss_osx
Feb 3, 2016
Merged

Measure USS on OSX#745
giampaolo merged 2 commits intogiampaolo:masterfrom
EricRahm:uss_osx

Conversation

@EricRahm
Copy link
Contributor

This adds a USS (unique set size) measurement to memory_info_ex on OSX. It is based on the implementation in Firefox's memory measurement subsystem.

@giampaolo
Copy link
Owner

I would be interested in seeing the impact on performances. Could you please measure it before and after the patch and paste the result?

python -m timeit -s "import psutil; p = psutil.Process()" "p.memory_info_ex()"

...also, while we're at it, is it possible to provide also PSS stats?

@EricRahm
Copy link
Contributor Author

before (master):

python -m timeit -s "import psutil; p = psutil.Process()" "p.memory_info_ex()"
100000 loops, best of 3: 2.17 usec per loop

after:

python -m timeit -s "import psutil; p = psutil.Process()" "p.memory_info_ex()"
10000 loops, best of 3: 113 usec per loop

So slower, but not as slow as on Linux.

@giampaolo
Copy link
Owner

Any idea how to determine PSS?

@EricRahm
Copy link
Contributor Author

I do not know of a method for calculating PSS on OSX.

@giampaolo
Copy link
Owner

Sorry for replying late. I was looking back at this and I wonder how this relates to Proces.memory_maps(). Currently memory_maps() implementation on OSX provides a "private" field. As such we should already be able to determine USS by doing:

>>> sum([x.private for x in psutil.Process().memory_maps()])

...but on the other hand the way you are determining USS (I'm talking about the implementation) is completely different. What's the difference between sum([x.private for x in psutil.Process().memory_maps()]) and psutil.Process().memory_info_ex().uss results? Are they equal?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • indentation should be 4 spaces
  • I don't like this being in a separate file, you can use `psutil/arch/osx/process_info.c
  • use // for all comments (instead of /*)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a second thought I would prefer you define this in psutil/_psutil_osx.c as proc_memory_uss.
From the python file (psutil/_psosx.py) after you call cext.psutil_proc_memory_info you will call cext.proc_memory_uss to get USS stats separately.

@giampaolo
Copy link
Owner

@EricRahm
Copy link
Contributor Author

EricRahm commented Feb 3, 2016

What's the difference between sum([x.private for x in psutil.Process().memory_maps()]) and psutil.Process().memory_info_ex().uss results? Are they equal?

They are not equal, memory_maps seems very low:

>>>print p.memory_info_ex(); print sum([x.private for x in psutil.Process().memory_maps()])
pextmem(rss=7880704L, vms=2534256640L, pfaults=9531392, pageins=1564672, uss=5849088L)
622592

The primary difference I can tell is memory_maps uses vm_region_recurse_64 and skips all submaps.

I found this:
https://gitlab.phusion.nl/passenger/temp-passenger/blob/99bf9b590e2a5c3620d5fc34f540b749452448ca/ext/common/Utils/ProcessMetricsCollector.h#L381
...and here the same operation is referenced as "PSS" not "USS". Weird...

They seem to be calculating some sort of USS/PSS hybrid.

@EricRahm
Copy link
Contributor Author

EricRahm commented Feb 3, 2016

@giampaolo Thanks for the review, I'll update per your style requests and merge into the main file.

@EricRahm
Copy link
Contributor Author

EricRahm commented Feb 3, 2016

@giampaolo Updated per requests, I think I covered everything:

  • 4 spaces, foo_bar naming, // for comments
  • Moved impl into _psutil_osx.c, renamed psutil_proc_memory_uss, exported in cext
  • Added function level docs
  • Use cext.proc_memory_uss to add uss to memory_info_ex result

If all is good I'd recommend squashing before merging if that's not in your standard workflow.

@giampaolo
Copy link
Owner

What do you mean by that?
On Feb 3, 2016 9:19 PM, "Eric Rahm" [email protected] wrote:

@giampaolo https://github.com/giampaolo Updated per requests, I think I
covered everything:

  • 4 spaces, foo_bar naming, // for comments
  • Moved impl into _psutil_osx.c, renamed psutil_proc_memory_uss,
    exported in cext
  • Added function level docs
  • Use cext.proc_memory_uss to add uss to memory_info_ex result

If all is good I'd recommend squashing before merging if that's not in
your standard workflow.


Reply to this email directly or view it on GitHub
#745 (comment).

@EricRahm
Copy link
Contributor Author

EricRahm commented Feb 3, 2016

@giampaolo I pushed an update that covered your code review changes, just wanted to list out what I changed and make sure you noticed.

giampaolo added a commit that referenced this pull request Feb 3, 2016
@giampaolo giampaolo merged commit fe2c677 into giampaolo:master Feb 3, 2016
@giampaolo
Copy link
Owner

Merged. Thanks a lot for this (I'll also update CREDITS).

@giampaolo
Copy link
Owner

The primary difference I can tell is memory_maps uses vm_region_recurse_64 and skips all submaps.

This made me think of #289.
If you understand the issue maybe you could fix memory_maps so that it takes submaps into account?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants