Why so many references to global variables?

Re: Why so many references to global variables?

Alexander Schmolck wrote:[color=blue]
> J-P <[email protected]> writes:
>
>[color=green]
>>It appears that every time the interpreter tests for
>>the value of 'myflag', it keeps a reference to it.[/color]
>
>
> What makes you think so? This seems rather unlikely to me (not that the
> references themselves should eat your memory anyway!).[/color]

Well, sys.getrefcount () does tell me there are 3 millions references
to it and other globals. Even though this doesn't eat up all my memory
(how large is a reference object in Python?), I definitely think there's
something fishy with keeping that many references to global variables
that appear here and there in the script.


[color=blue]
> Chances are, the C extension code doesn't work correctly (C extensions to
> python code have to do memory management by hand; increasing and decreasing
> reference counts for the python objects they deal with as appropriate; so if a
> bit of code forgets to decrease the refcount, the object will stay alive
> forever; my guess would be that this it what happens here).[/color]

Might be, but the ref count for the objects interacting with the C
library are pretty much what I expect them to be, i.e. a few dozens.
I don't think there are other memory leaks in the bindings to the
library. I've passed it through Purify a couple of times and everything
seems clean.


J-P

Re: Why so many references to global variables?

J-P wrote:
[color=blue]
> Well, sys.getrefcount () does tell me there are 3 millions references
> to it and other globals. Even though this doesn't eat up all my memory
> (how large is a reference object in Python?), ...[/color]

Reference counts are just maintained internally with a single number
that's incremented or decremented.
[color=blue]
> I definitely think
> there's
> something fishy with keeping that many references to global variables
> that appear here and there in the script.[/color]

Yes, it does. It strongly suggests that the fishiness is in your C
extension.

--
Erik Max Francis && [email protected] && http://www.alcyone.com/max/
__ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
/ \ Now I must follow them!
\__/ Beowulf, King of the Geats

Re: Why so many references to global variables?

J-P <[email protected]> writes:
[color=blue]
> Alexander Schmolck wrote:[color=green]
> > J-P <[email protected]> writes:
> >[/color]
>[color=green][color=darkred]
> >>It appears that every time the interpreter tests for
> >>the value of 'myflag', it keeps a reference to it.[/color]
> > What makes you think so? This seems rather unlikely to me (not that the[/color]
>[color=green]
> > references themselves should eat your memory anyway!).[/color]
>
>
> Well, sys.getrefcount () does tell me there are 3 millions references
> to it and other globals. Even though this doesn't eat up all my memory
> (how large is a reference object in Python?), I definitely think there's
> something fishy with keeping that many references to global variables that
> appear here and there in the script.[/color]


Erik Max Francis has hopefully already sorted your reference count confusion
out (if not, maybe a look under "reference counts" in the C API/extending bits
of the python docu might clarify matters), so I'll just give you a simple
practical tip:

Take one of the suspicious C extension functions, and call it repeatedly from
python (passing and returning data structures that are as large as possible
and that you discard immediately aftewerwards). Then using 'top' or something
equivalent, look at how the memory consumption of your program changes: if you
find that with each couple of calls python swallows a few megabytes, you can
be pretty sure that something is going wrong (at least if you force gc with
gc.collect()).

Once you've isolated the function(s) you're in for some fun debugging the
corresponding C code, paying particular attention to PY_DECREFs and
PY_INCREFs.
[color=blue]
>
>
>
>[color=green]
> > Chances are, the C extension code doesn't work correctly (C extensions to
> > python code have to do memory management by hand; increasing and decreasing
> > reference counts for the python objects they deal with as appropriate; so if a
> > bit of code forgets to decrease the refcount, the object will stay alive
> > forever; my guess would be that this it what happens here).[/color]
>
> Might be, but the ref count for the objects interacting with the C library are
> pretty much what I expect them to be, i.e. a few dozens.
>
> I don't think there are other memory leaks in the bindings to the
> library. I've passed it through Purify a couple of times and everything
> seems clean.[/color]

Purify is unlikely to have a deep understanding of python's internal reference
counting (memory management) mechanism, right? So while it will be helpful for
finding memory *violations* (and leaks not due to refcounts) it's quite
unlikely to find problems due to the C extension not *decreasing reference
counts*, which is what I bet is happening in your case.

'as

Re: Why so many references to global variables?

Thank you both for the advice, I appreciate it!

J-P