Skip to content

Add a --use-system-expat option to use the system's expat library. - #316

Merged
mdboom merged 11 commits into
astropy:masterfrom
mdboom:cextern/expat
Jul 13, 2012
Merged

mdboom merged 11 commits into
astropy:masterfrom
mdboom:cextern/expat

Conversation

@mdboom

@mdboom mdboom commented Jul 11, 2012

Copy link
Copy Markdown
Contributor

This makes is easier for packagers to specify that they want to use the system's expat library.

It also lays the groundwork for doing this more generally, namely:

  1. Easily adding build arguments from a packages setup_package.py

  2. Making pkg-config calls, if necessary and available, to get the compiler arguments to link to a particular library.

@iguananaut may want to look at this since io.fits also has some external packages that may need similar functionality.

Comment thread docs/development/building_packaging.rst Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It says "get_extra_build_options" here, but in xml/setup_package.py you have get_build_options. Is one of these a typo?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. I think the "extra" is a typo -- sort of redundant. I'll fix this in the docs.

@eteq

eteq commented Jul 11, 2012

Copy link
Copy Markdown
Member

Hmm... I'm a little hesitant on this one, because we definitely want to implement a general way of specifying whether or not to use system libraries. (This may or may not be connected into the (also to-be-implemented) python-level optional dependencies.) If we do that, this option might be confusing because it might conflict with that framework (unless the "--use-system-`` scheme is what we use for that).

The get_extra_build_options is potentially useful, regardless.

@mdboom

mdboom commented Jul 11, 2012

Copy link
Copy Markdown
Contributor Author

Even if we provide a general way to "use no internally shipped libraries", which perhaps is useful for end users, but not terribly useful for packagers, we still need a way to specify each individually. Imagine a scenario (not too far from the truth) where Debian ships an old version of wcslib that won't work with astropy, but its expat is compatible. The packager will want to specify using one and not the other.

I think it's simple enough to add another build option "use-system-libraries", and change the check from

if get_distutils_build_option("use_system_expat"):

to

if get_distutils_build_option("use_system_expat") or get_distutils_build_option("use_system_libraries")

How these flags are ultimately handled is necessarily library-specific anyway, so I don't know how much further one could generalize this.

Comment thread astropy/setup_helpers.py

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this, though the way it updates locals() at the end seems unnecessarily hackish. Could it not just return maybe a dict and let the caller decide what it wants to do with the results? This works for expat, but if there isn't already a local variable called library_dirs referencing a list this will fail.

I'd rather just do something like:

flag_updates = pkg_config(...)
if 'library_dirs' in flag_updates:
    library_dirs.extend(flag_updates['library_dirs'])

(maybe the if statement could be skipped if the returned dict is guaranteed to contain certain keywords, which may just have empty lists as their values).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea was that this function will be used in one context: to add additional elements to the include_dirs, library_dirs and libraries kwargs of a call to distutils.Extension. Requiring all of the callers to do that work, which admittedly amounts to three lines of code appending lists together, just seemed like unnecessary duplication.

But I do think, given your question, that it could be made more obvious what it is doing. For example, there will always be a local variable named library_dirs, because it is an argument to the function -- and yes, it must be a list, but that's mentioned in the docstring. Maybe having an if statement in the loop rather than using a mapping dictionary to sort the arguments into categories would be more obvious?

@eteq

eteq commented Jul 11, 2012

Copy link
Copy Markdown
Member

Oh, I agree the actual implementation of what to do if someone wants the system version has to be specified - I'm more thinking of putting in infrastructure to make it clear for every package how to say "always use the system version of library x".

The sort of scenario I'm thinking of is that some future subpackage might also want to use expat, but they might add a build option "--use-local-expat" or something like that. We don't want users (or packagers) to have to sort through that, so instead we should include a standard way of looking for which libraries should use external vs. bundled versions. Ideally, this is the sort of thing we could tell people to add to their setup.cfg.

This is already pretty close to what you have implemented, though... So could you use a get_external_libraries function instead of get_build_options that just returns ['expat'] for this case? (may as well leave in get_build_options, though - could be useful in the future.) Then in setup_helpers.py, perhaps change it so that we get an option of the form --use-system-library="expat", and that can be used straightforwardly from either the command line or setup.cfg.

Does that sound reasonable?

@mdboom

mdboom commented Jul 11, 2012

Copy link
Copy Markdown
Contributor Author

@eteq: I've added another commit that I think addresses your concerns. There are two new functions add_external_library and use_system_library that can be used to declare a new external library and query whether it should be used, respectively. This way, packages can declare what external library dependencies they have and later query whether the user wants to the system or internal version of that library and act accordingly. Duplicates are automatically filtered out, so multiple packages can declare the same external dependency and not have to worry. All of this stuff works either from the commandline or setup.cfg (thanks to how distutils works).

I don't like the idea of having a single option --use-system-library that would take a list. That will be difficult for some packaging tools to merge and compare across versions and to programmatically add and remove libraries from based on something dynamic (remember, most packaging system are written in something a little more cumbersome than Python, so we have to remain shell-scripting-friendly). Instead, each library has a --use-system-X option, and there is also a global --use-system-libraries option that will override everything and try to use all system libraries wherever possible and available. Does that address all the use cases we can think of?

@eteq

eteq commented Jul 12, 2012

Copy link
Copy Markdown
Member

@mdboom - yep, with these changes it looks perfect!

You might add a brief section in docs/install.rstthat mentions that users or packagers can use these options to link to their system libraries (and I suppose note that python setup.py build --help is currently the way to see which libraries can be used in this way). Note that I recently made a change to install.rst in master (to fix some messed-up sectioning), so you will probably want to rebase against master first.

Other than that, it looks set to merge to me.

…guments are listed as being from the "build" command in the help.
Comment thread docs/install.rst

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe throw in a sentence at the end here that says something like "you can also put use-system-expat=1 in setup.cfg to make this the default behavior." Or not, if you think that makes it too confusing.

@eteq

eteq commented Jul 12, 2012

Copy link
Copy Markdown
Member

Feel free to incorporate that one minor comment (or not if you prefer) and merge!

mdboom added a commit that referenced this pull request Jul 13, 2012
Add a `--use-system-expat` option to use the system's expat library.
@mdboom
mdboom merged commit 3cb7fc1 into astropy:master Jul 13, 2012
keflavich pushed a commit to keflavich/astropy that referenced this pull request Oct 9, 2013
Add a `--use-system-expat` option to use the system's expat library.
@mdboom
mdboom deleted the cextern/expat branch May 21, 2014 23:54
jeffjennings pushed a commit to jeffjennings/astropy that referenced this pull request Jul 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants