Repository navigation
Use traitlets for our configuration. #16476
Description
What is the problem this feature will solve?
We currently bundle configobj with Astropy and use it to do our configuration. It's old, not well supported / actively developed, and doesn't do nice things like load configurations from JSON files.
Describe the desired outcome
We replace configobj with traitlets, which is the IPython and Jupyter configuration system (but that depends on neither). It's stable, well-supported, actively developed, supports a lot of modern conveniences, and has 0 dependencies.
Additional context
See the Design Requirements!
I wish our config system supported this.
Here are the main requirements we wanted our configuration system to have:
Support for hierarchical configuration information.
Full integration with command line option parsers. Often, you want to read
a configuration file, but then override some of the values with command line
options. Our configuration system automates this process and allows each
command line option to be linked to a particular attribute in the
configuration hierarchy that it will override.Configuration files that are themselves valid Python code. This accomplishes
many things. First, it becomes possible to put logic in your configuration
files that sets attributes based on your operating system, network setup,
Python version, etc. Second, Python has a super simple syntax for accessing
hierarchical data structures, namely regular attribute access
(Foo.Bar.Bam.name). Third, using Python makes it easy for users to
import configuration attributes from one configuration file to another.
Fourth, even though Python is dynamically typed, it does have types that can
be checked at runtime. Thus, a1in a config file is the integer '1',
while a'1'is a string.A fully automated method for getting the configuration information to the
classes that need it at runtime. Writing code that walks a configuration
hierarchy to extract a particular attribute is painful. When you have
complex configuration information with hundreds of attributes, this makes
you want to cry.Type checking and validation that doesn't require the entire configuration
hierarchy to be specified statically before runtime. Python is a very
dynamic language and you don't always know everything that needs to be
configured when a program starts.
Activity
neutrinoceros commentedon May 21, 2024
Vendoring something that isn't actively developed indeed sounds worse than adding a dependency on a package that doesn't have any.
Are you proposing to vendor traitlets or to add it as a dependency ? If we vendor it, could it create incompatibilities with IPython/Jupyter at runtime (I assume not) ? If we depend on it, how likely is it that future versions might break astropy in a significant way ?
nstarman commentedon May 21, 2024
Dependency.
Nothing can be promised, but it's been stable. And the config system in particular is stable because it's isolated: we would only use it for our config.
eerovaher commentedon May 21, 2024
Could switching to
traitletshelp with #9893?pllim commentedon May 21, 2024
I think you are talking about gutting out something fundamental in
astropy.config. I see your point but the impact is very high, as the configuration system is also widely used downstream. I wonder if this warrants an APE.nstarman commentedon May 21, 2024
@eerovaher.
Yes! Traitlets supports custom types. It's quite easy to do (https://traitlets.readthedocs.io/en/stable/defining_traits.html)
@pllim
Yes.
🎉
There's two different uses:
configuring Astropy itself. This is a breaking change in Astropy so downstream libraries would have to modify their code.
Downstream libraries building their own config by inheriting from
astropy.config. The current config in extern has seen very little change so it would be relatively easy to spin it out into a standalone legacy library that we basically don't touch except to add a deprecation warning. Then libraries that are building their own config on top ofastropy.configwould just have a trivial change to their import, e.g.legacy_astropy_config_system(or whatever we name it).Sadly, probably.
jdavies-st commentedon Oct 1, 2026
Agree that
traitletsis very nice.Several years ago @eslavich wrote up a nice issue showing how the JWST science calibration pipeline could move from
configobjtotraitlets, showing how you got the CLI for free, and how validation worked.spacetelescope/stpipe#10
We ran out of steam and never got further. But it was a nice idea.
astrofrog commentedon Oct 1, 2026
I'm a bit confused, I thought we mainly used configobj to deal with I/O to .cfg files? I don't think traitlets does I/O so even if we used traitlets for in-memory validation we would still need ConfigObj for I/O?
nstarman commentedon Oct 1, 2026
Traitlets does not natively write read/cfg files, in part for the same reasons Python switched from setup.cfg to a more structured format. Traitlets uses JSON.
Our non-mutually exclusive options are:
IMO we should do both, setting a long horizon on deprecating cfg.
neutrinoceros commentedon Oct 1, 2026
if we're planning to sunset
cfgsupport why go with json now that we have widely available toml support ?saimn commentedon Oct 1, 2026
Independently from the config file format (json is not the greatest for a config file) and from the library we use to read/write config files, the problem is that our configuration system is a complicated custom beast, supporting some weird features (e.g. hot reloading or temporarily modifying config items). And iirc config items are stored deep into the ConfigObj instances... (which has some performance impact: #12504).
I learned that painfully when #17461 was reverted because of one of those features (#18138).
So I think the discussion needs to start with questioning the usefulness of those features, otherwise good luck with changing anything here.
pllim commentedon Oct 1, 2026
I don't see the point of breaking so many things for no apparent benefit now that we no longer bundle configobj. I think the existing config system will die when we archive this repo. Can I close this as won't fix?
nstarman commentedon Oct 1, 2026
Not bundling configobj is better than bundling, but see the release cadence. If we're ever interested in supporting IMO near-tablestakes features for 2020+ (see Additional context in #16476 (comment)) this is a must fix.
traitletssupports this relatively easily.See how I do it in https://unxt.readthedocs.io/en/stable/guides/configuration.html#temporary-configuration-with-context-manager
We can do that easily too! Anything that can be mapped to a dict is supported by traitlets, which ingests dicts. (That's why JSON is the easiest format, it basically is a dict.)