Software Irresponsibility
In the way of achieving a healthy software development environment,
a lot of projects fall in one of these two DON'Ts:
irresponsibility and territoriality.
Irresponsibility is when there is no one in charge
of making things right (with respect to a whole category of problems).
Territoriality is when there is someone in charge,
and he won't let anyone else touch the code without him.
They may sound opposite to one another, but often
irresponsibility is a result of territoriality, where
the person in charge just isn't interested in the kind of problem
you're experiencing, and so any consideration for such problems
gets disregarded in favor of whatever fits the interests of the maintainer.
As a notorious example of irresponsible software design,
I will cite PHP, whose maintainer has repeatedly claimed
he was never interested in designing a good programming language.
It shows.
But I'm sure you'll find plenty of familiar examples
in your own experience,
from the irresponsive provider of some proprietary software
you've been locked into using,
to maybe someone from some other department of your company
that you've had to deal with,
and perhaps even to yourself with respect to your own users.
Of course, it is always possible to start a new project
to replace the software you're dissatisfied with.
But the costs associated to such endeavour are often huge.
You will have to reimplement not just the parts
that the replaced software did wrong,
but also the parts it did right.
While by assumption you can clearly see parts of the design that are wrong
and do better with respect to those parts, you may be lacking
the proficiency, time and interest to do as well on these other parts.
Last but not least, the community around the original software,
that made it valuable, will not switch to the new one for merely small gains,
and the transition from the old system to the new system
may impose demanding constraints of backwards compatibility.
Free Software can drastically reduce the cost of starting a new project,
by allowing you to fork the existing code base, copy large chunks of it,
or at the very least read the code of what you may have to be compatible with.
It does not make the cost go down to zero, however.
There again, cruft that made the original software bad enough
that you felt like changing it
will be a heavy burden upon you if you keep it,
and a huge setback to reimplement if you decide to throw it away.
And even in the world of free software, some licenses can incur
a higher cost to forking than others: for instance,
licenses designed to be incompatible with other licenses,
licences that force you to distribute the code as source,
that prevent you from making any adaptation and keep the name,
or otherwise put hurdles in the way of forkers.
In the world of Common Lisp, an interesting case
of not-so-responsible software is
ASDF.
Some of you might be tempted to sneer,
is he going to blame the problem with ASDF on
the original author who is long gone,
and the current maintainer who isn't committing enough energy
and availability to making the project go forward?
No, not at all.
Dan Barlow actually did a great job while he was there,
and if no one is stepping up to offer more than Gary King provides,
it's certainly not Gary who is to blame;
he is getting blatant bugs fixed and accepting consensual patches,
and at this point it would be trivial to fork the git repo
if anyone were dissatisfied with Gary's performance.
Rather, I'll argue that any problem with ASDF is a symptom of the way
responsibilities are distributed with respect to its update,
and propose a small technical change that can hopefully improve things a lot.
When Dan Barlow wrote ASDF in 2001, he only wanted a better
semi-portable CL build system that he could use
on top of SBCL for his own projects—something that would be more declarative
than the previous best alternative MK:DEFSYSTEM,
and would only need to support modern ANSI CL implementations
not old forgotten Lisp variants.
In developing ASDF, he loosely followed the better design principles
once proposed by Kent Pitman for an extensible CL build system.
In all those objectives, ASDF was a tremendous success,
and CL users should all be grateful to Dan for his achievement.
However, as usual the problem lies with issues he didn't try to tackle.
Problems with ASDF include
the painful way you have to configure it,
deficient support for
documentation
or testing,
broken support for conditional compilation,
missing support for generated files,
its weird TRAVERSE algorithm,
etc.
Unlike the issue of
deterministic incremental compilation
that prompted me to start XCVB,
none of the above issues is unsolvable in itself within ASDF.
But the constraints within which ASDF is developed make it hard to solve them.
Happily, some of these constraints can be removed with a little bit of hackery.
Most importantly, ASDF has a distribution bootstrap problem:
it specifies how to load other systems, but
nothing specifies how ASDF itself is loaded into your system.
That problem is left to whoever writes the Lisp implementation,
distributes the Lisp package, or maintains a larger project
within which ASDF is used.
And in the world of Common Lisp, there are tens of different implementations,
many of them packaged in various ways.
To make the problem tractable, ASDF follows the constraint that it
comes as a single file that may be loaded independently of its location,
and doesn't depend on any third party library beside what is included
by each supported implementation.
This means that the provider of libraries that use ASDF
must rely on the least common denominator of ASDF features,
unless he wants to force his users to make sure they upgrade their ASDF.
Upgrading the current ASDF could conceivably be easy, but is actually hard,
because of the
inability to hot-upgrade a loaded ASDF.
It is still possible to remove an old ASDF and replace it,
and XCVB does it in an ugly way
(see how xcvb/no-asdf deletes the ASDF package and anything that uses it);
however, files compiled against the old ASDF may not work with the new one
and vice versa
(I've had "interesting" issues that way
while using xcvb-master in slightly different setups).
This inability to upgrade further
makes the pre-packaging of a Lisp implementation
with a given version ASDF a bit of a damned if you do, damned if you don't
proposition to Lisp implementers and distributors.
Happily, implementers have the option of only providing ASDF when REQUIRE'd,
and this is the preferred way for users to specify they want to use ASDF,
when it is available.
Distributors may not have this option inasmuch as
there is no standard mechanism to hook into an implementation's REQUIRE.
An example of a failed attempt at distributing Common Lisp with ASDF is
common-lisp-controller. It tried to provide a way to make sure there is always
a configured ASDF, but their solution only works if everyone uses only
debian or another supported OS distribution,
and only if the debian maintainers keep up with all
the common-lisp software (implementations and libraries)
that one may want to use;
two hypotheses that have been disproved in practice
with respect to the needs of many users.
To step back to the problem of responsibility,
the current constraints of ASDF are such that a consistent upgrade of ASDF
requires action on the part of tens of people, each in his own territory:
ASDF contributors, Lisp implementers, package distributors.
If only one hacker alone decides to upgrade ASDF,
he has to incur the cost of the associated work of modifying ASDF
and the risk that this will introduce incompatibilities,
for little overall progress in terms of what users may rely upon
unless he convinces tens of other people also do the same.
Thus, the costs of adoption are high, the benefits of development are low,
development is slow, and a cycle of stagnation continues.
If someone cares about the future of ASDF, I recommend they should address
the upgradability bug, which will itself unlock a big hurdle on the way
of further evolution of ASDF. Note that this upgradability includes both
the technical ability to load a new ASDF on top of an old one, but probably
also the proper configuration of ASDF, to load systems from a series of
configured paths declared in user and system preference files.
Personally, I'd rather work on XCVB, and invite
people interested in build systems to work on it instead of ASDF.
But if XCVB is to prevail, it may as well prevail
against the best ASDF that ASDF can be.
XCVB tries to avoid all the problems of ASDF by stepping back
from the assumption that everything happens
in a one Lisp world that has already been setup.
Instead, XCVB will manage many Lisp worlds that it will help you setup.
It doesn't live inside your Lisp image, and thus doesn't depend
on Lisp implementers or distributors to provide it for you;
it can therefore evolve fast without any issue due to slow distribution.
It is also liberated from the constraint ASDF has
of itself having no dependencies,
because these dependencies would both make bootstrap more difficult
and interfere with other versions of same software
as part of the software being built.
XCVB thus reuses plenty of libraries and is richer in features,
and will use even more libraries in the future.
All this in addition to its original essential distinguishing feature
of providing deterministic incremental compilation.
XCVB isn't complete yet, but I believe that on supported implementations
(currently only SBCL, CCL, CLISP, but I could add more on demand,
or you could do it yourself), it is already better than ASDF.
If you care for CL build systems, please fix ASDF;
but please give XCVB a try, too.
Even if you don't care for CL build systems,
organize your software developments so that
every upgrade step can be done by the single maintainer of a single project,
without requiring the coordination
between many people in many different projects.
This lesson should be particularly dear to those who dream of
making a Standard for Lisp, Scheme, Javascript, Java
or any other language or piece of software:
if you need to agree on something that will require
future coordination between plenty of people,
it is almost sure to fail to evolve fast enough to remain relevant,
even though one or two iterations might make it live
(see e.g. R6RS and its current attempted successor).
Instead, you should offer software that already runs
with a free software reference implementation that
others can trivially adapt to their systems if needed
(also in Scheme, compare the SRFI process).
After your code has become de facto standard, you might not even care
for it being any kind of de jure standard anymore.
Whereas if there's going to be a de jure standard,
it will be blood and conflict as long as there's anything to disagree about,
and still resentment at the unsatisfying compromise that results.
The same idea also accounts for
Conway's Law:
if you divide your software project between many teams,
each team will build its own code base,
and the structure of the software will end up following
the structure of the teams that build it.
Parts of the software where the ownership is not clearly assigned
but where authority is divided will be the scene for
conflict, slow evolution, bad design (if design at all), etc.
Vertical disintegration, where independent teams
manage layers of the software and its life-cycle,
will in particular lead to a multiplication of such dysfunctional interfaces
between entrenched overlapping code bases
the global architecture of which cannot be refactored.
Happily, solving the problem of irresponsible software
is much easier than solving the problem of
irresponsible government.
PS: I've since accepted co-maintainership of ASDF.
If problems persist beyond March 2010, blame me.