Strangely, I'm seeing the following error on my machine;

ERROR: TypeError: must be unicode, not str [sphinx.util]
Traceback (most recent call last):
  File "<stdin>", line 26, in <module>
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/sphinx/application.py", line 138, in __init__
    self._init_env(freshenv)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/sphinx/application.py", line 184, in _init_env
    return self._init_env(freshenv=True)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/sphinx/application.py", line 166, in _init_env
    self.env.find_files(self.config)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/sphinx/environment.py", line 345, in find_files
    self.srcdir, config.source_suffix, exclude_matchers=matchers))
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/sphinx/util/__init__.py", line 94, in get_matching_docs
    for filename in get_matching_files(dirname, exclude_matchers):
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/sphinx/util/__init__.py", line 78, in get_matching_files
    qdirs = [entry for entry in qdirs if not matcher(entry[1])]
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/sphinx/util/__init__.py", line 74, in <genexpr>
    for dn in dirs)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/sphinx/util/__init__.py", line 57, in path_stabilize
    newpath = unicodedata.normalize('NFC', newpath)
TypeError: must be unicode, not str
Sphinx Documentation subprocess failed with return code 1

@mdboom - do you agree with the change in this PR, and do you have any idea what is going on above?

See:

https://bitbucket.org/birkenfeld/sphinx/issue/1272/failure-when-running-sphinx-12b2#comment-6154427

for a bug report I filed.

@mdboom - what do you think we should do? Should we roll back to not requiring Sphinx 1.2 for the build?

So this is fixed in the latest developer version of Sphinx. Now it looks like it's possible to install the latest dev version of Sphinx in travis using pip, so will try that and if it works we could use it as a temporary measure until the next beta release.

@mdboom - what do you think?

Ok, so the tests are passing now with the latest dev version of Sphinx. Not too happy about having to do this, but should be temporary until 1.2b3 or 1.2 are out.

@mdboom - ok to merge?

I'm actually going to go ahead and merge this because c58b4d7 is needed to get affiliated package builds to work. Also annoying that all Astropy builds are failing because of 0.2b2 not working. We can update Sphinx back to the non-dev version as soon as there is a release.

(I asked this in #936, but it's probably better discussed here as this is more up to date)

@astrofrog and @mdboom - is there a reason why we need the minimum version to be 1.2? I'm fine with having travis build from the dev version, but we're not actually using any features from 1.2, and it's rather annoying to not be able to build with 1.1.3 now... (and I don't like the idea of shipping a version that requires a dev build of sphinx to make the docs, although that's debatable, I suppose)

I agree, maybe we should lower the requirement back down to 1.1 in conf.py. @mdboom - what do you think?

We need 1.2b2 or later specifically, because earlier versions do not work with current versions of graphviz. (They generate .dot files with syntax errors).

I don't think we should downgrade the Sphinx requirement to 1.1. In order to build our docs, we either need a dev version of Sphinx or an old version of graphviz. On most Linux systems, it's far easier to upgrade Sphinx than downgrade graphviz, therefore, I think it's best to leave this as-is, though the minimum version should be 1.2b2, not 1.2, which does not have the graphviz fix.

I guess it comes down to what needs_sphinx is to be interpreted as. To me, that means "this is the lowest version of sphinx that can compile these docs based on the features they use." In that case, 1.1 is the right answer. But the alternative that I think @mdboom is thinking is "this is the lowest version that runs correctly on a typical sphinx install". I tend towards the former, because on a lot of systems, the version of graphviz will not be easily upgradable if you use the package manager, so now the docs won't build only because we set the version to 1.2, not because it actually won't work. But once the current release cycle of Sphinx is done, it will go back to working for everyone even if we leave needs_sphinx at 1.1, as the relevant packages will update their Sphinx versions to work correctly with graphviz.

Also, given where this change occurred, it propagates to affiliated packages. So now people working on affiliated packages that aren't necessarily working much on the core will suddenly be unable to build their affiliated package docs. That suggests a middle way of requiring only 1.1 in astropy/sphinx/conf.py and 1.2 in docs/conf.py. That gets around the affiliated package issues, at least.

Many of the packaging systems have already upgraded graphviz to the newer (stricter) version, that breaks when used with any Sphinx prior to 1.2.b2. And because Sphinx still has the "beta" label, none of the packagers are including it yet, even though it's "released" on PyPI (and even though PyPI explicitly recommends not posting beta releases). Ugh.

Perhaps the best solution here is to detect the version of graphviz dot. If too new, require Sphinx 1.2b2 or later. If old enough, we can leave the Sphinx requirement at 1.1. I'm happy to develop a PR for that if we agree it makes sense.

👍 to @mdboom's idea of detecting the dot version. That addresses all the use cases I can think of.

If it helps any, the last "known good" graphviz for me (macports and Sphinx 1.1.3) was 2.28.0