I believe this PR is finally ready for a first round of review!

Pinging some folks who have contributed to our GDAL recipe in the past: @Sinan81 @snehring @haralmha @rouault @amaji @payerle @michaelkuhn @iarspider @neilflood

Internally, this PR makes a ton of changes so that we can build with CMake or Autotools depending on GDAL version. Externally, the main user-facing change is that all variants have been renamed from the dependencies they add to the drivers that require them. Instead of saying spack install gdal+openjpeg, you would use spack install gdal raster=jp2openjpeg to install GDAL's "JPEG2000 driver based on the OpenJPEG library".

The main benefit of this is that we can now run spack external find gdal and detect the variants of the package from gdalinfo --formats and ogrinfo --formats. I also think it may be more intuitive to specify the drivers you want than to specify the dependencies you want to add. One downside is that spack install gdal raster=jp2openjpeg will only build the jp2openjpeg driver, you have to add the entire list from spack info gdal if you want to build that driver + the defaults.

In hindsight, this solution may be more trouble than it's worth. There isn't a one-to-one mapping between many of the CMake/Autotools flags and the drivers they enable, so the package became really complicated (1,000+ lines). I'm happy to maintain this, but I'm curious what Spack + GDAL users think about dependency variants vs. driver variants. If we switch back to dependency variants we could re-add GDAL 1 and 2, although I'm not sure if anyone is still using them.

I also changed the default drivers to build everything with no/required/recommended dependencies, so hopefully that will make @jcphill happy. Would love to hear more feedback on which drivers/dependencies we should build by default.

One downside is that spack install gdal raster=jp2openjpeg will only build the jp2openjpeg driver, you have to add the entire list from spack info gdal if you want to build that driver + the defaults.

It would be clearer to just have the user specify the changes to the default driver set, perhaps as with_raster=jp2openjpeg (and maybe also without_raster=... to disable defaults).

Thanks for the ping. I am a bit out of this loop these days, but you are welcome to any thoughts I have.

I think overall I agree with the deprecation of the old versions <3.0, no problem.

The intuitiveness or otherwise of the new driver selection is probably outside my expertise. I come from having spent many years building GDAL using the configure script, so having Spack's options look similar to that never bothered me. However, I can see the point, and doubly so if the new CMake build has changed its approach to this, too. Probably good to have Spack reproduce something similar.

Your warning about raster= selecting ONLY that driver is much appreciated. I think that this would be difficult. I agree with @jcphill that something which allowed adding/subtracting to the default set of drivers would be easier to use.

re: the default drivers set. In the past I have thought it would be better to include as many drivers as possible by default. My general thought would be only to exclude things which had requirements for libraries existing outside of Spack, e.g. proprietary libraries like the FileGDB and OCI drivers, etc. The joy of GDAL is that it makes it easy to read/write any format, so it is usually just one more irritation to find that the build one is using did not include the driver one suddenly needs. There are trade-offs, of course, so I don't feel strongly about it.

I'll second @neilflood that including as many drivers as possible would minimize user frustration/astonishment. Having an installation that "just works" without needing to recompile for small workload modifications is well worth a much longer but low-effort compile. Let the people who want a fast, minimal, single-workload build go through the effort of listing the exact drivers they need.

I'll second @neilflood that including as many drivers as possible would minimize user frustration/astonishment.

as upstream GDAL developer, I'd seoncd that: a build with as many drivers as possible is best to avoid users filing tickets to upstream "I don't understand why file in format X cannot be opened despite GDAL documentation advertizing support for it".

Sorry I've been busy. Won't be able to review till the week of June 20th.

Thanks for the reviews everyone! It seems like the raster/vector driver multi-valued variants might be more trouble than they're worth. We could do something like provide a list of drivers to add and a list of drivers to remove, or turn every value into a separate variant, but this will increase the complexity of an already 1,000+ line build recipe.

Given the above feedback and the confusion I myself had when trying to map the drivers documented online to the CMake variable names, I'm starting to think it might be better to keep the recipe as it was (variants control whether or not dependencies are built, drivers are chosen automatically based on available deps). This also allows us to retain support for GDAL 1 and 2 (although I hope no one still relies on them). I'll try to add CMake support only and keep the drivers as is in a second PR so that we can compare the two solutions.