Closes #1762

Changes proposed in this Pull Request

This request fixes FileNotFoundError bugs preventing pypsa from being run as a Snakemake module.

The cause of this bug was that intermediate zip files in rules were being saved in directories that didn't exist yet (without creating the parent directories). This didn't fail when using PyPSA-Eur as a standalone module, because the directory was the same as the rule's output file. However, when using PyPSA-Eur as a Snakemake module, this was not the case as Snakemake prepends a prefix to all the input and output files, but not to any file locations listed as parameters. The fix was to use Snakemake's built-in shadow directories and save intermediate zip files at the top level within these. This was fixed for many rules in retrieve.smk, i.e., retrieve_eez, retrieve_nuts_2021_shapes, retrieve_nuts_2013_shapes, retrieve_worldbank_urban_population, retrieve_co2stop, download_wdpa, download_wdpa_marine.

Fixes detail

• When calling a rule imported using a snakemake module, before it is run, snakemake will prepend the given prefix (in our case, “pypsa-eur”) to all the input and output files of the rule. However, snakemake does not prepend the prefix for any file or directory locations listed as params. Therefore, any rules using params to list file or directory locations will return a FileNotFoundError as soon as a function looks for a file based on the location given in a param. Concretely, snakemake saves any input (including those fetched using the storage function) in the prefix directory (e.g., pypsa-eur/data/your/file.here), and if you try to access this file using a location derived from a param (e.g., data/your/file.here), you’ll get a FileNotFoundError. The fix is to never indicate a path relative to a param.
• Use Snakemake’s shadow directories (with setting “minimal”) to enable the temporary saving of zip files that are cleaned up automatically.
• For rules using a script that was not robust to Snakemake module use, the solution was similarly to make filepaths relative to any provided output files. Any intermediate files were processed using tempfile.
• Replace os.rename with shutil.copy2 to ensure the code has explicit permission to read zip files
• Change shutil.copy to shutil.copy2 to preserve the file metadata
• Change potentially problematic variable name zip to zip_file

Commands that didn’t work that now work

• snakemake pypsa-eur/data/eez/World_EEZ_v12_20231025_LR/eez_v12_lowres.gpkg --cores 1 (calls rule retrieve_eez)
• snakemake pypsa-eur/data/nuts/NUTS_RG_01M_2021_4326_LEVL_3.geojson --cores 1 (calls rule retrieve_nuts_2021_shapes)
• snakemake pypsa-eur/data/nuts/NUTS_RG_03M_2013_4326_LEVL_3.geojson --cores 1 (calls rule retrieve_nuts_2013_shapes)
• snakemake pypsa-eur/data/worldbank/API_SP.URB.TOTL.IN.ZS_DS2_en_csv_v2.csv --cores 1 (calls rule retrieve_worldbank_urban_population)
• snakemake pypsa-eur/data/CO2JRC_OpenFormats/CO2Stop_DataInterrogationSystem/Hydrocarbon_Storage_Units.csv --cores 1 (calls rule retrieve_co2stop)
• snakemake pypsa-eur/data/WDPA.gpkg --cores 1 (calls rule download_wdpa)
• snakemake pypsa-eur/data/WDPA_WDOECM_marine.gpkg --cores 1 (calls rule download_wdpa_marine)
• This rule ran successfully before, since all filepaths were already relative to outputs, not params: snakemake pypsa-eur/data/bgr/ihme1500_aquif_ec4060_v12_poly.shp --cores 1 (calls rule retrieve_aquifer_data_bgr)
• snakemake pypsa-eur/data/eurostat/Balances-April2023 --cores 1 (calls rule retrieve_eurostat_data)

More generally, the following overall commands now work

• snakemake _pypsa_solve_sector_networks and snakemake _pypsa_solve_elec_networks

Checklist

• I tested my contribution locally and it works as intended.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in config/config.default.yaml.
• Changes in configuration options are documented in doc/configtables/*.csv.
• Sources of newly added data are documented in doc/data_sources.rst.
• A release note doc/release_notes.rst is added.