Error opening a WarpedVRT that was created in the same process and stored in Google Cloud Store #3254
Description
I'm trying to create a WarpedVRT file and copy it into a Google Cloud Store bucket to be read later. I'm running into a strange problem where if I try to open the created VRT file in the same process that created it, I get a RasterIOError:
Traceback (most recent call last):
File "rasterio/_base.pyx", line 310, in rasterio._base.DatasetBase.__init__
File "rasterio/_base.pyx", line 221, in rasterio._base.open_dataset
File "rasterio/_err.pyx", line 359, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: '/vsigs/some-bucket/test/test.vrt' does not exist in the file system, and is not recognized as a supported dataset name.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jzxu/test.py", line 35, in <module>
r2 = rasterio.open(vrt_path) # Throws RasterioIOError
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jzxu/testenv/lib/python3.11/site-packages/rasterio/env.py", line 463, in wrapper
return f(*args, **kwds)
^^^^^^^^^^^^^^^^
File "/home/jzxu/testenv/lib/python3.11/site-packages/rasterio/__init__.py", line 355, in open
dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "rasterio/_base.pyx", line 312, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: '/vsigs/some-bucket/test/test.vrt' does not exist in the file system, and is not recognized as a supported dataset name.
However, I am able to open the same file using both gdalinfo and rasterio running in another Python interpreter.
Here's a minimal test case:
import os
import rasterio
import rasterio.shutil
import shutil
import tensorflow as tf
image_path = 'gs://some-bucket/test/image.tif'
vrt_path = 'gs://some-bucket/test/test.vrt'
temp_vrt_path = '/tmp/test.vrt'
with rasterio.open(image_path) as raster:
with rasterio.vrt.WarpedVRT(raster) as vrt:
rasterio.shutil.copy(vrt, temp_vrt_path, driver='VRT')
r1 = rasterio.open(temp_vrt_path) # This works fine.
print(r1.crs)
with open(temp_vrt_path, 'r') as f:
temp_contents = f.read()
# Have to copy the VRT from local filesystem to GCS manually because
# rasterio doesn't support creating the VRT directly in GCS.
with open(temp_vrt_path, 'r') as source, tf.io.gfile.GFile(vrt_path, 'w') as dest:
shutil.copyfileobj(source, dest)
with tf.io.gfile.GFile(vrt_path, 'r') as f:
dest_contents = f.read()
assert temp_contents == dest_contents # This passes.
r2 = rasterio.open(vrt_path) # Throws RasterioIOError
print(r2.crs)
I've attached the contents of the VRT here: test.vrt.txt
A few more observations:
- This doesn't happen if the files are written to the local filesystem, only on GCS. I didn't try with S3 though.
- If I change the "folder" of the VRT file to be different from the GeoTiff file, the error doesn't happen. So if I change
vrt_pathto'gs://some-bucket/test-123/test.vrt', the program will correctly open the VRT file. This led me to think this has something to do with the source image path being set as a relative path, but I checked the VRT contents and it is an absolute path.- I put "folder" in quotes because Google Cloud Storage doesn't have real folder, only files with slashes in their names and tool support simulating the existence of folders.
- If I rerun the script a second time without deleting the VRT file in between, it will successfully open the VRT file. This is all the more strange considering that the script overwrites the VRT file in GCS with a new copy before trying to open it. So the fact that the file previously existed should not affect the
opencall.
Any help would be appreciated. Thanks!
rasterio info:
rasterio: 1.4.2
GDAL: 3.9.3
PROJ: 9.4.1
GEOS: 3.11.1
PROJ DATA: /home/jzxu/testenv/lib/python3.11/site-packages/rasterio/proj_data
GDAL DATA: /usr/share/gdal/2.2
System:
python: 3.11.9 (main, Jun 19 2024, 00:38:48) [GCC 13.2.0]
executable: /home/jzxu/testenv/bin/python3
machine: Linux-6.9.10-1rodete5-amd64-x86_64-with-glibc2.39
Python deps:
affine: 2.4.0
attrs: 24.2.0
certifi: 2024.08.30
click: 8.1.7
cligj: 0.7.2
cython: None
numpy: 2.0.2
click-plugins: None
setuptools: 70.3.0