Conversation
Codecov Report
@@ Coverage Diff @@
## golden #149 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 35 38 +3
Lines 2173 2242 +69
Branches 477 489 +12
=========================================
+ Hits 2173 2242 +69
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
|
Development on https://gitlab.com/potato-oss/google-cloud/gcloud-storage-emulator seems slowed so I tried the https://github.com/oittaa/gcp-storage-emulator fork, which supports a bit more. However, I still had a couple issues (eg: oittaa/gcp-storage-emulator#147). A quick test with https://github.com/fsouza/fake-gcs-server (as GCSFS uses) seemed to work but it's not available as a python lib so would need to shell out/distribute somehow. The gcp-storage-emulator maintainer has been quite responsive, so I'll go with that and fall back to fake-gcs-server. -- For posterity, here's a few different (over-engineered) versions of the emulator fixture I've tried while going back and forth here... gcp-storage-emulator@pytest.fixture(scope="session")
def gcs_emulator() -> Generator[tuple[str, int], None, None]:
# port=0 -> run on a random open port; though we have to lookup later.
server = gcp_storage_emulator.server.create_server("localhost", 0, in_memory=True)
server.start()
try:
host, port = server._api._httpd.socket.getsockname()
with mock.patch.dict(os.environ, {"STORAGE_EMULATOR_HOST": f"http://{host}:{port}"}):
yield host, port
finally:
server.stop()fake-gcs-server dockerI originally tried this Docker version (similar to GCSFS's test suite), but the GitHub Mac runner doesn't have docker-for-mac installed due to EULA issues. @pytest.fixture(scope="session")
def gcs_emulator(
fake_gcs_server_version: str = "1.37.0", port: int = 2784
) -> Generator[str, None, None]:
if not sh.which("docker"):
raise pytest.skip("docker not available to run fake-gcs-server")
container = "arti-test-gcs-emulator"
url = f"http://localhost:{port}"
sh.docker.run(
"-d",
f"--name={container}",
f"-p={port}:{port}",
f"fsouza/fake-gcs-server:{fake_gcs_server_version}",
"-backend=memory",
"-scheme=http",
f"-external-url={url}",
f"-port={port}",
f"-public-host={url}",
)
try:
time.sleep(0.25)
requests.get(url + "/storage/v1/b").raise_for_status()
with mock.patch.dict(os.environ, {"STORAGE_EMULATOR_HOST": url}):
yield url
finally:
sh.docker.rm("-fv", container)fake-gcs-server binariesMACHINE_MAP = {
"aarch64": "arm64",
"arm64": "arm64",
"x86_64": "amd64",
}
BIN_CACHE_DIR = Path(__file__).parent / ".bin_cache"
BIN_MACHINE = MACHINE_MAP[platform.machine()]
BIN_SYSTEM = platform.system()
def _get_fake_gcs_server_cmd(
machine: str = BIN_MACHINE, system: str = BIN_SYSTEM, version: str = "1.37.0"
) -> sh.Command:
binpath = BIN_CACHE_DIR / f"fake-gcs-server-{version}-{system}-{machine}"
if not binpath.exists():
tgz_name = f"{binpath}.tgz"
url = f"https://github.com/fsouza/fake-gcs-server/releases/download/v{version}/fake-gcs-server_{version}_{system}_{machine}.tar.gz"
with requests.get(url, stream=True) as resp:
if resp.status_code != requests.codes.ok:
pytest.skip(f"fake-gcs-server for {system} {machine} is not available")
with open(tgz_name, "wb") as tgz:
shutil.copyfileobj(resp.raw, tgz)
sh.tar("-xf", tgz_name, "fake-gcs-server")
sh.mv("fake-gcs-server", binpath)
sh.rm(tgz_name)
return sh.Command(binpath)
@pytest.fixture(scope="session")
def gcs_emulator(port: int = 2784) -> Generator[str, None, None]:
url = f"http://localhost:{port}"
fake_gcs_server_proc = _get_fake_gcs_server_cmd()(
"-backend=memory",
"-scheme=http",
f"-external-url={url}",
f"-port={port}",
f"-public-host={url}",
_bg=True,
)
try:
time.sleep(0.25)
requests.get(url + "/storage/v1/b").raise_for_status()
with mock.patch.dict(os.environ, {"STORAGE_EMULATOR_HOST": url}):
yield url
finally:
fake_gcs_server_proc.terminate()
fake_gcs_server_proc.wait() |
685edd4 to
b005093
Compare
Signed-off-by: Jacob Hayes <[email protected]>
Signed-off-by: Jacob Hayes <[email protected]>
Signed-off-by: Jacob Hayes <[email protected]>
Signed-off-by: Jacob Hayes <[email protected]>
Adds
GCSFileandGCSFilePartitionStorage types + I/O for JSON and Pickle formats.Tests for external resources can be a bit tricky, but
gcsfs(and originalgoogle-cloud-storagelib) support aSTORAGE_EMULATOR_HOSTenv var. During gcs test setup, we spin up an instance ofgcp-storage-emulator(there are a couple alternatives, namelyfake-gcs-server) and give each test a separate bucket for isolation.Closes #121