Fix Data Lake tests concurrent run by alexey-milovidov · Pull Request #80386 · ClickHouse/ClickHouse

alexey-milovidov · 2025-05-17T03:11:47Z

Changelog category (leave one):

CI Fix or Improvement (changelog entry is not required)

clickhouse-gh · 2025-05-17T03:12:13Z

Workflow [PR], commit [0744072]

bharatnc · 2025-05-17T06:07:07Z

I realized that one more common cause of flake is multiple tests all use the same bind ports which leads to errors similar to the following:

E               Error response from daemon: driver failed programming external connectivity on endpoint roottestdatabaseglue-gw1-minio-1 (086121f6bcda800827d82dce9519f6b87f7d76ad5f5610258b96a82aacd909ea): Bind for 0.0.0.0:9002 failed: port is already allocated

alesapin · 2025-05-23T11:33:50Z

@alexey-milovidov FYI, it's not allowed to have _ in bucket name https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html.

azat · 2025-05-23T16:23:08Z

tests/integration/compose/docker_compose_iceberg_hms_catalog.yml

-      /usr/bin/mc policy set public minio/warehouse;
+      until (/usr/bin/mc config host add minio http://minio:9000 minio ClickHouse_Minio_P@ssw0rd) do echo '...waiting...' && sleep 1; done;
+      /usr/bin/mc rm -r --force minio/warehouse-hms;
+      /usr/bin/mc mb minio/warehouse-hms --ignore-existing;


Maybe we should move bucket creation to the cluster.py as for other MinIOs?

ClickHouse/tests/integration/helpers/cluster.py

Lines 2607 to 2659 in 51be0b3

def wait_minio_to_start(self, timeout=180, secure=False):

self.minio_ip = self.get_instance_ip(self.minio_host)

self.minio_redirect_ip = self.get_instance_ip(self.minio_redirect_host)

os.environ["SSL_CERT_FILE"] = p.join(

self.base_dir, self.minio_dir, "certs", "public.crt"

)

minio_client = Minio(

f"{self.minio_ip}:{self.minio_port}",

access_key=minio_access_key,

secret_key=minio_secret_key,

secure=secure,

http_client=urllib3.PoolManager(cert_reqs="CERT_NONE"),

) # disable SSL check as we test ClickHouse and not Python library

start = time.time()

while time.time() - start < timeout:

try:

minio_client.list_buckets()

logging.debug("Connected to Minio.")

buckets = [self.minio_bucket, self.minio_bucket_2]

for bucket in buckets:

if minio_client.bucket_exists(bucket):

delete_object_list = map(

lambda x: x.object_name,

minio_client.list_objects_v2(bucket, recursive=True),

)

errors = minio_client.remove_objects(bucket, delete_object_list)

for error in errors:

logging.error(

f"Error occurred when deleting object {error}"

)

minio_client.remove_bucket(bucket)

minio_client.make_bucket(bucket)

logging.debug("S3 bucket '%s' created", bucket)

self.minio_client = minio_client

return

except Exception as ex:

logging.debug("Can't connect to Minio: %s", str(ex))

time.sleep(1)

try:

with open(os.path.join(self.minio_dir, "docker.log"), "w+") as f:

subprocess.check_call( # STYLE_CHECK_ALLOW_SUBPROCESS_CHECK_CALL

self.base_minio_cmd + ["logs"], stdout=f

)

except Exception as e:

logging.debug("Unable to get logs from docker.")

raise Exception("Can't wait Minio to start")

I've add a todo for now.

tests/integration/helpers/cluster.py

alesapin · 2025-05-23T17:42:46Z

E               Exception: Command [docker compose --project-name roottestdatabasehms-gw3 --env-file /ClickHouse/tests/integration/test_database_hms/_instances-0-gw3/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_iceberg_hms_catalog.yml --verbose up -d] return non-zero code 17: time="2025-05-23T16:01:23Z" level=trace msg="Docker Desktop integration not enabled"
E               time="2025-05-23T16:01:23Z" level=debug msg="using default config store \"/root/.docker/buildx\""
E               time="2025-05-23T16:01:23Z" level=debug msg="using default config store \"/root/.docker/buildx\""
E               time="2025-05-23T16:01:24Z" level=debug msg="serving grpc connection" spanID=1a08c18adf73eb92 traceID=688283e6744b3c5733acdc15fee8c146
E               time="2025-05-23T16:01:24Z" level=debug msg="stopping session" span="load buildkit capabilities" spanID=1a08c18adf73eb92 traceID=688283e6744b3c5733acdc15fee8c146
E               time="2025-05-23T16:01:24Z" level=debug msg="otel error" error="<nil>"
E               time="2025-05-23T16:01:24Z" level=debug msg="otel error" error="<nil>"
E               unable to prepare context: path "/ClickHouse/tests/hms_extensions" not found

What is this???

azat · 2025-05-23T17:45:14Z

What is this???

Egh, do we build the hive image every time?

$ grep '^  hive:' -A2 tests/integration/compose/docker_compose_iceberg_hms_catalog.yml
  hive:
    build: ./../../hms_extensions/
    restart: unless-stopped

We should use pre-built image!

azat · 2025-05-25T17:33:00Z

We need this very much, here it fails all/most of the time - https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=80777&sha=2a795af2631742290a640597a6e129685a1211fe&name_0=PR

alexey-milovidov · 2025-05-25T18:18:00Z

Please commit directly to this branch, let's merge it together!

azat · 2025-05-25T21:06:35Z

For now I've fixed with "hot fix"

We should use pre-built image!

For this I need access to docker registry to create a new image

E unable to prepare context: path "/ClickHouse/tests/hms_extensions" not found

I have no idea how this was possible for now, only a theory, at some point this image had been built and used from cache (but it should be only local on machine, so again, no idea how it was possible), now this file has been modified and compose tries to rebuild the image

azat · 2025-05-25T21:12:58Z

tests/integration/compose/docker_compose_iceberg_hms_catalog.yml

      - minio
-    image: minio/mc
+    # Stick to version with "mc config"
+    image: minio/mc:RELEASE.2025-04-16T18-13-26Z


I've also pinned all other docker images that was using in this tests

And about this place, it fails for me, since new mcli does not config command (alias should be used instead, but we stick with older version for now)

Cherry pick #80386 to 25.4: Fix Data Lake tests concurrent run

Cherry pick #80386 to 25.5: Fix Data Lake tests concurrent run

Backport #80386 to 25.4: Fix Data Lake tests concurrent run

Backport #80386 to 25.5: Fix Data Lake tests concurrent run

Cherry pick #80386 to 25.3: Fix Data Lake tests concurrent run

azat · 2025-06-01T18:32:41Z

removed the pr-backports-created label

Let's see if this will help for #81015 to processed

Backport #80386 to 25.3: Fix Data Lake tests concurrent run

Fix Data Lake

6948833

alexey-milovidov added the 🍃 green ci 🌿 Fixing flaky tests in CI label May 17, 2025

clickhouse-gh bot added the pr-ci label May 17, 2025

alexey-milovidov mentioned this pull request May 17, 2025

Allow compatible types in StripeLog tables #80304

Merged

alexey-milovidov mentioned this pull request May 18, 2025

Add asynchronous metrics about sockets #80188

Merged

bharatnc mentioned this pull request May 21, 2025

tests: fix flaky hms/glue/iceberg tests due to port overlaps #80646

Merged

azat mentioned this pull request May 22, 2025

Fix possible SSH protocol hung on startup (due to hang in replxx) #80688

Merged

alesapin added 2 commits May 23, 2025 13:03

Merge remote-tracking branch 'origin' into fix-data-lakes

4286208

Fix bucket naming

61ee0dc

alesapin changed the title ~~Fix Data Lake~~ Fix Data Lake tests concurrent run May 23, 2025

alesapin added 2 commits May 23, 2025 13:34

Followup

e78e92e

Followup

51be0b3

azat self-assigned this May 23, 2025

alesapin and others added 2 commits May 23, 2025 16:46

Better

40aa096

Update core-site.xml

c75e81f

azat approved these changes May 23, 2025

View reviewed changes

azat added 3 commits May 25, 2025 22:38

tests: fix typo in the message

0a6dab2

tests: fix build context or docker_compose_iceberg_hms_catalog.yml

cda61a6

tests: pin minio/mc tag to some older version with config

58ea6c7

tests: pin other docker images for data lakes tests

00b286b

azat reviewed May 25, 2025

View reviewed changes

tests: add a comments for data lake tests

0744072

robot-ch-test-poll4 added the pr-synced-to-cloud The PR is synced to the cloud repo label May 26, 2025

This was referenced May 26, 2025

Cherry pick #80386 to 25.4: Fix Data Lake tests concurrent run #80841

Merged

Cherry pick #80386 to 25.5: Fix Data Lake tests concurrent run #80842

Merged

This was referenced May 26, 2025

Update python dependencies in CI images to address Dependabot alerts #80658

Merged

Add IF EXISTS modifier to SYSTEM SYNC REPLICA #80810

Merged

This was referenced May 26, 2025

Forbid ATTACH AS [NOT] REPLICATED if table already exists #80816

Merged

Do not complain about unsupported MADV_POPULATE_WRITE for older kernels #80704

Merged

Fix crash in Kafka due to get_member_id() was creating std::string from NULL #80793

Merged

robot-clickhouse-ci-1 added a commit that referenced this pull request May 28, 2025

Merge pull request #80841 from ClickHouse/cherrypick/25.4/80386

cd84640

Cherry pick #80386 to 25.4: Fix Data Lake tests concurrent run

robot-clickhouse added a commit that referenced this pull request May 28, 2025

Backport #80386 to 25.4: Fix Data Lake tests concurrent run

fbf7fab

robot-clickhouse-ci-1 mentioned this pull request May 28, 2025

Backport #80386 to 25.4: Fix Data Lake tests concurrent run #80956

Merged

robot-clickhouse-ci-1 added a commit that referenced this pull request May 28, 2025

Merge pull request #80842 from ClickHouse/cherrypick/25.5/80386

94f181e

Cherry pick #80386 to 25.5: Fix Data Lake tests concurrent run

robot-clickhouse added a commit that referenced this pull request May 28, 2025

Backport #80386 to 25.5: Fix Data Lake tests concurrent run

295ac92

robot-clickhouse-ci-1 mentioned this pull request May 28, 2025

Backport #80386 to 25.5: Fix Data Lake tests concurrent run #80957

Merged

robot-ch-test-poll1 added the pr-backports-created-cloud deprecated label, NOOP label May 28, 2025

robot-clickhouse-ci-1 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label May 28, 2025

azat added a commit that referenced this pull request May 28, 2025

Merge pull request #80956 from ClickHouse/backport/25.4/80386

bf06157

Backport #80386 to 25.4: Fix Data Lake tests concurrent run

azat added a commit that referenced this pull request May 28, 2025

Merge pull request #80957 from ClickHouse/backport/25.5/80386

82b46e6

Backport #80386 to 25.5: Fix Data Lake tests concurrent run

Algunenano added v25.3-must-backport and removed pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore labels May 29, 2025

robot-ch-test-poll2 mentioned this pull request May 29, 2025

Cherry pick #80386 to 25.3: Fix Data Lake tests concurrent run #81015

Merged

robot-clickhouse-ci-1 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label May 31, 2025

azat added a commit that referenced this pull request Jun 1, 2025

Merge pull request #81015 from ClickHouse/cherrypick/25.3/80386

2accd58

Cherry pick #80386 to 25.3: Fix Data Lake tests concurrent run

azat removed the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Jun 1, 2025

robot-clickhouse added a commit that referenced this pull request Jun 1, 2025

Backport #80386 to 25.3: Fix Data Lake tests concurrent run

415d916

robot-ch-test-poll2 mentioned this pull request Jun 1, 2025

Backport #80386 to 25.3: Fix Data Lake tests concurrent run #81125

Merged

robot-ch-test-poll3 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Jun 1, 2025

azat added a commit that referenced this pull request Jun 1, 2025

Merge pull request #81125 from ClickHouse/backport/25.3/80386

d1a9c3b

Backport #80386 to 25.3: Fix Data Lake tests concurrent run

robot-clickhouse added the pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR label Jul 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Data Lake tests concurrent run#80386

Fix Data Lake tests concurrent run#80386
azat merged 12 commits intomasterfrom
fix-data-lakes

alexey-milovidov commented May 17, 2025 •

edited by azat

Loading

Uh oh!

clickhouse-gh bot commented May 17, 2025 •

edited

Loading

Uh oh!

bharatnc commented May 17, 2025

Uh oh!

alesapin commented May 23, 2025

Uh oh!

azat May 23, 2025

Uh oh!

azat May 25, 2025

Uh oh!

Uh oh!

alesapin commented May 23, 2025 •

edited

Loading

Uh oh!

azat commented May 23, 2025 •

edited

Loading

Uh oh!

azat commented May 25, 2025

Uh oh!

alexey-milovidov commented May 25, 2025

Uh oh!

azat commented May 25, 2025

Uh oh!

azat May 25, 2025

Uh oh!

azat commented Jun 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

	def wait_minio_to_start(self, timeout=180, secure=False):
	self.minio_ip = self.get_instance_ip(self.minio_host)
	self.minio_redirect_ip = self.get_instance_ip(self.minio_redirect_host)

	os.environ["SSL_CERT_FILE"] = p.join(
	self.base_dir, self.minio_dir, "certs", "public.crt"
	)
	minio_client = Minio(
	f"{self.minio_ip}:{self.minio_port}",
	access_key=minio_access_key,
	secret_key=minio_secret_key,
	secure=secure,
	http_client=urllib3.PoolManager(cert_reqs="CERT_NONE"),
	) # disable SSL check as we test ClickHouse and not Python library
	start = time.time()
	while time.time() - start < timeout:
	try:
	minio_client.list_buckets()

	logging.debug("Connected to Minio.")

	buckets = [self.minio_bucket, self.minio_bucket_2]

	for bucket in buckets:
	if minio_client.bucket_exists(bucket):
	delete_object_list = map(
	lambda x: x.object_name,
	minio_client.list_objects_v2(bucket, recursive=True),
	)
	errors = minio_client.remove_objects(bucket, delete_object_list)
	for error in errors:
	logging.error(
	f"Error occurred when deleting object {error}"
	)
	minio_client.remove_bucket(bucket)
	minio_client.make_bucket(bucket)
	logging.debug("S3 bucket '%s' created", bucket)

	self.minio_client = minio_client
	return
	except Exception as ex:
	logging.debug("Can't connect to Minio: %s", str(ex))
	time.sleep(1)

	try:
	with open(os.path.join(self.minio_dir, "docker.log"), "w+") as f:
	subprocess.check_call( # STYLE_CHECK_ALLOW_SUBPROCESS_CHECK_CALL
	self.base_minio_cmd + ["logs"], stdout=f
	)
	except Exception as e:
	logging.debug("Unable to get logs from docker.")

	raise Exception("Can't wait Minio to start")

Conversation

alexey-milovidov commented May 17, 2025 • edited by azat Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Uh oh!

clickhouse-gh bot commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bharatnc commented May 17, 2025

Uh oh!

alesapin commented May 23, 2025

Uh oh!

azat May 23, 2025

Choose a reason for hiding this comment

Uh oh!

azat May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alesapin commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azat commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azat commented May 25, 2025

Uh oh!

alexey-milovidov commented May 25, 2025

Uh oh!

azat commented May 25, 2025

Uh oh!

azat May 25, 2025

Choose a reason for hiding this comment

Uh oh!

azat commented Jun 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

alexey-milovidov commented May 17, 2025 •

edited by azat

Loading

clickhouse-gh bot commented May 17, 2025 •

edited

Loading

alesapin commented May 23, 2025 •

edited

Loading

azat commented May 23, 2025 •

edited

Loading