Skip to content

Commit 33e09cf

Browse files
authored
JITSU-269: fix ClickHouse upload stalls and lock rotor runtime dependencies (#1612)
JITSU-269 Node 24.22.x can stall chunked HTTP writes containing non-ASCII strings while waiting for a missing `drain` event. Rotor then hits ClickHouse's 30-second receive timeout and loses the affected log batch. Patch `@clickhouse/client` 1.18.3 to emit already-serialized JSON rows as UTF-8 Buffers, preserving the payload and streaming behavior. This works around nodejs/node#66781 while its upstream fix remains pending. The patch is applied through pnpm and included before Docker dependency fetch/install. A real HTTP regression test covers `JSONCompactEachRow` and `JSONEachRow`, checking request chunk types and exact payload bytes. Also make rotor's external runtime dependencies reproducible: move their manifest into `runtime-deps`, commit its npm lockfile, and copy both into `dist` for `npm ci --omit=dev`. The initial lock preserves all 77 installed runtime package versions from beta.96 and includes optional esbuild packages for other platforms. The README documents updates. Validation: - Node 24.22.1: both Unicode streaming tests time out with the original SDK and pass with the patch. - Production consumer-group canary using the exact beta.96 image with only its bundled encoder patched: two pods completed 914 inserts over 20 minutes with zero ClickHouse errors; the identical unpatched control reproduced the 30-second timeout in about four minutes. Test resources were removed after validation. - Clean Linux ARM64 and AMD64 runtime installs passed, with no dependency version drift; Kafka/MongoDB/prom-client loading, Zstd roundtrip, and esbuild smoke tests passed. - Full rotor build (all four bundles plus `npm ci`), focused TypeScript checks, formatting, and local pre-push reviews passed. The live test used the same temporary functions signing key in all variants against the existing beta.80 functions servers. Provisioning the production signing keys remains a separate prerequisite for rolling forward from beta.38.
2 parents fc1da6c + ac31b38 commit 33e09cf

10 files changed

Lines changed: 1474 additions & 16 deletions

File tree

‎all.Dockerfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ WORKDIR /app
4747
# STEP 1: Copy lockfiles and workspace config (smallest possible layer)
4848
# This layer only invalidates when dependency versions change
4949
COPY pnpm-lock.yaml pnpm-workspace.yaml package.json ./
50+
COPY patches ./patches
5051

5152
# STEP 2: Fetch dependencies into pnpm store
5253
# The builder image already has most packages cached at /pnpm-store

‎builder.Dockerfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ ENV NPM_CONFIG_STORE_DIR=/pnpm-store
2525

2626
# Copy only the files needed for dependency fetching and Playwright version extraction
2727
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
28+
COPY patches ./patches
2829
COPY libs/jitsu-js/package.json ./libs/jitsu-js/package.json
2930

3031
# Extract Playwright version before cleanup
@@ -37,7 +38,7 @@ RUN PLAYWRIGHT_VERSION=$(jq -r '.devDependencies["@playwright/test"]' ./libs/jit
3738
# The store will be available to all.Dockerfile when it uses this image as builder
3839
# Use --ignore-scripts to skip compilation - only fetch source packages
3940
RUN pnpm fetch --ignore-scripts --loglevel debug --verbose
40-
RUN rm -rf /package.json /pnpm-lock.yaml /pnpm-workspace.yaml /libs
41+
RUN rm -rf /package.json /pnpm-lock.yaml /pnpm-workspace.yaml /patches /libs
4142

4243
# Install Playwright globally with pnpm
4344
RUN PLAYWRIGHT_VERSION=$(cat /tmp/playwright-version.txt) && \

‎patches/@[email protected]‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
diff --git a/dist/utils/encoder.js b/dist/utils/encoder.js
2+
--- a/dist/utils/encoder.js
3+
+++ b/dist/utils/encoder.js
4+
@@ -24,7 +24,9 @@ class NodeValuesEncoder {
5+
return values;
6+
}
7+
// JSON* formats streams
8+
- return stream_1.default.pipeline(values, (0, stream_2.mapStream)((value) => (0, client_common_1.encodeJSON)(value, format, this.json.stringify)), pipelineCb);
9+
+ // Node 24.22.x can lose 'drain' for chunked non-ASCII strings.
10+
+ // Keep UTF-8 byte accounting consistent: https://github.com/nodejs/node/issues/66781
11+
+ return stream_1.default.pipeline(values, (0, stream_2.mapStream)((value) => Buffer.from((0, client_common_1.encodeJSON)(value, format, this.json.stringify), 'utf8')), pipelineCb);
12+
}
13+
// JSON* arrays
14+
if (Array.isArray(values)) {

‎pnpm-lock.yaml‎

Lines changed: 9 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pnpm-workspace.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
patchedDependencies:
2+
"@clickhouse/[email protected]": patches/@[email protected]
3+
14
packages:
25
- e2e
36
- examples/*
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
import { createClient } from "@clickhouse/client";
2+
import { ClientRequest, createServer } from "node:http";
3+
import { AddressInfo } from "node:net";
4+
import { Readable } from "node:stream";
5+
import { describe, expect, it, vi } from "vitest";
6+
7+
describe("ClickHouse streamed inserts", () => {
8+
it.each(["JSONCompactEachRow", "JSONEachRow"] as const)(
9+
"sends %s as UTF-8 buffers without stalling or changing rows",
10+
async format => {
11+
// Large enough to cross Node's HTTP high-water mark repeatedly. Non-ASCII
12+
// strings trigger nodejs/node#66781; ASCII-only payloads miss the regression.
13+
const rows = Array.from({ length: 4000 }, (_, i) => {
14+
const message = `Accrued payroll — café 中文 😀 ${i} ` + "ع".repeat(100);
15+
return format === "JSONCompactEachRow" ? [i, message] : { i, message };
16+
});
17+
const expected = Buffer.from(rows.map(row => JSON.stringify(row) + "\n").join(""));
18+
const chunks: Buffer[] = [];
19+
const server = createServer((req, res) => {
20+
req.on("data", chunk => chunks.push(chunk));
21+
req.on("end", () => {
22+
res.setHeader("X-ClickHouse-Query-Id", "buffer-regression");
23+
res.end();
24+
});
25+
});
26+
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
27+
const client = createClient({
28+
url: `http://127.0.0.1:${(server.address() as AddressInfo).port}`,
29+
request_timeout: 2000,
30+
});
31+
// Observe the real SDK-to-HTTP boundary, not the received socket data:
32+
// Node always delivers received bytes as Buffers, even for string writes.
33+
const writes = vi.spyOn(ClientRequest.prototype, "write");
34+
try {
35+
const result = await client.insert({
36+
table: "events_log",
37+
format,
38+
values: Readable.from(rows),
39+
});
40+
expect(result.executed).toBe(true);
41+
expect(Buffer.concat(chunks).equals(expected)).toBe(true);
42+
expect(writes.mock.calls.length).toBeGreaterThan(0);
43+
expect(writes.mock.calls.every(([chunk]) => Buffer.isBuffer(chunk))).toBe(true);
44+
} finally {
45+
writes.mockRestore();
46+
await client.close();
47+
server.closeAllConnections();
48+
await new Promise<void>((resolve, reject) => server.close(err => (err ? reject(err) : resolve())));
49+
}
50+
},
51+
10000
52+
);
53+
});

‎services/rotor/build.mts‎

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,14 @@
11
import * as esbuild from "esbuild";
2-
import { writeFileSync, mkdirSync, rmSync, statSync, readdirSync } from "fs";
2+
import { copyFileSync, readFileSync, mkdirSync, rmSync, statSync, readdirSync } from "fs";
33
import { execSync } from "child_process";
44
import { join } from "path";
55

6-
// Native modules that need to be external and installed with their versions
7-
const nativeDeps = {
8-
"@confluentinc/kafka-javascript": "1.9.0",
9-
"@mongodb-js/zstd": "2.0.0",
10-
esbuild: "0.27.0",
11-
mongodb: "6.12.0",
12-
"prom-client": "15.1.3",
13-
};
6+
// Dependencies kept outside the bundle have their own lockfile, including
7+
// transitive dependencies. Install them for the target platform with npm ci.
8+
const runtimeDepsDir = new URL("./runtime-deps/", import.meta.url);
9+
const nativeDeps: Record<string, string> = JSON.parse(
10+
readFileSync(new URL("package.json", runtimeDepsDir), "utf8")
11+
).dependencies;
1412

1513
// External deps for Deno functions-server.
1614
// Only runtime-compiled code and native binaries stay external.
@@ -179,11 +177,12 @@ esbuild
179177
mkdirSync("dist", { recursive: true });
180178
// Remove old node_modules to force recompilation of native addons for current platform
181179
rmSync("dist/node_modules", { recursive: true, force: true });
182-
writeFileSync("dist/package.json", JSON.stringify({ dependencies: nativeDeps }, null, 2));
180+
copyFileSync(new URL("package.json", runtimeDepsDir), "dist/package.json");
181+
copyFileSync(new URL("package-lock.json", runtimeDepsDir), "dist/package-lock.json");
183182

184183
// Install native deps
185184
console.log("Installing native dependencies...");
186-
execSync("cd dist && npm install --prod --no-package-lock", { stdio: "inherit" });
185+
execSync("npm ci --omit=dev --no-audit --no-fund", { cwd: "dist", stdio: "inherit" });
187186

188187
// Show sizes
189188
function getDirSize(dirPath: string): number {
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Rotor runtime dependencies
2+
3+
These packages stay outside the JavaScript bundle. `build.mts` reads this
4+
manifest to choose the external modules, copies both manifests into `dist`, and
5+
runs `npm ci --omit=dev` there. Native binaries are installed for the build
6+
platform; dependency versions come from this lockfile.
7+
8+
To update a dependency, change its version in `package.json`, then regenerate
9+
the lockfile from this directory without installing platform-specific binaries:
10+
11+
```sh
12+
npm install --package-lock-only --ignore-scripts --workspaces=false
13+
```
14+
15+
Commit both files and validate a clean `npm ci --omit=dev` on the target build
16+
platform. Do not commit `node_modules`. The initial lockfile preserves the
17+
installed dependency versions from rotor `2.15.0-beta.96`.

0 commit comments

Comments
 (0)