-
Notifications
You must be signed in to change notification settings - Fork 56
JSONEachRowWithProgress return types no longer accurate after ClickHouse v25.1 #438
Description
Describe the bug
ClickHouse v25.1 extended the JSONEachRowWithProgress response format in ClickHouse/ClickHouse#74181. The format used to stream back objects with either a row or a progress key, and nothing else. Now it can stream back objects with other keys (i.e. meta).
This change broke the client's definition of the RowOrProgress type (here), which encodes that either row or progress will be defined in each object. It's possible to work around by double-checking that .row is != null wherever it's used, but it's not obvious for users. I am assuming the server-side change also broke this project's tests, since they were disabled in 3c914ac around the time of the v25.1 release.
Expected behaviour
The type returned from a client call using the JSONEachRowWithProgress format should accurately describe all the possible top-level keys in the objects streamed back from the server.
Code example
import { createClient } from '@clickhouse/client'
import { isProgressRow } from '@clickhouse/client-common'
type Data = { number: string }
void (async () => {
const client = createClient()
const rs = await client.query({
query: 'SELECT number FROM system.numbers LIMIT 100',
format: 'JSONEachRowWithProgress',
})
let totalRows = 0
let totalProgressRows = 0
const stream = rs.stream<Data>()
for await (const rows of stream) {
for (const row of rows) {
const decodedRow = row.json()
if (isProgressRow(decodedRow)) {
console.log('Got progress:', decodedRow.progress)
totalProgressRows++
} else {
totalRows++
if (totalRows % 100 === 0) {
// Bug is here! Types say "row" is always defined, but it can now be undefined
console.log('Sample row:', decodedRow.row)
}
}
}
}
console.log('Total rows:', totalRows)
console.log('Total progress rows:', totalProgressRows)
await client.close()
})()ClickHouse server
- ClickHouse Server version: 25.1 (and later)