Skip to content

JSONEachRowWithProgress return types no longer accurate after ClickHouse v25.1 #438

@danxmoran

Description

@danxmoran

Describe the bug

ClickHouse v25.1 extended the JSONEachRowWithProgress response format in ClickHouse/ClickHouse#74181. The format used to stream back objects with either a row or a progress key, and nothing else. Now it can stream back objects with other keys (i.e. meta).

This change broke the client's definition of the RowOrProgress type (here), which encodes that either row or progress will be defined in each object. It's possible to work around by double-checking that .row is != null wherever it's used, but it's not obvious for users. I am assuming the server-side change also broke this project's tests, since they were disabled in 3c914ac around the time of the v25.1 release.

Expected behaviour

The type returned from a client call using the JSONEachRowWithProgress format should accurately describe all the possible top-level keys in the objects streamed back from the server.

Code example

import { createClient } from '@clickhouse/client'
import { isProgressRow } from '@clickhouse/client-common'

type Data = { number: string }

void (async () => {
  const client = createClient()
  const rs = await client.query({
    query: 'SELECT number FROM system.numbers LIMIT 100',
    format: 'JSONEachRowWithProgress',
  })

  let totalRows = 0
  let totalProgressRows = 0

  const stream = rs.stream<Data>()
  for await (const rows of stream) {
    for (const row of rows) {
      const decodedRow = row.json()
      if (isProgressRow(decodedRow)) {
        console.log('Got progress:', decodedRow.progress)
        totalProgressRows++
      } else {
        totalRows++
        if (totalRows % 100 === 0) {
          // Bug is here! Types say "row" is always defined, but it can now be undefined
          console.log('Sample row:', decodedRow.row)
        }
      }
    }
  }

  console.log('Total rows:', totalRows)
  console.log('Total progress rows:', totalProgressRows)

  await client.close()
})()

ClickHouse server

  • ClickHouse Server version: 25.1 (and later)

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions