refactor(LogQueries): wrap row-extraction with jsonb_build_array#422
refactor(LogQueries): wrap row-extraction with jsonb_build_array#422tonyalaribe wants to merge 8 commits into
Conversation
executeArbitraryQuery previously wrapped each row with
SELECT (SELECT json_agg(x.value ORDER BY x.ordinality)::jsonb
FROM json_each(row_to_json(sub.*)) WITH ORDINALITY AS x) FROM (...) sub
to coerce arbitrary rows into a single jsonb-array column for hasql to decode
into Vector AE.Value.
That wrapper is portable to Postgres only. TimeFusion (DataFusion-backed)
ships neither json_each, row_to_json, WITH ORDINALITY, nor row-correlated
lateral UDTFs — the wrapper made TF unable to serve log-explorer queries
through this path. The wrapper also tripped PostgreSQL's own JSON parser on
TEXT columns containing NULL bytes or lone surrogates (see prior comment at
the RawSessionRow site), forcing typed-decoder workarounds.
Replaced with the equivalent SQL:
SELECT jsonb_build_array(c1,...,cN) FROM (<inner>) sub(c1,...,cN)
This works natively on both backends:
- Postgres: jsonb_build_array(VARIADIC any) -> jsonb is built-in.
- TimeFusion: json_build_array UDF now exposes a jsonb_build_array alias.
executeArbitraryQuery gains a column-count parameter; the two callers
(selectLogTable, fetchEventExamples) already know it via
queryComponents.toColNames / colsNoAsClause cols. Derived-column aliases
sub(c1,...,cN) decouple the wrapper from the inner SELECT's expression shapes
so we never have to introspect those.
Side effect: the JSON-parser hazard documented near RawSessionRow is also
gone for the Postgres path, since jsonb_build_array builds JSON from typed
values instead of re-parsing a Postgres-side text serialization.
executeSecuredQuery (Dashboards / AI raw-SQL path) is left on the legacy
wrapper for now — its callers don't have column-count info and never worked
against TimeFusion anyway.
Helper exposed as Pkg.Parser.wrapForRowExtraction for future callers that
want to wrap their own SQL.
Code ReviewOverall: The motivation is solid and the approach is correct — 1.
|
- Drop unused 'wrapForRowExtraction' export from Pkg.Parser (YAGNI — exporting before a real caller exists expands the API surface and forces a future breaking rename). Inline the small alias-formula in executeArbitraryQuery; re-extract a HI.Sql-level helper if a second caller appears. - Update executeArbitraryQuery docstring: drop the dangling cross-reference, spell out the exact legacy expression we're replacing. - Guard 'colCount <= 0' — the previous code would emit 'SELECT jsonb_build_array() FROM (...) sub()' which is not valid SQL. Empty-projection inputs now short-circuit to V.empty. - Add TODO on executeSecuredQuery so the legacy wrapper there isn't overlooked once user/LLM-SQL column counts become derivable.
Code ReviewOverviewThis PR replaces the 🐛 Critical Bug — column alias count mismatch when
|
|
Found 1 test failure on Blacksmith runners: Failure
|
Critical bug flagged in PR review: selectLogTable was passing length toColNames as the wrapper's column count, but toColNames returns finalColumns (display columns only). When hasCountOver = True (aggregations, service-entry-spans target, summarize path), the inner SELECT also emits count(*) OVER() as _total_count — so PG sees N+1 output columns with only N sub() aliases and rejects with 'column alias list must be the same length as the number of output columns'. Pass length toColNames + (1 if hasCountOver else 0) so the alias list matches the actual emitted column count. Also bind colsNoAsClause cols once in fetchEventExamples (rawCols) so it's reused for both processedCols and the colCount.
Code ReviewThe PR makes sense and the motivation is solid — replacing the
|
Code ReviewOverviewClean, focused fix that replaces the Suggestions1. Over-verbose Haddock block (executeArbitraryQuery) The 8-line comment block spends most of its words describing the old implementation. Per the project's comment policy, only the non-obvious -- Wraps the inner SELECT so each row arrives as a jsonb array of column values.
-- Uses jsonb_build_array(c1,…,cN) rather than json_each/row_to_json/WITH ORDINALITY
-- — the latter is unsupported by TimeFusion and trips PG's JSON parser on NULL bytes.2. Overlong call-site comment in The 3-line comment before the -- +1 for count(*) OVER() appended to SELECT but absent from toColNames
$ executeArbitraryQuery (length queryComponents.toColNames + fromEnum queryComponents.hasCountOver) (rawSql q)3.
$ executeArbitraryQuery (length queryComponents.toColNames + bool 0 1 queryComponents.hasCountOver) (rawSql q)Minor style point; either is defensible. 4. Silent Dropping the query silently when 5. Intermediate list allocation in pure $ V.mapMaybe jsonArrayToVector (V.fromList results)
What's good
Test coverageThe PR test plan notes manual verification against a dev TimeFusion cluster is still pending (blocked on companion TF PR deploy). The PG path should be covered by existing regression tests, but it would be worth adding an automated integration test that exercises |
…ents Per CLAUDE.md "compactness is a priority" and PR review feedback.
Contributing to MonoscopeThank you for your interest in contributing! This guide covers development setup, running the project locally, testing, code style, and the PR workflow. Development SetupPrerequisites
Initial Setup# Clone the repository (fork first if contributing)
git clone https://github.com/monoscope-tech/monoscope.git
cd monoscope
# Install Haskell dependencies
cabal update
# Activate the pre-commit hook for auto-formatting
git config core.hooksPath .githooksRunning Locally with Docker ComposeThe fastest way to get a running instance: docker-compose upVisit Optional ProfilesStart additional services with Docker Compose profiles: # pgAdmin for database management (http://localhost:5050)
docker-compose --profile dev up
# Ephemeral test database (port 5433)
docker-compose --profile test upEnvironment VariablesCopy Building from Source# Build all targets
cabal build all -j
# Run the application
cabal run
# Or with live-reload using ghcid
make live-reloadRunning TestsUnit Tests (fast, no database required)cabal test unit-tests --ghc-options="-O0" --test-show-details=directOr via Makefile: make test-unitDoctestscabal test doctests --ghc-options="-O0" --test-show-details=directOr via Makefile: make test-doctestsIntegration Tests (requires external PostgreSQL)USE_EXTERNAL_DB=true cabal test integration-tests --ghc-options="-O0" --test-show-details=directOr via Makefile: make test-integrationAll Testsmake testRun Specific Tests# Match by test name pattern
cabal test unit-tests --test-options='--match="/pattern/"'
# Run with live file-watching
make live-test-reload-unitCode StyleFormattingFourmolu is used for Haskell code formatting. It is automatically run via a GitHub Actions workflow on every push — any formatting changes are auto-committed to your branch. You don't need to format manually before opening a PR. To format locally: make fmtTo format on commit (pre-commit hook is in # Already activated via git config core.hooksPath .githooks
# To bypass for a single commit:
SKIP_FOURMOLU=1 git commit ...LintingHLint checks for common Haskell code smells and style issues. make lintTo auto-fix lint issues: make fix-lintFrontendJavaScript/TypeScript uses Prettier (configured in npx prettier --write .Tailwind CSS is processed during the build — run it manually with: make post-cssPR WorkflowFork & Branch
Making Changes
Opening a Pull Request
Auto-Formatting on PushThe
Project StructureGetting Help
|
Code ReviewOverviewClean, well-motivated refactor. The switch from Issues / Suggestions1. Unnecessary intermediate vector in The new form allocates a full -- new (worse — intermediate vector):
pure $ V.mapMaybe jsonArrayToVector (V.fromList results)
-- old pattern (still used in executeSecuredQuery — prefer this):
pure $ V.fromList $ mapMaybe jsonArrayToVector results2. Alias construction — list comprehension that is just let aliases = T.intercalate "," ["c" <> show i | i <- [1 .. colCount]]With let aliases = T.intercalate "," $ map (("c" <>) . show) [1 .. colCount]3. Doc comment over-explains the guard The sentence about 4. Fragile caller-side column-count arithmetic (maintenance risk) executeArbitraryQuery (length queryComponents.toColNames + fromEnum queryComponents.hasCountOver) (rawSql q)If emittedColCount :: QueryComponents -> Int
emittedColCount qc = length qc.toColNames + fromEnum qc.hasCountOver5. Positives
|
Code ReviewOverviewClean, well-motivated fix. Replaces the IssuesRegression: intermediate vector allocation in
|
…ector first The earlier V.mapMaybe (V.fromList results) suggestion materializes the full vector before filtering. The original V.fromList $ mapMaybe results is cheaper: mapMaybe traverses the list and filters in one pass, then V.fromList packs once. Also stays consistent with executeSecuredQuery.
Code ReviewOverviewReplaces the Issues & Suggestions1. -- current
length queryComponents.toColNames + fromEnum queryComponents.hasCountOver
-- cleaner — Relude exports bool :: b -> b -> Bool -> b
length queryComponents.toColNames + bool 0 1 queryComponents.hasCountOver
2. Fragile column-count invariant — worth a type-level assertion or closer coupling The caller arithmetic 3. -- current — fine
let aliases = T.intercalate "," ["c" <> show i | i <- [1 .. colCount]]
-- marginally shorter using point-free
let aliases = T.intercalate "," $ map (("c" <>) . show) [1 .. colCount]Neither is wrong; just noting the alternative if point-free style is preferred elsewhere. 4. rawSql ("SELECT jsonb_build_array(" <> aliases <> ") FROM (")
<> querySql
<> rawSql (") sub(" <> aliases <> ")")Minor: the full SQL template is split across three fragments. A single-line helper or let wrap inner = rawSql ("SELECT jsonb_build_array(" <> aliases <> ") FROM (") <> inner <> rawSql (") sub(" <> aliases <> ")")5. Test coverage No new tests are added. Would be useful to have:
The 6. The inline TODO is fine for now, but since Minor / Nits
SummaryThe core change is correct and solves a real compatibility problem. Main asks: (1) switch |
|
Superseded by #423 and its follow-ups on master (b68260d, 0078dc5, 89817bf, 4acc4e7): the jsonb_build_array wrap is now inlined in SQL codegen, and executeSecuredQuery is covered via jsonb_build_array(sub.*) (TF WildcardFnArgExpander / PG record overload from migration 0099), so the manual column-count threading here is no longer needed. |
Summary
Replaces the legacy
json_each(row_to_json(*)) WITH ORDINALITYrow-extraction wrapper inexecuteArbitraryQuerywith the equivalentSELECT jsonb_build_array(c1,…,cN) FROM (<inner>) sub(c1,…,cN).Why:
json_each,row_to_json,WITH ORDINALITY, and row-correlated lateral subqueries — none of which TimeFusion (DataFusion-backed) supports today. So log-explorer queries against the TF backend fail at parse time withtable function 'json_each' not found.RawSessionRow);jsonb_build_arraybuilds JSON from typed values and avoids that.Companion PR in TimeFusion exposes a
jsonb_build_arrayalias on the existingjson_build_arrayUDF so the same wrapper works on both backends.Changes
executeArbitraryQuery :: HI.Sql -> …→Int -> HI.Sql -> …. NewIntis the column count of the inner SELECT. Wrapper buildsc1,…,cNaliases inline.colCount <= 0short-circuits toV.emptysincesub()with zero aliases is invalid SQL.selectLogTablepasseslength toColNames + fromEnum hasCountOversince thecount(*) OVER()extra column isn't intoColNamesbut is emitted in the SELECT.fetchEventExamplespasseslength rawCols(bindscolsNoAsClause colsonce and reuses).executeSecuredQueryleft on the legacy wrapper for now — its callers (Dashboards, AI raw-SQL) take raw user/LLM SQL without column info and never worked against TimeFusion anyway. TODO inline.Test plan
cabal build lib:monoscope— clean compile.jsonb_build_arrayis native in PG).