Conversation
Replace ASOF JOIN with UNION ALL + ORDER BY for querying multiple command/telemetry tables in Data Extractor. ASOF JOIN only returned rows from T0 (losing other tables' data) and SPLICE JOIN cannot chain 3+ tables. UNION ALL queries each table independently with positional NULLs for other tables' columns, then sorts by timestamp. Also adds tsdb_table_has_data? helper to filter non-existent tables and scopes calculated timestamp items to the row's source table so the frontend correctly identifies the packet name. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Rewrite stream_items to query each table independently and merge results by timestamp, avoiding JOIN issues with multi-table queries. Add multi-table integration tests covering interleaved merges, CMD+TLM mixing, table filtering, cross-table timestamps, batch boundaries, and multi-item streaming. Fix before(:suite) QuestDB availability check. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2900 +/- ##
==========================================
+ Coverage 78.28% 78.31% +0.03%
==========================================
Files 673 673
Lines 55204 55196 -8
Branches 728 728
==========================================
+ Hits 43216 43227 +11
+ Misses 11910 11891 -19
Partials 78 78
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ryanmelt
left a comment
ryanmelt
left a comment
There was a problem hiding this comment.
Can you move all uses of creating TSDB connections into the QuestDBClient helper file? That will make sharding it much easier, and consolidate all the TSDB stuff together into one file better.
![aikido-pr-checks[bot]](https://avatars.githubusercontent.com/in/898896?s=60&v=4){kind=small}
| # Returns true if the given TSDB table exists and has at least one row in the | ||
| # time range. Tables that have never received data don't exist in QuestDB. | ||
| def tsdb_table_has_data?(table_name, start_time, end_time) | ||
| query = "SELECT 1 FROM #{table_name}" |
There was a problem hiding this comment.
Potential SQL injection via string-based query concatenation - high severity
SQL injection might be possible in these locations, especially if the strings being concatenated are controlled via user input.
Show fix
Remediation: If possible, rebuild the query to use prepared statements or an ORM. If that is not possible, make sure the user input is allowlisted or sanitized. As an added layer of protection, we also recommend installing a WAF that blocks SQL injection attacks.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
|
2 participants
Replace UNION ALL with per-table k-way merge for TSDB streaming
Rewrite stream_items to query each table independently and merge results
by timestamp, avoiding JOIN issues with multi-table queries. Add
multi-table integration tests covering interleaved merges, CMD+TLM
mixing, table filtering, cross-table timestamps, batch boundaries, and
multi-item streaming. Fix before(:suite) QuestDB availability check.