Skip to content

refactor: remove ParseSpec#19239

Merged
clintropolis merged 3 commits into
apache:masterfrom
clintropolis:remove-parse-spec
Mar 31, 2026
Merged

refactor: remove ParseSpec#19239
clintropolis merged 3 commits into
apache:masterfrom
clintropolis:remove-parse-spec

Conversation

@clintropolis

Copy link
Copy Markdown
Member

changes:

  • ParseSpec and all of its implementations have been removed, including JavascriptParseSpec and JSONLowercaseParseSpec, which have no InputFormat equivalents
  • migrated all tests which were still using ParseSpec to convert into a normal schema + input format to just use InputFormat directly instead
  • FlattenJSONBenchmark now measures the JsonInputFormat instead of using a json parse spec/parser

changes:
* `ParseSpec` and all of its implementations have been removed, including `JavascriptParseSpec` and `JSONLowercaseParseSpec`, which have no `InputFormat` equivalents
* migrated all tests which were still using `ParseSpec` to convert into a normal schema + input format to just use `InputFormat` directly instead
* `FlattenJSONBenchmark` now measures the `JsonInputFormat` instead of using a json parse spec/parser
@github-actions github-actions Bot added Area - Documentation Area - Batch Ingestion Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Mar 31, 2026
@clintropolis clintropolis changed the title remove ParseSpec refactor: remove ParseSpec Mar 31, 2026
@clintropolis
clintropolis merged commit f4f9c06 into apache:master Mar 31, 2026
87 of 90 checks passed
@clintropolis
clintropolis deleted the remove-parse-spec branch March 31, 2026 23:46
@github-actions github-actions Bot added this to the 37.0.0 milestone Mar 31, 2026
@renatocron

Copy link
Copy Markdown

Hello, Good work on release 37!

Was dropping JavaScriptParseSpec purely a "no one was using it" call, or is there a longer-term plan to phase JS out more broadly?

Asking because I'm currently relying on JS at the aggregation layer for things like a bitwise day-of-week mask during rollup:

{
  "type": "javascript",
  "name": "days_mask",
  "fieldNames": ["day_of_week"],
  "fnAggregate": "function(current, dow) { return current | (1 << dow); }",
  "fnCombine":   "function(a, b) { return a | b; }",
  "fnReset":     "function() { return 0; }"
}

No native longBitwiseOr aggregator exists afaik, so JS is the only real option here.
Would love to know if this is on borrowed time or not

@gianm

gianm commented May 19, 2026

Copy link
Copy Markdown
Contributor

Was dropping JavaScriptParseSpec purely a "no one was using it" call, or is there a longer-term plan to phase JS out more broadly?

ParseSpec in general had been deprecated for a long time, and in this release we removed all implementations. It wasn't part of a plan to phase JS out more broadly. Although, JS isn't generally recommended due to the lack of sandboxing. (The docs mention this is why it is off by default.)

I think in your case you could use an expression aggregator: https://druid.apache.org/docs/latest/querying/aggregations/#expression-aggregator. The functions bitwiseOr and bitwiseShiftLeft are there.

@clintropolis

Copy link
Copy Markdown
Member Author

I think in your case you could use an expression aggregator...

just in case you need that expression aggregator at ingest time: #19508

@renatocron

Copy link
Copy Markdown

Thanks both!
Good to know JS aggregators aren't going anywhere imminently, and the expression aggregator path is actually cleaner for this use case anyway.
Will migrate to bitwiseOr once 38 drops, appreciate the quick turnaround on #19508

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - Batch Ingestion Area - Documentation Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 Refactoring Release Notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants