test(verify): hand-write the configurations a schema cannot describe - #8363

…ython (apache#8709)

### What changes were proposed in this PR?

Aggregate and the sort family (Sort, Stable Merge Sort and Sort
Partitions) implement `StandaloneCodeGenerator`. The sort family's row
order is its contract, so `LogicalOp` gains `orderSensitive`, which
those three override. Anything comparing two runs of an operator reads
it to decide whether the rows have to arrive in the same order or only
be the same rows.

Review then found five places where the block answered a question the
engine answers differently. Aggregate summed an INTEGER column without
the wrap Java ints have, refused a TIMESTAMP column outright, answered a
timestamp average with a timestamp where AVERAGE is declared DOUBLE, and
spelled a boolean in Python's capitals rather than Java's lower case.
The sort put a NaN last on the way down, where the engine puts it first.

The edge values the shared table now carries found one more. Aggregate's
concatenation wrote a double past 1e7 or under 1e-3, and a timestamp, as
Python prints them, where the engine writes what Java's toString gives.
It now writes every value, the lower-case boolean the review asked for
included, through the helper apache#8597 adds for casting to text.

One change reaches code the engine runs. `getPhysicalPlan` swapped
Aggregate's aggregations for their global-stage form and left them on
the operator. The export compiles a plan and then translates the same
operators, so the script grouped on the result column names and failed.
The operator now keeps what the user set. The desc the global stage
receives is unchanged.

### Any related issues, documentation, discussions?

Part of apache#8325, step 11b; that issue lists the set in order. Aggregate's
concatenation calls a helper apache#8597 adds.

Sorting a NaN apart from a null, and grouping on one, need the nullable
dtypes apache#8704 reads an Arrow file into. Without it a null and a NaN share
a slot and both behave as they did before, so this PR is no worse on its
own and only complete alongside that one.

Closes apache#8707, the task this change is the whole of.

### How was this PR tested?

Each operator asserts the block it emits in its own spec. The Aggregate
cases the review named run in the verify tables of apache#8363, which put the
same input through the engine and through the script and compare the
two. What those tables cannot reach stays here as executed tests: a
timestamp at two resolutions and in a zone other than UTC, and a null
kept apart from a NaN in the sort and the group key.

`AggregateOpDescSpec` builds the physical plan and then checks that the
operator still holds the user's aggregations and that its generated code
still reads the input column.

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>

…as Python (apache#8705)

### What changes were proposed in this PR?

File Scan, the file-scan source, Text Input and URL Fetcher implement
`StandaloneCodeGenerator`. Each slices the raw lines before parsing, as
the engine does, so a line outside the window is never converted. A
boolean line is read as `parseField` reads one, where 1 is true and
other text ends the run. A timestamp line is kept as text and parsed as
a column with the helper apache#8597 adds, as the CSV readers parse theirs.
Every file is read as bytes and decoded the way the engine's reader
decodes it. Bytes the charset cannot hold become U+FFFD, and UTF-16 with
no byte-order mark reads as big-endian. Python's codecs refuse both, so
a file read under the wrong Encoding used to end the script while the
engine read on.

URL Fetcher is reported as unverifiable rather than exported blind. It
reads a live URL, so two runs are not required to agree and a comparison
would only measure the network. File Scan takes its filenames from an
input port, so the verification feeds it a table that names the files to
read, as it feeds a transform. That lands with apache#8364 and apache#8363.

### Any related issues, documentation, discussions?

Part of apache#8325, step 12c; that issue lists the set in order. It needs
apache#8341 for the shared source code and apache#8597 for the timestamp helpers, so
both merge first.

Closes apache#8703, the task this change is the whole of.

One engine defect was found while matching these readers, and it is
fixed here rather than worked around: the export alone could not close
it. apache#8598, where a file scan extracting an archive could not also
include the filename, because the line-by-line branch emitted a
one-field row against the two-column schema its own flag declares. A
second, apache#8596, where the Encoding field never reached the executor, was
fixed upstream by apache#8605, and this branch takes that fix.

Closes apache#8598.

### How was this PR tested?

Each operator asserts the block it emits in its own spec. Where a
difference is one of behaviour rather than of text, the spec runs the
engine's own executor and the generated script over the same input and
compares what each produced: the boolean parsing rules and the entries
read out of an archive. Once the verification lands, every operator is
also run both ways on every configuration its schema offers; this branch
is cut from main and does not carry that machinery, so those runs are
not on this diff's CI.

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>

…pache#8506)

### What changes were proposed in this PR?

Six operators implement `StandaloneCodeGenerator`: the three joins and
the three set operations. What they have in common is that they combine
or compare tables rather than read what is inside a row.

apache#8340 keeps the operators that work on text. Aggregate and the sort
family follow in apache#8709, and the samplers and control flow in apache#8710.

Review then found two places where the block answered a question the
engine answers differently. The hash join asked pandas for a suffix,
which appends once and then refuses the duplicate it just made, and let
an outer join's holes cost an integer column its type.

Running the engine on those cases found the platform out of step with
itself. HashJoin declared a right column's new name by checking only the
left names, while the executor also avoids the other right names. So a
right `x` beside a right `x#@1` was declared as a second `x#@1`, and one
value was lost. An unmatched right row in an outer join kept its raw
names and landed in the left column. Both now go through one renaming
rule in `JoinUtils`.

The shared table then found one more. In a right or full outer join on
keys both sides name alike, the engine writes an unmatched right row's
key into the left key column, and the export left it empty. The engine
had its own half: an unmatched right key named like a left payload
column landed in that payload. The export now fills the key, and the
engine leaves it out of the payload.

The set operations matched rows the way pandas does. The engine's
HashSet compares rows with Scala's `==`, under which NaN equals nothing,
and hashes them by each double's bits, which tell 0.0 from -0.0. The
export now selects rows by a key that follows both rules.

A probe worker that received no build rows could not name an unmatched
right row's columns, because it read the left names off the first build
row. This happens when the left input is empty or no left key hashes to
that worker. The runtime now records each input port's declared schema
on the executor before the port finishes
(`OperatorExecutor.inputSchemas`, filled in `EndChannelHandler`), and
the probe reads the left names from it. This is the Scala counterpart of
what apache#8766 does for Python operators.

### Any related issues, documentation, discussions?

Part of apache#8325, step 11a; apache#8709 and apache#8710 follow. That issue lists the
set in order.

The set operations and joins keyed on a null or a NaN need the nullable
dtypes apache#8704 reads an Arrow file into. Without it a null and a NaN share
a slot and both behave as they did before, so this PR is no worse on its
own and only complete alongside that one.

Closes apache#8505, the task this change is the whole of.

### How was this PR tested?

Each operator asserts the block it emits in its own spec. The interval
join reads its keys off the merged frame rather than writing over an
input column that shares a temporary's name.

The join cases the review named run in the verify tables of apache#8363, which
put the same input through the engine and through the script and compare
the two. What those tables cannot reach stays here as executed tests: a
null kept apart from a NaN in the set operations and the join key.
`HashJoinProbeOpExecSpec` pins the platform fixes.

Each set operation's spec runs its native executor and the export over
NaN, signed-zero, finite and null rows. `HashJoinProbeOpExecSpec` covers
an empty build side, and `EndChannelHandlerSpec` checks that the schema
is in place before the operator finishes the port.

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>