A file scan reads every file as UTF-8, whatever its Encoding field says #8596

What happened?

File Scan offers an Encoding field. FileScanSourceOpDesc stores it in its own encoding property, but FileScanSourceOpExec decodes with fileEncoding, the property inherited from ScanSourceOpDesc. The class carries @JsonIgnoreProperties(Array("limit", "offset", "fileEncoding")), so fileEncoding never survives serialization into the executor and is always its default, UTF_8.

Choosing any other charset therefore changes nothing. A UTF-16 file comes back decoded as UTF-8 rather than as its text.

Expected: the executor decodes with the charset the Encoding field names.

How to reproduce?

Deserialize a File Scan descriptor carrying "encoding":"UTF_16", write it out the way getPhysicalOp does, and read it back the way FileScanSourceOpExec does. fileEncoding comes back UTF_8, and the encoding the user chose is the only place UTF-16 survives.

In the UI: upload a UTF-16 text file, drop a File Scan on it, set Encoding to UTF_16 and run. The rows hold the file's bytes read as UTF-8, not its lines.

Version/Branch

1.4.0-incubating-SNAPSHOT (main)

Commit Hash (Optional)

2ab8ee0

What browsers are you seeing the problem on?

No response

Relevant log output

JSON = {"operatorType":"FileScan","dummyPropertyList":[],"encoding":"UTF_16","extract":false,"outputFileName":false,"attributeType":"string","attributeName":"line",...}
exec reads desc.fileEncoding = UTF_8