You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Avoid FFI import/export when passing batches between two native plans #3925
When Comet has a native ShuffleWriter and a native child plan, batches get created in native code, then imported into JVM, then exported back to native. The JVM side does not need to access the data so the import/export is unnecessary overhead.
I would like to explore a simpler approach where a simple pointer (for the whole batch) is passed back and forth without any additional processing.
Activity
docs: add design spec for batch stash shuffle optimization