Refactor SessionContext, BallistaContext to support multi-tenancy configurations - Part 3#2091
Conversation
|
Hi, @alamb @yjshen @andygrove @xudong963 @thinkharderdev @yahoNanJing Please help to take a look. |
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
I didn't review all the Ballista changes carefully, but I did review the changes to DataFusion and they look good to me.
Epic work @mingmwang
| } | ||
|
|
||
| /// Default session builder using the provided configuration | ||
| pub fn default_session_builder(config: SessionConfig) -> SessionState { |
There was a problem hiding this comment.
Since there is already a SessionContext::with_config_rt maybe we could name this function SessionState::with_config and the existing SessionState::with_config to SessionState::with_config_rt for consistency
| } | ||
| } | ||
|
|
||
| impl FunctionRegistry for TaskContext { |
| pub fn parse_expr( | ||
| proto: &protobuf::LogicalExprNode, | ||
| ctx: &SessionContext, | ||
| registry: &dyn FunctionRegistry, |
| // /// Batch size when reading CSV or Parquet files | ||
| // #[structopt(short = "s", long = "batch-size", default_value = "8192")] | ||
| // batch_size: usize, | ||
| /// Batch size when reading CSV or Parquet files |
|
Thanks @mingmwang. I do agree to remove the SessionContext from the Ballista executor and to serialize session related things in the task definition.😄🐂 |
|
I'll plan to merge this PR on Monday unless we hear anything else -- this PR is likely to accumulate conflicts quickly. In fact it looks like it has accumulated a few conflicts already which need to be resolved |
I will resolve the conflicts today. |
mingmwang
force-pushed
the
issue-1862-part3
branch
from
abee72c to
43ab96e
Compare
…figurations - Part 3 resolve review comments
mingmwang
force-pushed
the
issue-1862-part3
branch
from
43ab96e to
539edbe
Compare
|
Rebased to master to resolve conflicts. |
|
Thanks again @mingmwang and everyone else who helped review this PR! |
5 participants
Which issue does this PR close?
Closes #1862.
Rationale for this change
What changes are included in this PR?
This is the part3 of the PR. It covers the below parts:
&mut` selfin SessionContext methods, trivial UT fixesoptional_session_idtoExecuteQueryParams, add session_id and props toTaskDefinition. Create new SessionContext for the first time of the query and reuse the existing SessionContext for ongoing queries for a users.Are there any user-facing changes?