-
Notifications
You must be signed in to change notification settings - Fork 257
Allow setting default ConnectedComponents configurations from spark config #563
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
rjurney
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
|
latm (= looks asymmetric to me) Why allow only the |
|
I like where you're going with this... |
Because I'm lazy and greedy. And I'm gauging interest in this kind of approach. |
cae580b to
633b3da
Compare
Didn't know lazy people actually propose new things ... 😂 |
|
@james-willis jfyi, you can use a Alternatively, just run |
|
Hey, and what about the Spark Connect feature in Graphframes? isn't it affected also by this change? |
The new interface here is just spark config so I don't think there is any need to change anything about the connect integration but if you think there is some change to be made please point me in the right direction. |
91dd0a0 to
c368a39
Compare
It isnt, there is no problem, it is just an alternative way of passing configs. |
|
@james-willis in the latest changes we introduced a new mixin (WithAlgorithmChoice) to avoid code repeating that was added to ConnectedComponents. So, instead of redefining the default you should probably just call a setter. |
|
Bogged down at work. will get back to this eventually. |
4fc31ce to
de50969
Compare
src/main/scala/org/apache/spark/sql/graphframes/GraphFramesConf.scala
Outdated
Show resolved
Hide resolved
SemyonSinchenko
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Nice work @james-willis !
What changes were proposed in this pull request?
Enable setting the connected components algorithm from spark configurations.
Why are the changes needed?
spark configurations are a standard way to control implementation details across a spark runtime. This pattern will help runtime environment owners control how their users use graphframes