Skip to content

Don't let loaders override hardcoded configuration#185

Closed
pietern wants to merge 11 commits intomainfrom
config-precedence
Closed

Don't let loaders override hardcoded configuration#185
pietern wants to merge 11 commits intomainfrom
config-precedence

Conversation

@pietern
Copy link
Copy Markdown
Contributor

@pietern pietern commented Nov 24, 2022

New approach of #170 (see comment).

The approach in this PR is simple: we load a zero-initialized configuration with 1) environment variables, 2) databrickscfg, 3) environment variables again, to make sure they have precedence over databrickscfgs.

This configuration is then merged into the target configuration, ignoring attributes that were already set.

The net result is that:

  • attributes that are set on the struct can never be overridden
  • attributes configured through environment variables take precedence over attributes loaded from a configuration file

@pietern pietern changed the title Don't let environment variables override hardcoded configuration [wip] Don't let environment variables override hardcoded configuration Nov 24, 2022
Copy link
Copy Markdown
Contributor

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this approach might work

// Read from ~/.databrickscfg if applicable.
KnownConfigLoader{},
// Load from environment variables again (they have precedence).
ConfigAttributes.EnvironmentLoader(),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that might work

@pietern pietern changed the title [wip] Don't let environment variables override hardcoded configuration Don't let loaders override hardcoded configuration Nov 24, 2022
@pietern
Copy link
Copy Markdown
Contributor Author

pietern commented Nov 24, 2022

Without this PR:

  1. Load env vars, but never overwrite stuff that's already set
  2. Load config file, but never overwrite stuff that's already set

With this PR:

  1. Load env vars, clobber existing
  2. Load config file, clobber existing
  3. Load env vars, clobber existing
  4. Take 1-3 and merge that into the config struct, ignoring entries that are already set

Without checking for dups on 4 this is equivalent to the current approach and ambiguity persists. It is still possible to hardcode host as well as specify DATABRICKS_HOST and nothing will complain.

config/config.go Outdated
if len(loaders) == 0 {
loaders = []Loader{
// Load from environment variables.
environmentVariableLoader{},
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

giving it another thought: can we create a loader just for DATABRICKS_CONFIG_FILE / DATABRICKS_CONFIG_PROFILE and use it as the first one?... otherwise, in bricks cli - can we just specify the list of custom loaders to use only direct configuration?...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

What do you mean with:

otherwise, in bricks cli - can we just specify the list of custom loaders to use only direct configuration?...

?

func createConfigurationFromLoaders(loaders []Loader) (*Config, error) {
if len(loaders) == 0 {
loaders = []Loader{
// Load from environment variables.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you also update the main readme file, because this behavior has to be documented on auth resolution. otherwise plenty of GH issues/ES tickets.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The auth flow doesn't change.

@pietern pietern requested a review from nfx December 2, 2022 15:04
@nfx
Copy link
Copy Markdown
Contributor

nfx commented Feb 17, 2023

i think this should be closed in favor of #306

@pietern pietern closed this Mar 14, 2024
@pietern pietern deleted the config-precedence branch March 14, 2024 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants