refactor(map): use `config.get` instead of `pillar.get` #95

myii · 2019-04-24T23:22:28Z

We've had some discussions in Slack/IRC/Matrix about moving on from pillar.get to config.get. libtofs.jinja is already using the latter. Selected WIP for this PR since this is something to consider across all formulas. The intention here is to open up the discussion about going forward with this.

CC: @gtmanfred.

gtmanfred

i like this, because it allows pulling from grains as well as resolves sdb:// uris, so if someone wants to store the data in sdb, they can.

daks · 2019-04-25T11:17:02Z

For now, I don't really see the use case for config.get instead of pillar.get. I read that it's to reduce load on busy masters, but I'm still not confronted to this problem.
And today all salt docs and 'litterature' (blog posts, ...) talk about pillar and none talk about anything else, like sdb. So I'm not sure if we already know what will replace pillars and there is no documentation.

In theory this has no impact because config.get also read pillars. But in reality I'm pretty sure that requet will take longer if all your data stays in pillars.

My 2 cents :)

in summary: I'm not opposed to this change even if this subject is still quite obscure for me, and I think we (and users above all) miss information about to do things differently.

gtmanfred · 2019-04-25T14:11:58Z

The time difference is negligible, since it just does a lookup for the key in the __grains__ __opts__ or __pillar__ object, and bails to the next one if it doesn't find it in the first.

vutny

I like config.get.

daks · 2019-04-26T13:39:52Z

The time difference is negligible, since it just does a lookup for the key in the __grains__ __opts__ or __pillar__ object, and bails to the next one if it doesn't find it in the first.

Good to know @gtmanfred

myii · 2019-04-27T07:58:35Z

Thanks everyone for your feedback. I've merged this PR.

saltstack-formulas-travis · 2019-04-27T08:12:23Z

🎉 This PR is included in version 2.0.4 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

myii · 2019-04-30T02:10:54Z

This commit may need to be reverted. config.get and pillar.get aren't 100% interchangeable, for example merge=True isn't available for the former. So if the result of the final config.get is an empty dict, the entire map ends up as an empty dict.

I've tried a workaround using defaults.merge again but I'm running into another problem:

TypeError: Cannot update using non-dict types in dictupdate.update()

Anyone know of how this can be resolved? Or is the only option to return back to pillar.get, for the time being at least?

myii · 2019-04-30T18:23:52Z

Update: I've been able to find a solution to this by building upon the defaults.merge strategy we used to use earlier on. The problem with this is that this doesn't work for salt-ssh users. So the dilemma:

Use config.get via. defaults.merge -- get the benefit but leave salt-ssh behind.
Use pillar.get -- works for everyone but limited to pillar only.

* Fix 5dc0b86 in saltstack-formulas#95 - No option `merge=True` for `config.get` * Use `pillar.get` for `salt-call` (for `salt-ssh`) * Use `config.get` otherwise

* Fix 5dc0b86 in saltstack-formulas#95 - No option `merge=True` for `config.get` * Use `pillar.get` for `salt-call` (i.e. `salt-ssh`) * Use `config.get` via. `defaults.merge` otherwise - Reintroduce based on 775a930 in saltstack-formulas#20

Co-Authored-By: Alexander Weidinger <[email protected]> * Fix 5dc0b86 in saltstack-formulas#95 - No option `merge=True` for `config.get` * Use `pillar.get` for `salt-call` (i.e. `salt-ssh`) * Use `config.get` otherwise

* Fix 5dc0b86 in saltstack-formulas#95 - No option `merge=True` for `config.get` * Use `pillar.get` for `salt-call` (i.e. `salt-ssh`) - Differentiate `salt-ssh`/`salt-call` via. `root_dir` * Use `config.get` via. `defaults.merge` otherwise - Reintroduce based on 775a930 in saltstack-formulas#20

* `merge` not available via. `salt-ssh` * Additionally, fix 5dc0b86 in saltstack-formulas#95 - No option `merge=True` for `config.get`

## [2.1.4](v2.1.3...v2.1.4) (2019-05-15) ### Bug Fixes * **`map.jinja`:** remove `merge` from `config.get` (for `salt-ssh`) ([00e474c](00e474c)), closes [#95](#95)

* saltstack-formulas/template-formula#95

## 1.0.0 (2025-08-26) ### ⚠ BREAKING CHANGES * **map:** `map.jinja` now exports a generic `mapdata` variable * **map:** The per grain parameter values are now under `TEMPLATE/parameters/` * changed all state names and ids * **pkgname:** the parameter `pkg` is now a dictionary. References to `template.pkg` should be changed to `template.pkg.name`. * **tofs:** every formula writer will need to change the import to use this new version. * template/libtofs.jinja: provides the “files_switch” macro. * docs/TOFS_pattern.rst: update documentation to use the new path. * template/config/clean.sls: change import from “macros.jinja” to “libtofs.jinja”. * template/config/file.sls: ditoo. * **states:** Wholesale state ID changes will break implementations that are relying on the previous state IDs for requisite purposes. * **pkg:** Changing the `pkg` directory to `package` will break implementations that are depending on `pkg` for `include` or `sls`-based requisite purposes. ### Features * add Gentoo support ([4c2f4ed](dafyddj/tmp-test-template-formula@4c2f4ed)) * add script to ease conversion from template to real formula ([edfa269](dafyddj/tmp-test-template-formula@edfa269)) * **authors:** update automatically alongside `semantic-release` ([8000098](dafyddj/tmp-test-template-formula@8000098)) * **centos-6:** reshape formula and tests for this platform ([a4b1608](dafyddj/tmp-test-template-formula@a4b1608)), closes [saltstack-formulas#104](https://github.com/dafyddj/tmp-test-template-formula/issues/104) * **convert-formula.sh:** assign `@NONE` as whole-formula owner ([cceffff](dafyddj/tmp-test-template-formula@cceffff)) * **kitchen+travis:** add `opensuse-leap` after resolving issues ([7614a3c](dafyddj/tmp-test-template-formula@7614a3c)) * **kitchen+travis:** conduct tests on a wider range of platforms ([1348078](dafyddj/tmp-test-template-formula@1348078)) * **m2r:** use `m2r` to convert automatic `.md` files to `.rst` ([b86ddf4](dafyddj/tmp-test-template-formula@b86ddf4)) * **macos:** basic package and group handling ([8c3fe22](dafyddj/tmp-test-template-formula@8c3fe22)) * **map:** generate a YAML file to validate `map.jinja` ([fc90075](dafyddj/tmp-test-template-formula@fc90075)) * **mapping:** introduce osarchmap per issue [saltstack-formulas#13](https://github.com/dafyddj/tmp-test-template-formula/issues/13) ([41ac40d](dafyddj/tmp-test-template-formula@41ac40d)) * **map:** update to v5 `map.jinja` ([42e1932](dafyddj/tmp-test-template-formula@42e1932)) * **pkg:** add `clean` states ([422c7ac](dafyddj/tmp-test-template-formula@422c7ac)) * **pkg:** use `require` requisite between `pkg` states ([6e7141b](dafyddj/tmp-test-template-formula@6e7141b)) * **rtd:** provide custom CSS file for overriding in-use Sphinx theme ([24bd338](dafyddj/tmp-test-template-formula@24bd338)) * run for real ([041e27b](dafyddj/tmp-test-template-formula@041e27b)) * **semantic-release:** configure for this formula ([cbcfd75](dafyddj/tmp-test-template-formula@cbcfd75)) * **sub-component:** manage a dedicated configuration file ([c4440d7](dafyddj/tmp-test-template-formula@c4440d7)) * **toc:** use `markdown-toc` directly to update inline ([a5bae1e](dafyddj/tmp-test-template-formula@a5bae1e)) * **tofs:** implement backwards-compatible TOFSv2 for configurability ([068a94d](dafyddj/tmp-test-template-formula@068a94d)) * **tofs:** lookup files directory in “tpldir” hierarchy ([5c495fb](dafyddj/tmp-test-template-formula@5c495fb)) * **workflows:** dry-run `semantic-release` in GitHub Actions ([764cd4c](dafyddj/tmp-test-template-formula@764cd4c)) * **yamllint:** include for this repo and apply rules throughout ([e76525f](dafyddj/tmp-test-template-formula@e76525f)) ### Bug Fixes * **_mapdata:** ensure map data is directly under `values` ([bcb8e29](dafyddj/tmp-test-template-formula@bcb8e29)) * **`config/file`:** add missing space before Jinja `}}` ([5cd08ab](dafyddj/tmp-test-template-formula@5cd08ab)) * **`libtofs`:** use `select` to deal with empty strings in path ([afe0751](dafyddj/tmp-test-template-formula@afe0751)) * **`libtofs`:** use `strip` to deal with leading/trailing slashes ([2563a46](dafyddj/tmp-test-template-formula@2563a46)) * **`map.jinja`:** _merge_ defaults and `config.get` ([91bc2f0](dafyddj/tmp-test-template-formula@91bc2f0)) * **`map.jinja`:** remove `merge` from `config.get` (for `salt-ssh`) ([00e474c](dafyddj/tmp-test-template-formula@00e474c)), closes [saltstack-formulas#95](https://github.com/dafyddj/tmp-test-template-formula/issues/95) * **`map.jinja`:** use tplroot ([b9c5e03](dafyddj/tmp-test-template-formula@b9c5e03)) * broken install-hooks due to saltlint v0.8.0 ([7da11c9](dafyddj/tmp-test-template-formula@7da11c9)) * **codeowners:** ensure `lib*` files are owned by `ssf` ([d60cc15](dafyddj/tmp-test-template-formula@d60cc15)) * **comments:** explain that at least an empty dict is required ([426f955](dafyddj/tmp-test-template-formula@426f955)), closes [saltstack-formulas#93](https://github.com/dafyddj/tmp-test-template-formula/issues/93) * **commitlint:** fix header length at 72 chars as agreed ([a95061d](dafyddj/tmp-test-template-formula@a95061d)) * **convert-formula.sh:** add -_ to allowed chars in formula name ([a999fee](dafyddj/tmp-test-template-formula@a999fee)) * **convert-formula.sh:** add `~` to reST underlining during conversion ([80ed8cd](dafyddj/tmp-test-template-formula@80ed8cd)) * **convert-formula.sh:** apply remaining suggestions from [saltstack-formulas#180](https://github.com/dafyddj/tmp-test-template-formula/issues/180) ([76ecd44](dafyddj/tmp-test-template-formula@76ecd44)) * **convert-formula.sh:** delete all existing tags ([7c33601](dafyddj/tmp-test-template-formula@7c33601)), closes [saltstack-formulas#210](https://github.com/dafyddj/tmp-test-template-formula/issues/210) * **convert-formula.sh:** fix reST underlining during conversion ([11068af](dafyddj/tmp-test-template-formula@11068af)) * **convert-formula.sh:** remove "Using this template" post-conversion ([55ab937](dafyddj/tmp-test-template-formula@55ab937)) * **convert-formula.sh:** remove `rubocop` override post-conversion ([aca4e44](dafyddj/tmp-test-template-formula@aca4e44)) * **convert-formula.sh:** remove CI test post-conversion ([06ec949](dafyddj/tmp-test-template-formula@06ec949)) * **convert-formula.sh:** replace instances of `template-formula` for CI ([537fe65](dafyddj/tmp-test-template-formula@537fe65)), closes [saltstack-formulas#231](https://github.com/dafyddj/tmp-test-template-formula/issues/231) * **convert-formula.sh:** reset version to `1.0.0` ([39889ce](dafyddj/tmp-test-template-formula@39889ce)) * **convert-formula.sh:** use portable sed function to make replacements ([41e10b5](dafyddj/tmp-test-template-formula@41e10b5)), closes [saltstack-formulas#192](https://github.com/dafyddj/tmp-test-template-formula/issues/192) * **convert-formula:** `_mapdata` control name must use the formula one ([1f3600d](dafyddj/tmp-test-template-formula@1f3600d)) * fix `CentOS Linux-7` and add `os` details from current CI setup ([4be16ca](dafyddj/tmp-test-template-formula@4be16ca)) * **formula:** update to current oldest supported version of Salt ([878eca1](dafyddj/tmp-test-template-formula@878eca1)) * **gitignore:** add Gemfile.lock to .gitignore ([87fa410](dafyddj/tmp-test-template-formula@87fa410)) * **grain:** fix grain value ([26edfa0](dafyddj/tmp-test-template-formula@26edfa0)) * **inspec:** validate `map.jinja` configuration ([41d222e](dafyddj/tmp-test-template-formula@41d222e)) * **libmapstack:** allow mapping by booleans and numbers ([bb3a7ea](dafyddj/tmp-test-template-formula@bb3a7ea)) * **libsaltcli+libmatchers:** ensure Salt client API detection [skip ci] ([6eb2073](dafyddj/tmp-test-template-formula@6eb2073)) * **libsaltcli:** update `salt-ssh` detection for `enable_ssh_minions` ([f0e7192](dafyddj/tmp-test-template-formula@f0e7192)) * **libtofs:** “files_switch” mess up the variable defined by “map.jinja” ([ab4ce75](dafyddj/tmp-test-template-formula@ab4ce75)) * **libtofs:** “files_switch” mess up the variable exported by “map.jinja” [skip ci] ([241646f](dafyddj/tmp-test-template-formula@241646f)) * **libtofs:** avoid using subpath by default ([c07471d](dafyddj/tmp-test-template-formula@c07471d)) * **libtofs:** don't crash if “tofs.files_switch” lookup a list ([0979d35](dafyddj/tmp-test-template-formula@0979d35)) * **pillar:** fix `os_family` typo ([3f89c12](dafyddj/tmp-test-template-formula@3f89c12)) * **release.config.js:** use full commit hash in commit link [skip ci] ([4ac8d92](dafyddj/tmp-test-template-formula@4ac8d92)) * **rubocop:** add fixes using `rubocop --safe-auto-correct` ([484ce24](dafyddj/tmp-test-template-formula@484ce24)) * **rubocop:** fix remaining errors manually ([9566b6f](dafyddj/tmp-test-template-formula@9566b6f)) * **running.sls:** use `watch` not `require` to ensure service restart ([3a1fc35](dafyddj/tmp-test-template-formula@3a1fc35)) * **subcomponent:** clean referencing wrong sls ([394808e](dafyddj/tmp-test-template-formula@394808e)) * **suse:** correct OS grain ([6aee580](dafyddj/tmp-test-template-formula@6aee580)) * **tofs:** prepend the config-based `source_files` to the default ([3483e76](dafyddj/tmp-test-template-formula@3483e76)), closes [saltstack-formulas#151](https://github.com/dafyddj/tmp-test-template-formula/issues/151) * **tofs:** update comments in `files_switch` macro for new method ([3fa3640](dafyddj/tmp-test-template-formula@3fa3640)) * **tofs:** update use of state ID in `config` and `pillar` ([3d9a24c](dafyddj/tmp-test-template-formula@3d9a24c)) * **tofs:** use `source_files` instead of `files` ([5110716](dafyddj/tmp-test-template-formula@5110716)) * **tofs:** use `tpldir` derivative `topdir` for pillar (config) paths ([5e9df00](dafyddj/tmp-test-template-formula@5e9df00)) * **travis:** don't install gems twice ([925d8e2](dafyddj/tmp-test-template-formula@925d8e2)) * **travis:** reinstate conversion test [skip ci] ([5d47fda](dafyddj/tmp-test-template-formula@5d47fda)) * **travis:** use version numbers in Gemfile to prevent failed builds ([35f7111](dafyddj/tmp-test-template-formula@35f7111)) * typo in the installation instructions ([306a0d9](dafyddj/tmp-test-template-formula@306a0d9)) * use correct repo ([ca96491](dafyddj/tmp-test-template-formula@ca96491)) ### Performance Improvements * **travis:** improve `salt-lint` invocation [skip ci] ([7a96cd7](dafyddj/tmp-test-template-formula@7a96cd7)) ### Reverts * **kitchen+travis:** disable `debian-8` due to `2019.2` bug ([e8f0f7e](dafyddj/tmp-test-template-formula@e8f0f7e)) * **kitchen+travis:** use `debian:jessie-backports` as `debian-8` ([dcd141a](dafyddj/tmp-test-template-formula@dcd141a)) ### Code Refactoring * improve reusability using an unique keyword TEMPLATE ([2e8ded6](dafyddj/tmp-test-template-formula@2e8ded6)) * **pkg:** change to `package` instead ([2cd82e5](dafyddj/tmp-test-template-formula@2cd82e5)) * **pkgname:** reserve 'pkg' as packaging dict ([c6ae81c](dafyddj/tmp-test-template-formula@c6ae81c)) * **states:** set state IDs based on a dependable structure ([6690ee6](dafyddj/tmp-test-template-formula@6690ee6)) * **tofs:** move “files_switch” macro to “libtofs.jinja” ([da7e692](dafyddj/tmp-test-template-formula@da7e692))

refactor(map): use config.get instead of pillar.get

5dc0b86

myii requested review from aboe76, daks, javierbertoli, noelmcloughlin and vutny April 24, 2019 23:22

gtmanfred approved these changes Apr 24, 2019

View reviewed changes

javierbertoli approved these changes Apr 25, 2019

View reviewed changes

vutny approved these changes Apr 26, 2019

View reviewed changes

myii changed the title ~~WIP: refactor(map): use config.get instead of pillar.get~~ refactor(map): use config.get instead of pillar.get Apr 27, 2019

myii merged commit 24bf7d0 into saltstack-formulas:master Apr 27, 2019

myii deleted the refactor/map-config-get branch April 27, 2019 07:58

myii mentioned this pull request Apr 27, 2019

Move to New Template saltstack-formulas/chrony-formula#16

Merged

2 tasks

saltstack-formulas-travis added the released label Apr 27, 2019

myii mentioned this pull request May 1, 2019

fix(map.jinja): use pillar/config .get according to __cli option #102

Merged

myii added a commit to myii/packages-formula that referenced this pull request Dec 6, 2019

refactor(map): use config.get instead of pillar.get

ca9bbd1

* saltstack-formulas/template-formula#95

myii mentioned this pull request Dec 6, 2019

refactor(map): use config.get instead of pillar.get saltstack-formulas/packages-formula#63

Merged

19 tasks

myii added a commit to myii/packages-formula that referenced this pull request Dec 6, 2019

refactor(map): use config.get instead of pillar.get

8781324

* saltstack-formulas/template-formula#95

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(map): use `config.get` instead of `pillar.get` #95

refactor(map): use `config.get` instead of `pillar.get` #95

Uh oh!

myii commented Apr 24, 2019

Uh oh!

gtmanfred left a comment

Uh oh!

daks commented Apr 25, 2019 •

edited

Loading

Uh oh!

gtmanfred commented Apr 25, 2019

Uh oh!

vutny left a comment

Uh oh!

daks commented Apr 26, 2019

Uh oh!

myii commented Apr 27, 2019

Uh oh!

saltstack-formulas-travis commented Apr 27, 2019

Uh oh!

myii commented Apr 30, 2019

Uh oh!

myii commented Apr 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

refactor(map): use config.get instead of pillar.get #95

refactor(map): use config.get instead of pillar.get #95

Uh oh!

Conversation

myii commented Apr 24, 2019

Uh oh!

gtmanfred left a comment

Choose a reason for hiding this comment

Uh oh!

daks commented Apr 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gtmanfred commented Apr 25, 2019

Uh oh!

vutny left a comment

Choose a reason for hiding this comment

Uh oh!

daks commented Apr 26, 2019

Uh oh!

myii commented Apr 27, 2019

Uh oh!

saltstack-formulas-travis commented Apr 27, 2019

Uh oh!

myii commented Apr 30, 2019

Uh oh!

myii commented Apr 30, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

refactor(map): use `config.get` instead of `pillar.get` #95

refactor(map): use `config.get` instead of `pillar.get` #95

daks commented Apr 25, 2019 •

edited

Loading