TL;DR This proposal changes the API but keeps most of the original hueristics completely intact. Tests were added and adopted, but with the exception of deprecated methods (e.g. util methods), no tests were removed and all previously defined tests still pass.

You can check the code here: https://github.com/iMoses/node-config/tree/next-gen

Disclaimer: some breaking changes can be reverted while others cannot. If you feel strongly about some of the changes let me know and let's discuss the available possibilities.

Motivation

About half of today's open issues are related to immutability inconsistencies and sub-modules support. while many closed issues are related to non-intuitive methods of initializing the library and means of adding support for this and that, which can come down to more control over the library's internal mechanisms.

While looking for possible solutions I came to realize that the main issue with sub-modules is our lack of support for multiple instances, and the main issue with that is our reliance on environment variables for initialize the config instance.

The second major issue is that we initialize the main instance ourselves when loaded, which prevents us from introducing new features to manipulate the library's options before initialization.

These problems can only be solved by changing the initialization logic of the library, which wouldn't have been much of a problem if we weren't expected to return an instance already initialized and containing the resulted configuration files on top of the instance's root... (backwards compatibility) Which brings us back to the immutability issues that are complex due to the same logical problems.

If we change the Config instance API and remove direct access to the configuration object (at least from the root) we can enforce mutability and delay files loading until the first access to the configuration object, instead of the config instance as we do today.

It all comes down to that:

• Allow the creation of multiple instances
• Remove reliance on environmental variables
• Better enforce immutability
• Lazy-load configuration

Along with changes to the parser to support validators and middleware we can cover most of today's open issues in one fine swipe, which will require a major version with many breaking changes effecting only advance users.

I think it's worth it :)

Breaking changes

• Removed config.util methods
• Removed runtime.json support
• Removed custom-environment-variables.* support
• Changed location of asyncConfig and deferConfig
• asyncConfig now uses config.whenReady to resolve
• get() and has() are available only on the config instance - removes all reserved words
• Changed Parser API to a programmatic solution, as opposed to an external file
• getArgv() slightly differs from its predecessor - in case of multiple matching arguments it returns the last (previous logic was to return the first occurrence)
• Removed raw() as it is now obsolete (complex objects are not extended by default)

[x] Removed config.util methods

We have many utilities which are exposed as part of our API and I claim that most of these shouldn't be exposed at all. This isn't the library's purpose, no one installs node-config so that they can use config.util.extendDeep, and exposing them as part of our API is forcing us to keep support for them or else we introduce a breaking change, which makes it a fragile API that is forced to carry legacy code.

My proposal removes config.util completely.

Internal utilities are placed at config/lib/utils with a disclaimer that we will not guarantee stability between versions. Users can decide to use them at their own risk. Anything worthwhile should be placed on top of the config instance, while we keep utils strictly internal.

[x] Removed runtime.json support

Support was removed in an effort to dispose of deprecated code and inconsistent heuristics, and can be simply restored by calling config.parseFile manually.

config.parseFile(process.env.NODE_CONFIG_DIR + '/runtime.json');

Removed custom-environment-variables.* support

Support was removed in an effort to dispose of inconsistent heuristics.

The new architecture adds middleware support to the parser which can be used to apply templates. A template engine can be used to apply the same logic in a consistent manner, unlike custom-environment-variables.*, and replace it with a cross-extension ability to set values from environment variables and much more.

See Parser.lib.middleware.configTemplate() for examples.

Changed location of asyncConfig and deferConfig

Previously these methods were in separate files outside of the /lib directory to allow easy access from configuration files without trigger the initialization of the config instance. With the new architecture this isn't necessary anymore and we can simply export these methods as part of out instance.

Previous version:

const { asyncConfig } = require('config/async');
const { deferConfig } = require('config/defer');

New version:

const { asyncConfig, deferConfig } = require('config');

I added shim files for now with a deprecation warning.

asyncConfig now uses config.whenReady to resolve

Previous version:

const config = require('config');
const { resolveAsyncConfigs } = require('config/async');
resolveAsyncConfigs(config)
  .then(() => require('./main'));

New version:

const config = require('config');
config.whenReady.then(() => require('./main'));

get() and has() are available only on the config instance

The main reason for this is to remove reserved words completely. I don't think the library should reserve any keys.

Previous version:

const config = require('config');
const db = config.get('services.db');
console.log(db.has('host'));  // true

New version:

const config = require('config');
const db = config.get('services.db');
console.log(db.has('host'));  // Error: method has doesn't exist

Changed Parser API to a programmatic solution

Explained in the Parser API reference.

It changes quite a lot, but I figured since it's a very new feature users who already use it are early adopters and won't mind as much, since it's a much cleaner API with validators and middleware :)

getArgv() slightly differs from its predecessor

In case of multiple matching arguments it returns the last (previous logic was to return the first occurrence). It actually seemed like a bug-fix to me, since that's what I would've expected to be the correct behavior. I also added support for boolean and space separated cli-args.

// in case of `node index.js --NODE_ENV=dev --NODE_ENV=prod`
console.log(utils.getArgv('NODE_ENV'));  // prod

Supported syntax:
    --VAR_NAME=value    value separated by an equality sign
    --VAR_NAME value    value separated by spaces
    --BOOL_VAR          boolean - no value automatically returns true

Config

Config is an internal class that's used to create configuration instances, as it did in previous versions.

Configuration instances are created by three means:

1. Config.create(options)
2. Config.subModule(moduleName)
3. A default instance that is initiated and returned by node-config

The default instance initialization options comes from env-vars and cli-args, using the same heuristics as previous versions.

module.exports = new Config(null, getDefaultInstanceOptions());

function getDefaultInstanceOptions() {
  return clearUndefinedKeys({
    configDir: utils.getOption('NODE_CONFIG_DIR', './config'),
    environment: utils.getOption('NODE_CONFIG_ENV') || utils.getOption('NODE_ENV'),
    hostname: utils.getOption('HOST') || utils.getOption('HOSTNAME'),
    appInstance: utils.getOption('NODE_APP_INSTANCE'),
    strict: Boolean(utils.getOption('NODE_CONFIG_STRICT_MODE')),
    freeze: !utils.getOption('ALLOW_CONFIG_MUTATIONS'),
    legacy: true,
  });
}

Common Usage

Users who've been using the library according to the Common Usage section recommendations won't be affected (in most cases) by these changes.

// autoload defaults on first access
const config = require('config');
console.log(config.get('some.key'));  // value
console.log(config.has('another.key'));  // boolean

You can also access the config property directly if you prefer it over the instance API.

const { config } = require('config');
console.log(config.some.key);  // value

Autoload and Mutation methods

Configuration instances will automatically load with the default options provided to the constructor (see Config.loadDefaults(legacy)), unless a mutation methods has been used in which case the autoload mechanism is disabled.

Mutation methods have access to modify the configuration object without accessing the external property. These are the only methods which are available to mutate the configuration object, and they are automatically locked once the configuration object is frozen.

• Config.loadFiles(options)
• Config.parseFile(filename)
• Config.extend(object, source)

That's important to keep in mind when using one of these methods. A common pitfall is to use a mutation method and forget about it canceling the autoload mechanism.

// overrides default options
// only the content of variables.yaml will be available
const config = require('config')
  .parseFile(__dirname + '/scripts/variables.yaml');

Which can be simply solved by executing the Config.loadDefault(legacy) manually.

// loads defaults before parsing variables.yaml
const config = require('config')
  .loadDefaults(true)
  .parseFile(__dirname + '/scripts/variables.yaml');

asyncConfig and deferConfig

All library files were moved to /lib, and since they can now be loaded directly from the config instance without triggering the autoload sequence it acts as a simpler API.

const { deferConfig, asyncConfig } = require('config');

Config.config

The is the jewel in the crown, the reason for this major change in the library's API.

In previous versions the configuration instance is initialized when the module is loaded and automatically reads options, loads files, run validations and freezes the configuration object, which caused many problems. Moving the configuration object to a separate property allows us to observe access and delay the autoload sequence until it is used.

This property is lazily loaded so until accessed the following will not be activated.

On first access this property triggers the resolve mechanism, composed of these steps:

• Executes loadDefaults(true) only in case none of the mutation methods were used
• Resolves all deferConfig values
• Disable all mutation methods
• Freeze sources and the configuration object

Once resolved the frozen configuration object is set to this property's value.

Note that Config.get() and Config.has() access this property, so it can trigger the resolve mechanism.

const config = require('config');
// only resolves once config.get is used
// until then we can use mutation methods (loadFiles, parseFile, extend)
console.log(config.get('services.db.host'));  // localhost

Config.whenReady

This property is lazily loaded so unless accessed the following will not take effect.

On first access this property triggers the resolve mechanism for both deferConfig and asyncConfig. The property returns a promise which resolves after all asyncConfig were replaced with their final values.

This is a replacement mechanism for resolveAsyncConfigs(config).

Config.options

A frozen copy of the options passed to the instance constructor, mainly for debugging purposes.

Config.parser

Property that's used to access the Parser instance used by the instance.

Config.sources

Returns an array of sources used to compose the configuration object. The array is composed of objects, each with a source string and a data object containing the source's content.

Note that when using sub-modules, any mutations caused by a sub-module will also contain the module property which hold the module name of the effecting sub-module. (see Config.subModule)

This is a replacement mechanism for config.util.getConfigSources().

const config = require('config');
config.parseFile('/absolute/path/to/file.js');
config.loadFiles({configDir: __dirname + '/ultra-config'});
config.extend(JSON.parse(process.env.MY_ENV_VAR), '$MY_ENV_VAR');
config.extend({defaultProp: 'MyValue'});
console.log(config.sources);

Would output:

[
  {source: '/absolute/path/to/file.js', data: {/* ... */}},
  {source: '/absolute/path/to/ultra-config/defaults.json', data: {/* ... */}},
  {source: '/absolute/path/to/ultra-config/local.yml', data: {/* ... */}},
  {source: '$MY_ENV_VAR', data: {/* ... */}},
  {source: 'config.extend()', data: {defaultProp: 'MyValue'}},
]

Config.create(options)

Create's a new configuration instance that's completely independent.

Should be used by package owners who wish to use node-config without effecting dependent packages who may use the library themselves. This method is also useful for testing purposes.

const myConfig = require('config').create({
  configDir: __dirname + '/config',
  appInstance: process.platform,
  environment: 'package',
});

options will be passed down to Config.loadFiles(options) in case of an autoload or a manual execution of Config.loadDefaults(legacy).

• configDir - path of the configuration dir
• appInstance - appInstance used to match files
• environment - environment used to match files, defaults to "development"
• hostname - hostname used to match files, defaults to os.hostname()

Additional options are:

• strict - passed down to strictValidations, replaces NODE_CONFIG_STRICT_MODE
• legacy - passed down to loadDefaults as part of the autoload mechanism
• freeze - if set to false the configuration object will never be locked

Config.get(key)

Same as in previous versions.

const config = require('config');
console.log(config.get('services.db.host'));  // localhost

Config.has(key)

Same as in previous versions.

const config = require('config');
console.log(config.has('services.db.host'));  // true

[x] Config.loadFiles(options)

Uses Config.loadConfigFiles(options) and passes matching files to Config.parseFile(filename).

Accepts an options object which is passed down to loadConfigFiles:

• configDir - path of the configuration dir
• appInstance - appInstance used to match files
• environment - environment used to match files, defaults to "development"
• hostname - hostname used to match files, defaults to os.hostname()

const config = require('config')
  .loadFiles({
    configDir: __dirname + '/resources/configuration',
    appInstance: process.platform,
    environment: 'override',
    hostname: 'node-config',
  });

Config.parseFile(filename)

Extends the internal configuration object with the parsed content of filename.

const config = require('config')
  .parseFile(__dirname + '/runtime.json');

Config.extend(object, source)

Merges object into the internal configuration object and puts a new source into Config.sources with the given source. (source is optional and defaults to config.extend())

const config = require('config')
  .extend({some: {key: 'value'}})
  .extend(JSON.parse(process.env.MY_APP_CONF), '$MY_APP_CONF');

Config.loadDefaults(legacy)

Applies the options provided to the instance constructor on Config.loadFiles(options) and run strictness validation. If legacy is true then it also extends the configuration object with $NODE_CONFIG_JSON and --NODE_CONFIG_JSON, if available.

const config = require('config')
  .loadDefaults()
  .parseFile(`${process.env.HOME}/.config/services.json`);
console.log(config.get('services.db.host'));  // localhost

[x] Config.collectConfigFiles(options)

Used internally by Config.loadFiles(options).

This method is the heart of the library which holds the heuristics for collecting configuration files given a set of options. It returns an array containing matching configuration files according to their resolution order.

Accepts an options object with the following keys:

• configDir - path of the configuration dir
• appInstance - appInstance used to match files
• environment - environment used to match files, defaults to "development"
• hostname - hostname used to match files, defaults to os.hostname()

See Config.loadFiles(options) for an example.

Config.subModule(moduleName)

Sub-modules is a mechanism to create new configuration instances which are derived from their initiating module instance. The moduleName is also the path in which our sub-module is located. If the path already exists it will be used, otherwise we create the path set an new object at the end of it.

Sub-modules shares reference between their config and their main-modules config properties.

const config = require('config');
const subModule = config.subModule('TestModule');
console.log(config.get('TestModule') === subModule.config);  // identical

Notice that moduleName can contain dot notations to describe a deeper path.

const config = require('config');
const subModule = config.subModule('TestModule.Suite1');
console.log(config.get('TestModule.Suite1') === subModule.config);  // identical

Sub-modules also share sources with their main-modules. Any mutation executed from a sub-module will register the module property on the source item containing the module's name, and the data propery will emulate the sub-module's path.

const config = require('config').loadDefaults();
const subModule = config.subModule('TestModule').extend({defaultKey: 'defaultValue'});
console.log(config.sources === subModule.sources);  // identical
console.log(config.sources);

Would output:

[
  {source: '/path/to/config/default.js', data: {/* ... */}},
  //  ...
  {module: 'TestModule',
   source: 'config.extend()', 
   data: {TestModule: {defaultKey: 'defaultValue'}}},
]

Parser

I took some opinionated decisions that we can open a discussion about.

I removed the register methods for TypeScript and CoffeeScript. I don't think that's an action we should take, interfering with the registration options of a transpiler. Users of TypeScript and CoffeeScript should be familiar with these concepts. We basically treat these extensions as regular JavaScript files, and we expect the users to register these before they call the node-config.

I removed our support for multiple packages at the same parser function. Since it's now very easy to override parsers I think we should only offer the most common and let anyone who wants another package simply switch the parser.

I added a POC of a template engine as a default middleware (also a way to show off middleware). I'd like to discuss the idea further since this is only a POC version.

Parser.parse(filename)

Used internally by the mutation method Config.parseFile(filename).

• Extracts the extension from filename
• Calls the matching parser according to the extension
• Runs the result through our middleware and reduce the results
• Returns the result

Parser.readFile(filename)

Reads a file and returns its content if all validators successfully passed, otherwise it returns null.

Used internally by parsers to read a file and pass it to the parsing library.

lib.iniParser = function(filename) {
  const content = this.readFile(filename);
  return content ? require('ini').parse(content) : null;
};

[x] Parser.readDir(dirname)

Returns an array containing the content of the dirname directory.

lib.dirParser = function(dirname) {
  return this.readDir(dirname).map(filename => {
    filename = Path.join(dirname, filename);
    return [filename, this.parse(filename)];
  });
};

[x] Parser.collectFiles(configDir, allowedFiles)

Used internally by Config.collectConfigFiles.

// returns an array of matching filenames within __dirname
// keeps files order even when multiple directories are given
config.parser.collectFiles(`${__dirname}/http:${__dirname}/www`, ['index.html', 'index.htm', 'index.php', 'index.js']);

Parser.validators

An array of validators that are executed as part of Parser.readFile(filename).

Defaults to:

validators = [
  lib.validators.gitCrypt(!utils.getOption('CONFIG_SKIP_GITCRYPT')),
]

You can override the array to replace or clear the active validators.

const config = require('config');
config.parser.validators = [];
config.parser.validators = [
  // validate content is unicode
  (filename, content) => isUnicode(content),
];

Parser.middleware

An array of middleware that are executed as part of Parser.parse(filename).

Defaults to:

middleware = [
  lib.middleware.configTemplate(lib.middleware.configTemplate.defaultHandlers),
]

Important! The configTemplate middleware is only a POC at this point and should not be counted on. It is used here only as an example, but structure and functionality (or existence) may change.

You can override the array to replace or clear the active middleware.

const config = require('config');
config.parser.middleware = config.parser.middleware.concat([
  // check if es-module and has default then return content.default
  (filename, content) => content.__esModule && content.default ? content.default : content,
]);

Parser.definition

An array of parser definitions that are used with both Config.collectConfigFiles(options) and Parser.parse(filename).

It is recommended not to handle this property directly. Instead you can use Parser.set(ext, parser) and Parser.unset(ext) to add, remove and override parser, and Parser.resolution to set the resolution order in a simple manner.

Defaults to:

definition = [
  {ext: 'js', parser: lib.jsParser},
  {ext: 'ts', parser: lib.jsParser},
  {ext: 'json', parser: lib.jsonParser},
  {ext: 'json5', parser: lib.jsonParser},
  {ext: 'hjson', parser: lib.hjsonParser},
  {ext: 'toml', parser: lib.tomlParser},
  {ext: 'coffee', parser: lib.jsParser},
  {ext: 'iced', parser: lib.jsParser},
  {ext: 'yaml', parser: lib.yamlParser},
  {ext: 'yml', parser: lib.yamlParser},
  {ext: 'cson', parser: lib.csonParser},
  {ext: 'properties', parser: lib.propertiesParser},
  {ext: 'xml', parser: lib.xmlParser},
  {ext: 'ini', parser: lib.iniParser},  // this is new!
  {ext: 'd', parser: lib.dirParser},  // this is new!
]

Parser.resolution

A dynamic property that returns an array of parser.definition extensions. It's the same as executing parser.definition.map(def => def.ext). What's special about this propery is its setter which allows for a simple way to filter out and reorder parser.definition.

const config = require('config');
config.parser.resolution = ['js', 'json', 'json5', 'yaml', 'd'];

This removes parsers who are not included in the array and set this order for the existing parsers. Note that setting an array with an undefined (unknown) extension would result in an exception.

Parser.set(extension, parser)

Adds, or overrides in case it exists, a parser to Parser.definition.

const config = require('config');
const hoconParser = require('hoconparser');
config.parser.set('hocon', function(filename) {
  const content = this.readFile(filename);
  return content ? hoconParser(content) : content;
});
// autoload will now support hocon files

Parser.unset(extension)

Removes a parser from Parser.definition by extension.

const config = require('config');
// remove support for yaml and yml files
config.parser.unset('yaml').unset('yml');

Parser.reset()

Clears all default

const config = require('config');
config.parser.reset();
console.log(config.parser.validators);  // []
console.log(config.parser.middleware);  // []
console.log(config.parser.definition);  // []
console.log(config.parser.resolution);  // []

Parser.lib

A frozen object containing all the default parsers, validators and middleware.

const config = require('config');
config.parser.set('rss', config.parser.lib.xmlParser);  // equals to
config.parser.set('rss', function(filename) {
  return this.lib.xmlParser(filename);
});

Parser.lib.validators.gitCrypt(strict)

lib.validators.gitCrypt = strict => function(filename, content) {
  if (/^.GITCRYPT/.test(content)) {
    if (strict) throw new Error(`Cannot read git-crypt file ${filename}`);
    console.warn(`WARNING: skipping git-crypt file ${filename}`);
    return false;
  }
  return true;
};

Parser.lib.middleware.configTemplate(commandHandlers)

Okay, first the repeating disclaimer that this is just a POC.

I'd created an extremely simple, case-insensitive DSL that allows piping of command and applying flag modifiers.

It currently contains two commands and three options.

• "File::" - Command used to read a file using Parser.parse(filename). Returns an object
  • "string" - Modifier used to read the file using Parser.readFile(filename) instead. Returns a string
• "Env::" - Command used to set value of corresponding environment variable
  • "json" - Modifier used to parse the value using JSON.parse(str)
• "secret" - Modifier that can be used on any value to defines its property as non-enumerable. Replaces what used to be config.util.makeHidden()

example.json5:

{
  dbHost: 'localhost',
  dbPass: 'ENV::DB_PASSWORD',
  httpServer: 'ENV::HTTP_SERVER|json',
  ssl: {
    key: 'FILE::/credentials/ssl.key|string',
    pem: 'FILE::/credentials/ssl.pem|string',
  },
  runtime: 'FILE::ENV::CUSTOM_RUNTIME_JSON_PATH',
  secretKey: 'ENV::SPECIAL_SECRET_KEY|secret'
}

example.yaml:

dbHost: localhost
dbPass: ENV::DB_PASSWORD
httpServer: ENV::HTTP_SERVER|json
ssl:
  key: FILE::/credentials/ssl.key|string
  pem: FILE::/credentials/ssl.pem|string
runtime: FILE::ENV::CUSTOM_RUNTIME_JSON_PATH
secretKey: ENV::SPECIAL_SECRET_KEY|secret

Related issues

Solved or no longer relevant due to new architecture:

• #549 - Immutability
• #546 - SubModules
• #543 - SubModules
• #517 - SubModules
• #514 - Immutability
• #478 - Obsolete
• #472 - Obsolete
• #471 - Immutability
• #424 - SubModules
• #412 - Obsolete
• #338 - Obsolete
• #329 - Immutability
• #226 - SubModules
• #225 - SubModules

We can provide a middleware solution:

• #539 - Middleware
• #536 - Middleware
• #521 - Middleware / Parser
• #400 - Middleware
• #372 - Middleware
• #324 - Parser