feat(config): allow wildcard matches in import statements - #1816

Allow importing files matching wildcard patterns using path/filepath#Match. For example:

bots:
  # Every rule file in the (data)/crawlers folder.
  - import: (data)/crawlers/*.yaml
  # Every rule file you put in /etc/anubis/known-good.
  - import: /etc/anubis/known-good/*.yaml

Supported wildcards

Anubis uses the Go standard library function filepath.Match for matching. It supports four wildcards:

Use this Go playground link to test pattern matches.

Matched files load in natural sort order

Anubis matches requests top to bottom, similar to Apache's DENY, ALLOW ACL. Anubis uses the first rule that matches the request exactly. A rule that loads early can shadow a rule that loads later.

When Anubis loads files imported with a wildcard match, it sorts the files in natural sort order. To control the order rules are loaded, put a number at the beginning of each filename:

/etc/anubis/known-good/
├── 00-allow-monitoring.yaml
├── 01-allow-git-clients.yaml
├── 02-allow-techaro-ci-bot.yaml
└── 10-weigh-everything-else.yaml

Natural sort compares numbers numerically and words alphabetically. This makes 02-deny.yaml load before 10-deny.yaml.

Every matched file must be a top level list

Anubis combines the matched files into one YAML document and then parses that document. Write each file as a list of rules at the top level, with no indentation and no bots: key.

For example, take a folder known-good with two files in it:

# known-good/techknowlogick.yaml
- name: allow-techknowlogick-gitea
  action: ALLOW
  expression:
    all:
      - host == "git.techaro.lol"
      - verifyFCrDNS(remoteAddress, "techknowlogick.com$")

# known-good/xe.yaml
- name: allow-xe-gitea
  action: ALLOW
  expression:
    all:
      - host == "git.techaro.lol"
      - verifyFCrDNS(remoteAddress, "xeiaso.net$")

When importing known-good/*.yaml, Anubis reads them as if you wrote both lists of rules into one file:

- name: allow-techknowlogick-gitea
  action: ALLOW
  expression:
    all:
      - host == "git.techaro.lol"
      - verifyFCrDNS(remoteAddress, "techknowlogick.com$")

- name: allow-xe-gitea
  action: ALLOW
  expression:
    all:
      - host == "git.techaro.lol"
      - verifyFCrDNS(remoteAddress, "xeiaso.net$")

Closes: #1815

Checklist:

• Added a description of the changes to the [Unreleased] section of docs/docs/CHANGELOG.md
• Added test cases to the relevant parts of the codebase
• Ran integration tests npm run test:integration (unsupported on Windows, please use WSL)
• All of my commits have verified signatures