Skip to content

question on test for "all" template files #320

Open
0 of 2 issues completed
Open

Description

@epDHowwD

To test that every object has an app label or something like a support link, I would like to formulate a test like this:

suite: test app label
templates:
  - templates/*.yaml
tests:
- it: object has app label
  asserts:
  - equal:
      path: metadata.labels.app
      value: myapp

According to documentSelector this should default select "all" documents within every (*.yaml) template file.

Now the document structure already has some template surrounded by feature flags, so some files might render empty, and thus do not comprise documents.

templates/file-v1.yaml:

{{- if .Values.features.magic }}
  # see file-v2.yaml
{{- else }}
---
kind: ...
{{- end }}

templates/file-v2.yaml:

{{- if .Values.features.magic }}
---
kind: ...
{{- end }}

I would expect that asserts on all documents in all templates would somehow accept empty templates, but the assert fails if not at least one document can be found per template. As I see certainly why asserts should raise if no document is fulfilling the assert, I would like to know if it is possible to test this similarly?

Activity

  1. quintush commented on May 10, 2024

    @quintush
    Contributor

    Hello @epDHowwD,

    The unittests are indeed very strict in validating "all" templates. So the validation will fail when an empty file is part of the assertions. This setup is by design.

    The best way to validate all templates is by ensure "all" template do get generated, which can be done by setting the features.magic values as part of the test. The following namesOveride test shows a similar approach where the assertions are all validated against to mention list of templates, where the ingess.enabled value has been set to true, otherwise the ingress template would result in an empty file (failing the assertion).

    Hopes this helps,

    Greetings,
    @quintush

  2. mszygenda commented on May 20, 2024

    @mszygenda

    I believe use-case described here is manifestation of the same issue described here:

    #308

    And related PR I prepared to tackle it.

    #335

    The idea is that you could change the default behavior requiring document in each template by using documentSelector.skipEmptyTemplates property.

    suite: Document Selector is matching many documents
    templates:
      - "*"
    tests:
      - it: deployment names should end with -deployment suffix
        documentSelector:
          path: kind
          value: Deployment
          matchMany: true
          skipEmptyTemplates: true
        asserts:
          - matchRegex:
              path: metadata.name
              pattern: -deployment$
    
  3. llavaud commented on Mar 14, 2025

    @llavaud

    @mszygenda that is a great addition but too bad we can't target all documents... :(
    I have same sort of assert to validate documents name but it apply to everything, that would have been great to be able to just add this:

        documentSelector:
          matchMany: true
          skipEmptyTemplates: true
    

    @quintush and the solution to set specific value as part of the test don't work because for a specific value I have mutual exclusive document generation, so I always have one or another that is empty...

  4. lucasfcnunes commented on Jul 9, 2025

    @lucasfcnunes

    It seems that documentSelector.value can be an object. How can a regex be made using it?

    "documentSelector": {
    "type": "object",
    "description": "The path of the key to be match and the match value to assert. Using this information, helm-unittest will automatically discover the documentIndex. Generally you can ignored this field if the template file render only one document.",
    "markdownDescription": "**documentSelector** (object) _optional_\n\nThe path of the key to be match and the match value to assert.\n\nUsing this information, helm-unittest will automatically discover the documentIndex.\n\nGenerally you can ignored this field if the template file render only one document.",
    "properties": {
    "path": {
    "type": "string",
    "description": "The `documentSelector` path to assert.",
    "markdownDescription": "**path** (string)\n\nThe `documentSelector` path to assert."
    },
    "value": {
    "type": [
    "string",
    "object"
    ],
    "description": "The expected value.",
    "markdownDescription": "**value** (string|object)\n\nThe expected value.",
    "additionalProperties": true
    },


    I achieved what I needed by using https://github.com/vmware-labs/yaml-jsonpath features.
    Without relying in jsonpath features the document selector is TOO STRICT #636 (comment)

    x-keys:
      documentSelector: &x-documentSelector
        # exists: metadata.name
        path: $.metadata.name~
        value: name
        matchMany: true
        skipEmptyTemplates: true
  5. quintush commented on Aug 2, 2025

    @quintush
    Contributor

    Hello @epDHowwD , @lucasfcnunes , @llavaud , @mszygenda

    With the restructuring of the test-job and assertion options, it is possible to use the documentSelector with only the 'skipEmptyTemplates' feature on test-job level.

    The jsonschema and documentation are not yet updated, but you can check if it resolves the issue.

    Greetings
    @quintush

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions