[TT-11746] Add linter for regression test names by titpetric · Pull Request #6191 · TykTechnologies/tyk

titpetric · 2024-03-27T17:59:01Z

User description

Tests only change, adds linting for regression test functions names. Imports docker/services for test env setup.

Intended for: master, release-5.3, release-5-lts.

https://tyktech.atlassian.net/browse/TT-11746

Type

tests, enhancement

Description

Introduced a new linting task (test:lint) to enforce naming conventions for regression tests.
The linting task checks if test names follow the specified regex pattern, aiming to prevent naming violations.
This linting step is integrated into the test workflow, ensuring that tests with improper names are flagged before execution.

Changes walkthrough

Relevant files

Enhancement

Taskfile.yml `Add Linting Task for Regression Test Names` tests/regression/Taskfile.yml Added a new task `test:lint` for linting test names. The `test:lint` task checks for naming violations against a regex pattern and exits with an error if violations are found. Integrated `test:lint` task into the test workflow before running tests.	+11/-0

✨ PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

github-actions · 2024-03-27T17:59:38Z

PR Description updated to latest commit (97ab0f0)

github-actions · 2024-03-27T17:59:45Z

API Changes

no api changes detected

github-actions · 2024-03-27T18:00:05Z

PR Review

⏱️ Estimated effort to review [1-5]	2, because the PR introduces a specific enhancement with a clear scope: adding a linting task for regression test names. The changes are localized to a single file (`Taskfile.yml`) and involve adding a new task with straightforward logic. The complexity is low, and the changes are well-described in the PR description, making it easier to review.
🧪 Relevant tests	No
🔍 Possible issues	Regex Pattern Limitation: The current regex pattern `egrep -v 'Test_Issue[0-9]+'` might be too restrictive or not comprehensive enough to cover all naming conventions that might be considered valid. This could lead to valid test names being flagged as violations or vice versa.
🔒 Security concerns	No

Code feedback:

relevant file	tests/regression/Taskfile.yml
suggestion	Consider extending the regex pattern or providing a way to update it easily to accommodate future naming conventions without requiring changes to the task itself. This could involve externalizing the regex pattern to a configuration file or environment variable. [medium]
relevant line	sh: ./regression.test -test.list=Test \| egrep -v 'Test_Issue[0-9]+' \|\| true

relevant file	tests/regression/Taskfile.yml
suggestion	It might be beneficial to provide a more detailed error message that includes guidelines or a link to documentation on the naming conventions. This would help developers quickly understand and fix naming violations. [medium]
relevant line	- cmd: '{{if ne .funcs ""}}echo -e "Naming violations:\n{{.funcs}}" && exit 1 {{end}}'

relevant file	tests/regression/Taskfile.yml
suggestion	Consider adding a success message or log for cases where no naming violations are found. This can improve clarity for developers by explicitly stating that the linting passed, rather than simply not failing. [medium]
relevant line	- cmd: '{{if ne .funcs ""}}echo -e "Naming violations:\n{{.funcs}}" && exit 1 {{end}}'

✨ Review tool usage guide:

Overview:
The review tool scans the PR code changes, and generates a PR review which includes several types of feedbacks, such as possible PR issues, security threats and relevant test in the PR. More feedbacks can be added by configuring the tool.

The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.

When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

See the review usage page for a comprehensive guide on using this tool.

github-actions · 2024-03-27T18:00:07Z

PR Code Suggestions

Category	Suggestions
Enhancement	Add a success message for cases with no naming violations. Consider adding a success message or log for cases where no naming violations are found. This can improve the clarity of the test output and provide immediate feedback that the linting process has completed successfully without finding issues. tests/regression/Taskfile.yml [50] -- cmd: '{{if ne .funcs ""}}echo -e "Naming violations:\n{{.funcs}}" && exit 1 {{end}}' +- cmd: '{{if ne .funcs ""}}echo -e "Naming violations:\n{{.funcs}}" && exit 1 {{else}}echo "No naming violations found."{{end}}'
Enhancement	Add a flag to control the verbosity of the linting task output for CI integration. To ensure that the linting task integrates well with continuous integration (CI) systems, consider adding a flag or environment variable to control the verbosity of the output. This can help in debugging issues in CI environments without overwhelming the log output. tests/regression/Taskfile.yml [45] -quiet: true +quiet: ${{env.CI_QUIET_MODE \|\| "true"}}
Maintainability	Break long shell command into multiple lines for better readability. To enhance readability and maintainability, consider breaking the long shell command into multiple lines. This can be achieved by using a YAML multiline string. tests/regression/Taskfile.yml [48] -sh: ./regression.test -test.list=Test \| egrep -v 'Test_Issue[0-9]+' \|\| true +sh: > + ./regression.test -test.list=Test \| + egrep -v 'Test_Issue[0-9]+' \|\| + true
Maintainability	Clarify the regex pattern used in the `sh` command. Consider using a more specific regex pattern or adding a comment explaining the regex used in the `sh` command. This can help future maintainers understand the intent and scope of the regex pattern. tests/regression/Taskfile.yml [48] -sh: ./regression.test -test.list=Test \| egrep -v 'Test_Issue[0-9]+' \|\| true +sh: ./regression.test -test.list=Test \| egrep -v 'Test_Issue[0-9]+' # Exclude tests named Test_Issue followed by numbers \|\| true
Best practice	Document the `test:lint` task in the project's README. Ensure that the `test:lint` task is documented in the project's README or relevant documentation. This helps new contributors understand the purpose of this task and how to use it effectively. tests/regression/Taskfile.yml [43] -desc: "Lint tests" +desc: "Lint tests (see README for usage details)"

✨ Improve tool usage guide:

Overview:
The improve tool scans the PR code changes, and automatically generates suggestions for improving the PR code. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on a PR.

When commenting, to edit configurations related to the improve tool (pr_code_suggestions section), use the following template:

/improve --pr_code_suggestions.some_config1=... --pr_code_suggestions.some_config2=...

With a configuration file, use the following template:

[pr_code_suggestions]
some_config1=...
some_config2=...

See the improve usage page for a comprehensive guide on using this tool.

github-actions · 2024-03-27T18:20:02Z

💥 CI tests failed 🙈

git-state

all ok

Please look at the run or in the Checks tab.

sonarqubecloud · 2024-03-27T18:35:29Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

titpetric · 2024-03-27T18:47:39Z

/release to release-5-lts

tykbot · 2024-03-27T18:47:53Z

Working on it! Note that it can take a few minutes.

## **User description** Tests only change, adds linting for regression test functions names. Imports docker/services for test env setup. Intended for: master, release-5.3, release-5-lts. https://tyktech.atlassian.net/browse/TT-11746 ___ ## **Type** tests, enhancement ___ ## **Description** - Introduced a new linting task (`test:lint`) to enforce naming conventions for regression tests. - The linting task checks if test names follow the specified regex pattern, aiming to prevent naming violations. - This linting step is integrated into the test workflow, ensuring that tests with improper names are flagged before execution. ___ ## **Changes walkthrough** <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>Taskfile.yml</strong><dd><code>Add Linting Task for Regression Test Names</code>                              </dd></summary> <hr> tests/regression/Taskfile.yml <li>Added a new task <code>test:lint</code> for linting test names.<br> <li> The <code>test:lint</code> task checks for naming violations against a regex <br>pattern and exits with an error if violations are found.<br> <li> Integrated <code>test:lint</code> task into the test workflow before running tests. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6191/files#diff-32da89368e3551ec281e80af08b54c3410bcea3257e75af6bac113a51d822a55">+11/-0</a>    </td> </tr> </table></td></tr></tr></tbody></table> ___ > ✨ **PR-Agent usage**: >Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions --------- Co-authored-by: Tit Petric <[email protected]> (cherry picked from commit 871158a)

titpetric · 2024-03-27T18:48:24Z

/release to release-5.3

tykbot · 2024-03-27T18:48:27Z

@titpetric Succesfully merged PR

tykbot · 2024-03-27T18:49:39Z

Working on it! Note that it can take a few minutes.

## **User description** Tests only change, adds linting for regression test functions names. Imports docker/services for test env setup. Intended for: master, release-5.3, release-5-lts. https://tyktech.atlassian.net/browse/TT-11746 ___ ## **Type** tests, enhancement ___ ## **Description** - Introduced a new linting task (`test:lint`) to enforce naming conventions for regression tests. - The linting task checks if test names follow the specified regex pattern, aiming to prevent naming violations. - This linting step is integrated into the test workflow, ensuring that tests with improper names are flagged before execution. ___ ## **Changes walkthrough** <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>Taskfile.yml</strong><dd><code>Add Linting Task for Regression Test Names</code>                              </dd></summary> <hr> tests/regression/Taskfile.yml <li>Added a new task <code>test:lint</code> for linting test names.<br> <li> The <code>test:lint</code> task checks for naming violations against a regex <br>pattern and exits with an error if violations are found.<br> <li> Integrated <code>test:lint</code> task into the test workflow before running tests. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6191/files#diff-32da89368e3551ec281e80af08b54c3410bcea3257e75af6bac113a51d822a55">+11/-0</a>    </td> </tr> </table></td></tr></tr></tbody></table> ___ > ✨ **PR-Agent usage**: >Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions --------- Co-authored-by: Tit Petric <[email protected]> (cherry picked from commit 871158a)

tykbot · 2024-03-27T18:50:04Z

@titpetric Succesfully merged PR

…es (#6191) [TT-11746] Add linter for regression test names (#6191) ## **User description** Tests only change, adds linting for regression test functions names. Imports docker/services for test env setup. Intended for: master, release-5.3, release-5-lts. https://tyktech.atlassian.net/browse/TT-11746 ___ ## **Type** tests, enhancement ___ ## **Description** - Introduced a new linting task (`test:lint`) to enforce naming conventions for regression tests. - The linting task checks if test names follow the specified regex pattern, aiming to prevent naming violations. - This linting step is integrated into the test workflow, ensuring that tests with improper names are flagged before execution. ___ ## **Changes walkthrough** <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>Taskfile.yml</strong><dd><code>Add Linting Task for Regression Test Names</code>                              </dd></summary> <hr> tests/regression/Taskfile.yml <li>Added a new task <code>test:lint</code> for linting test names.<br> <li> The <code>test:lint</code> task checks for naming violations against a regex <br>pattern and exits with an error if violations are found.<br> <li> Integrated <code>test:lint</code> task into the test workflow before running tests. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6191/files#diff-32da89368e3551ec281e80af08b54c3410bcea3257e75af6bac113a51d822a55">+11/-0</a>    </td> </tr> </table></td></tr></tr></tbody></table> ___ > ✨ **PR-Agent usage**: >Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions --------- Co-authored-by: Tit Petric <[email protected]>

…ames (#6191) [TT-11746] Add linter for regression test names (#6191) ## **User description** Tests only change, adds linting for regression test functions names. Imports docker/services for test env setup. Intended for: master, release-5.3, release-5-lts. https://tyktech.atlassian.net/browse/TT-11746 ___ ## **Type** tests, enhancement ___ ## **Description** - Introduced a new linting task (`test:lint`) to enforce naming conventions for regression tests. - The linting task checks if test names follow the specified regex pattern, aiming to prevent naming violations. - This linting step is integrated into the test workflow, ensuring that tests with improper names are flagged before execution. ___ ## **Changes walkthrough** <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>Taskfile.yml</strong><dd><code>Add Linting Task for Regression Test Names</code>                              </dd></summary> <hr> tests/regression/Taskfile.yml <li>Added a new task <code>test:lint</code> for linting test names.<br> <li> The <code>test:lint</code> task checks for naming violations against a regex <br>pattern and exits with an error if violations are found.<br> <li> Integrated <code>test:lint</code> task into the test workflow before running tests. </details> </td> <td><a href="https://github.com/TykTechnologies/tyk/pull/6191/files#diff-32da89368e3551ec281e80af08b54c3410bcea3257e75af6bac113a51d822a55">+11/-0</a>    </td> </tr> </table></td></tr></tr></tbody></table> ___ > ✨ **PR-Agent usage**: >Comment `/help` on the PR to get a list of all available PR-Agent tools and their descriptions --------- Co-authored-by: Tit Petric <[email protected]>

Add linter for regression test names

97ab0f0

github-actions Bot added enhancement Tests labels Mar 27, 2024

github-actions Bot added the Review effort [1-5]: 2 label Mar 27, 2024

titpetric requested a review from jeffy-mathew March 27, 2024 18:01

Import docker services used for regression tests

5482227

mac: rm -f may be positional

35289c9

jeffy-mathew approved these changes Mar 27, 2024

View reviewed changes

titpetric enabled auto-merge (squash) March 27, 2024 18:28

titpetric merged commit 871158a into master Mar 27, 2024

titpetric deleted the test/add-regression-test-naming-lint branch March 27, 2024 18:45

buger mentioned this pull request Mar 27, 2024

Merging to release-5-lts: [TT-11746] Add linter for regression test names (#6191) #6192

Merged

buger mentioned this pull request Mar 27, 2024

Merging to release-5.3: [TT-11746] Add linter for regression test names (#6191) #6193

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[TT-11746] Add linter for regression test names#6191

[TT-11746] Add linter for regression test names#6191
titpetric merged 3 commits into
masterfrom
test/add-regression-test-naming-lint

titpetric commented Mar 27, 2024 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 27, 2024

Uh oh!

github-actions Bot commented Mar 27, 2024

Uh oh!

github-actions Bot commented Mar 27, 2024

Uh oh!

github-actions Bot commented Mar 27, 2024 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 27, 2024

Uh oh!

sonarqubecloud Bot commented Mar 27, 2024

Uh oh!

titpetric commented Mar 27, 2024

Uh oh!

tykbot Bot commented Mar 27, 2024

Uh oh!

titpetric commented Mar 27, 2024

Uh oh!

tykbot Bot commented Mar 27, 2024

Uh oh!

tykbot Bot commented Mar 27, 2024

Uh oh!

tykbot Bot commented Mar 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

titpetric commented Mar 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User description

Type

Description

Changes walkthrough

Uh oh!

github-actions Bot commented Mar 27, 2024

Uh oh!

github-actions Bot commented Mar 27, 2024

Uh oh!

github-actions Bot commented Mar 27, 2024

PR Review

Uh oh!

github-actions Bot commented Mar 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Code Suggestions

Uh oh!

github-actions Bot commented Mar 27, 2024

💥 CI tests failed 🙈

git-state

Uh oh!

sonarqubecloud Bot commented Mar 27, 2024

Quality Gate passed

Uh oh!

titpetric commented Mar 27, 2024

Uh oh!

tykbot Bot commented Mar 27, 2024

Uh oh!

titpetric commented Mar 27, 2024

Uh oh!

tykbot Bot commented Mar 27, 2024

Uh oh!

tykbot Bot commented Mar 27, 2024

Uh oh!

tykbot Bot commented Mar 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

titpetric commented Mar 27, 2024 •

edited

Loading

github-actions Bot commented Mar 27, 2024 •

edited

Loading