Skip to content

[SYSE-332] Fail tests reliably#6130

Merged
alephnull merged 1 commit into
masterfrom
releng/master
Mar 11, 2024
Merged

[SYSE-332] Fail tests reliably#6130
alephnull merged 1 commit into
masterfrom
releng/master

Conversation

@ermirizio

@ermirizio ermirizio commented Mar 8, 2024

Copy link
Copy Markdown
Collaborator

User description

Apply latest template updates


Type

enhancement, documentation


Description

  • Added a new AWS BYOL AMI Packer configuration (ci/aws/byol.pkr.hcl) for creating custom AMIs.
  • Updated generation dates and simplified the publishers section in the GoReleaser configuration (ci/goreleaser/goreleaser.yml).
  • Enhanced the release GitHub Actions workflow (.github/workflows/release.yml) with new steps, including fetching the commit author and handling Docker logs.
  • Updated generation dates in various scripts and Dockerfiles to reflect the latest changes.

Changes walkthrough

Relevant files
Enhancement
3 files
byol.pkr.hcl
Add AWS BYOL AMI Packer Configuration                                       

ci/aws/byol.pkr.hcl

  • Added a new Packer configuration for BYOL (Bring Your Own License)
    AMIs on AWS.
  • Defined variables for AMI configuration, including OS flavor, source
    AMI owner, and search string.
  • Configured the Amazon EBS source with settings for the AMI, including
    instance type and region.
  • Added provisioners for setting up the Tyk Gateway environment.
  • +105/-0 
    goreleaser.yml
    Update GoReleaser Configuration and Simplify Publishers Section

    ci/goreleaser/goreleaser.yml

  • Updated the generation date in the GoReleaser configuration file
    header comment.
  • Modified the publishers section to use the packagecloud publish
    command directly.
  • +2/-5     
    release.yml
    Enhance Release Workflow with Additional Steps and Updates

    .github/workflows/release.yml

  • Updated the generation date in the workflow header comment.
  • Added a step to get the commit author.
  • Updated actions/cache to v4.
  • Updated actions/upload-artifact to v4.
  • Renamed the test-controller job to test-controller-api and updated its
    container version.
  • Added steps for handling Docker logs and uploading them as artifacts
    in case of failure.
  • Removed the step for commenting on PRs.
  • +52/-32 
    Documentation
    7 files
    Dockerfile.std
    Update Generation Date in Dockerfile                                         

    ci/Dockerfile.std

    • Updated the generation date in the Dockerfile header comment.
    +1/-1     
    pc.sh
    Update Generation Date in Shell Script                                     

    ci/bin/pc.sh

    • Updated the generation date in the script header comment.
    +1/-1     
    unlock-agent.sh
    Update Generation Date in Unlock Agent Script                       

    ci/bin/unlock-agent.sh

    • Updated the generation date in the script header comment.
    +1/-1     
    before_install.sh
    Update Generation Date in Before Install Script                   

    ci/install/before_install.sh

    • Updated the generation date in the script header comment.
    +1/-1     
    post_install.sh
    Update Generation Date in Post Install Script                       

    ci/install/post_install.sh

    • Updated the generation date in the script header comment.
    +1/-1     
    post_remove.sh
    Update Generation Date in Post Remove Script                         

    ci/install/post_remove.sh

    • Updated the generation date in the script header comment.
    +1/-1     
    post_trans.sh
    Update Generation Date in Post Trans Script                           

    ci/install/post_trans.sh

    • Updated the generation date in the script header comment.
    +1/-1     

    PR-Agent usage:
    Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

    @ermirizio
    ermirizio requested a review from a team as a code owner March 8, 2024 17:35
    @github-actions

    github-actions Bot commented Mar 8, 2024

    Copy link
    Copy Markdown
    Contributor

    API Changes

    no api changes detected

    @github-actions

    github-actions Bot commented Mar 8, 2024

    Copy link
    Copy Markdown
    Contributor

    PR Description updated to latest commit (5e4d557)

    @github-actions

    github-actions Bot commented Mar 8, 2024

    Copy link
    Copy Markdown
    Contributor

    PR Review

    ⏱️ Estimated effort to review [1-5]

    2, because the PR mainly involves updates to generated files, addition of a new Packer configuration, and minor workflow adjustments. The complexity is not high, but verifying the correctness of the Packer configuration and the impact of workflow changes requires some domain knowledge.

    🧪 Relevant tests

    No

    🔍 Possible issues

    Possible Bug: The goreleaser.yml file simplifies the publishers command to use packagecloud publish directly. This change assumes that the environment variables DEBVERS and RPMVERS are correctly interpreted by packagecloud publish, which might not be the case if the command expects a different format or additional arguments.

    🔒 Security concerns

    No

    Code feedback:
    relevant fileci/goreleaser/goreleaser.yml
    suggestion      

    Ensure that the packagecloud publish command correctly interprets the DEBVERS and RPMVERS environment variables. If these variables contain spaces or special characters, they might need to be quoted or processed differently. Testing this command manually with actual values for these variables can help ensure that the publishing process will not break after this change. [important]

    relevant linecmd: packagecloud publish --debvers "{{ .Env.DEBVERS }}" --rpmvers "{{ .Env.RPMVERS }}" tyk/tyk-gateway-unstable {{ .ArtifactPath }}

    relevant file.github/workflows/release.yml
    suggestion      

    Consider adding error handling or a retry mechanism for steps that involve network operations, such as Docker login, Docker push, and artifact uploads. Network operations can sometimes fail due to transient issues, and having a retry mechanism can make the workflow more robust. For instance, you could use a step that retries the command on failure up to a certain number of attempts. [medium]

    relevant lineif: startsWith(github.ref, 'refs/tags')

    relevant fileci/aws/byol.pkr.hcl
    suggestion      

    Validate the Packer configuration by running packer validate ci/aws/byol.pkr.hcl to ensure that all required variables are defined and that there are no syntax errors. This is especially important because the configuration introduces several new variables and a data source for fetching the base AMI. Ensuring the configuration is valid before merging can prevent runtime errors during AMI creation. [important]

    relevant linepacker {

    relevant fileci/aws/byol.pkr.hcl
    suggestion      

    Consider parameterizing the instance_type in the source "amazon-ebs" "byol" block to allow for flexibility in testing or production scenarios. Using a variable for the instance type instead of hardcoding it to t3.micro can make the configuration more adaptable to different use cases or performance requirements. [medium]

    relevant lineinstance_type = "t3.micro"


    ✨ Review tool usage guide:

    Overview:
    The review tool scans the PR code changes, and generates a PR review. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.
    When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

    /review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...
    

    With a configuration file, use the following template:

    [pr_reviewer]
    some_config1=...
    some_config2=...
    
    Utilizing extra instructions

    The review tool can be configured with extra instructions, which can be used to guide the model to a feedback tailored to the needs of your project.

    Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify the relevant sub-tool, and the relevant aspects of the PR that you want to emphasize.

    Examples for extra instructions:

    [pr_reviewer] # /review #
    extra_instructions="""
    In the 'possible issues' section, emphasize the following:
    - Does the code logic cover relevant edge cases?
    - Is the code logic clear and easy to understand?
    - Is the code logic efficient?
    ...
    """
    

    Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

    How to enable\disable automation
    • When you first install PR-Agent app, the default mode for the review tool is:
    pr_commands = ["/review", ...]
    

    meaning the review tool will run automatically on every PR, with the default configuration.
    Edit this field to enable/disable the tool, or to change the used configurations

    Auto-labels

    The review tool can auto-generate two specific types of labels for a PR:

    • a possible security issue label, that detects possible security issues (enable_review_labels_security flag)
    • a Review effort [1-5]: x label, where x is the estimated effort to review the PR (enable_review_labels_effort flag)
    Extra sub-tools

    The review tool provides a collection of possible feedbacks about a PR.
    It is recommended to review the possible options, and choose the ones relevant for your use case.
    Some of the feature that are disabled by default are quite useful, and should be considered for enabling. For example:
    require_score_review, require_soc2_ticket, and more.

    Auto-approve PRs

    By invoking:

    /review auto_approve
    

    The tool will automatically approve the PR, and add a comment with the approval.

    To ensure safety, the auto-approval feature is disabled by default. To enable auto-approval, you need to actively set in a pre-defined configuration file the following:

    [pr_reviewer]
    enable_auto_approval = true
    

    (this specific flag cannot be set with a command line argument, only in the configuration file, committed to the repository)

    You can also enable auto-approval only if the PR meets certain requirements, such as that the estimated_review_effort is equal or below a certain threshold, by adjusting the flag:

    [pr_reviewer]
    maximal_review_effort = 5
    
    More PR-Agent commands

    To invoke the PR-Agent, add a comment using one of the following commands:

    • /review: Request a review of your Pull Request.
    • /describe: Update the PR title and description based on the contents of the PR.
    • /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback.
    • /ask <QUESTION>: Ask a question about the PR.
    • /update_changelog: Update the changelog based on the PR's contents.
    • /add_docs 💎: Generate docstring for new components introduced in the PR.
    • /generate_labels 💎: Generate labels for the PR based on the PR's contents.
    • /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component.

    See the tools guide for more details.
    To list the possible configuration parameters, add a /config comment.

    See the review usage page for a comprehensive guide on using this tool.

    @github-actions

    github-actions Bot commented Mar 8, 2024

    Copy link
    Copy Markdown
    Contributor

    PR Code Suggestions

    CategorySuggestions                                                                                                                                                       
    Best practice
    Use a more specific version constraint for the amazon plugin.

    Consider using a more specific version constraint for the amazon plugin to ensure
    compatibility and predictability of your builds. Using a version range that is too broad,
    like ">= 0.0.2", can lead to unexpected changes if a new version introduces breaking
    changes.

    ci/aws/byol.pkr.hcl [7-9]

     amazon = {
    -  version = ">= 0.0.2"
    +  version = "~> 0.0.2"
       source  = "github.com/hashicorp/amazon"
     }
     
    Validate non-empty values for critical variables.

    It's a good practice to validate the flavour, source_ami_owner, and ami_search_string
    variables to ensure they are not empty. This helps prevent runtime errors and ensures that
    the script fails early if required variables are not provided.

    ci/aws/byol.pkr.hcl [14-17]

     variable "flavour" {
       description = "OS Flavour"
    -  type    = string
    +  type        = string
    +  validation {
    +    condition     = length(var.flavour) > 0
    +    error_message = "The flavour variable must not be empty."
    +  }
     }
     
    Specify more restrictive tags for resources.

    For better security and to avoid potential conflicts, consider specifying a more
    restrictive set of permissions for the tags in the source "amazon-ebs" "byol" block.
    Overly broad permissions can lead to unintended access or modifications.

    ci/aws/byol.pkr.hcl [76-81]

     tags = {
       Component = "tyk"
       Flavour   = "${var.flavour}"
       Product   = "byol"
       Version   = "${var.version}"
    +  Environment = "production" # Example tag for environment
    +  Owner       = "team-name"  # Example tag for ownership
     }
     
    Add validation for geoip_license and version variables.

    To ensure that the geoip_license and version variables are always set to meaningful
    values, consider adding a validation block for these variables. This can help catch
    configuration errors early in the build process.

    ci/aws/byol.pkr.hcl [26-28]

     variable "geoip_license" {
       type    = string
       default = "${env("GEOIP_LICENSE")}"
    +  validation {
    +    condition     = length(var.geoip_license) > 0
    +    error_message = "The geoip_license variable must not be empty."
    +  }
     }
     
    Use specific versions for GitHub Actions to ensure consistent behavior.

    Consider using a more specific tag than v4 for actions/download-artifact and
    actions/upload-artifact to ensure consistent behavior across runs. Using a specific
    version helps in avoiding unexpected changes when new versions are released.

    .github/workflows/release.yml [325-277]

    -- uses: actions/download-artifact@v4
    -- uses: actions/upload-artifact@v4
    +- uses: actions/[email protected]
    +- uses: actions/[email protected]
     
    Maintainability
    Use consistent indentation for better readability.

    To improve the maintainability and readability of your HCL code, consider using a
    consistent indentation style. For example, align the equals signs (=) within blocks to
    make the code easier to read.

    ci/aws/byol.pkr.hcl [14-17]

     variable "flavour" {
       description = "OS Flavour"
    -  type    = string
    +  type        = string
     }
     
    Use YAML anchors to reduce duplication of environment variables.

    To improve the maintainability and readability of the workflow, consider using a YAML
    anchor for the repeated environment variables (TYK_DB_LICENSEKEY, TYK_MDCB_LICENSE, ECR,
    VERSIONS) in the "Docker logs for all components" and "Upload Artifact" steps. This will
    help reduce duplication and make future updates easier.

    .github/workflows/release.yml [256-262]

    -env:
    +common_env: &common_env
       TYK_DB_LICENSEKEY: ${{ secrets.DASH_LICENSE }}
       TYK_MDCB_LICENSE: ${{ secrets.MDCB_LICENSE }}
       ECR: ${{ steps.ecr.outputs.registry }}
       VERSIONS: ${{ needs.test-controller-api.outputs.versions }}
     
    +env: *common_env
    +
    Enhancement
    Use descriptive names for workflow steps to enhance readability.

    For the "Get commit author" step, consider using a more descriptive step name that clearly
    indicates its purpose, such as "Extract and Set Commit Author". This makes the workflow
    easier to understand at a glance.

    .github/workflows/release.yml [51]

    -- name: Get commit author
    +- name: Extract and Set Commit Author
     
    Add error handling for the "Run tests" step to improve workflow robustness.

    To ensure the robustness of the workflow, consider adding error handling for the "Run
    tests" step. Specifically, you can capture the exit code of the pytest command and use it
    to conditionally perform cleanup or additional diagnostics. This approach provides more
    control over the workflow's behavior in case of test failures.

    .github/workflows/release.yml [244-246]

    -if docker run --rm --network auto_default --env-file pytest.env -v ${{ github.workspace }}/reports:/app/reports \
    +set +e
    +docker run --rm --network auto_default --env-file pytest.env -v ${{ github.workspace }}/reports:/app/reports \
       ${{ steps.ecr.outputs.registry }}/tyk-automated-tests:${{ needs.test-controller-api.outputs.gd_tag }} \
    -  pytest -c pytest_ci.ini --ci -m "not local and not dind ${{ matrix.markers }}" | tee tests.out; then
    +  pytest -c pytest_ci.ini --ci -m "not local and not dind ${{ matrix.markers }}" | tee tests.out
    +TEST_EXIT_CODE=$?
    +set -e
    +if [ $TEST_EXIT_CODE -eq 0 ]; then
     

    ✨ Improve tool usage guide:

    Overview:
    The improve tool scans the PR code changes, and automatically generates suggestions for improving the PR code. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on a PR.
    When commenting, to edit configurations related to the improve tool (pr_code_suggestions section), use the following template:

    /improve --pr_code_suggestions.some_config1=... --pr_code_suggestions.some_config2=...
    

    With a configuration file, use the following template:

    [pr_code_suggestions]
    some_config1=...
    some_config2=...
    
    Enabling\disabling automation

    When you first install the app, the default mode for the improve tool is:

    pr_commands = ["/improve --pr_code_suggestions.summarize=true", ...]
    

    meaning the improve tool will run automatically on every PR, with summarization enabled. Delete this line to disable the tool from running automatically.

    Utilizing extra instructions

    Extra instructions are very important for the improve tool, since they enable to guide the model to suggestions that are more relevant to the specific needs of the project.

    Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify relevant aspects that you want the model to focus on.

    Examples for extra instructions:

    [pr_code_suggestions] # /improve #
    extra_instructions="""
    Emphasize the following aspects:
    - Does the code logic cover relevant edge cases?
    - Is the code logic clear and easy to understand?
    - Is the code logic efficient?
    ...
    """
    

    Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

    A note on code suggestions quality
    • While the current AI for code is getting better and better (GPT-4), it's not flawless. Not all the suggestions will be perfect, and a user should not accept all of them automatically.
    • Suggestions are not meant to be simplistic. Instead, they aim to give deep feedback and raise questions, ideas and thoughts to the user, who can then use his judgment, experience, and understanding of the code base.
    • Recommended to use the 'extra_instructions' field to guide the model to suggestions that are more relevant to the specific needs of the project, or use the custom suggestions 💎 tool
    • With large PRs, best quality will be obtained by using 'improve --extended' mode.
    More PR-Agent commands

    To invoke the PR-Agent, add a comment using one of the following commands:

    • /review: Request a review of your Pull Request.
    • /describe: Update the PR title and description based on the contents of the PR.
    • /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback.
    • /ask <QUESTION>: Ask a question about the PR.
    • /update_changelog: Update the changelog based on the PR's contents.
    • /add_docs 💎: Generate docstring for new components introduced in the PR.
    • /generate_labels 💎: Generate labels for the PR based on the PR's contents.
    • /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component.

    See the tools guide for more details.
    To list the possible configuration parameters, add a /config comment.

    See the improve usage page for a more comprehensive guide on using this tool.

    @ermirizio

    Copy link
    Copy Markdown
    Collaborator Author

    All Required checks passing, head in sync, ready for review
    Uploading image.png…

    @alephnull alephnull changed the title releng/master [SYSE-332] Fail tests reliably Mar 11, 2024
    @buger
    buger force-pushed the releng/master branch 3 times, most recently from 87ee357 to bdcce62 Compare March 11, 2024 09:51
    @alephnull
    alephnull disabled auto-merge March 11, 2024 11:45
    @alephnull
    alephnull enabled auto-merge (squash) March 11, 2024 11:46
    @sonarqubecloud

    Copy link
    Copy Markdown

    Quality Gate Passed Quality Gate passed

    Issues
    0 New issues
    0 Accepted issues

    Measures
    0 Security Hotspots
    No data about Coverage
    0.0% Duplication on New Code

    See analysis details on SonarCloud

    @alephnull
    alephnull merged commit 855682d into master Mar 11, 2024
    @alephnull
    alephnull deleted the releng/master branch March 11, 2024 12:17
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Projects

    None yet

    Development

    Successfully merging this pull request may close these issues.

    2 participants