Repository navigation
[MAINT]: Add GitHub Actions workflow for integration testing #2425
Description
Describe the need
When pushing a change to the code I'd like the integration tests to be able to be run. This doesn't have to be automatic as there are a number of models which could be applied to stop integration credentials being abused.
- Auto-run all actions (DON'T DO THIS)
- Auto-run actions from repeat contributors (DON'T DO THIS)
- Auto-run actions from organisation members
- Put integration test credentials behind an environment and require approval to run
I'd be happy to contribute the code to do this.
SDK Version
No response
API Version
No response
Relevant log output
No response
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
nickfloyd commentedon Oct 17, 2024
@stevehipwell Thanks for the thoughts and input here! ❤
If you're up for it, this idea would be really great to see in code so that we can get a more clear picture of your thoughts here. I've also added the up for grabs label just incase others are up for working on this as well.
stevehipwell commentedon Oct 17, 2024
@nickfloyd I was working from a memory that there weren't automated tests in GH Actions and from the fact that my PRs never get automated tests. However I just checked out what's there and I can see automated tests for branches with a
test/prefix and for PRs labeled in a specific way; but they seem to be broken? Do you have more context on this?stevehipwell commentedon Oct 17, 2024
OK, I can see what the issue is. I'll have a go at fixing this.
stevehipwell commentedon Nov 25, 2024
@nickfloyd @kfcampbell I've got #2476 running and would expect the anonamous tests to work as is. Do you have a list of the currently broken acceptance tests as that'll help me figure out what should be working? Based on running some of the tests it looks like there are tests being run in anonymous mode which wont work.
kfcampbell commentedon Nov 26, 2024
We don't have a comprehensive list of broken acceptance tests unfortunately. As the suite hasn't been run front-to-back (maybe ever?), this is not readily available information. Also, it depends heavily on a really organized test setup that we just don't have (or have documented) at the moment: enterprise, organizations, test accounts, GitHub Apps, Enterprise Server instances, EMUs, Actions runners, various GitHub plans, etc., and many of those resources cost money in addition to the overhead in maintaining the test setup. #1414 is an attempt to quantify some of that setup, but hasn't taken off due to a lack of funding.
Agreed that we cannot run all tests in anonymous mode. Running any tests with anonymous mode would be a step up.
However, the test setup is stale, as you've noticed in your PR. I just checked and I'm an admin on github.com/terraformtesting, which is cool, but I have no idea where these
https://terraformtesting-ghe.eastus.cloudapp.azure.com/resources are configured (or if indeed they still exist at all).In order for this to be successful long-term, we'll need organizational buy-in from GitHub to support time and money into testing the provider so that we can get resources deployed and maintained for testing purposes. I unfortunately have been unsuccessful in my quest to sell this as a valid use of company resources thus far; hence my ad-hoc testing before merges with my own testing org/accounts.
stevehipwell commentedon Nov 26, 2024
@kfcampbell could you provide the list of tests that pass locally in your manual testing for each type (anon, user, org) when targeting github.com? Once I have these I'll look at skipping the currently failing tests conditionally based on type. I aim to have the tests where anonymous is valid running in GitHub Actions ASAP.
Could you please add
dotcom&ghesenvironments but don't add protection yet? Could you also add aacctest(or equivalent) label? With these I can finish testing the test triggering logic and any credentials added can be gated.RE GHES - the cron trigger works for anonymous tests. For now I'll disable automatic tests for GHES but we can revisit later.Ignore that, they're all being skipped.After the above we will need to work on the authenticated tests. I suspect that as well as a test user we need a free organisation, a team organisation and an enterprise organisation for the testing to be valid, does that ring true? The process here will be to have the credentials stored as environment secrets on the environments created above (
dotcomfor now) and to add required approvers to the environment.stevehipwell commentedon Dec 4, 2024
I've captured the following issues based on the work to complete the acceptance test refactoring, which has now been completed.
github_repositoryresource churns onvulnerability_alertsif not explicitly set #2489github_organization_teamsdata source doesn't populateparent.idcorrectly #2491github_organization_teamsdata source doesn't populateprivacycorrectly #2492github_branch_protection_v3resource churns when usingchecksorcontextsinrequired_status_checks#2493github_repositoryresource churns if pages are enabled andbuild_typeis set toworkflow#2495github_teamdata sourcerepositories_detaileddoes not directly correlate repo name with role #2496Other than #2490 which is a docs issue, the issues have code fixes in #2476 and I'm planning on opening individual PRs to resolve them.
github-actions commentedon Sep 1, 2025
👋 Hey Friends, this issue has been automatically marked as
stalebecause it has no recent activity. It will be closed if no further activity occurs. Please add theStatus: Pinnedlabel if you feel that this issue needs to remain open/active. Thank you for your contributions and help in keeping things tidy!stevehipwell commentedon Sep 1, 2025
This issue isn't stale! IMHO using a stale bot for a repository that hasn't had any meaningful interactions from the maintainers for almost a year is user hostile.
nickfloyd commentedon Oct 20, 2025
You are absolutely correct. I have disabled the stale workflow.
nickfloyd commentedon Oct 21, 2025
@stevehipwell I am going to start working on getting this batch of changes in - there are a few with merge conflicts but we should be able to work through those.