NEW FEATURE: Gather data for providers concurrently#2873
Merged
tlimoncelli merged 55 commits intomainfrom Mar 27, 2024
Merged
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
tlimoncelli
changed the title
tlimoncelli
changed the title
tlimoncelli
pushed a commit
that referenced
this pull request
Dec 1, 2025
Tested dnscontrol with `CanConcur()` enabled and seems to work fine. Read #2873 to see what to do, and hope below is the right way to test. ```text $ go build -race -o dnscontrol-race $ ./dnscontrol-race version v4.27.2-0.20251127184623-cf6b870052c0+dirty $ dnscontrol-race preview CONCURRENTLY checking for 16 zone(s) SERIALLY checking for 6 zone(s) Serially checking for zone: "domainX.tld" Serially checking for zone: "domainX.tld" Serially checking for zone: "domainX.tld" Serially checking for zone: "domainX.tld" Serially checking for zone: "domainX.tld" Serially checking for zone: "domainX.tld" Waiting for concurrent checking(s) to complete...DONE CONCURRENTLY gathering records of 16 zone(s) SERIALLY gathering records of 6 zone(s) Serially Gathering: "domainX.tld" Serially Gathering: "domainX.tld" Serially Gathering: "domainX.tld" Serially Gathering: "domainX.tld" Serially Gathering: "domainX.tld" Serially Gathering: "domainX.tld" Waiting for concurrent gathering(s) to complete...DONE ******************** Domain: domainX.tld INFO#1: 4 records not being deleted because of NO_PURGE: [...] ******************** Domain: domainX.tld ******************** Domain: domainX.tld INFO#1: 4 records not being deleted because of NO_PURGE: [...] ******************** Domain: domainX.tld ******************** Domain: domainX.tld ******************** Domain: domainX.tld 1 correction (PK-INWX) INFO#1: 1 records not being deleted because of IGNORE*(): [...] ******************** Domain: domainX.tld ******************** Domain: domainX.tld ******************** Domain: domainX.tld ******************** Domain: domainX.tld 30 corrections (PK-INWX) [...] ******************** Domain: domainX.tld ******************** Domain: domainX.tld 2 corrections (PK-INWX) [...] ******************** Domain: domainX.tld ******************** Domain: domainX.tld ******************** Domain: domainX.tld ******************** Domain: domainX.tld 2 corrections (PK-INWX) [...] ******************** Domain: domainX.tld ******************** Domain: domainX.tld Done. 37 corrections. ``` Unfortunately INWX sandbox is sporadically still broken so `go test` is of limited help: ```text $ go test -v -verbose -profile INWX === RUN TestDNSProviders Testing Profile="INWX" (TYPE="INWX") helpers_test.go:122: INWX: Unable to login --- FAIL: TestDNSProviders (30.03s) === RUN TestDualProviders Testing Profile="INWX" (TYPE="INWX") provider_test.go:50: Clearing everything provider_test.go:57: Adding test nameservers provider_test.go:44: #1: + CREATE dnscontrol-inwx.com NS ns1.example.com. ttl=300 provider_test.go:44: #2: + CREATE dnscontrol-inwx.com NS ns2.example.com. ttl=300 provider_test.go:60: Running again to ensure stability provider_test.go:76: Removing test nameservers provider_test.go:44: #1: - DELETE dnscontrol-inwx.com NS ns1.example.com. ttl=300 provider_test.go:44: #2: - DELETE dnscontrol-inwx.com NS ns2.example.com. ttl=300 --- PASS: TestDualProviders (2.44s) === RUN TestNameserverDots Testing Profile="INWX" (TYPE="INWX") === RUN TestNameserverDots/No_trailing_dot_in_nameserver --- PASS: TestNameserverDots (0.30s) --- PASS: TestNameserverDots/No_trailing_dot_in_nameserver (0.00s) === RUN TestDuplicateNameservers Testing Profile="INWX" (TYPE="INWX") provider_test.go:145: Skipping. Deduplication logic is not implemented for this provider. --- SKIP: TestDuplicateNameservers (0.35s) FAIL exit status 1 FAIL github.com/StackExchange/dnscontrol/v4/integrationTest 33.127s ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal: collect "preview" information in parallel to improve dnscontrol peformance.
dnscontrol ppreview-- Like preview, but collects data in parallel for providers that permit it.dnscontrol ppush-- Like ppreview but pushes--cmode=noneDon't do any work concurrently (useful when you find a bug)--cmode=allRun all providers concurrently (dangerous!)NOTE: Eventually
ppreviewandppushwill replacepreviewandpush. These are temporary commands. We'll probably do 1-2 public releases with theppversions before we remove the non-parallel verions.How to test this
Here's how to try the changes:
Is your provider safe to run concurrently?
Look for this line in the source file:
If you see
providers.Can()then your provider has already been tested and verified to be working.If you'd like to test your provider, change
Cannot()toCan()and compile with the-raceflag. This will attempt to detect any race conditions.Please report any bugs, suggestions, or performance results in the comments below.