Skip to content

GCLOUD: Support "private domains" plus many bugfixes#2482

Merged
tlimoncelli merged 6 commits intoStackExchange:masterfrom
asn-iac:gcloud-changes
Aug 8, 2023
Merged

GCLOUD: Support "private domains" plus many bugfixes#2482
tlimoncelli merged 6 commits intoStackExchange:masterfrom
asn-iac:gcloud-changes

Conversation

@asn-iac
Copy link
Copy Markdown
Contributor

@asn-iac asn-iac commented Jul 13, 2023

Hi all,
First I'd like to acknowledge and thank the dnscontrol community for contributing to such a useful project.
We needed to implement some changes with the GCLOUD provider in a private fork of dnscontrol a few years back and finally have had time to submit some of these back to the public project.
This PR addresses a few existing bugs and adds some new features:

  1. Simplify GCLOUD authentication logic and remove dependency on separate http.Client (looks related to GCLOUD: Stop using deprecated New() call #1409
    // FIXME(tlim): Is it a problem that ctx is included with hc and in
    // the call to NewService? Seems redundant.
    )
  2. extend rate limit retry to additional API calls GCLOUD: add retry support to additional API calls #2478
  3. address an issue executing changes after a new zone is created via 'dnscontrol push' GCLOUD: inserting new zones in dnsconfig blocks changes on subsequent zones #2479
  4. add support for the creation of zones with "private" visibility GCLOUD: support creation of zones with private visibility #2481
  5. implement batching of corrections to resolve 403 error when executing corrections on zones w/ >1000 changes GCLOUD: Google will return an error if too many changes are specified in a single request  #2480

Happy to break these up into smaller PRs if needed.
TIA

@TomOnTime
Copy link
Copy Markdown
Collaborator

Wow! What a generous contribution!

I'm on vacation now but will get to this when I return.

@TomOnTime
Copy link
Copy Markdown
Collaborator

@riyadhalnur could you please review?

Copy link
Copy Markdown
Collaborator

@riyadhalnur riyadhalnur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for these changes 😃. I'll test this branch against real data and get back soon

}

func (g *gcloudProvider) loadZoneInfo() error {
// TODO: In order to fully support split horizon domains within the same GCP project,
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably a good idea to create an issue for this for future action

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created #2483

var opt option.ClientOption
if key, ok := cfg["private_key"]; ok {
cfg["private_key"] = strings.Replace(key, "\\n", "\n", -1)
printer.Debugf("GCLOUD :private_key configured, attempting key-based authentication\n")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove this

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

if err != nil {
return nil, fmt.Errorf("no creds.json private_key found and ADC failed with:\n%s", err)
}
printer.Debugf("GCLOUD :using application default credentials\n")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove this

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

printer.Printf("GCLOUD :visibility %s configured\n", g.Visibility)
}
for i, v := range g.Networks {
networkURLCheck := regexp.MustCompile("^" + selfLinkBasePath + "[a-z][-a-z0-9]{4,28}[a-z0-9]/global/networks/[a-z]([-a-z0-9]{0,61}[a-z0-9])?$")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer to compile this once at the top rather than compiling it on every run of the loop

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved all of the regexp.MustComplie() calls to var block at the top - hope that was what you were looking for. let me know if you had something else in mind.

if networkURLCheck.MatchString(v) {
// the user specified a fully qualified network url
} else {
networkNameCheck := regexp.MustCompile("^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer to compile this once at the top rather than compiling it on every run of the loop

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved all of the regexp.MustComplie() calls to var block at the top - hope that was what you were looking for. let me know if you had something else in mind.

}
for i, v := range g.Networks {
networkURLCheck := regexp.MustCompile("^" + selfLinkBasePath + "[a-z][-a-z0-9]{4,28}[a-z0-9]/global/networks/[a-z]([-a-z0-9]{0,61}[a-z0-9])?$")
if networkURLCheck.MatchString(v) {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this not be simpler to just continue if value is a fully qualified URL rather than an if-else statement

if ok := networkURLCheck.MatchString(v); ok {
    continue
}

if ok := networkNameCheck.MatchString(v); !ok {
    return nil, fmt.Errorf("GCLOUD :networks set but %s does not appear to be a valid network name or url", v)
}

...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated logic to remove if-else

return nil, err
}
if len(g.Visibility) != 0 {
visibilityCheck := regexp.MustCompile("^(public|private)$")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer to compile this once at the top

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved all of the regexp.MustComplie() calls to var block at the top - hope that was what you were looking for. let me know if you had something else in mind.

if len(g.Visibility) != 0 {
visibilityCheck := regexp.MustCompile("^(public|private)$")
if ok := visibilityCheck.MatchString(g.Visibility); !ok {
return nil, fmt.Errorf("GCLOUD :visibility set but does not match ^(public|private)$")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better to say GCLOUD :visibility set but not one of public or private"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated message

Additions, Deletions []string
}

type orderedChanges struct {
Copy link
Copy Markdown
Collaborator

@riyadhalnur riyadhalnur Jul 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The struct below duplicates this. We can just have one of them

@asn-iac
Copy link
Copy Markdown
Contributor Author

asn-iac commented Jul 14, 2023

Thanks team! I'm a bit tied up today but will review comments early next week

@tlimoncelli
Copy link
Copy Markdown
Contributor

Hi! I'm back from sabbatical/vacation!

Code looks great! Thank you so much for this contribution!

Merging soon.

@tlimoncelli tlimoncelli changed the title GCLOUD: features + bugfixes GCLOUD: Support "private" domains plus many bugfixes Aug 8, 2023
@tlimoncelli tlimoncelli changed the title GCLOUD: Support "private" domains plus many bugfixes GCLOUD: Support "private domains" plus many bugfixes Aug 8, 2023
@tlimoncelli tlimoncelli merged commit 1ea9f4c into StackExchange:master Aug 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants