Skip to content

feat(datadog)!: Add SLO tables, housekeeping #11166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
kodiakhq merged 8 commits into from
Jun 1, 2023
Merged

feat(datadog)!: Add SLO tables, housekeeping #11166

kodiakhq merged 8 commits into from
Jun 1, 2023

Conversation

@disq
Copy link
Member

@disq disq commented Jun 1, 2023 •
edited
Loading

Adds datadog_slos and datadog_slo_corrections tables for SLO info.

Housekeeping:

  • Updates the Datadog SDK used from 2.9.0 to 2.13.0
  • Updates all PKs to include account_name and id, plus parent id if exists
  • Updates plugin-sdk version to 3.9.0
  • Removes all manually added fields, now using transformers exclusively
  • Datadog NullableTypes mapped to proper types

Closes #10780

@disq disq requested review from a team and bbernays and removed request for a team June 1, 2023 11:58
@github-actions
Copy link

github-actions bot commented Jun 1, 2023 •
edited
Loading

This PR has the following changes to source plugin(s) tables:

  • Table datadog_dashboard_lists: column order changed for id
  • ⚠️ BREAKING CHANGE: Table datadog_dashboards: column type changed from json to utf8 for description
  • Table datadog_dashboards: column order changed for id
  • ⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int32 for updater_id
  • ⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int64 for canceled
  • ⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int64 for end
  • ⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int64 for monitor_id
  • ⚠️ BREAKING CHANGE: Table datadog_downtimes: column type changed from json to int64 for parent_id
  • Table datadog_downtimes: column added with name notify_end_states and type list<item: utf8, nullable>
  • Table datadog_downtimes: column added with name notify_end_types and type list<item: utf8, nullable>
  • Table datadog_downtimes: column order changed for id
  • ⚠️ BREAKING CHANGE: Table datadog_global_variables: primary key constraint added to column account_name
  • Table datadog_global_variables: column order changed for id
  • Table datadog_hosts: column order changed for id
  • ⚠️ BREAKING CHANGE: Table datadog_incident_attachments: column added with name incident_id (PK) and type utf8
  • ⚠️ BREAKING CHANGE: Table datadog_incident_attachments: primary key constraint added to column account_name
  • ⚠️ BREAKING CHANGE: Table datadog_incident_attachments: primary key constraint added to column id
  • ⚠️ BREAKING CHANGE: Table datadog_incident_attachments: primary key constraint removed from column _cq_id
  • Table datadog_incident_attachments: column added with name additional_properties and type json
  • Table datadog_incident_attachments: column order changed for type
  • Table datadog_incidents: column order changed for id
  • ⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int32 for updater_id
  • ⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int64 for canceled
  • ⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int64 for end
  • ⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int64 for monitor_id
  • ⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: column type changed from json to int64 for parent_id
  • ⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: primary key constraint added to column account_name
  • ⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: primary key constraint added to column id
  • ⚠️ BREAKING CHANGE: Table datadog_monitor_downtimes: primary key constraint removed from column _cq_id
  • Table datadog_monitor_downtimes: column added with name additional_properties and type json
  • Table datadog_monitor_downtimes: column added with name notify_end_states and type list<item: utf8, nullable>
  • Table datadog_monitor_downtimes: column added with name notify_end_types and type list<item: utf8, nullable>
  • Table datadog_monitors: column added with name matching_downtimes and type json
  • Table datadog_monitors: column order changed for deleted
  • Table datadog_monitors: column order changed for id
  • Table datadog_monitors: column order changed for priority
  • Table datadog_notebooks: column order changed for id
  • Table datadog_permissions: column order changed for id
  • ⚠️ BREAKING CHANGE: Table datadog_role_permissions: column added with name role_id (PK) and type utf8
  • ⚠️ BREAKING CHANGE: Table datadog_role_permissions: primary key constraint added to column account_name
  • ⚠️ BREAKING CHANGE: Table datadog_role_permissions: primary key constraint added to column id
  • ⚠️ BREAKING CHANGE: Table datadog_role_permissions: primary key constraint removed from column _cq_id
  • ⚠️ BREAKING CHANGE: Table datadog_role_users: column added with name role_id (PK) and type utf8
  • ⚠️ BREAKING CHANGE: Table datadog_role_users: primary key constraint added to column account_name
  • ⚠️ BREAKING CHANGE: Table datadog_role_users: primary key constraint added to column id
  • ⚠️ BREAKING CHANGE: Table datadog_role_users: primary key constraint removed from column _cq_id
  • Table datadog_roles: column order changed for id
  • Table datadog_slo_corrections was added
  • Table datadog_slos was added
  • Table datadog_synthetics: column order changed for public_id
  • ⚠️ BREAKING CHANGE: Table datadog_user_permissions: column added with name user_id (PK) and type utf8
  • ⚠️ BREAKING CHANGE: Table datadog_user_permissions: primary key constraint added to column account_name
  • ⚠️ BREAKING CHANGE: Table datadog_user_permissions: primary key constraint added to column id
  • ⚠️ BREAKING CHANGE: Table datadog_user_permissions: primary key constraint removed from column _cq_id
  • Table datadog_users: column order changed for id

@disq disq linked an issue Jun 1, 2023 that may be closed by this pull request
feat: datadog source plugin to include SLO data #10780
Closed
1 task
hermanschaaf
hermanschaaf reviewed Jun 1, 2023
View reviewed changes
website/tables/datadog/datadog_slo_corrections.md Outdated
@@ -0,0 +1,19 @@
# Table: datadog_slo_corrections

This table shows data for Datadog Slo Corrections.
Copy link
Member

@hermanschaaf hermanschaaf Jun 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, but would be nice to add an exception for SLO in the title here

hermanschaaf
hermanschaaf approved these changes Jun 1, 2023
View reviewed changes
Copy link
Member

@hermanschaaf hermanschaaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - could you maybe just add some more details about the changes to the PR description for future readers?

candiduslynx
candiduslynx reviewed Jun 1, 2023
View reviewed changes
candiduslynx
candiduslynx reviewed Jun 1, 2023
View reviewed changes
candiduslynx
candiduslynx reviewed Jun 1, 2023
View reviewed changes
candiduslynx
candiduslynx reviewed Jun 1, 2023
View reviewed changes
plugins/source/datadog/resources/services/downtimes/downtimes.go
Resolver: fetchDowntimes,
Multiplex: client.AccountMultiplex,
Transform: transformers.TransformWithStruct(&datadogV1.Downtime{}),
Transform: client.TransformWithStruct(&datadogV1.Downtime{}, transformers.WithPrimaryKeys("Id")),
Copy link
Contributor

@candiduslynx candiduslynx Jun 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

transformers.WithPrimaryKeys("Id") might be moved to client.TransformWithStruct

Copy link
Member Author

@disq disq Jun 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that's wise, we'd need to revert it all for a single struct later on :)

candiduslynx
candiduslynx reviewed Jun 1, 2023
View reviewed changes
Copy link
Contributor

@candiduslynx candiduslynx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a note about WithTitleTransformer

candiduslynx
candiduslynx reviewed Jun 1, 2023
View reviewed changes
candiduslynx
candiduslynx approved these changes Jun 1, 2023
View reviewed changes
Copy link
Contributor

@candiduslynx candiduslynx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only a smallish transformer comment (might be not required, just took k8s title replacer as source)

@disq
Copy link
Member Author

disq commented Jun 1, 2023

only a smallish transformer comment (might be not required, just took k8s title replacer as source)

I don't think it's required as I don't see any rogue/trailing spaces in the SLO table markdown files.

@disq disq changed the title feat(datadog)!: Update Datadog SDK, fix types, add SLO data feat(datadog)!: Add SLO tables, housekeeping Jun 1, 2023
@disq
Copy link
Member Author

disq commented Jun 1, 2023

future readers?

@hermanschaaf Do you think that's enough? I've updated the PR title as well, but I don't think a "commit override" is necessary.

@hermanschaaf
Copy link
Member

hermanschaaf commented Jun 1, 2023

@disq Looks good, thanks. I agree, don't think a commit override is necessary here

@disq disq added the automerge Automatically merge once required checks pass label Jun 1, 2023
@kodiakhq kodiakhq bot merged commit 103a6bf into cloudquery:main Jun 1, 2023
@cq-bot cq-bot mentioned this pull request Jun 1, 2023
Merged
@disq disq deleted the feat/datadog-updates branch June 1, 2023 20:58
kodiakhq bot pushed a commit that referenced this pull request Jun 6, 2023
@cq-bot
chore(main): Release plugins-source-datadog v3.0.0 (#11122)
31de02f 
🤖 I have created a release *beep* *boop*
---


## [3.0.0](plugins-source-datadog-v2.0.0...plugins-source-datadog-v3.0.0) (2023-06-06)


### ⚠ BREAKING CHANGES

* **datadog:** Add SLO tables, housekeeping ([#11166](#11166))

### This Release has the Following Changes to Tables
- Table `datadog_dashboards`: column type changed from `json` to `utf8` for `description` (:warning: breaking)
- Table `datadog_downtimes`: column type changed from `json` to `int32` for `updater_id` (:warning: breaking)
- Table `datadog_downtimes`: column type changed from `json` to `int64` for `canceled` (:warning: breaking)
- Table `datadog_downtimes`: column type changed from `json` to `int64` for `end` (:warning: breaking)
- Table `datadog_downtimes`: column type changed from `json` to `int64` for `monitor_id` (:warning: breaking)
- Table `datadog_downtimes`: column type changed from `json` to `int64` for `parent_id` (:warning: breaking)
- Table `datadog_downtimes`: column added with name `notify_end_states` and type `list<item: utf8, nullable>`
- Table `datadog_downtimes`: column added with name `notify_end_types` and type `list<item: utf8, nullable>`
- Table `datadog_global_variables`: primary key constraint added to column `account_name` (:warning: breaking)
- Table `datadog_incident_attachments`: column added with name `incident_id (PK)` and type `utf8` (:warning: breaking)
- Table `datadog_incident_attachments`: primary key constraint added to column `account_name` (:warning: breaking)
- Table `datadog_incident_attachments`: primary key constraint added to column `id` (:warning: breaking)
- Table `datadog_incident_attachments`: primary key constraint removed from column `_cq_id` (:warning: breaking)
- Table `datadog_incident_attachments`: column added with name `additional_properties` and type `json`
- Table `datadog_monitor_downtimes`: column type changed from `json` to `int32` for `updater_id` (:warning: breaking)
- Table `datadog_monitor_downtimes`: column type changed from `json` to `int64` for `canceled` (:warning: breaking)
- Table `datadog_monitor_downtimes`: column type changed from `json` to `int64` for `end` (:warning: breaking)
- Table `datadog_monitor_downtimes`: column type changed from `json` to `int64` for `monitor_id` (:warning: breaking)
- Table `datadog_monitor_downtimes`: column type changed from `json` to `int64` for `parent_id` (:warning: breaking)
- Table `datadog_monitor_downtimes`: primary key constraint added to column `account_name` (:warning: breaking)
- Table `datadog_monitor_downtimes`: primary key constraint added to column `id` (:warning: breaking)
- Table `datadog_monitor_downtimes`: primary key constraint removed from column `_cq_id` (:warning: breaking)
- Table `datadog_monitor_downtimes`: column added with name `additional_properties` and type `json`
- Table `datadog_monitor_downtimes`: column added with name `notify_end_states` and type `list<item: utf8, nullable>`
- Table `datadog_monitor_downtimes`: column added with name `notify_end_types` and type `list<item: utf8, nullable>`
- Table `datadog_monitors`: column added with name `matching_downtimes` and type `json`
- Table `datadog_role_permissions`: column added with name `role_id (PK)` and type `utf8` (:warning: breaking)
- Table `datadog_role_permissions`: primary key constraint added to column `account_name` (:warning: breaking)
- Table `datadog_role_permissions`: primary key constraint added to column `id` (:warning: breaking)
- Table `datadog_role_permissions`: primary key constraint removed from column `_cq_id` (:warning: breaking)
- Table `datadog_role_users`: column added with name `role_id (PK)` and type `utf8` (:warning: breaking)
- Table `datadog_role_users`: primary key constraint added to column `account_name` (:warning: breaking)
- Table `datadog_role_users`: primary key constraint added to column `id` (:warning: breaking)
- Table `datadog_role_users`: primary key constraint removed from column `_cq_id` (:warning: breaking)
- Table `datadog_rum_events` was added
- Table `datadog_slo_corrections` was added
- Table `datadog_slos` was added
- Table `datadog_user_permissions`: column added with name `user_id (PK)` and type `utf8` (:warning: breaking)
- Table `datadog_user_permissions`: primary key constraint added to column `account_name` (:warning: breaking)
- Table `datadog_user_permissions`: primary key constraint added to column `id` (:warning: breaking)
- Table `datadog_user_permissions`: primary key constraint removed from column `_cq_id` (:warning: breaking)

### Features

* Add `datadog_rum_events` table ([#11229](#11229)) ([e2a07a2](e2a07a2)), closes [#11044](#11044)
* **datadog:** Add SLO tables, housekeeping ([#11166](#11166)) ([103a6bf](103a6bf))


### Bug Fixes

* **deps:** Update github.com/apache/arrow/go/v13 digest to e07e22c ([#11151](#11151)) ([5083cf7](5083cf7))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 20b0de9 ([#11199](#11199)) ([dc3565d](dc3565d))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 88d5dc2 ([#11226](#11226)) ([9f306bc](9f306bc))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to a7aad4c ([#11184](#11184)) ([8a0822e](8a0822e))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to c67fb39 ([#11169](#11169)) ([dcb0f92](dcb0f92))
* **deps:** Update golang.org/x/exp digest to 2e198f4 ([#11155](#11155)) ([c46c62b](c46c62b))
* **deps:** Update google.golang.org/genproto digest to e85fd2c ([#11156](#11156)) ([dbe7e92](dbe7e92))
* **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.0.9 ([#11240](#11240)) ([f92cd4b](f92cd4b))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.3 ([#11150](#11150)) ([dc00994](dc00994))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.4 ([#11244](#11244)) ([8fceef6](8fceef6))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.7.0 ([#11113](#11113)) ([487bf87](487bf87))
* **pagination:** Fix pagination for `datadog_incidents` table ([#11230](#11230)) ([b0eae04](b0eae04))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hermanschaaf hermanschaaf hermanschaaf approved these changes

@bbernays bbernays Awaiting requested review from bbernays

+1 more reviewer

@candiduslynx candiduslynx candiduslynx approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

automerge Automatically merge once required checks pass

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: datadog source plugin to include SLO data

4 participants

@disq @hermanschaaf @candiduslynx @cq-bot