Replace standalone processors with processor groups.

Before:

resource "datadog_observability_pipeline" "test" {
  name = "test pipeline"
  config {

    sources {
      kafka {
        id       = "source-1"
        group_id = "my-consumer-group"
        topics   = ["my-topic-1", "my-topic-2"]
      }
    }

    processors {
      parse_json {
        id      = "filter-1"
        include = "service:nginx"
        field   = "message2"
        inputs  = ["source-1"]
      }

      filter {
        id      = "filter-2"
        include = "service:nginx"
        inputs  = ["filter-1"]
      }
    }

    destinations {
      datadog_logs {
        id     = "sink-1"
        inputs = ["filter-3"]
      }
      
      amazon_s3 {
        id = "sink-2"
        inputs = ["filter-3]
        bucket = "my-bucket"
        region = "us-east-1"
      }
    }

  }
}

After:

resource "datadog_observability_pipeline" "test" {
  name = "test pipeline"
  config {

    source {
      id       = "source-1"
      kafka = {
        group_id = "my-consumer-group"
        topics   = ["my-topic-1", "my-topic-2"]
      }
    }

    processor_group {
      id = "group-1"
      inputs  = ["source-1"]
      include = "*"
      enabled = true
   
      processor {
        id      = "filter-1"
        include = "service:nginx"
        enabled = true
        parse_json = {
          field   = "message2"
        }
      }
      
      processor {
        id      = "filter-2"
        include = "service:nginx"
        enabled = true
        filter = {
        }
      }
    }

    destination {
       id     = "sink-1"
       inputs = ["filter-3"]
       datadog_logs = {
       }
     }
      
     destination {
       id = "sink-2"
       inputs = ["filter-3]
       amazon_s3 = {
         bucket = "my-bucket"
         region = "us-east-1"
      }
    }
  }
}

Important context:

• processors is replaced with the list of processor_group's, which have individual processor blocks inside. Both processor_group and processor blocks have id, enabled and include;
• processor blocks don't have inputs anymore, only processor groups;
• processor wrappers are required to maintain the order of processors(TF tracks the order only for the blocks of the same type). See TestAccDatadogObservabilityPipeline_multipleProcessorGroups as an example test with reordering.
• processor blocks(filter, parse_json etc.) must be ListNestedBlock with the size validation(<=1), because of the way TF handles optional blocks. If we use SingleNestedBlock(which makes more sense) for an optional block, which contains any required field, TF considers this block as required. This is a recommended workaround(see Disallow use of SingleNestedBlock in resource schemas hashicorp/terraform-provider-aws#35813 for example).

Replacing sources, destinations with source, destination

As in the example above, we have replaced single sources and destinations blocks with multiple source and destination blocks. This approach follows the TF convention for list blocks and lets us share common fields(id, inputs) at the component level.

Additional breaking changes combined together to avoid further breaking PRs

• all ListNestedBlock's are renamed from plural forms to singular ones: remaps -> remap, variables -> variable etc. to follow the TF convention for list blocks;
• all SingleNestedBlocks replaced with ListNestedBlocks to make the TF schema more robust, future-proof and eliminate potential breaking changes related to required/optional blocks and fields. See Disallow use of SingleNestedBlock in resource schemas hashicorp/terraform-provider-aws#35813 as an example of the same approach.

Suggested migration flow for existing resources

If you need to make immediate changes, you can safely downgrade your Datadog Terraform provider to 3.83.0 and make updates as before.

1. Detach the pipeline from the Terraform state (this does not delete the pipeline!):

terraform state rm datadog_observability_pipeline.test_pipeline

2. Open the pipeline in the UI and click Edit/Deploy (no actual changes required) - this will make the necessary config transformations on the API side. This step will make the pipeline incompatible with the previous Terraform provider versions.
3. Import the existing pipeline back into the Terraform state:

terraform import datadog_observability_pipeline.test_pipeline 4bcf83a6-dcdc-11f0-8624-da7ad0900002

4. Export the imported pipeline state into a HCL configuration:

terraform state show datadog_observability_pipeline.test_pipeline >> config.tf

5. Review the imported configuration and remove the old one.
6. Remove id field from the imported configuration.
7. Verify there is no diff for the pipeline:

terraform plan --target=datadog_observability_pipeline.test_pipeline