Skip to content

[BUG] [Serverless] failed to send traces to Datadog Agent at http://localhost:8126/v0.5/traces: HTTP error status 400, reason Bad Request #15900

Description

@pvicente

Agent Environment

Serverless dd_extension_version:35

datadog-lambda 4.67.0 The Datadog AWS Lambda Library
├── datadog >=0.41.0,<1.0.0
│   └── requests >=2.6.0 
│       ├── certifi >=2017.4.17 
│       ├── charset-normalizer >=2,<4 
│       ├── idna >=2.5,<4 
│       └── urllib3 >=1.21.1,<1.27 
├── ddtrace >=1.6.4,<2.0.0
│   ├── attrs >=20 
│   ├── bytecode * 
│   │   └── typing-extensions * 
│   ├── cattrs * 
│   │   ├── attrs >=20 (circular dependency aborted here)
│   │   └── exceptiongroup * 
│   ├── ddsketch >=2.0.1 
│   │   ├── protobuf >=3.0.0 
│   │   └── six * 
│   ├── envier * 
│   ├── jsonschema * 
│   │   ├── attrs >=17.4.0 (circular dependency aborted here)
│   │   └── pyrsistent >=0.14.0,<0.17.0 || >0.17.0,<0.17.1 || >0.17.1,<0.17.2 || >0.17.2 
│   ├── packaging >=17.1 
│   │   └── pyparsing >=2.0.2,<3.0.5 || >3.0.5 
│   ├── protobuf >=3 (circular dependency aborted here)
│   ├── six >=1.12.0 (circular dependency aborted here)
│   ├── tenacity >=5 
│   ├── typing-extensions * (circular dependency aborted here)
│   └── xmltodict >=0.12 
└── wrapt >=1.11.2,<2.0.0

with following env_vars:

    "DD_SITE" : "datadoghq.com"
    "DD_API_KEY_SECRET_ARN" : "xxx" 
    "DD_TRACE_ENABLED" : true
    "DD_APPSEC_ENABLED" : true
    "DD_FLUSH_TO_LOG" : true
    "DD_LOG_LEVEL" : "debug"
    "DD_TRACE_DEBUG" : true 
    "DD_PROFILING_ENABLED" : true
    "DD_PROFILING_IGNORE_PROFILER" : true
    "DD_PROFILING_ENABLE_CODE_PROVENANCE" : true
    "DD_CAPTURE_LAMBDA_PAYLOAD" : true
    "DD_ENV" : var.environment
    "DD_SERVICE" : var.service

Describe what happened:
Received logs but no APM traces with error log from dd

[ERROR]	2023-02-28T19:07:03.541Z	5bee0301-f72a-490a-8f3e-f33918628dbf	[dd.trace_id=0 dd.span_id=0]	failed to send traces to Datadog Agent at http://localhost:8126/v0.5/traces: HTTP error status 400, reason Bad Request

and before error log I've got another log:

[DEBUG]	2023-02-28T19:07:03.541Z	5bee0301-f72a-490a-8f3e-f33918628dbf	[dd.trace_id=0 dd.span_id=0]	sent 11.28KB in 0.00100s to http://localhost:8126/v0.5/traces

Describe what you expected:
See the log and APM trace related.

Steps to reproduce the issue:

Error happens with some API requests while "DD_CAPTURE_LAMBDA_PAYLOAD" : true, with same parameter set to false same API request doesn't show error. Is the problem a request size issue in the agent? I've seen how to set some limits in here

Additional environment details (Operating System, Cloud provider, etc):

AWS lambda running docker image with lambda:python 3.9:

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions