Skip to content

Sporadic lambda timeouts after migration to "datadog-lambda-extension" #191

Description

@hrist0stoichev

Hello! We were using the lambda forwarder for a long time and recently migrated to using the extension for sending logs to Datadog. I'm not 100% sure that this is due to the extension but we've been seeing sporadic timeouts for some of our lambda functions. I'll attach a few examples and try to explain what's wrong with them.

image Starting simple, here are the logs for 1 particular execution. Note the following things
  1. There are no logs whatsoever even though we expect to see some logs.
  2. The runtime duration is negative.
  3. The post runtime duration is huge. This is a function with 6s timeout.
  4. The timestamps indicate that only ~300ms have passed.
image
  1. Runtime duration is negative
  2. Post runtime duration is huge.
  3. A lot of "double logging"
  4. The logs prior to the START log indicate that this is a cold start but there is no Init Duration in the REPORT log.
  5. "Invalid trace id" logs
image Here actually the timeout is real and is indeed caused by the function being slow. The runtime duration and the post runtime durations look fine but note:
  1. The function actually timed out all 3 times (1 when trigger + 2 from the automatic async retry), however, there is only 1 log for the timeout. It looks like the other 2 executions were successful but they weren't (actually the third execution is what made me think "wait, why we haven't stopped after the second one?" and then I realized that all 3 have timed out).
  2. A very strange consequence of START -> REPORT -> START -> END -> REPORT -> START -> END
  3. Timestamps are wrong. First execution correctly shows ~15m but the next two are showing 2-3 minutes of execution when in fact the functions were timing out after 15 minutes (pretty sure about that since they were hitting an infinite loop)

All in all, as said before, I'm not 100% sure that it's all caused by the extension but there's definitely something wrong going on with the logs. The examples I provided are from different functions and different times. It feels like there's some issue in sending logs to Datadog which causes our functions to time out and drop logs. Do you think that's possible?

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions