{"@attributes":{"version":"2.0"},"channel":{"title":"Datadog | Engineering blog","description":"Check out the Datadog Engineering blog to discover posts written by Datadog\u2019s very own engineers.","link":"https:\/\/www.datadoghq.com\/","lastBuildDate":"Sun, 19 Jul 2026 10:14:25 GMT","language":"en-us","copyright":"Datadog, Inc. All rights reserved.","item":[{"title":"How we measure data completeness at scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/data-pipeline-completeness\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/data-pipeline-completeness\/","description":"Learn how Datadog helps ensure data completeness at scale, enabling accurate alerts and safer automated decisions across distributed pipelines.","pubDate":"Wed, 01 Jul 2026 00:00:00 GMT"},{"title":"How we migrated a live routing system using AI-assisted refactoring","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/ai-assisted-storage-migration\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/ai-assisted-storage-migration\/","description":"Using AI-assisted refactoring, we migrated our live routing brain to a relational model, safely validating changes against live production traffic.","pubDate":"Tue, 23 Jun 2026 00:00:00 GMT"},{"title":"When failover isn\u2019t safe: Building high-availability PostgreSQL on Kubernetes","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/postgresql-ha-kubernetes\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/postgresql-ha-kubernetes\/","description":"During a reliability gameday, Datadog engineers discovered that their PostgreSQL clusters couldn\u2019t safely fail over. Here\u2019s how the team redesigned them for high availability using Patroni and synchronous replication.","pubDate":"Thu, 04 Jun 2026 00:00:00 GMT"},{"title":"From single pull requests to full software packages: Detecting malicious code at scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/scaling-malicious-code-detection\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/scaling-malicious-code-detection\/","description":"By combining stacked LLM evaluations with tool-driven investigation, we scaled malicious code detection from pull requests to dependency packages without sacrificing accuracy or cost control.","pubDate":"Tue, 02 Jun 2026 00:00:00 GMT"},{"title":"Steganography at scale: Embedding share URLs in Datadog widget screenshots","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/steganography-at-scale\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/steganography-at-scale\/","description":"Learn how we embed widget metadata into screenshots using invisible, resilient watermarks, enabling self-describing visualizations at scale.","pubDate":"Tue, 21 Apr 2026 00:00:00 GMT"},{"title":"How we built a real-world evaluation platform for autonomous SRE agents at scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/bits-ai-eval-platform\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/bits-ai-eval-platform\/","description":"Find out how we built a scalable evaluation platform for Datadog\u2019s Bits AI SRE agent that replays real incidents, detects regressions, and measures agent performance across production scenarios.","pubDate":"Tue, 07 Apr 2026 00:00:00 GMT"},{"title":"When upserts don\u2019t update but still write: Debugging Postgres performance at scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/debugging-postgres-performance\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/debugging-postgres-performance\/","description":"When a high-volume upsert doubled disk writes, Datadog engineers traced the issue to Postgres WAL behavior and rewrote the query to eliminate hidden costs.","pubDate":"Mon, 23 Mar 2026 00:00:00 GMT"},{"title":"When an AI agent came knocking: Catching malicious contributions in Datadog\u2019s open source repos","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/stopping-hackerbot-claw-with-bewaire\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/stopping-hackerbot-claw-with-bewaire\/","description":"Learn how Datadog detected and resolved issues from hackerbot-claw, an AI-powered automated attack campaign.","pubDate":"Mon, 09 Mar 2026 00:00:00 GMT"},{"title":"Designing MCP tools for agents: Lessons from building Datadog\u2019s MCP server","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/mcp-server-agent-tools\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/mcp-server-agent-tools\/","description":"We share lessons learned building Datadog\u2019s MCP server, from designing agent-friendly tools and managing context windows to using queries instead of raw data retrieval.","pubDate":"Wed, 04 Mar 2026 00:00:00 GMT"},{"title":"How we reduced the size of our Agent Go binaries by up to 77%","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/agent-go-binaries\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/agent-go-binaries\/","description":"Get an inside look at shrinking large Go binaries in the Datadog Agent through dependency and linker analysis.","pubDate":"Wed, 18 Feb 2026 00:00:00 GMT"},{"title":"Hardening eBPF for runtime security: Lessons from Datadog Workload Protection","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/ebpf-workload-protection-lessons\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/ebpf-workload-protection-lessons\/","description":"Learn field-tested lessons for eBPF-powered workload protection.","pubDate":"Wed, 07 Jan 2026 00:00:00 GMT"},{"title":"Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/workload-protection-ebpf-fim\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/workload-protection-ebpf-fim\/","description":"Learn how Datadog scaled eBPF-powered file monitoring to handle more than 10 billion kernel events per minute while preserving full detection coverage.","pubDate":"Tue, 18 Nov 2025 00:00:00 GMT"},{"title":"Replication redefined: How we built a low-latency, multi-tenant data replication platform","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/cdc-replication-search\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/cdc-replication-search\/","description":"Discover how Datadog engineered a scalable Change Data Capture (CDC) platform to replicate data across systems in near real time\u2014reducing search latency by 87%, increasing availability, and powering diverse, multi-tenant use cases across the company.","pubDate":"Tue, 04 Nov 2025 00:00:00 GMT"},{"title":"Detecting malicious pull requests at scale with LLMs","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/malicious-pull-requests\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/malicious-pull-requests\/","description":"Learn how Datadog\u2019s SDLC Security team built an LLM-powered system to detect malicious pull requests at scale\u2014without sacrificing developer velocity.","pubDate":"Tue, 21 Oct 2025 00:00:00 GMT"},{"title":"Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/rethinking-reliability\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/rethinking-reliability\/","description":"After a major outage, we re-architected Datadog systems to degrade gracefully under failure. Here\u2019s what we learned\u2014and how we\u2019re building forward.","pubDate":"Wed, 15 Oct 2025 00:00:00 GMT"},{"title":"Inside Husky\u2019s query engine: Real-time access to 100 trillion events","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/husky-query-architecture\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/husky-query-architecture\/","description":"See how Husky enables interactive querying across 100 trillion events daily by combining caching, smart indexing, and query pruning.","pubDate":"Wed, 01 Oct 2025 00:00:00 GMT"},{"title":"From hand-tuned Go to self-optimizing code: Building BitsEvolve","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/self-optimizing-system\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/self-optimizing-system\/","description":"Learn how we turned hot-path optimizations into a system for continuous, AI-assisted performance improvements and saved thousands of cores in the process.","pubDate":"Thu, 18 Sep 2025 00:00:00 GMT"},{"title":"Scaling down to speed up: How we improved efficiency of live process metrics by 100x","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/scaling-process-pipeline-efficiency\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/scaling-process-pipeline-efficiency\/","description":"We re-architected the real-time data pipeline for Datadog\u2019s Processes and Containers views\u2014cutting traffic by 100x and infrastructure use by 98%. This post explores the system challenges, architectural changes, and their impact.","pubDate":"Tue, 12 Aug 2025 00:00:00 GMT"},{"title":"Evolving our real-time timeseries storage again: Built in Rust for performance at scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/rust-timeseries-engine\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/rust-timeseries-engine\/","description":"Discover how we reengineered our metrics storage engine for massive scale with Rust, a shard-per-core model, and real-time performance.","pubDate":"Mon, 04 Aug 2025 00:00:00 GMT"},{"title":"How we tracked down a Go 1.24 memory regression across hundreds of pods","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/go-memory-regression\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/go-memory-regression\/","description":"We rolled out Go 1.24 and saw a memory regression. Here\u2019s how we dug into system metrics, uncovered a bug in the runtime allocator, and worked with the Go team to help fix it.","pubDate":"Thu, 17 Jul 2025 00:00:00 GMT"},{"title":"How Go 1.24\u2019s Swiss Tables saved us hundreds of gigabytes","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/go-swiss-tables\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/go-swiss-tables\/","description":"Go 1.24\u2019s Swiss Tables cut our map memory usage by up to 70% in high-traffic workloads. Here\u2019s how we profiled the savings and improved performance.","pubDate":"Thu, 17 Jul 2025 00:00:00 GMT"},{"title":"How we built a real-time, client-side noise suppression library without server dependencies","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/noise-suppression-library\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/noise-suppression-library\/","description":"Learn how we implemented and open-sourced a noise filter for real-time audio chat without compromising performance. Better yet, try the demo and add it to your own project today.","pubDate":"Tue, 24 Jun 2025 00:00:00 GMT"},{"title":"How we built reliable log delivery to thousands of unpredictable endpoints","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/reliable-log-delivery\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/reliable-log-delivery\/","description":"Learn how we built Datadog\u2019s Log Forwarding system for low-latency, high-throughput delivery to thousands of unreliable third-party endpoints.","pubDate":"Mon, 23 Jun 2025 00:00:00 GMT"},{"title":"How we scaled fast, reliable configuration distribution to thousands of workload containers","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/scaling-config-delivery-containers\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/scaling-config-delivery-containers\/","description":"Learn how Datadog engineered a highly reliable, low-latency system to distribute per-tenant configuration data across thousands of containers, enabling real-time log processing at scale.","pubDate":"Tue, 17 Jun 2025 00:00:00 GMT"},{"title":"Breaking up a monolith: How we\u2019re unwinding a shared database at scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/unwinding-shared-database\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/unwinding-shared-database\/","description":"Learn how Datadog is breaking up a shared production database at scale\u2014defining clear ownership boundaries, minimizing migration risk, and building the tooling to make decoupling safe, automated, and sustainable.","pubDate":"Tue, 17 Jun 2025 00:00:00 GMT"},{"title":"Detecting faulty deployments: Our journey from unlabeled data to supervised learning","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/detecting-faulty-deployments\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/detecting-faulty-deployments\/","description":"Learn how we developed Datadog Automatic Faulty Deployment Detection and improved precision, recall, and time to detection along the way.","pubDate":"Tue, 03 Jun 2025 00:00:00 GMT"},{"title":"Squeezing every millisecond: How we rebuilt the Datadog Lambda Extension in Rust","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/datadog-lambda-extension-rust\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/datadog-lambda-extension-rust\/","description":"Learn how we rewrote the Datadog Lambda Extension in Rust, cutting cold starts by 82 percent, shrinking the binary 87 percent, and slashing memory use without sacrificing observability.","pubDate":"Wed, 09 Apr 2025 00:00:00 GMT"},{"title":"Achieving relentless Kafka reliability at scale with the Streaming Platform","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/streaming-platform-kafka-custom-abstractions\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/streaming-platform-kafka-custom-abstractions\/","description":"Learn how we built the Streaming Platform at Datadog to provide more resilience and flexibility to our Kafka infrastructure.","pubDate":"Wed, 19 Feb 2025 00:00:00 GMT"},{"title":"Husky: Efficient compaction at Datadog scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/husky-storage-compaction\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/husky-storage-compaction\/","description":"A deep dive into Husky\u2019s underlying data storage and compaction system.","pubDate":"Wed, 29 Jan 2025 00:00:00 GMT"},{"title":"Unraveling a Postgres segfault that uncovered an Arm64 JIT compiler bug","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/unraveling-a-postgres-segfault\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/unraveling-a-postgres-segfault\/","description":"Learn how an unassuming Postgres error led us to discover a bug in Postgres for Arm.","pubDate":"Wed, 15 Jan 2025 00:00:00 GMT"},{"title":"Effective habits of remote workers","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/effective-habits-remote-workers\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/effective-habits-remote-workers\/","description":"Learn some tips and strategies to stay connected, visible, and effective as a remote worker in a predominantly office-based company.","pubDate":"Mon, 23 Dec 2024 00:00:00 GMT"},{"title":"How we use formal modeling, lightweight simulations, and chaos testing to design reliable distributed systems","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/formal-modeling-and-simulation\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/formal-modeling-and-simulation\/","description":"Learn how we used formal modeling and simulation to analyze a distributed, multi-tenant queueing system.","pubDate":"Wed, 20 Nov 2024 00:00:00 GMT"},{"title":"How we built a Ruby library that saves 50% in testing time","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/ruby-test-impact-analysis\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/ruby-test-impact-analysis\/","description":"Learn how we built a test impact analysis library in Ruby, the challenges we faced, the solutions we found, and what we discovered about tracing the Ruby VM.","pubDate":"Fri, 01 Nov 2024 00:00:00 GMT"},{"title":"How we optimized LLM use for cost, quality, and safety to facilitate writing postmortems","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/llms-for-postmortems\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/llms-for-postmortems\/","description":"Learn about the challenges and solutions we discovered while using LLMs to automate writing postmortems.","pubDate":"Mon, 23 Sep 2024 00:00:00 GMT"},{"title":"Timeseries indexing at scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/timeseries-indexing-at-scale\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/timeseries-indexing-at-scale\/","description":"Learn how we implemented a new timeseries indexing strategy when the amount of data we ingested increased significantly.","pubDate":"Fri, 28 Jun 2024 00:00:00 GMT"},{"title":"How we migrated our static analyzer from Java to Rust","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-migrated-our-static-analyzer-from-java-to-rust\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-migrated-our-static-analyzer-from-java-to-rust\/","description":"Learn how we enhanced our static analyzer by migrating from Java to Rust, tripling performance improvements and a 10x reduction in memory usage.","pubDate":"Thu, 23 May 2024 00:00:00 GMT"},{"title":"Engineering VP spotlight: Ivo Dimitrov","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-vp-spotlight-ivo-dimitrov\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-vp-spotlight-ivo-dimitrov\/","description":"In this interview, Ivo Dimitrov, Distributed Data Systems VP of Engineering, describes the engineering career that led him to Datadog and his committment to helping build out our core backend platforms.","pubDate":"Tue, 21 May 2024 00:00:00 GMT"},{"title":".NET Continuous Profiler: Memory usage","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/dotnet-continuous-profiler-part-4\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/dotnet-continuous-profiler-part-4\/","description":"How we handle memory usage in our .NET continuous profiler.","pubDate":"Mon, 20 May 2024 00:00:00 GMT"},{"title":"How we built the Datadog heatmap to visualize distributions over time at arbitrary scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-built-the-datadog-heatmap-to-visualize-distributions-over-time-at-arbitrary-scale\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-built-the-datadog-heatmap-to-visualize-distributions-over-time-at-arbitrary-scale\/","description":"Learn how we used DDSketch to enhance our heatmap visualizations, allowing us to represent and analyze high cardinality data distributions at scale.","pubDate":"Wed, 01 May 2024 00:00:00 GMT"},{"title":"How we brought Datadog\u2019s data visualization to iOS: A focus on performance","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-brought-datadog-data-visualization-to-ios-performance\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-brought-datadog-data-visualization-to-ios-performance\/","description":"Learn how we built an internal graphing library to support data visualization in iOS using Swift and SwiftUI.","pubDate":"Thu, 18 Apr 2024 00:00:00 GMT"},{"title":".NET Continuous Profiler: Exception and lock contention","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/dotnet-continuous-profiler-part-3\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/dotnet-continuous-profiler-part-3\/","description":"How we handle exceptions and lock contention in our .NET continuous profiler.","pubDate":"Thu, 04 Apr 2024 00:00:00 GMT"},{"title":"Engineering spotlight: Marie-Laure Bardonnet","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-marie-laure-bardonnet-part-2\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-marie-laure-bardonnet-part-2\/","description":"In this interview, Marie-Laure Bardonnet, Log Management Senior Engineering Manager, describes the journey of learning, growing, and scaling a team of 4 backend engineers to over 30 frontend and backend engineers at Datadog.","pubDate":"Tue, 26 Mar 2024 00:00:00 GMT"},{"title":"How we use Vale to improve our documentation editing process","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-use-vale-to-improve-our-documentation-editing-process\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-use-vale-to-improve-our-documentation-editing-process\/","description":"Learn how Datadog\u2019s Documentation team uses a linter to shift quality left.","pubDate":"Wed, 14 Feb 2024 00:00:00 GMT"},{"title":".NET Continuous Profiler: CPU and wall time profiling","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/dotnet-continuous-profiler-part-2\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/dotnet-continuous-profiler-part-2\/","description":"How we implemented CPU and wall time profiling in our .NET continuous profiler.","pubDate":"Tue, 13 Feb 2024 00:00:00 GMT"},{"title":".NET Continuous Profiler: Under the hood","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/dotnet-continuous-profiler\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/dotnet-continuous-profiler\/","description":"Our .NET profiler was designed and implemented to run 24\/7 in production, at any scale, with negligible impact. Here are the details of how we built it.","pubDate":"Tue, 09 Jan 2024 00:00:00 GMT"},{"title":"Scaling self-serve analytics: The tools empowering 5,000 employees","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/crunchconf-talk-self-serve-analytics\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/crunchconf-talk-self-serve-analytics\/","description":"In this video, Jean-Mathieu Saponaro, Data & Analytics Senior Engineering Manager, describes the journey of leading, growing, and scaling self-serve analytics within Datadog.","pubDate":"Thu, 21 Dec 2023 00:00:00 GMT"},{"title":"Engineering spotlight: Jeromy Carriere","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-jeromy-carriere\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-jeromy-carriere\/","description":"Engineering spotlight: Jeromy Carriere","pubDate":"Fri, 27 Oct 2023 00:00:00 GMT"},{"title":"How we migrated our acceptance tests to use Synthetic Monitoring","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/migrating-acceptance-tests-to-synthetic-monitoring\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/migrating-acceptance-tests-to-synthetic-monitoring\/","description":"How Datadog\u2019s Frontend DevX team migrated a codebase from flaky, hard-to-maintain acceptance testing with Puppeteer to more robust Synthetic tests.","pubDate":"Fri, 30 Jun 2023 00:00:00 GMT"},{"title":"2023-03-08 incident: A deep dive into the platform-level recovery","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/2023-03-08-deep-dive-into-platform-level-recovery\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/2023-03-08-deep-dive-into-platform-level-recovery\/","description":"This post walks through how we restored our platform after it was affected by the outage of March 8, 2023.","pubDate":"Fri, 16 Jun 2023 00:00:00 GMT"},{"title":"2023-03-08 incident: A deep dive into our incident response","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/2023-03-08-deep-dive-into-incident-response\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/2023-03-08-deep-dive-into-incident-response\/","description":"This post sketches out our incident response process, where it succeeded and where it stumbled on March 8, and what we learned along the way.","pubDate":"Thu, 01 Jun 2023 00:00:00 GMT"},{"title":"Not just another network latency issue: How we unraveled a series of hidden bottlenecks","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/not-just-another-network-latency-issue\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/not-just-another-network-latency-issue\/","description":"Learn how we tackled a case of high network-latency in our usage estimation platform that required a multi-layered solution.","pubDate":"Fri, 26 May 2023 00:00:00 GMT"},{"title":"2023-03-08 incident: A deep dive into the platform-level impact","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/2023-03-08-deep-dive-into-platform-level-impact\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/2023-03-08-deep-dive-into-platform-level-impact\/","description":"A deep dive into what happened at the platform level during the outage of March 8, 2023.","pubDate":"Wed, 24 May 2023 00:00:00 GMT"},{"title":"Making fetch happen: Building a general-purpose query and render scheduler","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/making-fetch-happen-building-a-general-purpose-query-and-render-scheduler\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/making-fetch-happen-building-a-general-purpose-query-and-render-scheduler\/","description":"Learn how we developed a new scheduling algorithm for data fetching and rendering and how we built it for use across our suite of Datadog products.","pubDate":"Mon, 17 Apr 2023 00:00:00 GMT"},{"title":"Husky: Exactly-once ingestion and multi-tenancy at scale","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/husky-deep-dive\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/husky-deep-dive\/","description":"A closer look at storage routing in Husky, Datadog\u2019s third-generation event storage system.","pubDate":"Wed, 22 Feb 2023 00:00:00 GMT"},{"title":"Performance improvements in the Datadog Agent metrics pipeline","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/performance-improvements-in-the-datadog-agent-metrics-pipeline\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/performance-improvements-in-the-datadog-agent-metrics-pipeline\/","description":"We\u2019ve recently improved the raw performance of the Datadog Agent, leading to 20% less CPU use on Agents flooded with custom metrics.","pubDate":"Tue, 31 Jan 2023 00:00:00 GMT"},{"title":"DRUIDS, the design system that powers Datadog","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/druids-the-design-system-that-powers-datadog\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/druids-the-design-system-that-powers-datadog\/","description":"Learn about Datadog\u2019s repeatable design elements that we\u2019ve documented in our design style guide called DRUIDS.","pubDate":"Thu, 29 Sep 2022 00:00:00 GMT"},{"title":"Engineering Spotlight: Tay Nishimura","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-tay-nishimura\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-tay-nishimura\/","description":"Engineering spotlight: Tay Nishimura","pubDate":"Mon, 06 Jun 2022 00:00:00 GMT"},{"title":"Introducing Husky, Datadog\u2019s third-generation event store","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/introducing-husky\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/introducing-husky\/","description":"Husky is an unbundled, distributed, schemaless, vectorized column store. Here\u2019s how we built it\u2014and why.","pubDate":"Tue, 17 May 2022 00:00:00 GMT"},{"title":"How Datadog\u2019s IT team automated account inactivity and SaaS spend management","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-datadogs-it-team-automated-account-inactivity-and-saas-spend-management\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-datadogs-it-team-automated-account-inactivity-and-saas-spend-management\/","description":"Employees at all modern software companies use a ton of outside pieces of software to do their jobs. Learn how Datadog\u2019s IT team expanded Clarity to automate monitoring these accounts for inactivity and optimizing how much we spend on them.","pubDate":"Thu, 12 May 2022 00:00:00 GMT"},{"title":"It\u2019s always DNS . . . except when it\u2019s not: A deep dive through gRPC, Kubernetes, and AWS networking","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/grpc-dns-and-load-balancing-incident\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/grpc-dns-and-load-balancing-incident\/","description":"The story of a seemingly simple issue that led us into the hidden complexities of gRPC, DNS, and Kubernetes.","pubDate":"Wed, 13 Apr 2022 00:00:00 GMT"},{"title":"Using the Dirty Pipe vulnerability to break out from containers","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/dirty-pipe-container-escape-poc\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/dirty-pipe-container-escape-poc\/","description":"See Datadog\u2019s proof of concept exploit for breaking out from unprivileged containers using the Dirty Pipe vulnerability.","pubDate":"Fri, 25 Mar 2022 00:00:00 GMT"},{"title":"Profiling improvements in Go 1.18","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/profiling-improvements-in-go-1-18\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/profiling-improvements-in-go-1-18\/","description":"How several patches and fixes in Go 1.18 bring improved profiling accuracy.","pubDate":"Mon, 28 Feb 2022 00:00:00 GMT"},{"title":"How Datadog uses Datadog to gain visibility into the Datadog user experience","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-datadog-uses-datadog-to-gain-visibility-into-the-datadog-user-experience\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-datadog-uses-datadog-to-gain-visibility-into-the-datadog-user-experience\/","description":"How the Datadog DesignOps team uses Datadog to understand our users and make well-informed design decisions","pubDate":"Tue, 22 Feb 2022 00:00:00 GMT"},{"title":"Our journey taking Kubernetes state metrics to the next level","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/our-journey-taking-kubernetes-state-metrics-to-the-next-level\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/our-journey-taking-kubernetes-state-metrics-to-the-next-level\/","description":"Our story of contributing to kube-state-metrics, a popular open source Kubernetes service.","pubDate":"Fri, 15 Oct 2021 00:00:00 GMT"},{"title":"How we optimized our Akka application using Datadog\u2019s Continuous Profiler","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-optimized-our-akka-application-using-datadogs-continuous-profiler\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-optimized-our-akka-application-using-datadogs-continuous-profiler\/","description":"We identified a performance issue caused by the `ForkJoinPool` in our Java application based on the Akka framework. This is how we solved our issue.","pubDate":"Thu, 30 Sep 2021 00:00:00 GMT"},{"title":"How Datadog\u2019s IT team automated monitoring third-party accounts","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-datadogs-it-team-automated-monitoring-third-party-accounts\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-datadogs-it-team-automated-monitoring-third-party-accounts\/","description":"Employees at all modern software companies use a ton of outside pieces of software to do their jobs. Learn how Datadog\u2019s IT team built a tool to automate monitoring these accounts for security and compliance.","pubDate":"Fri, 30 Apr 2021 00:00:00 GMT"},{"title":"How we minimized the overhead of Kubernetes in our job system","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/moving-a-jobsystem-to-kubernetes\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/moving-a-jobsystem-to-kubernetes\/","description":"Solving performance problems when moving an application to Kubernetes","pubDate":"Mon, 22 Feb 2021 00:00:00 GMT"},{"title":"Engineering spotlight: Ma\u00ebl Nison","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-mael-nison\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-mael-nison\/","description":"Engineering spotlight: Ma\u00ebl Nison, maintainer of Yarn","pubDate":"Wed, 10 Feb 2021 00:00:00 GMT"},{"title":"PHP 8: Observability baked right in","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/php-8-observability-baked-right-in\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/php-8-observability-baked-right-in\/","description":"What the observer API means for PHP 8 and the future of observability","pubDate":"Tue, 02 Feb 2021 00:00:00 GMT"},{"title":"Introducing Glommio, a thread-per-core crate for Rust and Linux","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/introducing-glommio\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/introducing-glommio\/","description":"Glommio (pronounced glom-io or |glomj\u0259\u028a|) is a cooperative thread-per-core crate for Rust & Linux based on io_uring. It allows you to write asynchronous code that takes advantage of rust async\/await, but it doesn\u2019t use helper threads anywhere.","pubDate":"Mon, 02 Nov 2020 00:00:00 GMT"},{"title":"How we wrote a Python profiler","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-wrote-a-python-profiler\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/how-we-wrote-a-python-profiler\/","description":"Adventures in developing a Python profiler","pubDate":"Wed, 07 Oct 2020 00:00:00 GMT"},{"title":"The Old Datadog and the Sea","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/using-datadog-to-monitor-my-sailboat\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/using-datadog-to-monitor-my-sailboat\/","description":"How I used Datadog to become a better sailor.","pubDate":"Wed, 23 Sep 2020 00:00:00 GMT"},{"title":"Computing accurate percentiles with DDSketch","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/computing-accurate-percentiles-with-ddsketch\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/computing-accurate-percentiles-with-ddsketch\/","description":"Introducing DDSketch, the first fully-mergeable, relative-error quantile sketching algorithm with formal guarantees.","pubDate":"Mon, 23 Sep 2019 00:00:00 GMT"},{"title":"Secure publication of Datadog Agent integrations with TUF and in-toto","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/secure-publication-of-datadog-agent-integrations-with-tuf-and-in-toto\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/secure-publication-of-datadog-agent-integrations-with-tuf-and-in-toto\/","description":"How to guarantee end-to-end security when using automation to package and publish Datadog Agent integrations","pubDate":"Mon, 03 Jun 2019 00:00:00 GMT"},{"title":"Building highly reliable data pipelines at Datadog","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/highly-reliable-data-pipelines\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/highly-reliable-data-pipelines\/","description":"A look at how Datadog builds and operates data pipelines reliably at scale.","pubDate":"Tue, 02 Apr 2019 00:00:00 GMT"},{"title":"Rethinking UX for AI-driven alerting","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/rethinking-ux-for-ai-driven-alerting\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/rethinking-ux-for-ai-driven-alerting\/","description":"The introduction of advanced statistical methods is reshaping the UX of alerts","pubDate":"Tue, 22 Jan 2019 00:00:00 GMT"},{"title":"Improving trust with Datadog Log Management","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/improving-trust-with-datadog-log-management\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/improving-trust-with-datadog-log-management\/","description":"Integrating Amazon Simple Email Service with Datadog to improve observability.","pubDate":"Thu, 01 Nov 2018 00:00:00 GMT"},{"title":"Introducing Kafka-Kit: Tools for scaling Kafka","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/introducing-kafka-kit-tools-for-scaling-kafka\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/introducing-kafka-kit-tools-for-scaling-kafka\/","description":"Today, we\u2019re open-sourcing Kafka-Kit, a toolset for scaling and recovering Kafka.","pubDate":"Mon, 13 Aug 2018 00:00:00 GMT"},{"title":"Using Datadog APM to improve the performance of Homebrew","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/using-datadog-apm-to-find-bottlenecks-and-performance-benchmarking\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/using-datadog-apm-to-find-bottlenecks-and-performance-benchmarking\/","description":"Using Datadog to find performance bottlenecks, and contrasting tentative solutions using performance benchmarks.","pubDate":"Tue, 24 Apr 2018 00:00:00 GMT"},{"title":"Cgo and Python","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/cgo-and-python\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/cgo-and-python\/","description":"How the new Datadog Agent written in Go runs Python checks.","pubDate":"Mon, 16 Apr 2018 00:00:00 GMT"},{"title":"What product designers can learn from explanatory journalism","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/what-product-designers-can-learn-from-explanatory-journalism\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/what-product-designers-can-learn-from-explanatory-journalism\/","description":"How to be a better designer by being a better explainer.","pubDate":"Fri, 23 Mar 2018 00:00:00 GMT"},{"title":"Secure (and usable) multi-AWS account IAM setup","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/secure-aws-account-iam-setup\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/secure-aws-account-iam-setup\/","description":"If you are part of the team managing the AWS infrastructure at your organization, you\u2019ve likely had to wrestle with the complexity of managing multiple accounts for some time now.","pubDate":"Wed, 20 Sep 2017 00:00:00 GMT"},{"title":"Robust statistical distances for machine learning","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/robust-statistical-distances-for-machine-learning\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/robust-statistical-distances-for-machine-learning\/","description":"Designing powerful outlier and anomaly detection algorithms requires using the right tools. Discover how robust statistical distances can help.","pubDate":"Wed, 06 Sep 2017 00:00:00 GMT"},{"title":"Scaling support with Vagrant and Terraform","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/scaling-support-with-vagrant-terraform\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/scaling-support-with-vagrant-terraform\/","description":"The Datadog Solutions Team reproduces problems that customers run into while they try using our many integrations in their own, always-unique environments.","pubDate":"Wed, 23 Aug 2017 00:00:00 GMT"},{"title":"Improving cloud security visibility with ChatOps","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/improving-cloud-security-with-chatops\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/improving-cloud-security-with-chatops\/","description":"Highlights of our recent work to improve our cloud-based monitoring and alerting pipeline.","pubDate":"Tue, 15 Aug 2017 00:00:00 GMT"},{"title":"Piecewise regression: When one line simply isn\u2019t enough","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/piecewise-regression\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/piecewise-regression\/","description":"A piecewise regression can model multiple trends in a single data set. Learn how Datadog automates piecewise regression on our timeseries data.","pubDate":"Tue, 11 Jul 2017 00:00:00 GMT"},{"title":"Hackathon project: Viewing Datadog metrics in Minecraft","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/hackathon-project-viewing-datadog-metrics-in-minecraft\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/hackathon-project-viewing-datadog-metrics-in-minecraft\/","description":"At Datadog we see and gather metrics everywhere by using Datadog to monitor our applications and infrastructure. So our team thought it\u2019d be fun to come up with creative solutions to \u201cwhere can we display metrics?\u201d","pubDate":"Thu, 15 Jun 2017 00:00:00 GMT"},{"title":"Protobuf parsing in Python","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/protobuf-parsing-in-python\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/protobuf-parsing-in-python\/","description":"Recently we extended the Datadog Agent to support extracting additional metrics from Kubernetes using the kube-state-metrics via protobufs.","pubDate":"Thu, 08 Jun 2017 00:00:00 GMT"},{"title":"Being a solutions engineer at Datadog","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/being-a-solutions-engineer-at-datadog\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/being-a-solutions-engineer-at-datadog\/","description":"Solutions Engineers at Datadog have to stay on top of what\u2019s going on within the company and outside.","pubDate":"Wed, 17 May 2017 00:00:00 GMT"},{"title":"The trouble with mounting","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/the-trouble-with-mounting\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/the-trouble-with-mounting\/","description":"When some of our customers reported that their agents were freezing, sometimes for hours at a time, we tracked down the issue to their disk mount options.","pubDate":"Mon, 23 Jan 2017 00:00:00 GMT"},{"title":"Engineering spotlight: Marie-Laure Bardonnet","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-marie-laure-bardonnet\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/engineering-spotlight-marie-laure-bardonnet\/","description":"It might surprise you to learn who built most of the prototype of the newest Datadog feature. Read more about Marie-Laure and her internship at Datadog.","pubDate":"Tue, 17 Jan 2017 00:00:00 GMT"},{"title":"Redux-Doghouse: Creating reusable React-Redux components through scoping","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/redux-doghouse-reusable-react-redux-components-through-scoping\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/redux-doghouse-reusable-react-redux-components-through-scoping\/","description":"Today, we\u2019re open-sourcing Redux-Doghouse, a library for Redux that helps you scope components so that they can be reused multiple times in multiple contexts without conflicting with one another.","pubDate":"Mon, 14 Nov 2016 00:00:00 GMT"},{"title":"Cheering on coworkers: Building culture with Datadog dashboards","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/cheering-on-coworkers-building-culture-with-datadog-dashboards\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/cheering-on-coworkers-building-culture-with-datadog-dashboards\/","description":"One of our colleagues, Christian, is participating in a tremendous 6-day-run challenge. Yes, you read that right, he will run around 850km (528 miles) over 6 days. As we like to graph everything, we thought it would be fun to cheer him on remotely and follow his progress in this crazy race via a Datadog dashboard.","pubDate":"Thu, 27 Oct 2016 00:00:00 GMT"},{"title":"Restroom hacks","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/restroom-hacks\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/restroom-hacks\/","description":"Do you ever walk to the bathroom across the office only to discover that it\u2019s in use? Then you\u2019ve got to decide if you want to awkwardly hover right outside, or hold it in for a while and try again later. This is obviously a first world problem, but bathroom contention was getting to be a challenge as we quickly outgrew our office space.","pubDate":"Tue, 23 Aug 2016 00:00:00 GMT"},{"title":"Consul at Datadog","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/consul-at-datadog\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/consul-at-datadog\/","description":"We\u2019ve been using Consul for about 18 months at Datadog and it\u2019s an important part of our production stack. In this post we will discuss some of the lessons we have learned.","pubDate":"Thu, 11 Aug 2016 00:00:00 GMT"},{"title":"Releasing czlib and zstd Go bindings","link":"https:\/\/www.datadoghq.com\/blog\/engineering\/releasing-czlib-and-zstd-go-bindings\/","guid":"https:\/\/www.datadoghq.com\/blog\/engineering\/releasing-czlib-and-zstd-go-bindings\/","description":"To commemorate the third annual GopherCon US in Denver this week, we\u2019re releasing cgo bindings to two compression libraries that we\u2019ve been using in production at Datadog for a while now: czlib and zstd.","pubDate":"Mon, 11 Jul 2016 00:00:00 GMT"}]}}