Data Engineering Daily
Real-time aggregation of data engineering, data warehouse, lakehouse, and infrastructure news.
Conformed Dimensions vs Dependency Explosion
Alright everyone, lower your voices. Bring it in. Let’s talk about the thing no one in data engineering is talking about right now. JUST KIDDING! Not another slop post. For years I build independent data marts and the Kimball strategy was clear, asso...
Read Full Article →FREE review copy of my new book "Data Engineering for Beginners"
submitted by /u/qptbook [link] [comments]
Read Full Article →How do you ensure that the data is 100% clean apart from manual review?
Hi! So I am working on cleaning up our customer data quality to arrive at a customer masterdata. I tried to check for duplicates, nulls, invalid email formats and phone numbers, etc. I also tried to review with business some logic, like an inactive c...
Read Full Article →How bad is the lock in for Azure Fabric?
I am working with a client and they got sold hard on the Azure Fabric platform. I am there to assist them and I am trying to point them in a direction where they are not as locked in with one vendor. So for those who have made the move to Fabric, how...
Read Full Article →Building a lightweight Kafka monitoring tool for small teams — worth paying for it
Been running Kafka in production for a while now and honestly the monitoring situation for small teams sucks. Confluent Control Center is way overkill/expensive, Datadog's Kafka integration is priced like you're a 200-person company, and the open sou...
Read Full Article →Fable Ban Reversed + Dr. Dana Suskind on Parenting With A.I. + Prediction Market Drama
What we learned from the government’s biggest attempt yet to control who can gain access to the most powerful new A.I. models.
Read Full Article →Hardwood 1.0: A Fast, Lightweight Apache Parquet Reader for the JVM
submitted by /u/gunnarmorling [link] [comments]
Read Full Article →What are the issues one could face with using SparkListener
I am thinking of adding SparkListener to our Spark Cluster and then get metrics and query plan for every job. This will be loaded to a database in Background thread in the driver. Are there any issues doing the above? Like will Driver face any perfor...
Read Full Article →Microsoft Disclosure Provides Rare Glimpse of Tax Haven Tactics
Other U.S. companies will soon need to provide similar reports as a new European directive takes effect.
Read Full Article →Open-source Airbyte MCP server for self-hosted ops
I just released an open-source MCP server for the Airbyte Public API and wanted to share it here because the main use case is operational data engineering work What’s included: 36 tools across workspaces, sources, destinations, connections, jobs, str...
Read Full Article →Tesla Driver Using Autopilot in Texas Crash Is Charged With Manslaughter
A front-door video camera of the crash, which killed a woman inside her home, showed the Tesla plowing into a house through its driveway.
Read Full Article →Event sourcing with Kafka
Does anyone have any good ideas/data streams for building an event sourcing kafka project. Similar to New York Times Kafka model, using logs in the topic as the source of truth. submitted by /u/pandagotthedoginhim [link] [comments]
Read Full Article →To Reduce Electrical Grid Strain Amid Heat Wave, Data Centers Are Ordered to Use Backup Power
As triple-digit temperatures engulf much of the United States, the Trump administration wants grid managers to require the use of backup power that often goes unused.
Read Full Article →1 Month Job Search | 5-7 YOE
Hello! I just wanted to share my own stats following being impacted by recent layoffs. Interviewing can be intense so I'm happy to answer any questions about the experience and what did/didn't work for me. Some notes: Most applications were done via ...
Read Full Article →Tesla Sales Surge 25% on Recovery in Europe
Price cuts helped the electric automaker rebound in Europe in the second quarter, offsetting declines in the United States.
Read Full Article →Does MM2 actually support exactly once semantics?
I have been trying to get a clear answer on whether MM2 supports EOS for cross cluster replication. I found KIP-618(Exactly once support for source connectors), which was introduced in Kafka 3.3. Since MM2 is a source connector, it should theoretical...
Read Full Article →is anyone else using scala?
hello devs, Just a question: are companies still using Scala in their businesses, or are they using a mix of scala and other codes/AI? Is it hard to hire in this area? Just looking for opinions on this, Thanks submitted by /u/AlexaG_2026 [link] [comm...
Read Full Article →Is a ‘Frictionless’ Society a Trap?
Eliminating whatever stands between you and a task sounds nice, but it probably has more to offer companies than users.
Read Full Article →Monthly General Discussion - Jul 2026
This thread is a place where you can share things that might not warrant their own thread. It is automatically posted each month and you can find previous threads in the collection. Examples: What are you working on this month? What was something you...
Read Full Article →The Yoto Music Box Is a Ray of Hope Amid the ‘Techlash’
Amid widespread “techlash” over addictive screens and apps, the Yoto, an audio player for children, shows there’s still a way to make money while doing something nice.
Read Full Article →U.S. Lifts Restrictions on Anthropic’s Most Powerful A.I. Models
The move allows Anthropic to bring its most powerful technologies back online, de-escalating a feud with the Trump administration.
Read Full Article →Neon Buys ‘Artificial,’ a Film About OpenAI, After Amazon Dropped It
Neon purchased “Artificial,” which focuses on OpenAI’s chief, Sam Altman, after Amazon walked away from it following an investment in the start-up.
Read Full Article →do you debug local kafka consumer issues by grepping logs manually?
I am a swe working remotely and i have daily things to observe kafka jobs and check if data is flowing well so for that I was trying to go through logs and its messy like i wasn't ab;e to check which consumer is taking which of the messages is this t...
Read Full Article →Bending Spoons, Owner of AOL and Other Old Internet Brands, Is Going Public
Bending Spoons, an Italian company that buys aging internet companies, is going public this week at a potential value of $19 billion.
Read Full Article →Why does Kafka allow writes when ISR < min.insync.replicas (with acks=all)?
I’m currently learning Kafka, and while learning about ISR (In-Sync Replicas), acks, and min.insync.replicas, I tried to demonstrate the behavior in a local multi-broker setup. I observed something that doesn’t match my understanding, so I wanted to ...
Read Full Article →[Design Help] Efficient key-based lookup on a large Kafka topic for a background verification workflow
We are building a background workflow where for a given input, we need to find the corresponding message in Kafka and verify some fields on it. Our Kafka setup: - compacted topic, 24 partitions, ~200M messages per partition (~2.5B unique keys total) ...
Read Full Article →Interesting Kafka Links - June 2026
submitted by /u/rmoff [link] [comments]
Read Full Article →How do you handle robust ingestion in your orgs?
Our product needs so scan cloud assets (e.g. from aws account) and product insights after all assets has been saves to our storage. Currently we scan the account and send every result to Kafka that in turn being consumed by s3 sink writing messages t...
Read Full Article →📣 AI-generated content must be disclosed
A couple of weeks ago I started a RFC regarding posts on this sub that are AI-generated, or about AI-created tools. There was a range of views as to how far to go, but broad support for at least requiring the labelling of such content. So, this is no...
Read Full Article →What's with these iptv posts?
submitted by /u/myfootsmells [link] [comments]
Read Full Article →Residential vs. ISP Proxies: Which one do you ACTUALLY need? 🧐
submitted by /u/Mammoth-Dress-7368 [link] [comments]
Read Full Article →What actually makes you a STRONG data engineer (not just “good”)? Share your hacks & tips!
I’ve been thinking a lot about what separates a good data engineer from a strong one, and I want to hear your real hacks and tips. For me, it all comes down to how well you design, build, and maintain data pipelines. A pipeline isn’t just a script mo...
Read Full Article →Opinions on the area: Data Analytics & Big Data
I’ve started thinking about changing my professional career and doing a postgraduate degree in Data Analytics & Big Data. What do you think about this field? Is it something the market still looks for, or will the AI era make it obsolete? Do you ...
Read Full Article →14 Spark & Hive Videos Every Data Engineer Should Watch
Hello, I’ve put together a curated learning list of 14 short, practical YouTube videos focused on Apache Spark and Apache Hive performance, optimization, and real-world scenarios. These videos are especially useful if you are: Preparing for Spark / H...
Read Full Article →The Data Engineer Role is Being Asked to Do Way Too Much
I've been thinking about how companies are treating data engineers like they're some kind of tech wizards who can solve any problem thrown at them. Looking at the various definitions of what data engineers are supposedly responsible for, here's what ...
Read Full Article →Real-life Data Engineering vs Streaming Hype – What do you think? 🤔
I recently read a post where someone described the reality of Data Engineering like this: Streaming (Kafka, Spark Streaming) is cool, but it’s just a small part of daily work. Most of the time we’re doing “boring but necessary” stuff: Loading CSVs Pu...
Read Full Article →If You Put Kafka on Your Resume but Never Built a Real Streaming System, Read This
submitted by /u/Gold-Survey5264 [link] [comments]
Read Full Article →Charts: Plot 100 million datapoints using Wasm memory
submitted by /u/SciChartGuide [link] [comments]
Read Full Article →Reorienting my career to big data?
Hi everyone, I'm a 30y woman who has worked in scientific research at college for 9 years. I'm in the field of developmental psychology, but I've been in a lot of projects managing the data processing, treatment, cleaning, coding/programming in stati...
Read Full Article →📣 If you are employed by a vendor you must add a flair to your profile
As the r/apachekafka community grows and evolves beyond just Apache Kafka it's evident that we need to make sure that all community members can participate fairly and openly. We've always welcomed useful, on-topic, content from folk employed by vendo...
Read Full Article →