ETL Testing with Kafka and Apache -
Client Round Interview Q&A
1. What is ETL Testing?
ETL Testing ensures that the data extracted from source systems is transformed as
expected and loaded correctly into the target system (usually a data warehouse) without
data loss, corruption, or inconsistency.
2. What types of testing do you perform in ETL?
- Data validation
- Source to target count checks
- Data completeness
- Data transformation logic validation
- Duplicate check
- Data quality and data integrity
- Reconciliation testing
3. How do you validate data in a Kafka topic during ETL testing?
Use Kafka console consumers to read data from specific topics. Deserialize JSON/Avro data
if needed. Compare consumed Kafka records with the source or expected output. Validate
message order, partitioning, and timestamp metadata.
4. How do you handle schema evolution in Kafka topics during testing?
Use Schema Registry with tools like Confluent. Validate Avro schema versions. Ensure
backward or forward compatibility. Write test cases for schema validation to ensure
compatibility doesn’t break downstream systems.
5. How would you test real-time data flow from Kafka to a data warehouse or
Hadoop?
Produce test data into Kafka topic. Validate that the consumer (like Spark, Flink, or NiFi)
processes and transforms the data. Check intermediate storage (e.g., HDFS, S3) if used.
Validate row count, data format, and transformations in the target.
6. How do you validate ETL jobs that use Apache Spark?
Check Spark logs and execution DAGs for failed stages. Validate intermediate datasets using
Spark SQL. Compare input/output data using DataFrames. Test transformation logic using
test scripts or PySpark notebooks.
7. How do you test ETL pipelines using Apache NiFi?
Enable data provenance to trace record-level data flow. Inject sample flowfiles and validate
processor behavior. Use NiFi Expression Language to test dynamic attributes. Validate
output files, records, or Kafka sinks.
8. How do you perform reconciliation testing in Hive or HDFS?
Use Hive queries to compare record counts and column values with source. Use
checksum/hash-based comparisons for large datasets. Use Sqoop or Spark for automated
validation in pipelines.
9. Have you faced any data loss issues in Kafka ETL? How did you debug them?
Yes, it happened due to: consumers not committing offsets, partition rebalancing issues, and
network lag. Fixes included enabling offset monitoring, implementing retry logic, and
checking consumer lag with tools like Burrow.
10. How do you ensure data quality in a streaming ETL pipeline?
Implement real-time validations in Spark or Flink. Use CDC logs to catch anomalies. Set up
alerting on schema mismatches, nulls, and threshold breaches. Use Apache Druid or
Elasticsearch for anomaly detection dashboards.
11. How do you communicate defects or data mismatches to developers or
stakeholders?
Provide detailed logs, record IDs, and mismatch samples. Use defect tracking tools like JIRA.
Attach test case evidence, screenshots, and transformation logic. Suggest fixes or coordinate
with devs in stand-ups or reviews.
12. What are some best practices you follow in Kafka-based ETL Testing?
- Validate each stage: producer, topic, consumer
- Use idempotent consumers for repeatable tests
- Test with high and low volume to check performance and consistency
- Maintain a reusable test framework for Kafka data validations
