{"id":109977,"date":"2021-05-11T07:00:00","date_gmt":"2021-05-11T04:00:00","guid":{"rendered":"https:\/\/www.javacodegeeks.com\/?p=109977"},"modified":"2021-05-05T09:59:57","modified_gmt":"2021-05-05T06:59:57","slug":"kafka-record-patterns-for-data-replication","status":"publish","type":"post","link":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html","title":{"rendered":"Kafka Record Patterns for Data Replication"},"content":{"rendered":"

Imagine going down to your local milkshake bar and signing a contract with the owner so that you could purchase bespoke drinks at a set price. Let’s say you agreed on fresh milk with 3.5% fat and one tablespoon of chocolate powder, per 500ml of milk.  Putting that into a table might look like this:<\/p>\n

\n\n\n\n\n\n
PK<\/th>\ncontract_number<\/th>\nstart<\/th>\nfat_content<\/th>\nchocolate_powder<\/th>\n<\/tr>\n<\/thead>\n
100<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n3.5%<\/td>\n1 tbsp<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

After a few weeks, your tastebuds become a little desensitised and you decide you want to add some more chocolate powder. The owner is agile, so he adjusts the contract, meaning we need to add a few columns in order to track validity:<\/p>\n

\n\n\n\n\n\n\n
PK<\/th>\ncontract_number<\/th>\ncontract_from<\/th>\nstart<\/th>\nend<\/th>\nfat_content<\/th>\nchocolate_powder<\/th>\n<\/tr>\n<\/thead>\n
100<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n0001-01-01<\/td>\n2021-01-31<\/td>\n3.5%<\/td>\n1 tbsp<\/td>\n<\/tr>\n
101<\/strong><\/td>\n12345678<\/strong><\/td>\n2021-01-01<\/strong><\/td>\n2021-02-01<\/strong><\/td>\n9999-12-31<\/strong><\/td>\n3.5%<\/strong><\/td>\n2 tbsp<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

Note two things: 1) this table is not normalised and 2) I used a low date (year 0001) and high date (year 9999) for the start of the first row and the end of the last row.<\/p>\n

In reality we would probably normalise this data. For the sake of this example, I won’t because it will make it more readable as I add more information below.<\/p>\n

The low and high dates are there, so that I can always find data, regardless of the date I use – I don’t have to know the contract termination date which is different for every contract, in order to be able to simply ask what the latest recipe is, for a given contract number:<\/p>\n

\n
\n\n\n\n
\n
1<\/div>\n
2<\/div>\n
3<\/div>\n
4<\/div>\n
5<\/div>\n<\/td>\n
\n
\n
select *<\/code><\/div>\n
from contracts<\/code><\/div>\n
where contract_number = <\/code>'12345678'<\/code><\/div>\n
  <\/code>and <\/code>'9999-12-31'<\/code> between start and end;<\/code><\/div>\n
--> returns row with primary key <\/code>101<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

After a few more weeks, I realise that I need to reduce my calorific intake, but I’m a complete chocoholic. We agree to reduce the fat content:<\/p>\n

\n\n\n\n\n\n\n\n
PK<\/th>\ncontract_number<\/th>\ncontract_from<\/th>\nstart<\/th>\nend<\/th>\nfat_content<\/th>\nchocolate_powder<\/th>\n<\/tr>\n<\/thead>\n
100<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n0001-01-01<\/td>\n2021-01-31<\/td>\n3.5%<\/td>\n1 tbsp<\/td>\n<\/tr>\n
101<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2021-02-01<\/td>\n2021-02-28<\/strong><\/td>\n3.5%<\/td>\n2 tbsp<\/td>\n<\/tr>\n
102<\/strong><\/td>\n12345678<\/strong><\/td>\n2021-01-01<\/strong><\/td>\n2021-03-01<\/strong><\/td>\n9999-12-31<\/strong><\/td>\n0.8%<\/strong><\/td>\n2 tbsp<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

At some point I get bored of milkshakes and I terminate the contract, but because I never purchased a milkshake with 0.8% fat, the owner lets me terminate it with a date in the past, say 2021-02-14, so that we can delete the last row:<\/p>\n

\n\n\n\n\n\n\n
PK<\/th>\ncontract_number<\/th>\ncontract_from<\/th>\ncontract_to<\/th>\nstart<\/th>\nend<\/th>\nfat_content<\/th>\nchocolate_powder<\/th>\n<\/tr>\n<\/thead>\n
100<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2021-02-14<\/strong><\/td>\n0001-01-01<\/td>\n2021-01-31<\/td>\n3.5%<\/td>\n1 tbsp<\/td>\n<\/tr>\n
101<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2021-02-14<\/strong><\/td>\n2021-02-01<\/td>\n9999-12-31<\/td>\n3.5%<\/td>\n2 tbsp<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

Note that it is a design choice whether or not we “shorten” the end date. We might want to do that in order to make such data not be found after the contract termination date. It depends on requirements more than anything.<\/p>\n

What has all this got to do with Kafka, and data replication?<\/u><\/b><\/p>\n

Imagine a self-contained microservice which needs to have an up to date copy of this data, in memory, in order to run lightning fast. Imagine you want that cache to be distributed across all of your service instances (Kubernetes pods). How about the following 7 lines of Kotlin code that use the nifty Kafka Streams API:<\/p>\n

\n
\n\n\n\n
\n
01<\/div>\n
02<\/div>\n
03<\/div>\n
04<\/div>\n
05<\/div>\n
06<\/div>\n
07<\/div>\n
08<\/div>\n
09<\/div>\n
10<\/div>\n<\/td>\n
\n
\n
val builder = StreamsBuilder()<\/code><\/div>\n
val globalStore = Materialized.`as`(globalStoreName)<\/code><\/div>\n
\/\/ global, so that every pod has access to all data from all partitions:<\/code><\/div>\n
builder.globalTable(CONTRACTS_TOPIC, globalStore)<\/code><\/div>\n
val streams = KafkaStreams(builder.build(), props)<\/code><\/div>\n
streams.start()<\/code><\/div>\n
val globalBausteinView = streams.store(fromNameAndType(globalStoreName, ...)<\/code><\/div>\n
 <\/div>\n
\/\/ REST Handler:<\/code><\/div>\n
val contractJson = globalBausteinView.get(contractNumber)<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

We need to publish the contract data to the topic used as the input, but before we do that, let’s think about the keys we use, in order to have the data survive log compaction<\/a>. It would be no good to publish three records, each using the contract number as the key, because as soon as the topic were compacted, only the data from the last row would survive, and any service replicating from scratch would have an incomplete dataset. The solution is to include the start date in the key, e.g. “12345678::2021-02-01”.

<\/div> <\/div><\/p>\n

We have a number of options regarding the values (payload). Let’s work through the examples.<\/p>\n

(Note: initially contracts are valid for 5 years, so the contract_to column always has a value)<\/p>\n

1) Denormalised Table, Variation 1 – One Event per Attribute Combination<\/b><\/u><\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Use Case<\/th>\nPK<\/th>\ncontract_number<\/th>\ncontract_from<\/th>\ncontract_to<\/th>\nstart<\/th>\nend<\/th>\nfat<\/p>\n

 <\/p>\n

content<\/p>\n<\/th>\n

chocolate<\/p>\n

 <\/p>\n

powder<\/p>\n<\/th>\n

records emitted<\/th>\n<\/tr>\n<\/thead>\n
Contract Creation<\/td>\n100<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2025-12-31<\/td>\n0001-01-01<\/td>\n9999-12-31<\/td>\n3.5%<\/td>\n1 tbsp<\/td>\nKey:  12345678::2021-01-01<\/p>\n

 <\/p>\n

Value: {cn: 12345678, from: “2021-01-01”, to: “2025-12-31”, start: “2021-01-01”, end: “2025-12-31”, fatContent: 3.5, choc: 1}<\/p>\n<\/td>\n<\/tr>\n

 <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n<\/tr>\n
Change choc powder<\/td>\n101<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2025-12-31<\/td>\n0001-01-01<\/td>\n2021-01-31<\/strong><\/td>\n3.5%<\/td>\n1 tbsp<\/td>\nKey:  12345678::2021-01-01<\/p>\n

 <\/p>\n

Value: {cn: 12345678, from: “2021-01-01”, to: “2025-12-31”, start: “2021-01-01”, end: “2021-01-31”<\/strong>, fatContent: 3.5, choc: 1}<\/p>\n<\/td>\n<\/tr>\n

102<\/td>\n12345678<\/td>\n2025-12-31<\/td>\n2025-12-31<\/td>\n2021-02-01<\/strong><\/td>\n9999-12-31<\/td>\n3.5%<\/td>\n2 tbsp<\/strong><\/td>\nKey:  12345678::2021-02-01<\/strong>
Value: {cn: 12345678, from: “2021-01-01”, to: “2025-12-31”, start: “2021-02-01”<\/strong>, end: “2025-12-31”, fatContent: 3.5, choc: 2<\/strong>}<\/td>\n<\/tr>\n
 <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n<\/tr>\n
Change fat content<\/td>\n101<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2025-12-31<\/td>\n0001-01-01<\/td>\n2021-01-31<\/td>\n3.5%<\/td>\n1 tbsp<\/td>\nnone – no changes made<\/td>\n<\/tr>\n
102<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2025-12-31<\/td>\n2021-02-01<\/td>\n2021-02-28<\/strong><\/td>\n3.5%<\/td>\n2 tbsp<\/td>\nKey:  12345678::2021-02-01
Value: {cn: 12345678, from: “2021-01-01”, to: “2025-12-31”, start: “2021-02-01”, end: “2021-02-28”<\/strong>, fatContent: 3.5, choc: 2}<\/td>\n<\/tr>\n
103<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2025-12-31<\/td>\n2021-03-01<\/strong><\/td>\n9999-12-31<\/td>\n0.8%<\/strong><\/td>\n2 tbsp<\/td>\nKey:  12345678::2021-03-01<\/strong>
Value: {cn: 12345678, from: “2021-01-01”, to: “2025-12-31”, start: “2021-03-01”<\/strong>, end: “2025-12-31”, fatContent: 0.8<\/strong>, choc: 2}<\/td>\n<\/tr>\n
 <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n<\/tr>\n
Contract Termination<\/td>\n101<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2021-02-14<\/strong><\/td>\n0001-01-01<\/td>\n2021-01-31<\/td>\n3.5%<\/td>\n1 tbsp<\/td>\nKey:  12345678::2021-01-01<\/p>\n

 <\/p>\n

Value: {cn: 12345678, from: “2021-01-01”, to: “2021-02-14”<\/strong>, start: “2021-01-01”, end: “2021-01-31”, fatContent: 3.5, choc: 1}<\/p>\n<\/td>\n<\/tr>\n

102<\/td>\n12345678<\/td>\n2021-01-01<\/td>\n2021-02-14<\/strong><\/td>\n2021-02-01<\/td>\n2021-02-14<\/strong><\/td>\n3.5%<\/td>\n2 tbsp<\/td>\nKey:  12345678::2021-02-01
Value: {cn: 12345678, from: “2021-01-01”, to: “2021-02-14”<\/strong>, start: “2021-02-01”, end: “2021-02-14”<\/strong>, fatContent: 3.5, choc: 2}<\/td>\n<\/tr>\n
103<\/td>\ndeleted<\/em><\/td>\nKey: 12345678:2021-03-01<\/p>\n

 <\/p>\n

Value: null (tombstone record)<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

Note how the key and start\/end dates are not the ugly technical dates but limited to the atual contract validity. That is a design choice where I chose not to expose technical details.<\/p>\n

In this variant, we publish a record for the “lowest common denominators” in terms of validity. There is an event for each time window in which values are constant. Each change, leads to a new record.<\/p>\n

Imagine viewing the validities of the values seperately, as they might be if we normalised the table:<\/p>\n

\n\n\n\n\n\n\n\n
Value<\/th>\nJanuary<\/th>\nFebruary<\/th>\nMarch<\/th>\nApril…<\/th>\n<\/tr>\n<\/thead>\n
Milk Fat Content<\/td>\n3.5<\/td>\n0.8<\/td>\n<\/tr>\n
Chocolate Powder<\/td>\n1<\/td>\n2<\/td>\n<\/tr>\n
Resulting Time Windows with constant values<\/td>\n3.5 & 1<\/td>\n3.5 & 2<\/td>\n0.8 & 2<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

Each change leads to a new row in the denormalised table and hence a new record in Kafka. The three events that are published are visible on that bottom row.<\/p>\n

As an alternative, we could publish one event per contract, with validities inside the payload, as follows.<\/p>\n

2) Denormalised Table, Variation 2 – One Event per Contract<\/u><\/b><\/p>\n

\n\n\n\n\n\n\n
Use Case<\/th>\nrecords emitted<\/th>\n<\/tr>\n<\/thead>\n
Contract Creation<\/td>\nKey:  12345678<\/p>\n

 <\/p>\n

Value: {cn: 12345678, from: “2021-01-01”, to: “2025-12-31”,<\/p>\n

    fatContent: [ {start: “2021-01-01”, end: “2025-12-31”, value: 3.5} ],<\/p>\n

    choc: [ {start: “2021-01-01”, end: “2025-12-31”, value: 1} ]<\/p>\n

}<\/p>\n<\/td>\n<\/tr>\n

Change chocolate powder<\/td>\nKey:  12345678
Value: {cn: 12345678, from: “2021-01-01”, to: “2025-12-31”,<\/p>\n

 <\/p>\n

    fatContent: [ {start: “2021-01-01”, end: “2025-12-31”, value: 3.5} ],<\/p>\n

    choc: [ {start: “2021-01-01”, end: “2021-01-31”<\/strong>, value: 1},<\/p>\n

                 {start: “2021-02-01”<\/strong>, end: “2025-12-31”, value: 2<\/strong>} ]<\/p>\n

}<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

With this variation, we end up having to publish a list of values together with their validities.<\/p>\n

3) Normalised Table, Each Attribute on its own Topic<\/u><\/b><\/p>\n

The next solution is to publish each attribute on its own topic.<\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n
Use Case<\/th>\nrecords emitted<\/th>\n<\/tr>\n<\/thead>\n
Contract Creation<\/td>\nTopic: Contract<\/strong><\/u><\/p>\n

 <\/p>\n

Key:  12345678<\/p>\n

Value: {cn: 12345678, from: “2021-01-01”, to: “2025-12-31”}<\/p>\n

Topic: Fat Content<\/strong><\/u><\/p>\n

Key: 12345678::2021-01-01<\/p>\n

Value: {start: “2021-01-01”, end: “2025-12-31”, value: 3.5}<\/p>\n

Topic: Chocolate Powder<\/strong><\/u><\/p>\n

Key: 12345678::2021-01-01<\/p>\n

Value: {start: “2021-01-01”, end: “2025-12-31”, value: 1}<\/p>\n<\/td>\n<\/tr>\n

 <\/td>\n <\/td>\n<\/tr>\n
Change choc powder<\/td>\nTopic: Chocolate Powder<\/strong><\/u><\/p>\n

 <\/p>\n

Key: 12345678::2021-01-01<\/p>\n

Value: {start: “2021-01-01”, end: “2021-01-31”<\/strong>, value: 1}<\/p>\n

Key: 12345678::2021-02-01<\/strong><\/p>\n

Value: {start: “2021-02-01”, end: “2025-12-31”, value: 2}<\/strong><\/p>\n<\/td>\n<\/tr>\n

 <\/td>\n <\/td>\n<\/tr>\n
Change fat content<\/td>\nTopic: Fat Content<\/strong><\/u><\/p>\n

 <\/p>\n

Key: 12345678::2021-01-01<\/p>\n

Value: {start: “2021-01-01”, end: “2021-02-28”<\/strong>, value: 3.5}<\/p>\n

Key: 12345678::2021-03-01<\/p>\n

Value: {start: “2021-03-01”, end: “2025-12-31”, value: 0.8}<\/strong><\/p>\n<\/td>\n<\/tr>\n

 <\/td>\n <\/td>\n<\/tr>\n
Contract Termination<\/td>\nTopic: Contract<\/strong><\/u><\/p>\n

 <\/p>\n

Key:  12345678<\/p>\n

Value: {cn: 12345678, from: “2021-01-01”, to: “2021-02-14”<\/strong>}<\/p>\n

Topic: Fat Content<\/strong><\/u><\/p>\n

Key: 12345678::2021-01-01<\/p>\n

Value: {start: “2021-01-01”, end: “2021-02-14”<\/strong>, value: 3.5}<\/p>\n

Key: 12345678::2021-03-01<\/p>\n

Value: null (tombstone record)<\/p>\n

Topic: Chocolate Powder<\/strong><\/u><\/p>\n

Key: 12345678::2021-01-01 –> no change, so no record emitted<\/p>\n

Key: 12345678::2021-02-01<\/strong><\/p>\n

Value: {start: “2021-02-01”, end: “2021-02-14”<\/strong>, value: 2}<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

4) Verticalised Table, One Topic for all Attributes<\/u><\/b><\/p>\n

The final solution is to use a verticalised table in order to store the data. This has the advantage that you can dynamically add new attributes, and in fact each contract could have different attributes. This is akin to a schemaless document. The publication of records in Kafka becomes quite generic.<\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n
Use Case<\/th>\nrecords emitted<\/th>\n<\/tr>\n<\/thead>\n
Contract Creation<\/td>\nKey:  12345678::fatContent::2021-01-01<\/p>\n

 <\/p>\n

Value: {start: “2021-01-01”, end: “2025-12-31”, value: 3.5}<\/p>\n

Key: 12345678::chocolatePowder::2021-01-01<\/p>\n

Value: {start: “2021-01-01”, end: “2025-12-31”, value: 1}<\/p>\n<\/td>\n<\/tr>\n

 <\/td>\n <\/td>\n<\/tr>\n
Change choc powder<\/td>\nKey:  12345678::fatContent::2021-01-01 –> no change, no event emitted<\/p>\n

 <\/p>\n

Key: 12345678::chocolatePowder::2021-01-01<\/p>\n

Value: {start: “2021-01-01”, end: “2021-01-31”<\/strong>, value: 1}<\/p>\n

Key: 12345678::chocolatePowder::2021-02-01<\/strong><\/p>\n

Value: {start: “2021-02-01”<\/strong>, end: “2025-12-31”, value: 2<\/strong>}<\/p>\n<\/td>\n<\/tr>\n

 <\/td>\n <\/td>\n<\/tr>\n
Change fat content<\/td>\nKey:  12345678::fatContent::2021-01-01<\/p>\n

 <\/p>\n

Value: {start: “2021-01-01”, end: “2021-02-28”<\/strong>, value: 3.5}<\/p>\n

Key:  12345678::fatContent::2021-03-01<\/strong><\/p>\n

Value: {start: “2021-03-01”<\/strong>, end: “2021-02-28”, value: 0.8}<\/p>\n

Key: 12345678::chocolatePowder::2021-01-01 –> no change, no event emitted<\/p>\n

Key: 12345678::chocolatePowder::2021-02-01 –> no change, no event emitted<\/p>\n<\/td>\n<\/tr>\n

 <\/td>\n <\/td>\n<\/tr>\n
Contract Termination<\/td>\nKey:  12345678::fatContent::2021-01-01<\/p>\n

 <\/p>\n

Value: {start: “2021-01-01”, end: “2021-02-14”<\/strong>, value: 3.5}<\/p>\n

Key:  12345678::fatContent::2021-03-01<\/p>\n

Value: null (tombstone record)<\/p>\n

Key: 12345678::chocolatePowder::2021-01-01 –> no change, no event emitted<\/p>\n

Key: 12345678::chocolatePowder::2021-02-01<\/p>\n

Value: {start: “2021-02-01”, end: “2021-02-14”<\/strong>, value: 2}<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

My favourite is the first solution, as I find it to be the closest to the functional business requirements.<\/p>\n

Another way to choose which solution to use might be to calculate the effect that the solution has on data volume (storage in Kafka; transport through your landscape; storage in replicates).<\/p>\n

If you have other solutions, please get in touch.<\/p>\n

\n\n\n\n
\n

Published on Java Code Geeks with permission by Ant Kutschera, partner at our JCG program<\/a>. See the original article here: Kafka Record Patterns for Data Replication<\/a><\/p>\n

Opinions expressed by Java Code Geeks contributors are their own.<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Imagine going down to your local milkshake bar and signing a contract with the owner so that you could purchase bespoke drinks at a set price. Let’s say you agreed on fresh milk with 3.5% fat and one tablespoon of chocolate powder, per 500ml of milk.  Putting that into a table might look like this: …<\/p>\n","protected":false},"author":176,"featured_media":112,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[940,402],"class_list":["post-109977","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-enterprise-java","tag-apache-kafka","tag-open-source"],"yoast_head":"\nKafka Record Patterns for Data Replication - Java Code Geeks<\/title>\n<meta name=\"description\" content=\"Interested to learn about Kafka Record Patterns? Check our article explaining about Kafka Record Patterns for Data Replication.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kafka Record Patterns for Data Replication - Java Code Geeks\" \/>\n<meta property=\"og:description\" content=\"Interested to learn about Kafka Record Patterns? Check our article explaining about Kafka Record Patterns for Data Replication.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html\" \/>\n<meta property=\"og:site_name\" content=\"Java Code Geeks\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/javacodegeeks\" \/>\n<meta property=\"article:published_time\" content=\"2021-05-11T04:00:00+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"150\" \/>\n\t<meta property=\"og:image:height\" content=\"150\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Ant Kutschera\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@http:\/\/twitter.com\/maxant_ch\" \/>\n<meta name=\"twitter:site\" content=\"@javacodegeeks\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ant Kutschera\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html\"},\"author\":{\"name\":\"Ant Kutschera\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#\\\/schema\\\/person\\\/500abbeb5c0d3d0caff55a34b71d836e\"},\"headline\":\"Kafka Record Patterns for Data Replication\",\"datePublished\":\"2021-05-11T04:00:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html\"},\"wordCount\":1737,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2012\\\/10\\\/enterprise-java-logo.jpg\",\"keywords\":[\"Apache Kafka\",\"Open Source\"],\"articleSection\":[\"Enterprise Java\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html\",\"name\":\"Kafka Record Patterns for Data Replication - Java Code Geeks\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2012\\\/10\\\/enterprise-java-logo.jpg\",\"datePublished\":\"2021-05-11T04:00:00+00:00\",\"description\":\"Interested to learn about Kafka Record Patterns? Check our article explaining about Kafka Record Patterns for Data Replication.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html#primaryimage\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2012\\\/10\\\/enterprise-java-logo.jpg\",\"contentUrl\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2012\\\/10\\\/enterprise-java-logo.jpg\",\"width\":150,\"height\":150,\"caption\":\"java-interview-questions-answers\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/2021\\\/05\\\/kafka-record-patterns-for-data-replication.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.javacodegeeks.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Java\",\"item\":\"https:\\\/\\\/www.javacodegeeks.com\\\/category\\\/java\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Enterprise Java\",\"item\":\"https:\\\/\\\/www.javacodegeeks.com\\\/category\\\/java\\\/enterprise-java\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Kafka Record Patterns for Data Replication\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#website\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/\",\"name\":\"Java Code Geeks\",\"description\":\"Java Developers Resource Center\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#organization\"},\"alternateName\":\"JCG\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.javacodegeeks.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#organization\",\"name\":\"Exelixis Media P.C.\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/exelixis-logo.png\",\"contentUrl\":\"https:\\\/\\\/www.javacodegeeks.com\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/exelixis-logo.png\",\"width\":864,\"height\":246,\"caption\":\"Exelixis Media P.C.\"},\"image\":{\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/javacodegeeks\",\"https:\\\/\\\/x.com\\\/javacodegeeks\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.javacodegeeks.com\\\/#\\\/schema\\\/person\\\/500abbeb5c0d3d0caff55a34b71d836e\",\"name\":\"Ant Kutschera\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/9bea9b00cc0d440b450b34f76b4076c8d4cb0a66187d719230c6567f3bf1fba9?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/9bea9b00cc0d440b450b34f76b4076c8d4cb0a66187d719230c6567f3bf1fba9?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/9bea9b00cc0d440b450b34f76b4076c8d4cb0a66187d719230c6567f3bf1fba9?s=96&d=mm&r=g\",\"caption\":\"Ant Kutschera\"},\"description\":\"Ant is a freelance Java architect and developer. He has been writing a blog and white papers since 2004 and writes about anything he finds interesting, related to Java or software. Most recently he has been working on enterprise systems involving Eclipse RCP, Google GWT, Hibernate, Spring and J2ME. He believes very strongly in being involved in all parts of the software life cycle.\",\"sameAs\":[\"http:\\\/\\\/blog.maxant.co.uk\\\/\",\"http:\\\/\\\/www.linkedin.com\\\/in\\\/maxant\",\"https:\\\/\\\/x.com\\\/http:\\\/\\\/twitter.com\\\/maxant_ch\"],\"url\":\"https:\\\/\\\/www.javacodegeeks.com\\\/author\\\/Ant-Kutschera\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kafka Record Patterns for Data Replication - Java Code Geeks","description":"Interested to learn about Kafka Record Patterns? Check our article explaining about Kafka Record Patterns for Data Replication.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html","og_locale":"en_US","og_type":"article","og_title":"Kafka Record Patterns for Data Replication - Java Code Geeks","og_description":"Interested to learn about Kafka Record Patterns? Check our article explaining about Kafka Record Patterns for Data Replication.","og_url":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html","og_site_name":"Java Code Geeks","article_publisher":"https:\/\/www.facebook.com\/javacodegeeks","article_published_time":"2021-05-11T04:00:00+00:00","og_image":[{"width":150,"height":150,"url":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","type":"image\/jpeg"}],"author":"Ant Kutschera","twitter_card":"summary_large_image","twitter_creator":"@http:\/\/twitter.com\/maxant_ch","twitter_site":"@javacodegeeks","twitter_misc":{"Written by":"Ant Kutschera","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html#article","isPartOf":{"@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html"},"author":{"name":"Ant Kutschera","@id":"https:\/\/www.javacodegeeks.com\/#\/schema\/person\/500abbeb5c0d3d0caff55a34b71d836e"},"headline":"Kafka Record Patterns for Data Replication","datePublished":"2021-05-11T04:00:00+00:00","mainEntityOfPage":{"@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html"},"wordCount":1737,"commentCount":0,"publisher":{"@id":"https:\/\/www.javacodegeeks.com\/#organization"},"image":{"@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html#primaryimage"},"thumbnailUrl":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","keywords":["Apache Kafka","Open Source"],"articleSection":["Enterprise Java"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html","url":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html","name":"Kafka Record Patterns for Data Replication - Java Code Geeks","isPartOf":{"@id":"https:\/\/www.javacodegeeks.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html#primaryimage"},"image":{"@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html#primaryimage"},"thumbnailUrl":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","datePublished":"2021-05-11T04:00:00+00:00","description":"Interested to learn about Kafka Record Patterns? Check our article explaining about Kafka Record Patterns for Data Replication.","breadcrumb":{"@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html#primaryimage","url":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","contentUrl":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2012\/10\/enterprise-java-logo.jpg","width":150,"height":150,"caption":"java-interview-questions-answers"},{"@type":"BreadcrumbList","@id":"https:\/\/www.javacodegeeks.com\/2021\/05\/kafka-record-patterns-for-data-replication.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.javacodegeeks.com\/"},{"@type":"ListItem","position":2,"name":"Java","item":"https:\/\/www.javacodegeeks.com\/category\/java"},{"@type":"ListItem","position":3,"name":"Enterprise Java","item":"https:\/\/www.javacodegeeks.com\/category\/java\/enterprise-java"},{"@type":"ListItem","position":4,"name":"Kafka Record Patterns for Data Replication"}]},{"@type":"WebSite","@id":"https:\/\/www.javacodegeeks.com\/#website","url":"https:\/\/www.javacodegeeks.com\/","name":"Java Code Geeks","description":"Java Developers Resource Center","publisher":{"@id":"https:\/\/www.javacodegeeks.com\/#organization"},"alternateName":"JCG","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.javacodegeeks.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.javacodegeeks.com\/#organization","name":"Exelixis Media P.C.","url":"https:\/\/www.javacodegeeks.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.javacodegeeks.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png","contentUrl":"https:\/\/www.javacodegeeks.com\/wp-content\/uploads\/2022\/06\/exelixis-logo.png","width":864,"height":246,"caption":"Exelixis Media P.C."},"image":{"@id":"https:\/\/www.javacodegeeks.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/javacodegeeks","https:\/\/x.com\/javacodegeeks"]},{"@type":"Person","@id":"https:\/\/www.javacodegeeks.com\/#\/schema\/person\/500abbeb5c0d3d0caff55a34b71d836e","name":"Ant Kutschera","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/9bea9b00cc0d440b450b34f76b4076c8d4cb0a66187d719230c6567f3bf1fba9?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/9bea9b00cc0d440b450b34f76b4076c8d4cb0a66187d719230c6567f3bf1fba9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9bea9b00cc0d440b450b34f76b4076c8d4cb0a66187d719230c6567f3bf1fba9?s=96&d=mm&r=g","caption":"Ant Kutschera"},"description":"Ant is a freelance Java architect and developer. He has been writing a blog and white papers since 2004 and writes about anything he finds interesting, related to Java or software. Most recently he has been working on enterprise systems involving Eclipse RCP, Google GWT, Hibernate, Spring and J2ME. He believes very strongly in being involved in all parts of the software life cycle.","sameAs":["http:\/\/blog.maxant.co.uk\/","http:\/\/www.linkedin.com\/in\/maxant","https:\/\/x.com\/http:\/\/twitter.com\/maxant_ch"],"url":"https:\/\/www.javacodegeeks.com\/author\/Ant-Kutschera"}]}},"_links":{"self":[{"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/posts\/109977","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/users\/176"}],"replies":[{"embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/comments?post=109977"}],"version-history":[{"count":0,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/posts\/109977\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/media\/112"}],"wp:attachment":[{"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/media?parent=109977"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/categories?post=109977"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.javacodegeeks.com\/wp-json\/wp\/v2\/tags?post=109977"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}