{"id":20653166,"date":"2022-01-10T08:08:30","date_gmt":"2022-01-10T16:08:30","guid":{"rendered":"https:\/\/thenewstack.io\/?p=20653166"},"modified":"2022-01-10T08:08:30","modified_gmt":"2022-01-10T16:08:30","slug":"redis-pub-sub-vs-apache-kafka","status":"publish","type":"post","link":"https:\/\/thenewstack.io\/redis-pub-sub-vs-apache-kafka\/","title":{"rendered":"Redis Pub\/Sub vs. Apache Kafka"},"content":{"rendered":"

Redis is the “Swiss Army knife” of in-memory databases with many data types<\/a>, and it’s often used for caching<\/a>, but it does even more. It can also function as a loosely coupled distributed message broker, so in this article, we’ll have a look at the original Redis messaging approach, Redis Pub\/Sub, explore some use cases and compare it with Apache Kafka.<\/p>\n

1. Redis Pub\/Sub<\/h2>\n
\"\"

A Beatles-inspired submarine cocktail. Image<\/a> used under license from Shutterstock.com. Contributor: Evlakhov Valerii<\/a><\/p><\/div>\n

The theme of “pub” pops up frequently in my articles. In a previous article, I wrote about a conversation in an outback pub, “Apache ZooKeeper Meets the Dining Philosophers,<\/a>” and in this article, we are investigating Redis Pub\/Sub. This sounds like some sort of Beatles-inspired exotic yellow submarine cocktail, but Pub\/Sub is actually short for publish\/subscribe, the well-known pattern for loosely-coupled distributed messaging systems.<\/p>\n\n\t\t\t\t

\n\t\t\t\t\t\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t
Paul Brebner<\/div>\n\t\t\t\t\t

\nPaul is the technology evangelist at Instaclustr. He\u2019s been learning new scalable big data technologies, solving real problems and building applications, and blogging about Apache Cassandra, Spark, Zeppelin, and Kafka. Paul has extensive R&D and industry experience in distributed systems, technology innovation, software architecture and engineering, software performance and scalability, grid and cloud computing, and data analytics and machine learning.<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/a>\n

Redis Pub\/Sub<\/a> is the oldest style of messaging pattern supported by Redis and uses a data type called a “channel,” which supports typical pub\/sub operations, such as publish and subscribe. It’s considered loosely coupled because publishers and subscribers don’t know about each other. Publishers publish messages to a channel, or multiple channels, and subscribers subscribe to one or more channels.<\/p>\n

A channel can have zero or more subscribers, and the messages are delivered to all the current connected subscribers. Redis Pub\/Sub is therefore flexible and supports multiple topologies including fan-in (multiple producers, single subscriber), fan-out (single producer, multiple subscribers), and 1-1 (one producer, one consumer).<\/p>\n

So far this sounds like a fairly typical pub\/sub system, however, one feature is important to highlight: “connected” delivery semantics.<\/p>\n

2. Connected Delivery Semantics<\/h2>\n

Connected delivery functions like radio. Radio stations are constantly broadcasting on different frequencies (channels), but listeners can only hear the broadcast while their receiver is plugged in, turned on, and they are tuned in to a station. (“Stay tuned” could be the motto for Redis Pub\/Sub).<\/p>\n

\"\"

Redis Pub\/Sub — to listen, plugin, turn on and tune in. Image<\/a> used under license from Shutterstock.com. Contributor: Everett Collection<\/a><\/p><\/div>\n

Connected delivery means that:<\/p>\n

    \n
  1. Only connected subscribers receive messages.<\/li>\n
  2. Every connected subscriber receives each message.<\/li>\n
  3. Once the message is delivered to all current subscribers, it is deleted from the channel (there’s no “memory” in the system).<\/li>\n<\/ol>\n

    This implies that:<\/p>\n

      \n
    1. If a subscriber unsubscribes (disconnects) and later subscribes to a channel again:\n
        \n
      1. It will not receive any of the intervening messages that it missed while disconnected, and<\/li>\n
      2. It doesn’t know if it’s missed any messages.<\/li>\n<\/ol>\n<\/li>\n
      3. If there are no current subscribers to the channel,  the message will simply be discarded and not delivered to any subscribers.<\/li>\n
      4. The delivery semantics are, therefore, “at-most-once” per subscriber.<\/li>\n
      5. Because the message must be delivered to all current subscribers before being deleted:\n
          \n
        1. This will take longer with more subscribers.<\/li>\n
        2. This is unlike a radio broadcast, which delivers content currently at the speed of light to every receiver in range.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

          Note that “disconnection” is intentional by design, but can also be due to network or client failures, so may be unexpected, and this will also result in potential message loss.<\/p>\n

          From reading the Redis Pub\/Sub documentation,<\/a> and other articles<\/a>, it appears that Redis uses push notification to ensure messages are delivered to all current subscribers, which has potential performance penalties for large numbers of subscribers.<\/p>\n

          3. Use Cases for Redis Pub\/Sub<\/h2>\n
          \"\"

          Redis Pub\/Sub channels can have multiple subscribers, but too many may have a performance impact (unlike real radio which works perfectly for unlimited receivers). Image<\/a> used under license from Shutterstock.com. Contributor: jimeone<\/a><\/p><\/div>\n

          What are appropriate use cases for the Redis Pub\/Sub “connected” delivery semantics?<\/p>\n

            \n
          1. Real-time, low-latency, urgent messages: If messages are short-lived and age rapidly, so, therefore, are only relevant to subscribers for a short time window (basically “immediately”).<\/li>\n
          2. Unreliable delivery\/lossy messaging: If it doesn’t matter if some messages are simply discarded (redundant messages of low importance rather than uniquely critical “business” messages) due to unreliable delivery. Failures in the network and subscribers, or failover from master to replicas<\/a>, may all result in discarded messages.<\/li>\n
          3. A requirement for at-most-once delivery per subscriber (subscribers are not capable of detecting duplicate messages and target systems are not idempotent.)<\/li>\n
          4. If subscribers have short-lived, evolving or dynamic interest in channels, and only want to receive messages from specific channels for finite periods of time. For instance, mobile IoT devices may only be intermittently connected, and only interested and able to respond to current messages at their location.<\/li>\n
          5. If subscribers — and channels and publishers too — are themselves potentially short-lived.<\/li>\n
          6. One or more subscribers per channel, but…<\/li>\n
          7. There are only a small number of subscribers and patterns per channel.<\/li>\n<\/ol>\n\n\n
            \n\t\t
            \n\t\t\t
            \n\t\t\t\t
            \n\t\t \n\t\t\t\t\t\t\"\"<\/span>\n\t\t <\/a>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\tInstaclustr delivers reliability at scale through an integrated data platform of open source technologies such as Apache Cassandra®, Apache Kafka®, Apache SparkTM, ElasticsearchTM, RedisTM, Apache ZooKeeperTM, and PostgreSQL®.\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\tLearn More\n \n<\/svg>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n <\/div>\n\n
            \n\t\t\t
            \n\t\t\t\tThe latest from Instaclustr\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t