Search before asking

• I had searched in the issues and found no similar issues.

Feature Request

EventMesh-Camel-SpringCloudStream

In order to implement event stream processing, we will integrate Camel and Spring Cloud Stream in EventMesh. This document provides the basic architecture of the stream processing system in EventMesh.

Spring-cloud-stream
Spring Cloud Stream is a framework for building message-driven microservices. Spring Cloud Stream provides an opinionated configuration of message brokers, introducing the concepts of persistent pub/sub semantics, consumer groups and partitions across several middleware vendors. This opinionated configuration provides the basis to create stream processing applications.

The two components of Spring Cloud Stream and Spring Cloud Binder always appear in pairs. The application interacts with the binder object in Spring Cloud Stream through inputs or outputs. Binding is performed through configuration, and the binder object of Spring Cloud Stream is responsible for interacting with the message middleware, such as Kafka, RabbitMQ and RocketMQ. So, we only need to figure out how to interact with Spring Cloud Stream to facilitate the use of message-driven methods. At its most basic, Stream comes with three predefined interfaces: Source, Processor and Sink. These interfaces have a combination of input and output bindings. Normally, for incoming messages, the sink and input channels are binding. For the message to be sent, the source and output channels are binding.Processor can process incoming messages and then send them to other applications. The channel for sending messages can be abstracted as MessageChannel:

• Producers/consumers rely on message media to transfer information content (Message)
• The message must go through a specific channel (MessageChannel)

The architectures of messaging middleware such as RabbitMQ and RocketMQ are usually different. The differences of these middleware have caused some difficulties in actual project development. The migration of middleware based on business needs is undoubtedly disastrous. At this time Spring Cloud Stream provides a way of decoupling. Through a unified channel, the application does not need to consider various message middleware implementations. The middleware we use as an example in the architecture is RocketMQ.

Apache Camel
Spring Cloud Stream provides personalized automation configuration implementations for some vendors' messaging middleware products. Usually, we can use Spring Integration to connect to the message broker middleware to achieve message event-driven. But based on the characteristics of Source, Processor and Sink, we can also have the power and convenience of Spring Cloud Stream combined with the simple, yet powerful ecosystem of Apache Camel.

Apache Camel abstracts a set of message interaction models for application integration scenarios, and accesses third-party systems through components. Apache Camel provides domain-specific languages (DSL) such as message routing and message conversion, which greatly reduces the difficulty of integrated application development. The application node information that needs to be integrated is defined by URI, and users can use DSL to quickly compile message routing rules according to business needs without paying attention to the details of the integration protocol. At the same time, the core library of Apache Camel is very small and can be easily integrated with other systems.

The innermost part in the picture is Camel. In the specific implementation, we can provide some Routes defined in the DSL language. These Routes define the direction of message sending. It can be sent from one topic to another or multiple topics. The code format is mainly as follows:

from("rocketmq:topic1?namesrvAddr=localhost:9876&consumerGroup=consumer")
.to("rocketmq:topic2?namesrvAddr=localhost:9876&producerGroup=producer");

Message filter processors are an important function in Camel. They are processors for message communication, which can be used for message format conversion or other transformations. The processor in Spring Cloud Stream handle with the message between input channel and output channel in binder. The processors in Apache Camel are used in each route, and multiple processors can be used in one route.

Middleware
The camel spring cloud stream system mentioned above can use a variety of messaging middleware such as RabbitMQ and Kafka. Here, combined with EventMesh, we use RocketMQ for message delivery.

RocketMQ cluster deployment usually includes four parts: Name server, Broker, Producer and Consumer, and supports distributed transaction messages. RocketMQ will receive messages from Spring Cloud Stream in the channel, and deliver the message.

Are you willing to submit PR?

• Yes I am willing to submit a PR!