Skip to main content

Module getting_started

Module getting_started 

Source
Expand description

§Getting Started with Openraft

In this chapter, we will build a key-value store cluster using Openraft.

examples/raft-kv-memstore is the canonical example application: a server, a client, and a demo cluster that keeps its data in memory. This chapter follows that example, so every code link below points at a file in raft-kv-memstore or in one of the crates it depends on. The example’s README maps each component to the file that implements it.

examples/raft-kv-rocksdb is the same application with RocksDB persistent storage; read it after this chapter, as a storage variation of raft-kv-memstore.


Raft is a distributed consensus protocol designed to manage a replicated log containing state machine commands from clients.

Raft includes two major parts:

  • Replicating logs consistently among nodes,
  • Consuming the logs, which is mainly defined in the state machine.

Implementing your own Raft-based application with Openraft is quite simple, and it involves:

  1. Defining client request and response,
  2. Implementing a storage for Raft to store its state,
  3. Implementing a network layer for Raft to transmit messages.

§1. Define client request and response

A request is some data that modifies the Raft state machine. A response is some data that the Raft state machine returns to the client.

Request and response can be any types that implement AppData and AppDataResponse, for example:

use std::fmt;

#[derive(Clone, Debug)]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
pub struct Request {
    pub key: String,
}

// `AppData` requires `Display` in addition to `Debug`.
impl fmt::Display for Request {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "Set({})", self.key)
    }
}

#[derive(Clone, Debug)]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
pub struct Response {
    pub value: Option<String>,
}

AppData requires OptionalFeatures, which requires Serialize and Deserialize whenever Openraft is built with its serde feature. The cfg_attr attribute above adds the two derives in exactly that case.

These two types are entirely application-specific and are mainly related to the state machine implementation in RaftStateMachine.

§2. Define types config for the application

Openraft is a generic implementation of Raft. It requires the application to define concrete types for its generic arguments. Most types are parameterized by RaftTypeConfig, as Raft itself is:

pub struct Raft<C: RaftTypeConfig> { .. }

The simplest way to define your types config for example TypeConfig is using declare_raft_types! macro:

openraft::declare_raft_types!(
   pub TypeConfig: D = Request, R = Response
);

This macro call adds the above Request and Response to the TypeConfig struct.

  • D = Request is the raft-log payload (usually some command to run) that will be replicated by the raft protocol, and will be applied to the state machine, i.e., your implementation of RaftStateMachine.
  • R = Response is the response that the state machine returns to the client after applying a Request.

There are several more generic types that could be defined in RaftTypeConfig. The macro call above leaves each of them at its default value, and declare_raft_types! documents the complete list of defaults:

  • NodeId is the identifier of a node in the cluster, which implements NodeId trait. A node ID identifies one node, holding one log, for the lifetime of the cluster: never give the ID of a removed node to a node that starts from an empty log, otherwise the cluster may split-brain. See: Node IDs must not be reused.
  • Node is the node type that contains the node’s address, etc., which implements Node trait.
  • Membership decides the read quorums that elect a leader and the write quorums that commit an entry, and MembershipEdit is the change that Raft::change_membership() takes. Both default to a majority: Membership and ChangeMembers. The membership guide compares them with the memberships of openraft-membership, and the raft-kv-memstore-weighted-membership example runs this application with a WeightedMembership.
  • Entry is the log entry type that will be stored in the raft log, which includes the payload and log id, which implements RaftEntry trait.
  • Responder<T> is the type that will be used to send responses to the client, which implements Responder trait.
  • AsyncRuntime is the async runtime that will be used to run the raft instance, which implements AsyncRuntime trait.

Openraft provides default implementations for mostly used types:

You can use these implementations directly or define your own custom types. The canonical example overrides one of these defaults, Node, because each node in that example carries two addresses, api_addr and raft_addr. The example takes D and R from the shared types-kv crate, while the snippet below reuses the Request and Response declared above:

openraft::declare_raft_types!(
    pub TypeConfig:
        D = Request,
        R = Response,
        Node = openraft::NodeInfo,
);

A RaftTypeConfig is also used by other components such as RaftLogStorage, RaftStateMachine, RaftNetworkFactory and RaftNetworkV2.

§3. Implement RaftLogStorage and RaftStateMachine

The trait RaftLogStorage defines how log data is stored and consumed. It could be a wrapper for a local key-value store like RocksDB.

The trait RaftStateMachine defines how log is interpreted. Usually it is an in memory state machine with or without on-disk data backed.

Snapshot data is configured by the SnapshotData associated type, because it is the handle produced and consumed by the state machine.

There is a good example, Mem KV Store, that demonstrates what should be done when a method is called. The storage methods are listed as the below. Follow the links to method documentations to see the details.

KindRaftLogStorage methodReturn valueDescription
Read log:get_log_reader()impl RaftLogReaderget a read-only log reader
↳ try_get_log_entries()get a range of logs
get_log_state()LogStateget first/last log id
Write log:append()()append logs
Write log:truncate_after()()delete logs (index, +oo)
Write log:purge()()purge logs (-oo, index]
Vote:save_vote()()save vote
KindRaftStateMachine methodReturn valueDescription
SM:applied_state()LogId, Membershipget last applied log id, membership
SM:apply()Vec of AppDataResponseapply logs to state machine
Snapshot:install_snapshot()()install snapshot
Snapshot:get_current_snapshot()Snapshotget current snapshot
Snapshot:get_snapshot_builder()impl RaftSnapshotBuilderget a snapshot builder
↳ build_snapshot()build a snapshot from state machine

Most of the APIs are quite straightforward, except two indirect APIs:

§Ensure the storage implementation is correct

There is a Test suite for RaftLogStorage and RaftStateMachine available in Openraft. If your implementation passes the tests, Openraft should work well with it. To test your implementation, run Suite::test_all() with a StoreBuilder implementation, as shown in the sm-rocks test.

Once all tests pass, you can ensure that your custom storage implementation can work correctly in a distributed system.

§An implementation has to guarantee data durability.

The caller always assumes a completed writing is persistent. The raft correctness highly depends on a reliable store.

§4. Implement RaftNetworkV2.

Raft nodes communicate with each other to achieve consensus about the logs. The trait RaftNetworkV2 defines the data transmission protocol.

// Abbreviated; see `RaftNetworkV2` for the full signature.
pub trait RaftNetworkV2<C: RaftTypeConfig>: Send + Sync + 'static {
    type SnapshotData: OptionalSend + 'static;

    async fn append_entries(&mut self, rpc: AppendEntriesRequest<C>, option: RPCOption) -> Result<..>;
    async fn vote(&mut self, rpc: VoteRequest<C>, option: RPCOption) -> Result<..>;
    async fn full_snapshot(&mut self, vote: Vote<C::NodeId>, snapshot: SnapshotOf<C, Self::SnapshotData>, cancel: impl Future<..>, option: RPCOption) -> Result<..>;

    // Optional: override for pipelined replication
    fn stream_append(&mut self, input: impl Stream<Item = AppendEntriesRequest<C>>, option: RPCOption) -> BoxFuture<Result<BoxStream<AppendResult<C>>>>;
}

An implementation of RaftNetworkV2 can be considered as a wrapper that invokes the corresponding methods of a remote Raft. It is responsible for sending and receiving messages between Raft nodes.

The RPCOption argument carries the timeout budget for an RPC. The network implementation is responsible for enforcing soft_ttl() with a transport timeout, deadline, or reconnect policy. Openraft may shut down an in-flight RPC once hard_ttl() has elapsed.

For streaming AppendEntries, hard_ttl() is not a lifetime limit for the whole stream. Use soft_ttl() for setup, idle timeout, keepalive, or per-response deadline policy to detect a stuck stream.

Here is the list of methods that need to be implemented for the RaftNetworkV2 trait:

append_entries() returns an AppendResult: the last log id the follower accepted, or the AppendError that explains why it rejected the request. A conflict carries a hint that helps the leader find the matching log with fewer probes. AppendResult does not implement serde, so the transport converts it to and from its own wire type, such as network-v2-http’s AppendReply. To keep the AppendEntriesResponse wire format instead, the follower sends AppendEntriesResponse::from(result) and the leader rebuilds the result with AppendEntriesResponse::into_append_result(), passing the log ids of the request it sent. That format carries no hint, and the leader then uses binary search.

§Optional: stream_append() for pipelined replication

stream_append() is an optional method that enables bidirectional streaming for efficient pipelined log replication. The default implementation uses append_entries() sequentially.

To enable pipelined replication, override stream_append() to forward the request stream to the remote node’s Raft::stream_append() and return the response stream.

The response stream yields the same AppendResult as append_entries(), one per request the follower answered.

RaftNetworkV2 methodforward requestto target
stream_append()AppendEntriesRequest streamremote node Raft::stream_append()

The canonical example gets both sides from the network-v2-http crate. Its client demonstrates how to forward messages to other Raft nodes using reqwest as network transport layer.

To receive and handle these requests, there should be a server endpoint for each of these RPCs. When the server receives a Raft RPC, it simply passes it to its raft instance and replies with the returned result: network-v2-http server.

For a real-world implementation, you may want to use Tonic gRPC to handle gRPC-based communication between Raft nodes. The databend-meta project provides an excellent real-world example of a Tonic gRPC-based Raft network implementation.

§Implement RaftNetworkFactory.

RaftNetworkFactory is a singleton responsible for creating RaftNetworkV2 instances for each replication target node.

// Abbreviated; see `RaftNetworkFactory` for the full signature.
pub trait RaftNetworkFactory<C: RaftTypeConfig>: Send + Sync + 'static {
    type Network: RaftNetworkV2<C>;
    async fn new_client(&mut self, target: C::NodeId, node: &C::Node) -> Self::Network;
}

This trait contains only one method:

  • RaftNetworkFactory::new_client() builds a new RaftNetworkV2 instance for a target node, intended for sending RPCs to that node. The associated type RaftNetworkFactory::Network represents the application’s implementation of the RaftNetworkV2 trait.

This function should not establish a connection; instead, it should create a client that connects when necessary.

§How RaftNetworkV2 and server interact

The RaftNetworkV2 implementation forwards Raft RPCs to the application-implemented server on another node. The server then forwards these RPCs to the corresponding Raft methods and returns the response.

Request flow:

  1. Client node: append_entries() sends AppendEntriesRequest to target node’s server
  2. Target server: Receives the RPC and calls local Raft::append_entries()
  3. Target server: Gets the response and sends it back to the client node
  4. Client node: Receives the response

.--------------------------.           .--------------------------------.
|    RaftCore              |           |                                |
| (8) ^   | (1)            |           |                                |
|     |   v                |           |                                |
|  ReplicationCore         |           |  Raft::append_entries          |
| (7) ^   | (2)            |           | (4) ^ | (5)                    |
|     |   | append_entries |           |     | | AppendResult           |
|     |   v                |    (3)    |     | v                        |
|  RaftNetworkV2 -----------------------> Application Server            |
|     ^---------------------------------- /append_entries               |
|  (HTTP/gRPC/etc client)  |    (6)    |  (HTTP/gRPC/etc endpoint)      |
'--------------------------'           '--------------------------------'
    Leader                                 Follower


Flow:
(1) RaftCore triggers ReplicationCore to replicate logs
(2) ReplicationCore calls append_entries on RaftNetworkV2
(3) RaftNetworkV2 sends RPC request over network (HTTP/gRPC/etc)
(4) Application Server receives request at /append_entries endpoint
(5) Application Server forwards to local Raft::append_entries
(6) Raft::append_entries returns an AppendResult, which the Application Server sends back
(7) RaftNetworkV2 receives the response
(8) ReplicationCore updates RaftCore with replication result

The same pattern applies to other RPC methods: vote(), full_snapshot().

Example server implementation. A handler takes the shape below; the compiled version is network-v2-http’s /append route:

// Pseudocode: `ServerError` stands for the application's own error type.
async fn handle_append_entries(
    raft: Arc<Raft<TypeConfig, StateMachineStore>>,
    req: AppendEntriesRequest<TypeConfig>,
) -> Result<AppendReply<TypeConfig>, ServerError> {
    let result = raft.append_entries(req).await?;
    // `AppendResult` does not implement serde: reply in the transport's wire type.
    Ok(AppendReply::from(result))
}

§Find the address of the target node.

In Openraft, an implementation of RaftNetworkV2 needs to connect to remote Raft peers. To store additional information about each peer, you need to specify the Node type in RaftTypeConfig:

openraft::declare_raft_types!(
    pub TypeConfig:
        D = Request,
        R = Response,
        Node = openraft::BasicNode,
);

Then use Raft::add_learner(node_id, BasicNode::new("127.0.0.1"), ...) to instruct Openraft to store node information in Membership. This information is then consistently replicated across all nodes, and will be passed to RaftNetworkFactory::new_client() to connect to remote Raft peers:

{
  "configs": [ [ 1, 2, 3 ] ],
  "nodes": {
    "1": { "addr": "127.0.0.1:21001" },
    "2": { "addr": "127.0.0.1:21002" },
    "3": { "addr": "127.0.0.1:21003" }
  }
}

§Caution: ensure that a connection to the right node

See: Ensure connection to the correct node

§5. Put everything together

Finally, we put these parts together and boot up a raft node in lib.rs. Each node listens on two addresses: raft_addr serves the Raft RPCs that peers send, and api_addr serves client and admin requests. Openraft requires only that an inbound Raft RPC reaches the matching Raft method; splitting the two kinds of traffic across two listeners is the example’s own choice, explained in Two servers per node.

The excerpt below abridges that file. CI builds and tests the complete file as part of the raft-kv-memstore crate, whose sibling crates supply the types named here.

ⓘ
pub async fn start_example_raft_node(node_id: NodeId, api_addr: String, raft_addr: String) -> std::io::Result<()> {
    let config = Arc::new(Config::default().validate().unwrap());

    let log_store = LogStore::default();
    let state_machine_store = StateMachineStore::default();
    let network = network_v2_http::NetworkFactory::new();

    let raft = openraft::Raft::new(node_id, config, network, log_store, state_machine_store.clone()).await.unwrap();

    let app = Arc::new(App {
        id: node_id,
        api_addr: api_addr.clone(),
        raft_addr: raft_addr.clone(),
        raft,
        data: state_machine_store,
    });

    // Raft RPCs from peer nodes: `/append`, `/vote`, `/snapshot`, ...
    let raft_server = network_v2_http::Server::new(app.raft.clone()).run(raft_addr);

    // Client and admin API: `/init`, `/add-learner`, `/write`, `/read`, ...
    let app_server = app_http::Server::new(app)
        .add_openraft_routes()
        .post("/read", http_api::read)
        .post("/linearizable_read", http_api::linearizable_read)
        .post("/follower_read", http_api::follower_read)
        .run(api_addr);

    tokio::try_join!(raft_server, app_server)?;
    Ok(())
}

add_openraft_routes() registers the admin and write endpoints that every example shares, defined in app-http; the three post() calls add raft-kv-memstore’s own read endpoints from http_api.rs.

§6. Run the cluster

To set up a demo Raft cluster, follow these steps:

  1. Bring up three uninitialized Raft nodes.
  2. Initialize a single-node cluster.
  3. Add more Raft nodes to the cluster.
  4. Update the membership configuration.

The examples/raft-kv-memstore directory provides a detailed description of these steps.

Additionally, two test scripts for setting up a cluster are available:

  • test-cluster.sh is a minimal Bash script that uses curl to communicate with the Raft cluster. It demonstrates the plain HTTP messages being sent and received.

  • test_basic.rs uses app_http::Client to set up a cluster, write data, and read it back. Its sibling tests in the same directory cover membership changes, snapshots, and the three read modes.

§7. Production checklist

A cluster built from the steps above runs. Staying correct across crashes and network partitions takes the guarantees below; each item states one requirement and links the trait, protocol page, or test suite that defines it.

  • Durable, ordered vote and log writes. save_vote() must return only after the vote is on disk, and append() must call its callback only after the entries are on disk. A vote write and a log write must never be reordered against each other, because a reorder can lose committed data: IO ordering in Raft. append(), truncate_after() and purge() must also leave no hole in the log, because Raft examines only the last log id.

  • Storage conformance. Run Suite::test_all(builder) from the storage test suite against the store the deployment actually uses, passing a StoreBuilder that constructs that store, as the sm-rocks test does.

  • Snapshot persistence and restart recovery. After a restart, get_current_snapshot() must return the snapshot the node last built or installed, and applied_state() must report a log id whose effects are already durable in the state machine, because Openraft re-applies only entries above the reported id. install_snapshot() replaces the state machine wholesale, so a half-installed snapshot must not survive a crash. See snapshot replication.

  • Peer identity and RPC timeouts. A RaftNetworkV2 implementation must confirm it reached the node it addressed rather than whichever node now holds that address (ensure connection to the correct node), must honor the deadlines in RPCOption, and must report a peer it cannot reach as Unreachable so Openraft backs off instead of retrying immediately.

  • Cluster initialization and membership changes. Call Raft::initialize() once, on one pristine node (cluster formation). A new voter then joins in two steps: Raft::add_learner() with blocking = true, which returns once the leader sees that node’s log caught up, and then Raft::change_membership(), which promotes the learner to voter. Promoting a learner that is still far behind puts a node into the new quorum that cannot acknowledge writes yet, and commits stall until that node catches up (dynamic membership).

  • Read semantics. A read served straight from the local state machine is not linearizable: it can return stale data from a deposed leader. A linearizable read calls Raft::ensure_linearizable() on the leader, or Raft::get_read_linearizer() on the leader followed by Linearizer::await_ready() on the follower that serves the read. The ReadPolicy chooses what the guarantee rests on: ReadPolicy::ReadIndex rests on a quorum round-trip, ReadPolicy::LeaseRead on bounded clock drift. See read operations; the canonical example serves all three modes as /read, /linearizable_read and /follower_read in http_api.rs.

  • Metrics, fatal errors and shutdown. Watch Raft::metrics() for leadership, replication lag and membership (monitoring and maintenance). A Fatal error means the Raft core has stopped and will not recover on its own: take the node out of service instead of retrying against it. Call Raft::shutdown() and await its completion before the process exits.

  • On-disk compatibility before an upgrade. Data written by one Openraft version is not automatically readable by the next. Before upgrading, read the upgrade guide: a DataChange: entry in the change log for the target version means the stored format changed and a migration is required.