Expand description
§Getting Started with Openraft
In this chapter, we will build a key-value store cluster using Openraft.
examples/raft-kv-memstore
is the canonical example application: a server, a client, and a demo cluster
that keeps its data in memory. This chapter follows that example, so every code
link below points at a file in raft-kv-memstore or in one of the crates it
depends on. The example’s
README
maps each component to the file that implements it.
examples/raft-kv-rocksdb
is the same application with RocksDB persistent storage; read it after this
chapter, as a storage variation of raft-kv-memstore.
Raft is a distributed consensus protocol designed to manage a replicated log containing state machine commands from clients.
Raft includes two major parts:
- Replicating logs consistently among nodes,
- Consuming the logs, which is mainly defined in the state machine.
Implementing your own Raft-based application with Openraft is quite simple, and it involves:
- Defining client request and response,
- Implementing a storage for Raft to store its state,
- Implementing a network layer for Raft to transmit messages.
§1. Define client request and response
A request is some data that modifies the Raft state machine. A response is some data that the Raft state machine returns to the client.
Request and response can be any types that implement AppData and AppDataResponse, for example:
use std::fmt;
#[derive(Clone, Debug)]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
pub struct Request {
pub key: String,
}
// `AppData` requires `Display` in addition to `Debug`.
impl fmt::Display for Request {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "Set({})", self.key)
}
}
#[derive(Clone, Debug)]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
pub struct Response {
pub value: Option<String>,
}AppData requires OptionalFeatures, which requires Serialize and
Deserialize whenever Openraft is built with its serde feature. The
cfg_attr attribute above adds the two derives in exactly that case.
These two types are entirely application-specific and are mainly related to the
state machine implementation in RaftStateMachine.
§2. Define types config for the application
Openraft is a generic implementation of Raft. It requires the application to define
concrete types for its generic arguments. Most types are parameterized by
RaftTypeConfig, as Raft itself is:
pub struct Raft<C: RaftTypeConfig> { .. }The simplest way to define your types config for example TypeConfig
is using declare_raft_types! macro:
openraft::declare_raft_types!(
pub TypeConfig: D = Request, R = Response
);This macro call adds the above Request and Response to the TypeConfig struct.
D = Requestis the raft-log payload (usually some command to run) that will be replicated by the raft protocol, and will be applied to the state machine, i.e., your implementation ofRaftStateMachine.R = Responseis the response that the state machine returns to the client after applying aRequest.
There are several more generic types that could be defined in RaftTypeConfig.
The macro call above leaves each of them at its default value, and
declare_raft_types! documents the complete list of defaults:
NodeIdis the identifier of a node in the cluster, which implementsNodeIdtrait. A node ID identifies one node, holding one log, for the lifetime of the cluster: never give the ID of a removed node to a node that starts from an empty log, otherwise the cluster may split-brain. See: Node IDs must not be reused.Nodeis the node type that contains the node’s address, etc., which implementsNodetrait.Membershipdecides the read quorums that elect a leader and the write quorums that commit an entry, andMembershipEditis the change thatRaft::change_membership()takes. Both default to a majority:MembershipandChangeMembers. The membership guide compares them with the memberships ofopenraft-membership, and theraft-kv-memstore-weighted-membershipexample runs this application with aWeightedMembership.Entryis the log entry type that will be stored in the raft log, which includes the payload and log id, which implementsRaftEntrytrait.Responder<T>is the type that will be used to send responses to the client, which implementsRespondertrait.AsyncRuntimeis the async runtime that will be used to run the raft instance, which implementsAsyncRuntimetrait.
Openraft provides default implementations for mostly used types:
Node:EmptyNode,BasicNodeandNodeInfo,- log
Entry:Entry, AsyncRuntime:TokioRuntime, which is a wrapper of tokio runtime,Responder:ProgressResponder, the default, which notifies the caller both when the entry is committed and when it is applied; andOneshotResponder, which is a wrapper of oneshot sender and receiver provided byAsyncRuntime.
You can use these implementations directly or define your own custom types.
The canonical example overrides one of these defaults, Node, because each
node in that example carries two addresses, api_addr and raft_addr. The
example takes D and R from the shared
types-kv
crate, while the snippet below reuses the Request and Response declared
above:
openraft::declare_raft_types!(
pub TypeConfig:
D = Request,
R = Response,
Node = openraft::NodeInfo,
);A RaftTypeConfig is also used by other components such as RaftLogStorage, RaftStateMachine,
RaftNetworkFactory and RaftNetworkV2.
§3. Implement RaftLogStorage and RaftStateMachine
The trait RaftLogStorage defines how log data is stored and consumed.
It could be a wrapper for a local key-value store like RocksDB.
The trait RaftStateMachine defines how log is interpreted. Usually it is an in memory state machine with or without on-disk data backed.
Snapshot data is configured by the SnapshotData associated type, because it is the handle produced and consumed by the state machine.
There is a good example,
Mem KV Store,
that demonstrates what should be done when a method is called. The storage methods are listed as the below.
Follow the links to method documentations to see the details.
| Kind | RaftLogStorage method | Return value | Description |
|---|---|---|---|
| Read log: | get_log_reader() | impl RaftLogReader | get a read-only log reader |
↳ try_get_log_entries() | get a range of logs | ||
get_log_state() | LogState | get first/last log id | |
| Write log: | append() | () | append logs |
| Write log: | truncate_after() | () | delete logs (index, +oo) |
| Write log: | purge() | () | purge logs (-oo, index] |
| Vote: | save_vote() | () | save vote |
| Kind | RaftStateMachine method | Return value | Description |
|---|---|---|---|
| SM: | applied_state() | LogId, Membership | get last applied log id, membership |
| SM: | apply() | Vec of AppDataResponse | apply logs to state machine |
| Snapshot: | install_snapshot() | () | install snapshot |
| Snapshot: | get_current_snapshot() | Snapshot | get current snapshot |
| Snapshot: | get_snapshot_builder() | impl RaftSnapshotBuilder | get a snapshot builder |
↳ build_snapshot() | build a snapshot from state machine |
Most of the APIs are quite straightforward, except two indirect APIs:
-
Read logs:
RaftLogStoragedefines a methodget_log_reader()to get log readerRaftLogReader:// Abbreviated; see `RaftLogStorage` for the full signature. trait RaftLogStorage<C: RaftTypeConfig> { type LogReader: RaftLogReader<C>; async fn get_log_reader(&mut self) -> Self::LogReader; }RaftLogReaderdefines the APIs to read logs, and is an also super trait ofRaftLogStorage:try_get_log_entries()get log entries in a range;read_vote()read vote;
// Abbreviated; see `RaftLogReader` for the full signature. trait RaftLogReader<C: RaftTypeConfig> { async fn try_get_log_entries<RB: RangeBounds<u64>>(&mut self, range: RB) -> Result<Vec<C::Entry>, ..>; async fn read_vote(&mut self) -> Result<Option<Vote<C::NodeId>>, ..>; }And
RaftLogStorage::get_log_state()get latest log state from the storage; -
Build a snapshot from the local state machine needs to be done in two steps:
§Ensure the storage implementation is correct
There is a Test suite for RaftLogStorage and RaftStateMachine available in Openraft.
If your implementation passes the tests, Openraft should work well with it.
To test your implementation, run Suite::test_all() with a StoreBuilder implementation,
as shown in the sm-rocks test.
Once all tests pass, you can ensure that your custom storage implementation can work correctly in a distributed system.
§An implementation has to guarantee data durability.
The caller always assumes a completed writing is persistent. The raft correctness highly depends on a reliable store.
§4. Implement RaftNetworkV2.
Raft nodes communicate with each other to achieve consensus about the logs.
The trait RaftNetworkV2 defines the data transmission protocol.
// Abbreviated; see `RaftNetworkV2` for the full signature.
pub trait RaftNetworkV2<C: RaftTypeConfig>: Send + Sync + 'static {
type SnapshotData: OptionalSend + 'static;
async fn append_entries(&mut self, rpc: AppendEntriesRequest<C>, option: RPCOption) -> Result<..>;
async fn vote(&mut self, rpc: VoteRequest<C>, option: RPCOption) -> Result<..>;
async fn full_snapshot(&mut self, vote: Vote<C::NodeId>, snapshot: SnapshotOf<C, Self::SnapshotData>, cancel: impl Future<..>, option: RPCOption) -> Result<..>;
// Optional: override for pipelined replication
fn stream_append(&mut self, input: impl Stream<Item = AppendEntriesRequest<C>>, option: RPCOption) -> BoxFuture<Result<BoxStream<AppendResult<C>>>>;
}An implementation of RaftNetworkV2 can be considered as a wrapper that invokes
the corresponding methods of a remote Raft. It is responsible for sending
and receiving messages between Raft nodes.
The RPCOption argument carries the timeout budget for an RPC. The network
implementation is responsible for enforcing soft_ttl() with a transport
timeout, deadline, or reconnect policy. Openraft may shut down an in-flight RPC
once hard_ttl() has elapsed.
For streaming AppendEntries, hard_ttl() is not a lifetime limit for the whole
stream. Use soft_ttl() for setup, idle timeout, keepalive, or per-response
deadline policy to detect a stuck stream.
Here is the list of methods that need to be implemented for the RaftNetworkV2 trait:
RaftNetworkV2 method | forward request | to target |
|---|---|---|
append_entries() | AppendEntriesRequest | remote node Raft::append_entries() |
vote() | VoteRequest | remote node Raft::vote() |
full_snapshot() | Snapshot | remote node Raft::install_full_snapshot() |
append_entries() returns an AppendResult: the last log id the follower
accepted, or the AppendError that explains why it rejected the request. A
conflict carries a hint that helps the leader find the matching log with fewer
probes. AppendResult does not implement serde, so the transport converts it
to and from its own wire type, such as network-v2-http’s
AppendReply.
To keep the AppendEntriesResponse wire format instead, the follower sends
AppendEntriesResponse::from(result) and the leader rebuilds the result with
AppendEntriesResponse::into_append_result(), passing the log ids of the
request it sent. That format carries no hint, and the leader then uses binary
search.
§Optional: stream_append() for pipelined replication
stream_append() is an optional method that enables bidirectional streaming
for efficient pipelined log replication. The default implementation uses
append_entries() sequentially.
To enable pipelined replication, override stream_append() to forward the
request stream to the remote node’s Raft::stream_append() and return the
response stream.
The response stream yields the same AppendResult as append_entries(),
one per request the follower answered.
RaftNetworkV2 method | forward request | to target |
|---|---|---|
stream_append() | AppendEntriesRequest stream | remote node Raft::stream_append() |
The canonical example gets both sides from the network-v2-http
crate. Its client
demonstrates how to forward messages to other Raft nodes using reqwest as network transport layer.
To receive and handle these requests, there should be a server endpoint for each of these RPCs.
When the server receives a Raft RPC, it simply passes it to its raft instance and replies with the returned result:
network-v2-http server.
For a real-world implementation, you may want to use Tonic gRPC to handle gRPC-based communication between Raft nodes. The databend-meta project provides an excellent real-world example of a Tonic gRPC-based Raft network implementation.
§Implement RaftNetworkFactory.
RaftNetworkFactory is a singleton responsible for creating RaftNetworkV2 instances for each replication target node.
// Abbreviated; see `RaftNetworkFactory` for the full signature.
pub trait RaftNetworkFactory<C: RaftTypeConfig>: Send + Sync + 'static {
type Network: RaftNetworkV2<C>;
async fn new_client(&mut self, target: C::NodeId, node: &C::Node) -> Self::Network;
}This trait contains only one method:
RaftNetworkFactory::new_client()builds a newRaftNetworkV2instance for a target node, intended for sending RPCs to that node. The associated typeRaftNetworkFactory::Networkrepresents the application’s implementation of theRaftNetworkV2trait.
This function should not establish a connection; instead, it should create a client that connects when necessary.
§How RaftNetworkV2 and server interact
The RaftNetworkV2 implementation forwards Raft RPCs to the application-implemented server on another node.
The server then forwards these RPCs to the corresponding Raft methods and returns the response.
Request flow:
- Client node:
append_entries()sendsAppendEntriesRequestto target node’s server - Target server: Receives the RPC and calls local
Raft::append_entries() - Target server: Gets the response and sends it back to the client node
- Client node: Receives the response
.--------------------------. .--------------------------------.
| RaftCore | | |
| (8) ^ | (1) | | |
| | v | | |
| ReplicationCore | | Raft::append_entries |
| (7) ^ | (2) | | (4) ^ | (5) |
| | | append_entries | | | | AppendResult |
| | v | (3) | | v |
| RaftNetworkV2 -----------------------> Application Server |
| ^---------------------------------- /append_entries |
| (HTTP/gRPC/etc client) | (6) | (HTTP/gRPC/etc endpoint) |
'--------------------------' '--------------------------------'
Leader Follower
Flow:
(1) RaftCore triggers ReplicationCore to replicate logs
(2) ReplicationCore calls append_entries on RaftNetworkV2
(3) RaftNetworkV2 sends RPC request over network (HTTP/gRPC/etc)
(4) Application Server receives request at /append_entries endpoint
(5) Application Server forwards to local Raft::append_entries
(6) Raft::append_entries returns an AppendResult, which the Application Server sends back
(7) RaftNetworkV2 receives the response
(8) ReplicationCore updates RaftCore with replication resultThe same pattern applies to other RPC methods: vote(), full_snapshot().
Example server implementation. A handler takes the shape below; the
compiled version is
network-v2-http’s /append route:
// Pseudocode: `ServerError` stands for the application's own error type.
async fn handle_append_entries(
raft: Arc<Raft<TypeConfig, StateMachineStore>>,
req: AppendEntriesRequest<TypeConfig>,
) -> Result<AppendReply<TypeConfig>, ServerError> {
let result = raft.append_entries(req).await?;
// `AppendResult` does not implement serde: reply in the transport's wire type.
Ok(AppendReply::from(result))
}§Find the address of the target node.
In Openraft, an implementation of RaftNetworkV2 needs to connect to remote Raft peers. To store additional information about each peer, you need to specify the Node type in RaftTypeConfig:
openraft::declare_raft_types!(
pub TypeConfig:
D = Request,
R = Response,
Node = openraft::BasicNode,
);Then use Raft::add_learner(node_id, BasicNode::new("127.0.0.1"), ...) to instruct Openraft to store node information in Membership. This information is then consistently replicated across all nodes, and will be passed to RaftNetworkFactory::new_client() to connect to remote Raft peers:
{
"configs": [ [ 1, 2, 3 ] ],
"nodes": {
"1": { "addr": "127.0.0.1:21001" },
"2": { "addr": "127.0.0.1:21002" },
"3": { "addr": "127.0.0.1:21003" }
}
}§Caution: ensure that a connection to the right node
See: Ensure connection to the correct node
§5. Put everything together
Finally, we put these parts together and boot up a raft node in
lib.rs.
Each node listens on two addresses: raft_addr serves the Raft RPCs that peers
send, and api_addr serves client and admin requests. Openraft requires only
that an inbound Raft RPC reaches the matching Raft method; splitting the two
kinds of traffic across two listeners is the example’s own choice, explained in
Two servers per node.
The excerpt below abridges that file. CI builds and tests the complete file as
part of the raft-kv-memstore crate, whose sibling crates supply the types
named here.
pub async fn start_example_raft_node(node_id: NodeId, api_addr: String, raft_addr: String) -> std::io::Result<()> {
let config = Arc::new(Config::default().validate().unwrap());
let log_store = LogStore::default();
let state_machine_store = StateMachineStore::default();
let network = network_v2_http::NetworkFactory::new();
let raft = openraft::Raft::new(node_id, config, network, log_store, state_machine_store.clone()).await.unwrap();
let app = Arc::new(App {
id: node_id,
api_addr: api_addr.clone(),
raft_addr: raft_addr.clone(),
raft,
data: state_machine_store,
});
// Raft RPCs from peer nodes: `/append`, `/vote`, `/snapshot`, ...
let raft_server = network_v2_http::Server::new(app.raft.clone()).run(raft_addr);
// Client and admin API: `/init`, `/add-learner`, `/write`, `/read`, ...
let app_server = app_http::Server::new(app)
.add_openraft_routes()
.post("/read", http_api::read)
.post("/linearizable_read", http_api::linearizable_read)
.post("/follower_read", http_api::follower_read)
.run(api_addr);
tokio::try_join!(raft_server, app_server)?;
Ok(())
}add_openraft_routes() registers the admin and write endpoints that every
example shares, defined in
app-http;
the three post() calls add raft-kv-memstore’s own read endpoints from
http_api.rs.
§6. Run the cluster
To set up a demo Raft cluster, follow these steps:
- Bring up three uninitialized Raft nodes.
- Initialize a single-node cluster.
- Add more Raft nodes to the cluster.
- Update the membership configuration.
The examples/raft-kv-memstore directory provides a detailed description of these steps.
Additionally, two test scripts for setting up a cluster are available:
-
test-cluster.sh is a minimal Bash script that uses
curlto communicate with the Raft cluster. It demonstrates the plain HTTP messages being sent and received. -
test_basic.rs uses
app_http::Clientto set up a cluster, write data, and read it back. Its sibling tests in the same directory cover membership changes, snapshots, and the three read modes.
§7. Production checklist
A cluster built from the steps above runs. Staying correct across crashes and network partitions takes the guarantees below; each item states one requirement and links the trait, protocol page, or test suite that defines it.
-
Durable, ordered vote and log writes.
save_vote()must return only after the vote is on disk, andappend()must call its callback only after the entries are on disk. A vote write and a log write must never be reordered against each other, because a reorder can lose committed data: IO ordering in Raft.append(),truncate_after()andpurge()must also leave no hole in the log, because Raft examines only the last log id. -
Storage conformance. Run
Suite::test_all(builder)from the storage test suite against the store the deployment actually uses, passing aStoreBuilderthat constructs that store, as thesm-rockstest does. -
Snapshot persistence and restart recovery. After a restart,
get_current_snapshot()must return the snapshot the node last built or installed, andapplied_state()must report a log id whose effects are already durable in the state machine, because Openraft re-applies only entries above the reported id.install_snapshot()replaces the state machine wholesale, so a half-installed snapshot must not survive a crash. See snapshot replication. -
Peer identity and RPC timeouts. A
RaftNetworkV2implementation must confirm it reached the node it addressed rather than whichever node now holds that address (ensure connection to the correct node), must honor the deadlines inRPCOption, and must report a peer it cannot reach asUnreachableso Openraft backs off instead of retrying immediately. -
Cluster initialization and membership changes. Call
Raft::initialize()once, on one pristine node (cluster formation). A new voter then joins in two steps:Raft::add_learner()withblocking = true, which returns once the leader sees that node’s log caught up, and thenRaft::change_membership(), which promotes the learner to voter. Promoting a learner that is still far behind puts a node into the new quorum that cannot acknowledge writes yet, and commits stall until that node catches up (dynamic membership). -
Read semantics. A read served straight from the local state machine is not linearizable: it can return stale data from a deposed leader. A linearizable read calls
Raft::ensure_linearizable()on the leader, orRaft::get_read_linearizer()on the leader followed byLinearizer::await_ready()on the follower that serves the read. TheReadPolicychooses what the guarantee rests on:ReadPolicy::ReadIndexrests on a quorum round-trip,ReadPolicy::LeaseReadon bounded clock drift. See read operations; the canonical example serves all three modes as/read,/linearizable_readand/follower_readin http_api.rs. -
Metrics, fatal errors and shutdown. Watch
Raft::metrics()for leadership, replication lag and membership (monitoring and maintenance). AFatalerror means the Raft core has stopped and will not recover on its own: take the node out of service instead of retrying against it. CallRaft::shutdown()and await its completion before the process exits. -
On-disk compatibility before an upgrade. Data written by one Openraft version is not automatically readable by the next. Before upgrading, read the upgrade guide: a
DataChange:entry in the change log for the target version means the stored format changed and a migration is required.