raftkv.

a fault-tolerant distributed key-value store. a C++ storage engine and a go raft sidecar, glued together with gRPC.

c++
go
hashicorp raft
gRPC
msgpack
boltdb
docker

what it is

raftkv is a distributed key-value store that refuses to lose your data. every acknowledged write is fsynced to a write-ahead log before the client hears "ok", so a kill -9 at any moment costs you nothing. three nodes hold an election, pick a leader, and agree on the order of every operation through raft consensus.

i built it to find out what the raft paper glosses over. (a lot, it turns out. mostly about what happens on the disk.)

two processes, one node

each node runs two processes. a C++ engine owns the data: it serves HTTP, keeps the map in memory, and writes the WAL. a go sidecar owns the cluster: leader election, log replication, and membership via hashicorp raft. they talk over gRPC on loopback, and the sidecar never learns what the keys mean. C++ where the bytes are, go where the raft is.

data plane (c++) consensus plane (go)

the write path

the interesting part of any consensus system is what one write goes through before the client is told it happened.

client sends PUT /kv/{key} to any node. followers forward the request to the leader instead of bouncing it.
the engine packs the command into a msgpack blob and proposes it to its sidecar over gRPC.
the leader appends the entry to its raft log and replicates it to the followers.
a quorum acknowledges. the entry is committed, and every node's sidecar calls back into its engine: apply this.
the engine appends to kv.wal, fsyncs, and only then mutates the in-memory map.
the client gets its 200. by that point the write already exists on disk on a majority of nodes.

durability, in three files

kv.db

the base file. full key-value state in a binary format (KVB1), replaced atomically via temp-file-and-rename. a crash mid-write can never leave it half-updated.

kv.wal

the write-ahead log. every applied command, CRC32-checksummed. torn tails from a mid-write crash are detected and truncated on replay.

logs.dat

the raft log itself, kept by the sidecar in boltdb. snapshots keep it bounded and make node recovery fast.

recovery is just: load the base file, replay the WAL over it. once the WAL passes 4 MiB or 10,000 records, compaction folds it into a fresh base file and starts over.

reads, and what they cost

by default a read is served straight from local memory: fast, but it can lag the leader by a beat. pass consistency=linearizable and the request routes to the leader and waits for a barrier, guaranteeing you see the latest committed write. certainty costs roughly 7x.

33,289ops/s

local reads

p50 0.33 ms

4,618ops/s

linearizable reads

p50 6.4 ms

3-node cluster, 32 clients, 128-byte values, apple m2 under docker.
workload	throughput	p50	p99
writes	1,930 ops/s	15.5 ms	31.6 ms
local reads	33,289 ops/s	0.33 ms	1.8 ms
linearizable reads	4,618 ops/s	6.4 ms	13.9 ms
mixed, 90% reads	13,865 ops/s	0.74 ms	15.7 ms

fresh connection per request, no keep-alive. failed requests excluded from latency. full methodology in the repo's benchmarks doc.

my favourite finding: replication costs about 40% of write throughput, because the leader waits for follower WAL fsyncs before acknowledging. durability is a line item and you can see it on the invoice.

hardening

auth is opt-in: set RAFTKV_ADMIN_PASSWORD and clients authenticate against per-user ACLs with glob key patterns and read/write/admin command classes. the trick i'm proud of: user records replicate through raft itself under a reserved __sys: prefix, so ACLs survive restarts with no extra machinery. raft peers can speak mutual TLS, client traffic terminates HTTPS at a caddy proxy, and the loopback gRPC pair stays plaintext because it never leaves the box.

testing runs in three layers: go unit tests for the sidecar, googletest with ASan/UBSan/TSan for the engine, and a pytest e2e suite that kills leaders mid-write and checks nothing was lost. CI also fuzzes the parsers with libFuzzer, and prometheus metrics expose apply throughput, request counts, and store size.

run it

shell 3 nodes, 1 leader

$ git clone https://github.com/burhankapadia18/raftkv && cd raftkv
$ docker compose up --build -d
# three nodes come up and elect a leader. then:
$ curl -X PUT localhost:8080/kv/hello -d 'world'
{"ok":true}
$ curl 'localhost:8080/kv/hello?consistency=linearizable'
world

now docker compose kill the leader and run the GET against another node. the remaining two elect a new leader and the answer is still world. that's the whole point.