Setting up a 3-node cluster¶

This tutorial brings up a 3-node SkeinDB cluster on one machine using different ports. The same commands work across three hosts — only the bind addresses change.

Prerequisite: Quickstart completed, skeindb binary available on $PATH.

1. Start the first node (seed)¶

skeindb serve \
  --data ./data/n1 \
  --http 8001 \
  --mysql 3301 \
  --node-id n1 \
  --cluster-bind 127.0.0.1:7001

The first node generates a cluster id and a join token on first boot. Copy the join token from the startup log (or from ./data/n1/cluster/join_token.txt).

cluster id: 8c2f…a9
join token: jt_a3d8c1…

2. Join two more nodes¶

On the same machine (different ports):

skeindb serve \
  --data ./data/n2 \
  --http 8002 \
  --mysql 3302 \
  --node-id n2 \
  --cluster-bind 127.0.0.1:7002 \
  --cluster-join 127.0.0.1:7001 \
  --cluster-join-token jt_a3d8c1…

skeindb serve \
  --data ./data/n3 \
  --http 8003 \
  --mysql 3303 \
  --node-id n3 \
  --cluster-bind 127.0.0.1:7003 \
  --cluster-join 127.0.0.1:7001 \
  --cluster-join-token jt_a3d8c1…

3. Verify the topology¶

curl -s -XPOST http://127.0.0.1:8001/api/v1/rpc \
  -H 'Content-Type: application/json' \
  -d '{"skeinql":"1.0","id":1,"method":"cluster.topology","params":{}}' | jq

Or open http://127.0.0.1:8001/admin → Cluster panel. You should see three nodes and one primary.

4. Write on the primary, read on a replica¶

# Write via primary (n1)
curl -s -XPOST http://127.0.0.1:8001/api/v1/rpc \
  -H 'Content-Type: application/json' \
  -d '{"skeinql":"1.0","id":1,"method":"schema.create_database","params":{"db":"shop"}}'

# Read via replica (n2)
curl -s -XPOST http://127.0.0.1:8002/api/v1/rpc \
  -H 'Content-Type: application/json' \
  -d '{"skeinql":"1.0","id":1,"method":"cluster.topology","params":{}}'

Replicas accept read queries but forward writes. RPC fanout is recursion-suppressed, so cluster-wide operations are exactly-once.

Replication is self-healing: each write carries a primary-assigned log position, and a replica that falls behind (a transient blip) or joins late automatically pulls the ops it missed from the primary and catches up — no manual rebuild. Each node's applied position shows up in cluster.replication.status and cluster.failover.status. (A replica that has fallen further behind than the primary's op-log buffer retains reports resync_required and is re-synced from a backup instead.)

5. Simulate a failure¶

Stop n1:

# Ctrl+C on n1, or:
skeindb admin cluster.demote --node-id n1 --http http://127.0.0.1:8002

Promote a replica from the admin UI or via SkeinQL:

curl -s -XPOST http://127.0.0.1:8002/api/v1/rpc \
  -H 'Content-Type: application/json' \
  -d '{"skeinql":"1.0","id":1,"method":"cluster.promote","params":{"node_id":"n2"}}'

6. Automated fenced failover (optional)¶

By default, promoting a replica after the primary fails is a manual step (and is quorum-gated — a promotion is refused unless the promoting node still sees a majority of the cluster, so a minority partition can't create a second primary). To let the cluster fail over on its own, start every node with:

SKEINDB_CLUSTER_AUTO_FAILOVER=1 skeindb serve --data ./n1 --http 8001 ...

With this enabled, each node runs a background tick that:

Heartbeats its peers so the cluster keeps a live health view (a node unseen for SKEINDB_CLUSTER_NODE_TIMEOUT_MS, default 15s, is considered offline).
Fences itself if it is the primary but has lost quorum — it refuses writes (clients get a fenced error) so it can't diverge from the new primary the majority side will elect.
Elects a new primary on the majority side: the most up-to-date online replica (the one with the highest applied replication progress — heartbeats carry each node's applied_ops) requests votes from its peers (cluster.request_vote) and promotes itself only if a majority grant it. Each node votes at most once per election term, so at most one candidate can win — no split-brain. A voter also refuses any candidate less caught up than itself (Raft's log-matching rule), so the winner is guaranteed to hold every committed write — failover can't lose acknowledged data.

Watch the failover decision live (read-only, safe to poll):

curl -s -XPOST http://127.0.0.1:8001/api/v1/rpc \
  -H 'Content-Type: application/json' \
  -d '{"skeinql":"1.0","id":1,"method":"cluster.failover.status","params":{}}'
# → primary_healthy, recommended_candidate, leadership_epoch, and a quorum block per node

Run 3+ nodes. Quorum is a majority, so a 2-node cluster loses write availability when either node is down (neither side is a majority). Three nodes tolerate one failure; five tolerate two.

Sharded clusters fail over per shard — each shard is its own replication group with an independent primary, quorum, leadership epoch, and election. With SKEINDB_CLUSTER_AUTO_FAILOVER=1 the failover tick evaluates every shard as well as the whole cluster: a shard whose primary is down has its most up-to-date online replica auto-promoted through a per-shard vote round (majority of that shard's node set, with the same log-matching guarantee), and a shard primary that loses its shard quorum is write-fenced. Watch every shard's readiness with cluster.shard.failover.status; promote manually with cluster.replica.promote + a shard_id (also quorum-gated). See Configuration → Automated fenced failover.

7. Production notes¶

Put each node on its own host with a fixed --cluster-bind that is reachable from the other nodes.
Rotate the join token regularly: cluster.rotate_join_token.
Use separate networks for --http (applications) and --cluster-bind (inter-node).
See Clustering for shard placement, replication factors, and rolling upgrade procedure.
See Observability to wire Prometheus + Grafana.

Next¶

Clustering reference
Observability
Audit WAL — tamper-evident audit chain across nodes.
CAS replication — dedup-aware replication internals.

Edit this page on GitHub · View source