HTTP-native durable event log / message bus — written in C

A single-binary event streaming system where curl is your client. No JVM, no ZooKeeper, no thick client libraries.

▶️ 1-minute demo: SIGKILL → restart → data still there https://www.youtube.com/watch?v=c-n0X5t-A9Y

# Produce
curl -X POST 'localhost:1109/broker/topics/orders/produce?partition=0' \
  -H 'Authorization: Bearer dev' \
  -d '{"item":"widget"}'

# Consume
curl 'localhost:1109/broker/consume/orders/mygroup/0?encoding=b64' \
  -H 'Authorization: Bearer dev'

• Sustained: ~50K msg/s (wrk2 @ 50K req/s)
• Client P99: 3.46ms
• Server P99.999: 1.22ms (handler only)
• Recovery after SIGKILL: 40–50s (8M offsets)

Kafka is battle-tested but operationally heavy. JVM tuning, partition rebalancing, and config sprawl add up.

Redis Streams is simple and fast, but replication is async-only — no majority quorum, no strong durability guarantees.

Ayder sits in the middle: Kafka-grade durability (Raft sync-majority) with Redis-like simplicity (single binary, HTTP API). Think of it as what Nginx did to Apache — same pattern applied to event streaming.

• Append-only logs with per-partition offsets
• Consumer groups with committed offsets
• Durability via sealed append-only files (AOF) + crash recovery
• HA replication with Raft consensus (3 / 5 / 7 node clusters)
• KV store with CAS and TTL
• Stream processing with filters, aggregations, and windowed joins (including cross-format Avro+Protobuf joins)

All benchmarks use real network (not loopback). Numbers are real, not marketing.

Setup:

• 3-node Raft cluster on DigitalOcean (8 vCPU AMD)
• Sync-majority writes (2/3 nodes confirm before ACK)
• 64B payload
• Separate machines, real network

Server-side breakdown at P99.999:

Handler:     1.22ms
Queue wait:  0.47ms
HTTP parse:  0.41ms

The 154ms client-side tail is network/kernel scheduling — the broker itself stays under 2ms even at P99.999. HTTP is not the bottleneck.

Running 1m test @ http://10.114.0.3:8001
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.26ms    2.97ms 224.51ms   93.76%
    Req/Sec     7.86k     1.19k   13.70k    67.61%
  Latency Distribution
     50%    3.78ms
     75%    4.93ms
     90%    6.44ms
     99%   10.22ms
  5634332 requests in 1.00m, 2.99GB read
Requests/sec:  93807.95
Transfer/sec:     50.92MB

Running 1m test @ http://10.114.0.2:9001
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.72ms    1.19ms 216.19ms   96.39%
    Req/Sec     4.35k     1.17k    7.89k    79.58%
  Latency Distribution (HdrHistogram - Recorded Latency)
 50.000%    1.60ms
 75.000%    2.03ms
 90.000%    2.52ms
 99.000%    3.46ms
 99.900%   12.94ms
 99.990%   31.76ms
 99.999%  154.49ms
100.000%  216.32ms

  2991950 requests in 1.00m, 1.80GB read
Requests/sec:  49871.12

SERVER  server_us p99.999=1219us (1.219ms)
SERVER  queue_us p99.999=473us (0.473ms)
SERVER  recv_parse_us p99.999=411us (0.411ms)

Ayder runs natively on ARM64. Here's a benchmark on consumer hardware:

Setup:

• Snapdragon X Elite laptop (1.42 kg)
• WSL2 Ubuntu, 16GB RAM
• Running on battery (unplugged)
• 3-node Raft cluster (same machine — testing code efficiency)
• wrk: 12 threads, 400 connections, 60 seconds

Server-side breakdown at P99.999:

Handler:     0.65ms
Queue wait:  0.29ms
HTTP parse:  0.29ms

Comparison: Snapdragon vs Cloud VMs (Server-side P99.999)

The laptop's server-side latency is 47% faster while handling 14% more throughput — on battery, in WSL2.

What this proves:

• ARM64 is ready for server workloads
• Efficient C code runs beautifully on Snapdragon
• WSL2 overhead is minimal for async I/O
• You can test full HA clusters on your laptop

Running 1m test @ http://172.31.76.127:7001
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.81ms    3.80ms 289.49ms   99.00%
    Req/Sec     8.94k     1.16k   22.81k    80.11%
  Latency Distribution
     50%    3.57ms
     75%    4.01ms
     90%    4.51ms
     99%    7.62ms
  6408525 requests in 1.00m, 3.80GB read
Requests/sec: 106645.65
Transfer/sec:     64.83MB

CLIENT  p99.999=250843us (250.843ms)  max=289485us (289.485ms)
SERVER  server_us p99.999=651us (0.651ms)  max=11964us (11.964ms)
SERVER  queue_us p99.999=285us (0.285ms)  max=3920us (3.920ms)
SERVER  recv_parse_us p99.999=293us (0.293ms)  max=4149us (4.149ms)

Kafka in 2025 is like starting a car with a hand crank. It works, but why are we still doing this?

Tested crash recovery:

# 3-node cluster with 8 million offsets
1. SIGKILL a follower mid-write
2. Leader continues, follower misses offsets
3. Restart follower
4. Follower replays local AOF → asks leader for missing offsets
5. Leader streams missing data → follower catches up
6. Cluster fully healthy in 40-50 seconds
7. Zero data loss

No manual intervention. No partition reassignment. No ISR drama.

# Clone and run with Docker Compose (includes Prometheus + Grafana)
git clone https://github.com/A1darbek/ayder.git
cd ayder
docker compose up -d --build

# Or build and run standalone
docker build -t ayder .
docker run -p 1109:1109 --shm-size=2g ayder

# That's it. Now produce:
curl -X POST localhost:1109/broker/topics \
  -H 'Authorization: Bearer dev' \
  -H 'Content-Type: application/json' \
  -d '{"name":"events","partitions":4}'

curl -X POST 'localhost:1109/broker/topics/events/produce?partition=0' \
  -H 'Authorization: Bearer dev' \
  -d 'hello world'

# Dependencies: libuv 1.51+, openssl, zlib, liburing
make clean && make
./ayder --port 1109

The included docker-compose.yml brings up:

• Ayder on port 1109
• Prometheus on port 9090 (metrics scraping)
• Grafana on port 3000 (dashboards, default password: admin)

# docker-compose.yml
services:
  ayder:
    build: .
    ports:
      - "1109:1109"
    shm_size: 2g
    environment:
      - RF_BEARER_TOKENS=dev

# Create a topic
curl -X POST localhost:1109/broker/topics \
  -H 'Authorization: Bearer dev' \
  -H 'Content-Type: application/json' \
  -d '{"name":"events","partitions":8}'

# Produce a message
curl -X POST 'localhost:1109/broker/topics/events/produce?partition=0' \
  -H 'Authorization: Bearer dev' \
  -d 'hello world'

# Consume messages (binary-safe with base64)
curl 'localhost:1109/broker/consume/events/mygroup/0?limit=10&encoding=b64' \
  -H 'Authorization: Bearer dev'

# Commit offset
curl -X POST localhost:1109/broker/commit \
  -H 'Authorization: Bearer dev' \
  -H 'Content-Type: application/json' \
  -d '{"topic":"events","group":"mygroup","partition":0,"offset":10}'

A topic contains N partitions. Each partition is an independent append-only log with its own offset sequence.

Consumers read from /broker/consume/{topic}/{group}/{partition}. Progress is tracked per (topic, group, partition) tuple via explicit commits.

If you consume without specifying ?offset=, Ayder resumes from the last committed offset for that consumer group.

Ayder acknowledges writes in two modes:

Use timeout_ms to wait for sync confirmation.

GET  /health      # → {"ok":true}
GET  /ready       # → {"ready":true}
GET  /metrics     # → Prometheus format
GET  /metrics_ha  # → HA cluster metrics

Create topic

POST /broker/topics
{"name":"events","partitions":8}

Response:

{"ok":true,"topic":"events","partitions":8}

Single message (raw bytes in body)

POST /broker/topics/{topic}/produce

Query parameters:

Response:

{
  "ok": true,
  "offset": 123,
  "partition": 0,
  "batch_id": 9991,
  "sealed": true,
  "durable": true,
  "mode": "sealed",
  "synced": true
}

Duplicate detection (when idempotency_key matches):

{"ok":true,"offset":123,"partition":0,"sealed":true,"synced":null,"duplicate":true}

Batch produce (NDJSON — one message per line)

POST /broker/topics/{topic}/produce-ndjson

Response:

{
  "ok": true,
  "first_offset": 1000,
  "count": 250,
  "partition": 0,
  "batch_id": 424242,
  "sealed": true,
  "durable": true,
  "mode": "sealed",
  "synced": false
}

GET /broker/consume/{topic}/{group}/{partition}

Query parameters:

Response:

{
  "messages": [
    {"offset": 0, "partition": 0, "value_b64": "aGVsbG8=", "key_b64": "a2V5"}
  ],
  "count": 1,
  "next_offset": 1,
  "committed_offset": 0,
  "truncated": false
}

Use next_offset as the ?offset= parameter for subsequent reads.

POST /broker/commit
{"topic":"events","group":"g1","partition":0,"offset":124}

Response:

Commits are stored per (topic, group, partition). Backward commits are ignored.

Delete before offset (hard floor)

POST /broker/delete-before
{"topic":"events","partition":0,"before_offset":100000}

Response:

{"ok":true,"deleted_count":12345,"freed_bytes":987654}

Set retention policy

Examples:

// TTL + size cap for specific partition
{"topic":"events","partition":0,"ttl_ms":60000,"max_bytes":104857600}

// TTL for all topics
{"topic":"*","ttl_ms":300000}

Ayder includes a key-value store with CAS (compare-and-swap) and TTL support.

Put

POST /kv/{namespace}/{key}?cas=<u64>&ttl_ms=<u64>

Body contains raw value bytes.

Response:

{"ok":true,"cas":2,"sealed":true,"durable":true,"mode":"sealed","synced":true,"batch_id":123}

Get

GET /kv/{namespace}/{key}

Response:

{"value":"<base64>","cas":2}

Get metadata

GET /kv/{namespace}/{key}/meta

Response:

Delete

DELETE /kv/{namespace}/{key}?cas=<u64>

Response:

{"ok":true,"deleted":true,"sealed":true,"durable":true,"mode":"sealed","synced":false,"batch_id":456}

Built-in stream processing — no separate service required.

Consume JSON objects from a topic/partition with:

• Row filtering (eq, ne, lt, gt, in, contains)
• group_by with aggregations (count, sum, avg, min, max)
• Field projection
• Tumbling windows

Windowed join between two sources:

• Join types: inner / left / right / full
• Composite keys
• Window size and allowed lateness
• Optional dedupe_once
• Cross-format support (Avro + Protobuf in same join)

Ayder supports 3, 5, or 7 node clusters with Raft-based replication.

Writes must go to the leader. If you send a write to a follower, it returns an HTTP redirect with the leader's address in the Location header.

Options:

1. Follow redirects automatically
2. Discover the leader via /metrics_ha and pin writes to it

When a follower rejoins after downtime:

1. Replays local AOF
2. Connects to leader
3. Requests missing offsets
4. Leader streams missing data
5. Follower catches up automatically

Example scenario:

# 3-node cluster
node1 (7001) = LEADER
node2 (8001) = FOLLOWER
node3 (9001) = FOLLOWER

# Write to leader
curl -X POST 'localhost:7001/broker/topics/test/produce?partition=0' \
  -H 'Authorization: Bearer dev' -d 'msg-0'
# → offset 0

# Kill node2
kill -9 $(pgrep -f "port 8001")

# Write while node2 is down
curl -X POST 'localhost:7001/broker/topics/test/produce?partition=0' \
  -H 'Authorization: Bearer dev' -d 'msg-1'
# → offset 1

curl -X POST 'localhost:7001/broker/topics/test/produce?partition=0' \
  -H 'Authorization: Bearer dev' -d 'msg-2'
# → offset 2

# Restart node2 — automatically catches up to offset 2

# Verify all data is present on recovered node
curl 'localhost:8001/broker/consume/test/g1/0?offset=0&limit=10' \
  -H 'Authorization: Bearer dev'
# → offsets 0, 1, 2 all present

# Default port is 1109
./ayder --port 1109

# Or specify custom port
./ayder --port 7001

Ayder uses Raft for consensus. Here's a complete 3-node setup:

Environment variables:

3-Node Example:

# Node 1 (bootstrap leader)
export RF_HA_ENABLED=1
export RF_HA_NODE_ID=node1
export RF_HA_BOOTSTRAP_LEADER=1
export RF_HA_NODES='node1:10.0.0.1:7000:100,node2:10.0.0.2:8000:50,node3:10.0.0.3:9000:25'
export RF_HA_WRITE_CONCERN=2  # sync-majority (2/3 nodes)
export RF_HA_DEDICATED_WORKER=0  # critical for low P99
export RF_BEARER_TOKENS='dev@scope:token2:token3'
export RF_HA_TLS=1
export RF_HA_TLS_CA=./certs/ca.crt
export RF_HA_TLS_CERT=./certs/node1.crt
export RF_HA_TLS_KEY=./certs/node1.key
./ayder --port 7001

# Node 2
export RF_HA_ENABLED=1
export RF_HA_NODE_ID=node2
export RF_HA_NODES='node1:10.0.0.1:7000:100,node2:10.0.0.2:8000:50,node3:10.0.0.3:9000:25'
export RF_HA_WRITE_CONCERN=2
export RF_HA_DEDICATED_WORKER=0
export RF_BEARER_TOKENS='dev@scope:token2:token3'
export RF_HA_TLS=1
export RF_HA_TLS_CA=./certs/ca.crt
export RF_HA_TLS_CERT=./certs/node2.crt
export RF_HA_TLS_KEY=./certs/node2.key
./ayder --port 8001

# Node 3 (same pattern, port 9001)

5-Node and 7-Node Clusters:

# 5-node topology
export RF_HA_NODES='node1:host1:7000:100,node2:host2:8000:80,node3:host3:9000:60,node4:host4:10000:40,node5:host5:11000:20'
export RF_HA_WRITE_CONCERN=3  # majority of 5

# 7-node topology
export RF_HA_NODES='node1:host1:7000:100,node2:host2:8000:90,node3:host3:9000:80,node4:host4:10000:70,node5:host5:11000:60,node6:host6:12000:50,node7:host7:13000:40'
export RF_HA_WRITE_CONCERN=4  # majority of 7

Generate TLS certificates:

# Create CA
openssl req -x509 -newkey rsa:4096 -keyout ca.key -out ca.crt \
  -days 365 -nodes -subj "/CN=ayder-ca"

# Create node certificate (repeat for each node)
openssl req -newkey rsa:2048 -nodes -keyout node1.key -out node1.csr \
  -subj "/CN=node1" -addext "subjectAltName=DNS:node1,IP:10.0.0.1"

openssl x509 -req -in node1.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out node1.crt -days 365 -copy_extensions copy

Write concern tradeoffs:

Built by Aydarbek Romanuly — solo founder from Kazakhstan 🇰🇿

Errors follow a consistent format:

{
  "ok": false,
  "error": "missing_topic",
  "message": "Topic name is required",
  "docs": "https://ayder.dev/docs/api/produce"
}

✅ HTTP-native event log with partitions and offsets
✅ Fast writes with cursor-based consumption
✅ Durable with crash recovery
✅ Horizontally scalable with Raft replication
✅ Built-in stream processing with cross-format joins
✅ ARM64-native (tested on Snapdragon X Elite)

❌ Kafka protocol compatible
❌ A SQL database
❌ Magic exactly-once without client-side idempotency discipline

MIT