session active · /dev/quix/04
06:42:18 utc
./present --deck=04 --mode=cinematic

streaming
at the speed of decisions

# Terminal aesthetic — scan lines, CRT vignette, blinking cursor.
audience = "engineers"
tone = "technical, dry-witted, confident"

when stateful stops scaling.

A walk through three failure modes nobody documented because nobody survived the postmortem.

The bottleneck isn't where the metrics say it is.

Dashboards show CPU. The problem is RocksDB compaction. The problem is always RocksDB compaction.

01$ kubectl logs ingest-7d8f --tail=8
02[ok] consumer lag: 82 msgs
03[ok] p50 latency: 18ms
04[warn] compaction backlog: 3.2GB
05[err] rocksdb write stall "L0 too many files"
06--
07$ echo "this is the actual bug"
rocksdb kafka-3.6 k8s page-cache

After the rewrite.

latency.p99
42ms
// -78% vs. v0
cost.per_1m
$0.18
// -64% vs. v0
recovery.sla
7.2s
// -91% vs. v0

if you can't tail it, you can't debug it. if you can't debug it, you can't ship it.

end-of-deck