mirror of
https://github.com/DeBrosOfficial/orama.git
synced 2026-08-06 15:49:26 +00:00
a96b79db6a
Devnet had TURN running on ZERO nodes for namespace anchat-test after a node replacement, with every node reporting failed=0 and six hours of logs containing no matching lines. Four defects, one outage. #173 (root cause) — namespace_cluster_nodes accumulated rows for permanently dead nodes; removeClusterNodeAssignment existed but was never called, and RepairCluster is add-only. With 4 members (2 corpses) the WebRTC reconciler computed 2*live > members => 2*2 > 4 => false: a permanent 50/50 deadlock that could never resolve. Adds pruneStaleClusterNodes, wired into RepairCluster and the 60s reconcile loop, keyed off dns_nodes staleness. Also explains why the ring health monitor never fired: startDNSHeartbeat flips a silent node to inactive at 120s, but getRingNeighbors only probes active nodes, so the node leaves every observer's set before the monitor's own ~120s threshold and its miss count is discarded. The DNS sweep almost always wins that race. The prune is independent of it. Second bug found: an unpruned corpse made RepairCluster count it as active, so a replacement was never triggered. #170 — viable and live member sets came from two separate rqlite queries, so live ⊆ viable was incidental, not structural; combined with webrtcReconcileQuorumOK(live, 0) returning true, an empty viable set passed quorum and deallocated every role while allocating nothing back. Now one merged query split in Go, plus an explicit empty-set guard. #171 — regression in the prior fix: past the grace window both numerator and denominator derive from the same liveness signal, so a lone node always had "quorum" and would strip every other node's roles onto itself. Reachable via our own serial rolling-restart runbook. Adds webrtcReconcileMajorityHeld (viable >= (raw+1)/2) and a 5m startup grace. #172 — shortfall log moved after the plan and conditioned on the viable set, stale invariant comments corrected, grace-boundary and end-to-end tests added. Verified on devnet 0.122.100: TURN 0 -> 2 nodes, SFU 1 -> 3, every allocation on a live node, both corpses gone from namespace_cluster_nodes. 183 tests, go vet and -race clean. Testnet untouched. Build note: vault/build.zig.zon declares minimum_zig_version 0.15.2; Zig 0.16 removed GeneralPurposeAllocator and process.argsAlloc. Build with zig@0.15. Co-Authored-By: Claude <noreply@anthropic.com>