The ntfy fan-out (publish each push to every active push node so a
round-robin-DNS-pinned subscriber receives it) was coded but INERT: the
gateway's cfg.NtfyBaseURL was never populated, so the fan-out resolver was
never built and pushes went single-host (the ~87% loss the bug describes).
The orchestrator already derives https://push.<dnsZone> for the ntfy server +
Caddy reverse-proxy but never put it in node.yaml's http_gateway. Same
regression class as the v0.122.42 secrets_encryption_key fix (consumer
landed; template + parse field + node->gateway mapping were missed).
Plumb it through all four layers: render it under http_gateway (derived as
push.<dnsZone>, matching the ntfy host), parse it in HTTPGatewayConfig, map
it onto gateway.Config. Rolling-upgrade safe: Phase 4 regen runs under the
new binary (post-swap), so an old binary never reads a node.yaml with the new
field. DecodeStrict regression guard added (mirrors secrets_encryption_key).
caddy/coredns declare Requires=orama-node.service, so stopping orama-node
cascade-stops them; Requires propagates STOP but not START, and
StartServicesOrdered only starts the orama services — so a bare
'orama node restart' left caddy dead and the node's :443 HTTPS frontend
offline until the next reboot. Capture the active frontend units before the
stop and restart exactly those after the gateway is healthy. Adds
ServiceUnitExists helper + selectFrontendToRestore policy test.
APIKeyToJWTHandler looked up the namespace by the raw api key, but keys are
stored HMAC-hashed (Service.HashAPIKey), so it always returned 'invalid API
key' for real keys — no api-key holder could exchange for a JWT. Resolve the
hashed key first with a raw-key fallback for legacy rows, mirroring the
gateway middleware's lookupAPIKeyNamespace. Adds args-aware mock + tests for
the hashed-key, raw-fallback, and unknown-key paths.
The base64 wrapper wasn't enough: DBSecretsManager scanned encrypted_value
into []byte, so the rqlite client applied base64 binary semantics on read and
the ciphertext never round-tripped — get_secret stayed empty. Mirror the
proven push-credentials store exactly: encrypt to a 'enc:'-prefixed base64
string via pkg/secrets and scan the column into a STRING for Decrypt. Text
round-trips cleanly through rqlite regardless of the BLOB column.
Second #837 root cause (the key-derivation fix was necessary but not
sufficient): DBSecretsManager stored the AES-GCM ciphertext as a raw []byte
parameter, but the rqlite client serializes []byte as base64 and reads it
back as that base64 TEXT — never the original bytes. So decrypt() always
received base64 ASCII instead of ciphertext and failed, making get_secret
return empty for every stored secret (exactly the reported symptom).
Encode the ciphertext to an explicit base64 string in Set and decode it in
Get (with a raw fallback), making the round-trip symmetric and
driver-independent. The test mock now emulates rqlite's blob->base64-text
behavior so it's a real regression guard.
The namespace-ownership middleware compared an api_key caller's RAW key
against namespace_ownership.owner_id, but api_keys are stored HMAC-hashed
(HashAPIKey). So every api_key-authenticated owner got a 403 on a namespace
they actually own — blocking function deploy and PUT /v1/push/config.
Hash the presented api_key before the ownership comparison (hashed first,
raw second as a rolling-upgrade legacy fallback), mirroring the existing
lookupAPIKeyNamespace pattern. The wallet path is unchanged (wallets stored
raw). Security-reviewed: grants only to the correct key holder, no
escalation.
The 5-10s RPCs that broke calling were not cold-start — they were
per-RPC sequential rqlite reads, each forwarded to a raft leader that
geography-blind election had placed on a 256ms-distant node.
Lever A (serverless): cache function metadata + env vars in-process
(5s TTL, invalidated on deploy/enable/disable/delete) and stop the hot
invoke path re-fetching the function for the authorization check —
removes ~820ms of leader-routed pre-flight reads from every op.
Lever B (namespace): a locality-aware leadership reconciler hands raft
leadership off a geographically-isolated namespace leader to the nearest
co-located voter, via rqlite's transfer-leadership API. All nodes stay
voters — membership, quorum and fault tolerance are unchanged. Cuts the
per-hop cost from ~274ms to ~20ms when a distant node had become leader.
- Add `ntfyFanoutResolver` to distribute push notifications across all active cluster nodes, ensuring delivery when nodes lack shared state.
- Refactor secrets encryption key derivation to use cluster-wide secrets via HKDF, replacing ephemeral per-node keys to fix cross-node decryption issues.
- Add unit tests for fan-out resolution logic and caching behavior.
Root cause of the recurring "turn.credentials → namespace_not_configured" on a
distant node: at converge the gateway resolves its TURN secret from the
namespace rqlite, and on a slow/just-restarted node that read fails ONCE, so
the gateway is written with TURN disabled. Removing the node is not a fix — the
software must tolerate a slow read.
Two-part fix (complements e7ed718's "don't blank a warm config"):
- RETRY the secret read (5×2s) at converge so a node whose rqlite is still
syncing waits for it to land instead of writing an empty block once. A
genuine decrypt failure still exhausts the retries → unresolved → the
running config is preserved.
- CACHE the resolved secret into the node's own cluster-state.json
(applyResolvedWebRTCToState), so the NEXT cold start reads it from disk —
chooseRestoreWebRTC is state-first and short-circuits before the DB. The
state struct already had TURNSharedSecret "for cold start" but nothing
populated it; now it's filled on every successful resolve (only rewritten
on change). Each node self-heals its own cache; nothing new is sent
cross-node.
cluster-state.json now carries the TURN secret, so both writers (local
saveLocalState and the remote SaveClusterState) are tightened to 0600 + chmod.
Stale-secret self-heals: disable/enable webrtc re-pushes every node's config
and the next converge re-caches the new value.
Dual-reviewed: code-quality APPROVED; security SECURE after the remote-write
0600 fix. Tests: cache populate + short-circuit, no-change, turn-only node.
VoIP call-invite pushes set no apns-expiration, so apns2 omits the header and
APNs store-and-forwards the push — delivering it minutes late and firing a
phantom "missed call" ring long after the call ended (and burning PushKit
goodwill, inviting throttling). Cap the VoIP apns-expiration to the ring window
(30s) so APNs delivers promptly or DISCARDS, never a stale invite. Alert pushes
keep the default store-and-forward so a message notification still lands after
the device reconnects.
Also surface HTTP 200 on a successful dispatch instead of leaving HTTPStatus at
0 — a successful push was logging "http=0", which reads like an opaque failure
and masked real false-success classes.
Tests: VoIP push carries an expiration within the ring-window cap; alert push
carries none. push package green.
A lost rotation response strands the client on a just-revoked token: the retry
hits res.Count==0 → genuine 401 → SIWE, which is impossible on a VoIP-woken
locked screen, so the call dies. This recurred under the reconnect storms from
today's gateway rolls.
Add an RFC 9700 §4.13.2 reuse grace: a refresh token revoked within 60s whose
grace_used_at is still NULL is accepted ONCE more and mints a fresh session.
The grace path skips the revoke CAS (the token is already revoked — the CAS
would 0-match and mis-fire the replay tripwire) and is locked instead by a
single-use CAS on grace_used_at, so a stolen token can't be replayed at
leisure. The window predicate is repeated on the CAS to close the
SELECT→UPDATE TOCTOU, and the grace SELECT excludes expired tokens.
Security (found + fixed in review): explicit revocation (RevokeToken /
/v1/auth/logout) now also stamps grace_used_at, so a deliberately-logged-out
token can never be grace-recovered — closes a logout-bypass where a just-
revoked token would otherwise be resurrectable for 60s. Transient rqlite
errors on the grace lookup/CAS surface as 503 (retryable), not 401, preserving
the #125 transient-vs-genuine distinction.
Migration 032 adds grace_used_at (additive ALTER, rolling-safe; NULL = grace
available, the window predicate keeps historically-revoked tokens ineligible).
Dual-reviewed: code-quality APPROVED; security SECURE after the logout-bypass
fix. Tests: lost-response recovery, single-use second-attempt 401, genuine bad
token 401, and the logout-bypass regression.
A freshly-joined namespace node could come up with TURN silently disabled
(turn.credentials -> namespace_not_configured) when GetWebRTCConfig errored:
the stored TURN shared secret was encrypted with a pre-rotation
cluster-secret-derived key and failed to decrypt, and the converge swallowed
that error into "WebRTC disabled", writing a TURN-disabled gateway config.
Distinguish "resolved but not enabled" (genuinely disabled, fine to write the
empty block) from "unresolved" (DB/decrypt error). chooseRestoreWebRTC's
dbFetch callback now returns a `resolved` bool; an unresolved lookup forces
enabled=false AND sets restoreWebRTC.unresolved. The converge then:
- logs the decrypt/read error loudly with the exact remediation
(`orama namespace disable webrtc` then `enable webrtc`) instead of
swallowing it;
- on the warm path, SKIPS ReconcileGateway so it doesn't rewrite a running
gateway's working WebRTC block to empty (preserves TURN);
- on the cold path, still spawns the gateway (the namespace needs one) but
warns loudly that TURN is degraded until the secret is regenerated.
Healthy nodes are unaffected: a node whose state file holds the secret
short-circuits before dbFetch, so a flaky/rotated DB cannot disable it.
Dual-reviewed (code-quality APPROVED, security SECURE — no secret material is
logged; operator disable still resolves to disabled, not unresolved).
Pure-function coverage in restore_webrtc_test.go: unresolved marking,
resolved-empty-is-disabled, and state-secret-wins-over-db-error.
- implement `wsJWTExpired` to validate token lifetime with a grace period
- capture jwt expiry at connection upgrade and update via auth.refresh
- close connections with custom code 4401 when tokens expire to force re-auth
- add unit tests to verify expiry logic and state transitions
Custom JWT claims survive token refresh: migration 031 adds the
custom-claims column to refresh tokens, the new gateway ClaimsProvider
re-resolves claims on refresh, and the serverless invoke path carries
them through. Includes refresh-rotation, WS-JWT middleware, and
claims-provider test coverage.
- refactor(turn): extract decodeTURNConfig for testability
- feat(turn): add stealth domain fields to config
- fix(apns): nest custom data under "body" for expo-notifications compatibility
- Implement `resolveStealthCert` to use existing `*.<baseDomain>` wildcard certificates instead of dynamic Caddyfile provisioning.
- Avoids EROFS errors caused by `ProtectSystem=strict` on the orama-node service.
- Add strict validation to ensure stealth hosts are single-label subdomains covered by the wildcard.
b9d5f54 (stealth TURN discovery) emits a top-level `sni_router:` block
into node.yaml unconditionally, but only added a lenient ad-hoc parse
in the carry-forward logic — not the field on config.Config that
orama-node strict-decodes (KnownFields(true)) at boot. Identical
failure mode to the v0.122.42 secrets_encryption_key incident: the
unknown key fails the whole node.yaml parse and orama-node crash-loops.
Caught pre-deploy this time by the strict-decode gate check; devnet
never saw it. Regression test added alongside the v0.122.42 one in
decode_test.go.
- Add `turn_stealth_domain` to gateway config for stealth TURN support
- Introduce `turn_discovery` in `sni-router` to auto-discover per-namespace routes
- Add database migration to enable stealth TURN per namespace
- Document ephemeral state API in `SERVERLESS.md`
- add `secrets_encryption_key` to gateway config for serverless secrets
- implement durable TURN secret persistence to prevent config regen outages
- add regression test for gateway config loading and field mapping
v0.122.42 (f412425, secrets encryption) shipped the template emission,
the per-cluster secret generator, and the gateway.Config consumer — but
NOT the parse field on config.HTTPGatewayConfig. Phase 4 writes
`secrets_encryption_key` into node.yaml under the http_gateway section,
and pkg/config/yaml.go decodes with KnownFields(true) (strict). The
unknown field made every node.yaml parse fail, so orama-node exited 1
on every start and systemd crash-looped it (restart counter hit 380+ on
the first upgraded devnet node before the rolling controller halted).
Root cause: a generated-config field with no matching struct field under
strict unmarshal. Fix is the missing field. The runtime key itself is
still consumed from ~/.orama/secrets/secrets-encryption-key (pkg/node/
gateway.go), which already worked — so this one-field addition fully
restores boot AND the feature.
The standalone gateway (cmd/gateway/config.go) uses lenient parsing and
was unaffected.
Regression test in pkg/config/decode_test.go decodes a node.yaml
carrying secrets_encryption_key under strict mode.
- Add `raw_http_response` configuration to functions to allow verbatim HTTP responses
- Implement cluster-wide secrets encryption key generation and distribution for serverless functions
- Update documentation with UnifiedPush support for ntfy on Android/GrapheneOS
- Add `FileRouteReloader` to watch and atomically update routes from disk
- Refactor `main` to support seamless configuration updates without restarts
- Ensure existing routes are preserved if a reload encounters an error
- Remove the 2-second polling wait for gossipsub mesh formation in `Publish`
to eliminate unnecessary latency, relying on `FloodPublish` for delivery.
- Introduce a per-invocation publish budget (1000 messages) to prevent
potential flooding of the shared gossipsub router by WASM functions.
- Add regression tests to ensure `Publish` remains non-blocking and that
the publish budget is strictly enforced.
- Introduce `BatchQueryConsistency` with `ReadConsistencyNone` to allow
local SQLite reads, bypassing leader round-trips for performance.
- Add `function_invoke_async` host function to support non-blocking
fire-and-forget function execution.
- split webrtc route gating into `webrtcServeTURNCredentials` and `webrtcServeSFURoutes` to allow non-SFU gateways to mint TURN credentials
- update `chooseRestoreWebRTC` to correctly resolve configurations for nodes without local SFU ports
- add unit tests to verify independent route registration logic (bugboard #25)
- Add logic to reconcile gateway configuration drift for running instances
- Prevent unnecessary restart loops by verifying on-disk config state
- Add unit tests to validate synchronization logic and prevent regressions
- Add internal WebRTC management endpoints to public path exemption list
- Implement DB fallback for WebRTC configuration during cluster restore
- Add unit tests to verify WebRTC config precedence and state self-healing
- opt into `WithSysWalltime` and `WithSysNanotime` to prevent wazero from using a frozen sentinel clock
- add regression tests to verify real-time clock behavior in wasm execution
- ensure serverless functions receive accurate timestamps for audit and cursor logic
- Add `enable` and `disable` commands to manage function status
- Implement process re-exec in the upgrade orchestrator to ensure
Phase 4 config generation uses the newly-installed binary version
(fixes bugboard #15)
- wire PubSubDispatcher to host functions to support local wildcard
triggers for WASM-published topics
- implement batch deduplication by topic to prevent redundant trigger
invocations and bound fan-out
- propagate trigger depth through function invocations to maintain
recursion limits during local dispatch
- Update route registration logic to rely solely on SFUPort > 0, resolving a silent 404 issue where gateways with valid SFU configurations were incorrectly disabled.
- Retain WebRTCEnabled in config for backward compatibility with existing operator YAML and request schemas.
- Add unit tests to pin registration behavior and prevent future regressions.
- register "apns_voip" provider to handle PushKit/CallKit signals
- implement target provider filtering in dispatcher to prevent cross-talk
between alert and VoIP push paths
- add comprehensive tests to ensure backward compatibility for fan-out
and correct filtering behavior
- Attach InvocationContext to the execution context in Engine.Execute to
ensure host functions resolve identity from the request context.
- Fixes a race condition where concurrent stateless invocations would
overwrite the global singleton, causing cross-tenant leaks or nil
namespace errors.
- Added a regression test to verify per-invocation isolation under load.
#348 - APNs silent-drop guard
Apple's APNs silently returns HTTP 200 for pushes with no visible
content (no title, no body, no badge, no sound, no
content-available=1) and then drops them — which looked to the WASM
caller like a successful delivery. Now rejected up-front with the new
push.ErrEmptyContent sentinel, and the APNs provider returns the
structured push.PushError shape (HTTPStatus, Reason, Unregistered,
Wrapped) so the dispatcher can branch on Unregistered to remove dead
tokens automatically. Legacy ErrDeviceUnregistered sentinel is
preserved for errors.Is compatibility (wrapped inside PushError).
Always logs APNs HTTP response (status, reason, apns_id, token prefix)
so future silent-drop classes show up in operator logs.
content-available is also now correctly mapped from snake_case
Data["content_available"] (any truthy variant) into Apple's
canonical "content-available": 1 inside the aps dictionary.
#321 - mid-session JWT refresh on persistent WS
Long-lived persistent WS connections used to have to close+reconnect
when the JWT rolled — losing per-instance state, message queues, and
subscriptions. The handler now accepts an "auth.refresh" control
frame: client sends the new token, the gateway re-verifies it via
the new JWTVerifier interface, updates the per-instance invCtx
in-place (persistent.Instance.UpdateInvCtx), and acks. No close, no
state loss.
JWTVerifier is optional — handlers set it via SetJWTVerifier at
gateway init. When unwired the handler nack's with a "not supported
on this gateway" response and clients fall back to the old
close+reconnect path, so older deploys don't break.
Other:
- push/dispatcher.go: SendToUserDetailed returns per-device PushError
shape so callers can act on Unregistered / HTTPStatus / Reason.
- serverless/hostfunctions/push.go: WASM host functions for the new
detailed-error shape.
- serverless/persistent/instance.go: UpdateInvCtx mid-session.
Tests:
- ws_persistent_control_test.go: auth.refresh ack/nack paths.
- apns_test.go: empty-content rejection, PushError shape on 410 +
generic non-200, content-available mapping.
- dispatcher_detailed_test.go: SendToUserDetailed result shape.
- instance_update_invctx_test.go: invCtx update is per-instance, not
cross-tenant.
VERSION bumped to 0.122.27.
- Integrate PubSubDispatcher to enable libp2p subscription for trigger patterns
- Add BatchQuery to rqlite client to reduce round-trips for multi-query operations
- Implement lifecycle management for dispatcher and add safety limits for batch queries
Two serious bugs found via cross-node behavior observation:
1. libp2p peer-discovery published wrong port
PeerDiscovery's multiaddr was using the gateway's HTTP API port (e.g.
10004), not the actual libp2p TCP port. Remote gateways dialed that
port, hit the HTTP server, received 400, and failed the libp2p
multistream handshake ("message did not have trailing newline").
Result: cluster-wide cross-node libp2p mesh had 0 connected peers
and cross-node pubsub silently dropped 100% of messages.
The libp2p port is OS-assigned at startup (client.go uses
/ip4/0.0.0.0/tcp/0). It's not anywhere in cfg — it's only on
host.Addrs(). Fix: drop the listenPort field from PeerDiscovery
entirely and derive the port live from host.Addrs() via
extractLibp2pTCPPort. WG IP still comes from getWireGuardIP
(libp2p filters its own enumeration so WG IPs don't appear in
host.Addrs(), but the listener is bound 0.0.0.0 so the port is
reachable on the WG interface).
2. System triggers silently blocked by CanInvoke (#264)
Cron, pubsub, database, timer, and job triggers all fire from
gateway-internal state with no caller identity. Invoke() ran every
request through CanInvoke(callerWallet) which returned false for
the empty wallet — every fire returned ErrUnauthorized. Reported as
a cron firing every minute with "unauthorized" for 19+ hours.
Auth boundary for system triggers belongs at REGISTRATION time
(POST /v1/functions/{name}/triggers, deploy-time auto-register
from function.yaml). Skip the per-invocation check for system
trigger types; user-driven triggers (HTTP, WebSocket) still gate
on caller identity as before.
Tests:
- gateway/peer_discovery_test.go covers extractLibp2pTCPPort.
- serverless/invoke_system_trigger_test.go covers the bypass and the
user-trigger gate.
VERSION bumped to 0.122.25.
HostFunctions is a process-wide singleton (one per gateway engine).
Its `invCtx` field is shared across all WASM instances. For STATELESS
execution the executor sets/clears it per-call but the lock is
released before WASM runs — two concurrent invocations can race on
the field and one's host call can read the other's identity. Window
is microseconds.
For PERSISTENT WS the bug was much worse: invCtx used to be bound
ONCE at instantiation and reused for the connection's lifetime. Two
simultaneous persistent WS connections from different namespaces /
wallets overwrote each other's invCtx, and EVERY subsequent
function_invoke / GetCallerJWTSubject / GetCallerWallet / GetSecret
call from inside the WASM read whatever was bound LAST. Result:
silent identity leak across tenants for as long as the connections
overlapped.
Fix: per-call invCtx propagation through Go's context.Context.
wazero passes the ctx given to api.Function.Call through to host
function callbacks, so every WASM-host hop carries its own invCtx.
- pkg/serverless/invocation_context.go (new): WithInvocationContext +
InvocationContextFromCtx helpers using an unexported invCtxKey.
- pkg/serverless/hostfunctions/invocation_context.go (new):
currentInvocationContext(ctx) — ctx-attached invCtx wins over the
singleton field.
- All host accessors (FunctionInvoke, GetEnv, GetSecret, GetRequestID,
GetCallerWallet, GetWSClientID, GetCallerClaim, GetCallerJWTSubject)
now route through currentInvocationContext(ctx).
- pkg/serverless/persistent/instance.go: every export call's ctx is
wrapped with the per-instance invCtx before being passed to wazero.
- pkg/gateway/handlers/serverless/ws_persistent_handler.go: invCtx is
built per-frame and attached to ctx, not stored on a shared field.
- pkg/serverless/engine.go: removed the SetInvocationContext call at
InstantiatePersistent (no longer needed; ctx carries it).
Stateless still uses the singleton field — its race is latent since
the host-functions split and migrating it is a separate scoped
change.
Tests:
- hostfunctions/invocation_context_test.go covers ctx-wins-over-singleton.
- gateway/handlers/serverless/ws_persistent_handler_test.go covers the
per-frame ctx wiring.
- cli/functions/build_test.go is new coverage for the build path
touched in this change.
VERSION bumped to 0.122.24.
The previous fix (v0.122.22) made `InstantiatePersistent` call `_start`
to bootstrap TinyGo's runtime, then catch the resulting ExitError(0).
That got past init, but the module STILL died — wazero's stock
`proc_exit` implementation calls `mod.CloseWithExitCode(exitCode)`
before panicking, which invalidates the module regardless of what
the caller does with the panic. Every subsequent call to ws_open /
ws_frame / ws_close / orama_alloc returned ExitError(0) ("module
already closed").
Wazero exposes no flag for this — the close is hard-coded. The only
intercept point is to override `proc_exit` at the WASI host-module
boundary. Documented pattern at imports/wasi_snapshot_preview1/wasi.go
lines 111-127.
Fix: build the WASI host module manually so we can override
`proc_exit`:
- exit code 0 → panic ExitError(0) BUT do NOT close the module.
This is TinyGo's "_start completed cleanly" signal; the module's
other exports must stay callable for the persistent lifecycle.
- exit code != 0 → preserve standard WASI behavior (close + panic).
A non-zero exit is a genuine app-signaled failure; we want
`proc_exit(N != 0)` to behave exactly as upstream does.
The InstantiatePersistent caller already distinguishes the two cases
via errors.As + ExitCode() check — added in v0.122.22, no change here.
Safe for stateless functions on the same runtime: the stateless
execution path closes its own module after each invocation, so the
"module stays alive on exit 0" override has no effect on that path.
VERSION bumped to 0.122.23.
InstantiatePersistent passed WithStartFunctions() with no args,
explicitly disabling both wasi entry points. The intent was to skip
main(); the side effect was leaving the TinyGo runtime
uninitialized. The first call to any export traps via
wasmExportCheckRun and managed-memory ops panic. Every persistent WS
function was effectively dead since plan #06 landed.
Earlier patch in this thread restored the call but only handled
wasi-reactor builds (_initialize). AnChat's rpc-router is a wasi
command build (`_start` export only, no `_initialize`) — wasm-objdump
confirms — so the reactor-only fix still left it broken.
This fix tries `_initialize` first, falls back to `_start`, and
bounds whichever runs with a 5s timeout so a buggy main() can't hang
instantiation forever. Logs the chosen hook at Debug, warns when
neither is exported.
Still pass WithStartFunctions() (no args) so wazero doesn't
auto-call `_start` during InstantiateModule — we want full control
over which hook runs and the timeout that bounds it.
VERSION bumped to 0.122.22.
Register() writes the four ws_* columns (ws_persistent,
ws_idle_timeout_sec, ws_max_frame_bytes, ws_max_inflight_per_conn) to
the functions table, but every read path — Get, List, GetByID,
GetByNameInternal — silently dropped them from the SELECT. functionRow
had no fields for them either. Result: fn.WSPersistent was always the
zero value (false) at runtime, no matter what the DB row said. Every
WS function ran in per-frame stateless mode regardless of its
`ws_persistent: true` config.
AnChat's rpc-router was the canary: it relies on per-connection
instance state (request_id ↔ reply correlation, subscription
bookkeeping) that the stateless model destroys every frame. The
gateway telemetry envelope still reached the client
({request_id, status, duration_ms}) so the failure looked like
"function works, frames don't" — every RPC timed out at 15 s.
Fix: include the four columns in every SELECT, add the matching
functionRow fields, and copy them into Function in rowToFunction.
No schema change (columns have been in migration 011 from the start).
Regression tests in registry_ws_columns_test.go cover the Get / List
paths against an in-memory SQLite that mirrors the production DDL.
VERSION bumped to 0.122.21.
handleNamespaceGatewayRequest rewrites r.Host to the backend target
IP:port (e.g. "10.0.0.6:10004") before forwarding. The original
public host (e.g. "ns-anchat-test.orama-devnet.network") is preserved
in X-Forwarded-Host. checkWSOrigin in both pubsub/ws_client.go and
serverless/ws_handler.go was comparing the client's Origin against
the proxied r.Host only — so every browser / RN-iOS WS upgrade was
rejected 403 because their Origin's public hostname can never match
10.0.0.6.
curl probes don't send Origin, so curl returned true unconditionally
and the bug was invisible to operator smoke tests. AnChat's iPhone
WS clients hit `code=1006 reason="Received bad response code from
server: 403"` for ~24h.
Fix: prefer X-Forwarded-Host (the original public host) when present,
fall back to r.Host for direct (non-proxied) connections. Applied
identically to both WS handlers. Regression test in
serverless/ws_origin_test.go covers the proxy-hop case, no-Origin
case, and direct-connection case.
This is the real fix; v0.122.19 only closed a separate silent-forward
auth hole that produced opaque 401s on a different code path.
VERSION bumped to 0.122.20.
Root-cause hardening for bug #240 and #249's "intermittent 401 over WS"
reports. handleNamespaceGatewayRequest previously had a third code
path beyond "auth ok" and "auth error": when validateAuthForNamespaceProxy
returned empty namespace AND empty error (i.e. "no credentials found"),
the request fell through to a silent forward to the namespace gateway
WITHOUT internal-auth headers. The namespace gateway then rejected
with 401 "missing API key" in ~60µs.
From the client's perspective: opaque 401.
From our side: only the namespace gateway logged it, and that tier
can't validate API keys (they live in the main cluster RQLite), so
the operator had no signal that the main gateway had even seen the
request. AnChat's intermittent 401-on-WS reports went unsolved for
this exact reason.
Fix:
- Explicit reject at main when no credentials extracted AND path
isn't public. Returns 401 with WWW-Authenticate: Bearer realm and a
clear message naming the three accepted credential sources.
- Rich structured logging on every WS upgrade auth outcome: presence
of api_key/token/jwt query params, Authorization + X-API-Key
headers, Connection/Upgrade headers, Origin, User-Agent, client IP,
raw query length. Steady-state stays low-noise: success path logs at
debug, reject paths log at warn.
- Namespace-mismatch reject (existing branch) now also logs.
VERSION bumped to 0.122.19.
Upstream publishes the checksums asset as a plain "checksums.txt" at
the release root, not "ntfy_<VER>_checksums.txt". The version-prefixed
URL we were constructing 404'd, so InstallNtfy bailed in the
download-binary step and ntfy never landed even after we wired
InstallNtfy into the pre-built install path.
Verified against the v2.11.0 release assets list. If a future version
changes the naming convention, the install will 404 loud and this URL
gets bumped in the same PR as ntfyVersion.
VERSION bumped to 0.122.18.
Phase 2b auto-detects pre-built archive mode and routes to
installFromPreBuilt(). That path copies bundled binaries (caddy, orama,
gateway, …) into place but never called InstallNtfy() — because ntfy
is downloaded from upstream github, not bundled. Result: on devnet
(which always uses pre-built mode), ntfy never installed even though
the always-on code path in installFromSource() was correctly wired up.
Fix: add InstallNtfy() call to installFromPreBuilt right after the
binary deploy + setCapabilities steps, before disableResolvedStub
runs. Ordering matters because Phase 4's ConfigureNtfy chowns
/etc/ntfy/server.yml to the ntfy user, which needs to exist.
VERSION bumped to 0.122.17.
ntfy is now part of the standard node install, just like Caddy. The
binary, /etc/ntfy/server.yml, and the Caddy push.<dnsZone> reverse-
proxy block are written unconditionally on every node, and the
ntfy.service starts as part of the standard service order.
Why uniform: ntfy listens on 127.0.0.1:NtfyListenPort only, reachable
exclusively via the local Caddy reverse-proxy block. Nodes that don't
serve a public push.* DNS entry just have an idle ntfy with no
inbound traffic — zero operational cost, zero attack surface change.
Removing the flag means no per-node toggling, no preference drift
between nodes, no "did we remember to set --with-ntfy" mistakes when
DNS topology changes (e.g. promoting a node to nameserver later).
Removed:
- NodePreferences.NtfyHost (yaml: ntfy_host)
- ProductionSetup.isNtfyHost field, SetNtfyHost, IsNtfyHost
- install/flags.go --with-ntfy + NtfyHost field
- upgrade/flags.go --with-ntfy + NtfyHost field + isFlagPassed helper
(was only used for --with-ntfy tri-state semantics)
- upgrade/orchestrator.go preference-load and persist for ntfy
- upgrade/remote.go --with-ntfy forwarding
Phase 2 always calls InstallNtfy.
Phase 4 always calls EnableCaddyNtfyProxy + ConfigureNtfy.
Phase 5 always enables ntfy.service.
Phase 5b always starts ntfy.service.
VERSION bumped to 0.122.16.
