handleNamespaceGatewayRequest rewrites r.Host to the backend target
IP:port (e.g. "10.0.0.6:10004") before forwarding. The original
public host (e.g. "ns-anchat-test.orama-devnet.network") is preserved
in X-Forwarded-Host. checkWSOrigin in both pubsub/ws_client.go and
serverless/ws_handler.go was comparing the client's Origin against
the proxied r.Host only — so every browser / RN-iOS WS upgrade was
rejected 403 because their Origin's public hostname can never match
10.0.0.6.
curl probes don't send Origin, so curl returned true unconditionally
and the bug was invisible to operator smoke tests. AnChat's iPhone
WS clients hit `code=1006 reason="Received bad response code from
server: 403"` for ~24h.
Fix: prefer X-Forwarded-Host (the original public host) when present,
fall back to r.Host for direct (non-proxied) connections. Applied
identically to both WS handlers. Regression test in
serverless/ws_origin_test.go covers the proxy-hop case, no-Origin
case, and direct-connection case.
This is the real fix; v0.122.19 only closed a separate silent-forward
auth hole that produced opaque 401s on a different code path.
VERSION bumped to 0.122.20.
Root-cause hardening for bug #240 and #249's "intermittent 401 over WS"
reports. handleNamespaceGatewayRequest previously had a third code
path beyond "auth ok" and "auth error": when validateAuthForNamespaceProxy
returned empty namespace AND empty error (i.e. "no credentials found"),
the request fell through to a silent forward to the namespace gateway
WITHOUT internal-auth headers. The namespace gateway then rejected
with 401 "missing API key" in ~60µs.
From the client's perspective: opaque 401.
From our side: only the namespace gateway logged it, and that tier
can't validate API keys (they live in the main cluster RQLite), so
the operator had no signal that the main gateway had even seen the
request. AnChat's intermittent 401-on-WS reports went unsolved for
this exact reason.
Fix:
- Explicit reject at main when no credentials extracted AND path
isn't public. Returns 401 with WWW-Authenticate: Bearer realm and a
clear message naming the three accepted credential sources.
- Rich structured logging on every WS upgrade auth outcome: presence
of api_key/token/jwt query params, Authorization + X-API-Key
headers, Connection/Upgrade headers, Origin, User-Agent, client IP,
raw query length. Steady-state stays low-noise: success path logs at
debug, reject paths log at warn.
- Namespace-mismatch reject (existing branch) now also logs.
VERSION bumped to 0.122.19.
Migration 028: namespace_push_credentials
- Per-(namespace, provider) AES-256-GCM encrypted credential blob.
- Generic schema — apns/ntfy/expo/future plug in with zero migration.
- Separated from migration 026's namespace_push_config (preferences vs
credentials, different access patterns).
pkg/push/credentials
- Manager + Registry + RQLite store; HKDF purpose "namespace-push-credentials"
via pkg/secrets. Provider Validator interface for per-provider schema.
pkg/push/providers/apns
- Apple Push Notification service direct provider (no Expo proxy).
- Validator + dispatcher; credentials are p8 signing key + key_id + team_id.
pkg/push/providers/ntfy/credentials.go
- ntfy credential schema (auth_token + default topic). Used both with
the public ntfy.sh and our self-hosted instance.
pkg/environments/production/installers/ntfy.go
- Self-hosted ntfy server installer. Binary, system user, hardened
/etc/ntfy/server.yml, systemd unit. Listens on 127.0.0.1:NtfyListenPort
only — Caddy is the only public path.
pkg/environments/production/installers/caddy.go
- Emit reverse_proxy block for push.<dnsZone> -> 127.0.0.1:NtfyListenPort
when operator enables ntfy on a node.
CLI: install/upgrade orchestrators learn a new "ntfy" install/preserve
phase; flag gating in install/flags.go + upgrade/flags.go.
Gateway handlers/push/credentials_handler.go
- GET/PUT/DELETE /v1/namespace/push-credentials/{provider}.
- PUT validates against provider Validator before encrypting and storing.
- GET returns a redacted view (booleans + non-secret fields only).
Push manager: provider resolution now also consults
namespace_push_credentials before falling back to YAML defaults.
Docs: core/docs/PUSH_NOTIFICATIONS.md walks through end-to-end setup.
VERSION bumped to 0.122.14.
Per-namespace rate-limit config (feature #69)
- Migration 027: new `namespace_rate_limit_config` table
(namespace PK, requests_per_minute, burst, audit metadata).
- pkg/ratelimit: Manager + RQLite ConfigStore + types. Same pattern
as the push config in bug #220's follow-up — LRU cache, invalidate
on PUT/DELETE, falls back to YAML defaults when no row exists.
- pkg/gateway/handlers/ratelimit: GET/PUT/DELETE /v1/namespace/rate-limit.
PUT requests are rejected if they exceed the operator's configured
ceiling (MaxRequestsPerMinute / MaxBurst) — tenants self-serve but
cannot raise their quota past the cap.
- pkg/gateway/rate_limiter.go: per-namespace lookup, default fallback.
- pkg/gateway/middleware.go: WS JWT middleware (middleware_ws_jwt_test.go).
- pkg/gateway/auth/service.go: refresh-token rotation hardening with
regression test in refresh_rotation_test.go.
AI agent instructions
- Add AGENTS.md, CLAUDE.md, .github/copilot-instructions.md (DeBros v0.2.0
baseline).
DeBros rules bumped to v0.2.0 (sha bb6e6ef).
VERSION bumped to 0.122.12.
- Change RQLite consistency level from `none` to `weak` to ensure reads
route to the leader and prevent stale data reads (fixes#235)
- Add `normalizeColumnKey` to allow snake_case SQL columns to map to
CamelCase Go struct fields automatically (fixes#65)
- Add comprehensive unit tests for DSN generation and column mapping
- Add `namespace_push_config` table for per-namespace provider settings
- Introduce `cluster_secret_path` to enable deterministic JWT signing and
AES-256-GCM encryption for push credentials
- Update gateway config to support per-namespace overrides of push
notification providers (ntfy/Expo)
- Bump version to 0.122.3
- bump version to 0.122.2
- document schema migration invariants and push notification configuration
- add serverless host function aliases and v2 database API documentation
- introduce schema roundtrip test to prevent migration drift
- Add `contract.go` to manage and validate embedded SQL migrations
- Introduce `AssertSchema` to verify database version at startup
- Include `SchemaMismatchError` with actionable recovery instructions
- Add comprehensive unit tests for version parsing and validation
- Add migrations for per-namespace publish sequences and persistent WebSocket function settings
- Integrate PersistentWSManager and WSBridge into the gateway dependency graph
- Upgrade serverless engine to use a multi-tier rate limiter
- Update JWT claims to support custom application-defined fields
- Replace CLI-based rootwallet calls with agent-based communication
- Update production provisioner to support sudo-based service management
- Add API key-to-wallet resolution for gateway operator handlers
- implement `nodes`, `rollout`, `ssh`, and `status` commands
- add `migrate-conf` utility to register existing nodes with the gateway
- update database schema to support operator wallet tracking for nodes
