orama/docs/STEALTH_TURN.md

# Stealth TURN Deployment Guide

## What this is

A TLS-level SNI router that lets Orama serve TURN-over-TLS on `:443`,
sharing the port with Caddy HTTPS. From a network observer's
perspective, TURN traffic is indistinguishable from ordinary HTTPS —
useful for users in regions that block standard VoIP ports (UAE, Saudi
Arabia, China, Iran).

## Architecture

```
                    Internet
                        │
                        ▼
                  TCP :443
                        │
              ┌─────────┴─────────┐
              │  orama-sni-router │   peeks SNI, forwards bytes
              └─────────┬─────────┘
                        │
        ┌───────────────┼────────────────┐
        ▼                                ▼
  cdn-<hash>.<base>            *.<base>, <base>
  turn.ns-<ns>.<base>           (everything else)
        │                                │
        ▼                                ▼
  Pion TURN-TLS                       Caddy
  127.0.0.1:<tls-port>            127.0.0.1:8443
  (per namespace)                  (moved from :443)
```

The router does **not** terminate TLS. It reads the unencrypted TLS
ClientHello (first ~5 KB), inspects the SNI extension, and dials the
matching backend. Encrypted bytes pass through verbatim.

The stealth hostname is `cdn-<12-hex-of-sha256(namespace)>.<base-domain>`
(`turn.StealthHostForNamespace`). The label deliberately does NOT contain
the namespace — an SNI string like `cdn.ns-anchat.…` would hand DPI the
exact app to block. It's deterministic, so the cluster manager, gateway,
SNI router, and DNS all derive the same value with no coordination. Each
namespace also gets a human-readable `turn.ns-<namespace>.<base>` alias
for operator UX.

## Components

- **Library:** `pkg/sniproxy/` — ClientHello parser, route table, TCP
  server, config-file hot-reload (`FileRouteReloader`), and namespace
  auto-discovery (`TURNRouteDiscoverer`)
- **Binary:** `cmd/sni-router/` (built as `bin/orama-sni-router` by
  `orama build`, installed to `/opt/orama/bin/orama-sni-router`)
- **Installer:** `pkg/environments/production/installers/sni_router.go`
  — writes the config + systemd unit and drives the unit lifecycle
- **Systemd unit:** `/etc/systemd/system/orama-sni-router.service`
  (runs as the `orama` user with `CAP_NET_BIND_SERVICE`, ordered
  `Before=caddy.service`)
- **Config:** `<oramaDir>/configs/sni-router.yaml` (e.g.
  `/opt/orama/.orama/configs/sni-router.yaml`) — generated by the
  installer; re-running install/upgrade overwrites it

## Enabling the router on a node

The router is an **operator opt-in per node**, controlled by the
top-level `sni_router` block in the node's `node.yaml`
(`<oramaDir>/configs/node.yaml`):

```yaml
sni_router:
  enabled: true
```

Set the flag, then re-run the normal node install/upgrade flow. The
orchestrator does the whole cutover — there are no manual steps:

- Moves Caddy's HTTPS listener from `:443` to `:8443`
  (`CaddyHTTPSPortBehindSNI`) via the `https_port` global option in the
  generated Caddyfile. Plain HTTP (`:80`) is unaffected.
- Writes `sni-router.yaml` (listen `:443`, fallback Caddy
  `127.0.0.1:8443`, `turn_discovery` scanning the node's namespaces
  dir).
- Writes and enables the `orama-sni-router.service` unit, starting it
  after Caddy has been restarted on `:8443`.

The flag is carried forward across config regenerations, so an upgrade
never silently disables the router. When the flag is off (or removed),
the same flow stops and disables the router unit and Caddy reclaims
`:443` — that is also the rollback path.

## Route discovery — no static routes

The generated config emits `routes: []`. Every TURNS route is
auto-discovered: the router rescans
`<oramaDir>/data/namespaces/*/configs/turn-*.yaml` every 30 seconds
(`turn_discovery.rescan_interval`) and, for each namespace with a TURNS
listener, installs two routes to `127.0.0.1:<tls-port>`:

- `cdn-<hash>.<base-domain>` (the stealth host)
- `turn.ns-<namespace>.<base-domain>` (operator alias)

Route tables are swapped atomically (`Router.Replace`) while connections
are in flight — new namespaces appear without a router restart.
Operator-set static routes in the config file win on an SNI conflict. A
failed scan keeps the previously-installed routes (a filesystem hiccup
never blackholes live `:443` traffic).

Config values written by the installer: `client_hello_timeout: 5s`,
`backend_dial_timeout: 5s`, `max_concurrent_conns: 10000`.

## Enabling stealth for a namespace

Per-namespace stealth is toggled via the CLI (requires WebRTC to already
be enabled for the namespace):

```bash
orama namespace enable webrtc-stealth --namespace myapp
orama namespace disable webrtc-stealth --namespace myapp
```

This drives `POST /v1/namespace/webrtc/stealth/{enable|disable}`, which:

1. Creates DNS A records for the stealth host pointing at the
   namespace's TURN nodes.
2. Flips `namespace_webrtc_config.stealth_enabled` (migration
   `030_webrtc_stealth.sql`).
3. Re-spawns the namespace's TURN servers with the stealth domain, so
   they carry a second Let's Encrypt certificate for the stealth
   hostname (cert provisioning may take up to ~2 minutes). On failure
   the enable is rolled back so the board never claims a stealth
   endpoint that doesn't terminate TLS.
4. Refreshes the namespace gateways so `turn.credentials` advertises
   `turns:<stealth-host>:443` as the final rung of the ICE URI ladder.

Disabling keeps TURN and the baseline ladder (udp/tcp 3478, turns:5349)
running — only the `:443` rung is removed.

The gateway side is config-driven (`Config.StealthCDNDomain` in
`pkg/gateway/config.go`) — no code edit is needed to advertise the
stealth URI.

## TLS certificates

TURN serves both its primary cert and the stealth-host cert through
hot-reloading `GetCertificate` callbacks (selection by SNI ServerName),
polling the cert files every 60 seconds (`pkg/turn/cert_reloader.go`).
Caddy-renewed certs are picked up in-process — no TURN restart, so
active relays are never dropped by a renewal.

## Monitoring

```bash
journalctl -u orama-sni-router.service -f
journalctl -u caddy.service -f
```

Watch for:
- `Connection limit reached` warnings (bump `max_concurrent_conns`)
- `backend dial failed` warnings (Caddy isn't listening on `:8443`, or the namespace's TURNS listener is down)
- `ClientHello peek failed` debugs (curious clients sending non-TLS to `:443` — usually port scanners)
- `TURN route discovery failed` warnings (namespaces dir unreadable — router keeps serving its current routes)

## Rollback

Set `sni_router.enabled: false` in the node's `node.yaml` and re-run the
node upgrade flow. The orchestrator stops and disables
`orama-sni-router.service` and regenerates the Caddyfile so Caddy
reclaims `:443` — the fastest way back to the previous topology.

## What clients see

Once enabled, the credentials response gains one entry:

```json
{
  "username": "...",
  "password": "...",
  "ttl": 600,
  "uris": [
    "turn:turn.example.com:3478?transport=udp",
    "turn:turn.example.com:3478?transport=tcp",
    "turns:turn.example.com:5349",
    "turns:cdn-1a2b3c4d5e6f.example.com:443"
  ]
}
```

Browsers iterate ICE candidates; users in restricted regions will silently
succeed via the `:443` URI when others fail. No client-side change is
required.