orama/docs/SANDBOX.md

# Sandbox: Ephemeral Hetzner Cloud Clusters

Spin up temporary 5-node Orama clusters on Hetzner Cloud for development and testing. Total cost: ~€0.04/hour.

## Quick Start

```bash
# One-time setup (API key, domain, floating IPs, SSH key)
orama sandbox setup

# Create a cluster (~5 minutes)
orama sandbox create --name my-feature

# Check health
orama sandbox status

# SSH into a node
orama sandbox ssh 1

# Deploy code changes
orama sandbox rollout

# Tear it down
orama sandbox destroy
```

## Prerequisites

### 1. Hetzner Cloud Account

Create a project at [console.hetzner.cloud](https://console.hetzner.cloud) and generate an API token with read/write permissions under **Security > API Tokens**.

### 2. Domain with Glue Records

You need a domain (or subdomain) that points to Hetzner Floating IPs. The `orama sandbox setup` wizard will guide you through this.

**Example:** Using `sbx.dbrs.space`

At your domain registrar:
1. Create glue records (Personal DNS Servers):
   - `ns1.sbx.dbrs.space` → `<floating-ip-1>`
   - `ns2.sbx.dbrs.space` → `<floating-ip-2>`
2. Set custom nameservers for `sbx.dbrs.space`:
   - `ns1.sbx.dbrs.space`
   - `ns2.sbx.dbrs.space`

DNS propagation can take up to 48 hours.

### 3. Binary Archive

Build the binary archive before creating a cluster:

```bash
orama build
```

This creates `/tmp/orama-<version>-linux-amd64.tar.gz` with all pre-compiled binaries.

## Setup

Run the interactive setup wizard:

```bash
orama sandbox setup
```

This will:
1. Prompt for your Hetzner API token and validate it
2. Ask for your sandbox domain
3. Create or reuse 2 Hetzner Floating IPs (~$0.005/hr each)
4. Create a firewall with sandbox rules
5. Create a rootwallet SSH entry (`sandbox/root`) if it doesn't exist
6. Upload the wallet-derived public key to Hetzner
7. Display DNS configuration instructions

Config is saved to `~/.orama/sandbox.yaml`.

## Commands

### `orama sandbox create [--name <name>]`

Creates a new 5-node cluster. If `--name` is omitted, a random name is generated (e.g., "swift-falcon").

**Cluster layout:**
- Nodes 1-2: Nameservers (CoreDNS + Caddy + all services)
- Nodes 3-5: Regular nodes (all services except CoreDNS)

**Phases:**
1. Provision 5 CX22 servers on Hetzner (parallel, ~90s)
2. Assign floating IPs to nameserver nodes (~10s)
3. Upload binary archive to all nodes (parallel, ~60s)
4. Install genesis node + generate invite tokens (~120s)
5. Join remaining 4 nodes (serial with health checks, ~180s)
6. Verify cluster health (~15s)

**One sandbox at a time.** Since the floating IPs are shared, only one sandbox can own the nameservers. Destroy the active sandbox before creating a new one.

### `orama sandbox destroy [--name <name>] [--force]`

Tears down a cluster:
1. Unassigns floating IPs
2. Deletes all 5 servers (parallel)
3. Removes state file

Use `--force` to skip confirmation.

### `orama sandbox list`

Lists all sandboxes with their status. Also checks Hetzner for orphaned servers that don't have a corresponding state file.

### `orama sandbox status [--name <name>]`

Shows per-node health including:
- Service status (active/inactive)
- RQLite role (Leader/Follower)
- Cluster summary (commit index, voter count)

### `orama sandbox rollout [--name <name>]`

Deploys code changes:
1. Uses the latest binary archive from `/tmp/` (run `orama build` first)
2. Pushes to all nodes
3. Rolling upgrade: followers first, leader last, 15s between nodes

### `orama sandbox ssh <node-number>`

Opens an interactive SSH session to a sandbox node (1-5).

```bash
orama sandbox ssh 1    # SSH into node 1 (genesis/ns1)
orama sandbox ssh 3    # SSH into node 3 (regular node)
```

## Architecture

### Floating IPs

Hetzner Floating IPs are persistent IPv4 addresses that can be reassigned between servers. They solve the DNS chicken-and-egg problem:

- Glue records at the registrar point to 2 Floating IPs (configured once)
- Each new sandbox assigns the Floating IPs to its nameserver nodes
- DNS works instantly — no propagation delay between clusters

### SSH Authentication

Sandbox uses a rootwallet-derived SSH key (`sandbox/root` vault entry), the same mechanism as production. The wallet must be unlocked (`rw unlock`) before running sandbox commands that use SSH. The public key is uploaded to Hetzner during setup and injected into every server at creation time.

### Server Naming

Servers: `sbx-<name>-<N>` (e.g., `sbx-swift-falcon-1` through `sbx-swift-falcon-5`)

### State Files

Sandbox state is stored at `~/.orama/sandboxes/<name>.yaml`. This tracks server IDs, IPs, roles, and cluster status.

## Cost

| Resource | Cost | Qty | Total |
|----------|------|-----|-------|
| CX22 (2 vCPU, 4GB) | €0.006/hr | 5 | €0.03/hr |
| Floating IPv4 | €0.005/hr | 2 | €0.01/hr |
| **Total** | | | **~€0.04/hr** |

Servers are billed per hour. Floating IPs are billed as long as they exist (even unassigned). Destroy the sandbox when not in use to save on server costs.

## Troubleshooting

### "sandbox not configured"

Run `orama sandbox setup` first.

### "no binary archive found"

Run `orama build` to create the binary archive.

### "sandbox X is already active"

Only one sandbox can be active at a time. Destroy it first:
```bash
orama sandbox destroy --name <name>
```

### Server creation fails

Check:
- Hetzner API token is valid and has read/write permissions
- You haven't hit Hetzner's server limit (default: 10 per project)
- The selected location has CX22 capacity

### Genesis install fails

SSH into the node to debug:
```bash
orama sandbox ssh 1
journalctl -u orama-node -f
```

The sandbox will be left in "error" state. You can destroy and recreate it.

### DNS not resolving

1. Verify glue records are configured at your registrar
2. Check propagation: `dig NS sbx.dbrs.space @8.8.8.8`
3. Propagation can take 24-48 hours for new domains

### Orphaned servers

If `orama sandbox list` shows orphaned servers, delete them manually at [console.hetzner.cloud](https://console.hetzner.cloud). Sandbox servers are labeled `orama-sandbox=<name>` for easy identification.