orama/docs/SANDBOX.md

Sandbox: Ephemeral Hetzner Cloud Clusters

Spin up temporary 5-node Orama clusters on Hetzner Cloud for development and testing. Total cost: ~€0.04/hour.

Quick Start

# One-time setup (API key, domain, floating IPs, SSH key)
orama sandbox setup

# Create a cluster (~5 minutes)
orama sandbox create --name my-feature

# Check health
orama sandbox status

# SSH into a node
orama sandbox ssh 1

# Deploy code changes
orama sandbox rollout

# Tear it down
orama sandbox destroy

Prerequisites

1. Hetzner Cloud Account

Create a project at console.hetzner.cloud and generate an API token with read/write permissions under Security > API Tokens.

2. Domain with Glue Records

You need a domain (or subdomain) that points to Hetzner Floating IPs. The orama sandbox setup wizard will guide you through this.

Example: Using sbx.dbrs.space

At your domain registrar:

1. Create glue records (Personal DNS Servers):
   • ns1.sbx.dbrs.space → <floating-ip-1>
   • ns2.sbx.dbrs.space → <floating-ip-2>
2. Set custom nameservers for sbx.dbrs.space:
   • ns1.sbx.dbrs.space
   • ns2.sbx.dbrs.space

DNS propagation can take up to 48 hours.

3. Binary Archive

Build the binary archive before creating a cluster:

orama build

This creates /tmp/orama-<version>-linux-amd64.tar.gz with all pre-compiled binaries.

Setup

Run the interactive setup wizard:

orama sandbox setup

This will:

1. Prompt for your Hetzner API token and validate it
2. Ask for your sandbox domain
3. Create or reuse 2 Hetzner Floating IPs (~$0.005/hr each)
4. Create a firewall with sandbox rules
5. Create a rootwallet SSH entry (sandbox/root) if it doesn't exist
6. Upload the wallet-derived public key to Hetzner
7. Display DNS configuration instructions

Config is saved to ~/.orama/sandbox.yaml.

Commands

orama sandbox create [--name <name>]

Creates a new 5-node cluster. If --name is omitted, a random name is generated (e.g., "swift-falcon").

Cluster layout:

• Nodes 1-2: Nameservers (CoreDNS + Caddy + all services)
• Nodes 3-5: Regular nodes (all services except CoreDNS)

Phases:

1. Provision 5 CX22 servers on Hetzner (parallel, ~90s)
2. Assign floating IPs to nameserver nodes (~10s)
3. Upload binary archive to all nodes (parallel, ~60s)
4. Install genesis node + generate invite tokens (~120s)
5. Join remaining 4 nodes (serial with health checks, ~180s)
6. Verify cluster health (~15s)

One sandbox at a time. Since the floating IPs are shared, only one sandbox can own the nameservers. Destroy the active sandbox before creating a new one.

orama sandbox destroy [--name <name>] [--force]

Tears down a cluster:

1. Unassigns floating IPs
2. Deletes all 5 servers (parallel)
3. Removes state file

Use --force to skip confirmation.

orama sandbox list

Lists all sandboxes with their status. Also checks Hetzner for orphaned servers that don't have a corresponding state file.

orama sandbox status [--name <name>]

Shows per-node health including:

• Service status (active/inactive)
• RQLite role (Leader/Follower)
• Cluster summary (commit index, voter count)

orama sandbox rollout [--name <name>]

Deploys code changes:

1. Uses the latest binary archive from /tmp/ (run orama build first)
2. Pushes to all nodes
3. Rolling upgrade: followers first, leader last, 15s between nodes

orama sandbox ssh <node-number>

Opens an interactive SSH session to a sandbox node (1-5).

orama sandbox ssh 1    # SSH into node 1 (genesis/ns1)
orama sandbox ssh 3    # SSH into node 3 (regular node)

Architecture

Floating IPs

Hetzner Floating IPs are persistent IPv4 addresses that can be reassigned between servers. They solve the DNS chicken-and-egg problem:

• Glue records at the registrar point to 2 Floating IPs (configured once)
• Each new sandbox assigns the Floating IPs to its nameserver nodes
• DNS works instantly — no propagation delay between clusters

SSH Authentication

Sandbox uses a rootwallet-derived SSH key (sandbox/root vault entry), the same mechanism as production. The wallet must be unlocked (rw unlock) before running sandbox commands that use SSH. The public key is uploaded to Hetzner during setup and injected into every server at creation time.

Server Naming

Servers: sbx-<name>-<N> (e.g., sbx-swift-falcon-1 through sbx-swift-falcon-5)

State Files

Sandbox state is stored at ~/.orama/sandboxes/<name>.yaml. This tracks server IDs, IPs, roles, and cluster status.

Cost

Resource	Cost	Qty	Total
CX22 (2 vCPU, 4GB)	€0.006/hr	5	€0.03/hr
Floating IPv4	€0.005/hr	2	€0.01/hr
Total			~€0.04/hr

Servers are billed per hour. Floating IPs are billed as long as they exist (even unassigned). Destroy the sandbox when not in use to save on server costs.

Troubleshooting

"sandbox not configured"

Run orama sandbox setup first.

"no binary archive found"

Run orama build to create the binary archive.

"sandbox X is already active"

Only one sandbox can be active at a time. Destroy it first:

orama sandbox destroy --name <name>

Server creation fails

Check:

• Hetzner API token is valid and has read/write permissions
• You haven't hit Hetzner's server limit (default: 10 per project)
• The selected location has CX22 capacity

Genesis install fails

SSH into the node to debug:

orama sandbox ssh 1
journalctl -u orama-node -f

The sandbox will be left in "error" state. You can destroy and recreate it.

DNS not resolving

1. Verify glue records are configured at your registrar
2. Check propagation: dig NS sbx.dbrs.space @8.8.8.8
3. Propagation can take 24-48 hours for new domains

Orphaned servers

If orama sandbox list shows orphaned servers, delete them manually at console.hetzner.cloud. Sandbox servers are labeled orama-sandbox=<name> for easy identification.