Added hatzhner support for clustering cli orama to spin up clusters

This commit is contained in:
anonpenguin23 2026-02-25 15:13:18 +02:00
parent 6898f47e2e
commit fade8f89ed
14 changed files with 3022 additions and 0 deletions

208
docs/SANDBOX.md Normal file
View File

@ -0,0 +1,208 @@
# Sandbox: Ephemeral Hetzner Cloud Clusters
Spin up temporary 5-node Orama clusters on Hetzner Cloud for development and testing. Total cost: ~€0.04/hour.
## Quick Start
```bash
# One-time setup (API key, domain, floating IPs, SSH key)
orama sandbox setup
# Create a cluster (~5 minutes)
orama sandbox create --name my-feature
# Check health
orama sandbox status
# SSH into a node
orama sandbox ssh 1
# Deploy code changes
orama sandbox rollout
# Tear it down
orama sandbox destroy
```
## Prerequisites
### 1. Hetzner Cloud Account
Create a project at [console.hetzner.cloud](https://console.hetzner.cloud) and generate an API token with read/write permissions under **Security > API Tokens**.
### 2. Domain with Glue Records
You need a domain (or subdomain) that points to Hetzner Floating IPs. The `orama sandbox setup` wizard will guide you through this.
**Example:** Using `sbx.dbrs.space`
At your domain registrar:
1. Create glue records (Personal DNS Servers):
- `ns1.sbx.dbrs.space``<floating-ip-1>`
- `ns2.sbx.dbrs.space``<floating-ip-2>`
2. Set custom nameservers for `sbx.dbrs.space`:
- `ns1.sbx.dbrs.space`
- `ns2.sbx.dbrs.space`
DNS propagation can take up to 48 hours.
### 3. Binary Archive
Build the binary archive before creating a cluster:
```bash
orama build
```
This creates `/tmp/orama-<version>-linux-amd64.tar.gz` with all pre-compiled binaries.
## Setup
Run the interactive setup wizard:
```bash
orama sandbox setup
```
This will:
1. Prompt for your Hetzner API token and validate it
2. Ask for your sandbox domain
3. Create or reuse 2 Hetzner Floating IPs (~$0.005/hr each)
4. Create a firewall with sandbox rules
5. Generate an SSH keypair at `~/.orama/sandbox_key`
6. Upload the public key to Hetzner
7. Display DNS configuration instructions
Config is saved to `~/.orama/sandbox.yaml`.
## Commands
### `orama sandbox create [--name <name>]`
Creates a new 5-node cluster. If `--name` is omitted, a random name is generated (e.g., "swift-falcon").
**Cluster layout:**
- Nodes 1-2: Nameservers (CoreDNS + Caddy + all services)
- Nodes 3-5: Regular nodes (all services except CoreDNS)
**Phases:**
1. Provision 5 CX22 servers on Hetzner (parallel, ~90s)
2. Assign floating IPs to nameserver nodes (~10s)
3. Upload binary archive to all nodes (parallel, ~60s)
4. Install genesis node + generate invite tokens (~120s)
5. Join remaining 4 nodes (serial with health checks, ~180s)
6. Verify cluster health (~15s)
**One sandbox at a time.** Since the floating IPs are shared, only one sandbox can own the nameservers. Destroy the active sandbox before creating a new one.
### `orama sandbox destroy [--name <name>] [--force]`
Tears down a cluster:
1. Unassigns floating IPs
2. Deletes all 5 servers (parallel)
3. Removes state file
Use `--force` to skip confirmation.
### `orama sandbox list`
Lists all sandboxes with their status. Also checks Hetzner for orphaned servers that don't have a corresponding state file.
### `orama sandbox status [--name <name>]`
Shows per-node health including:
- Service status (active/inactive)
- RQLite role (Leader/Follower)
- Cluster summary (commit index, voter count)
### `orama sandbox rollout [--name <name>]`
Deploys code changes:
1. Uses the latest binary archive from `/tmp/` (run `orama build` first)
2. Pushes to all nodes
3. Rolling upgrade: followers first, leader last, 15s between nodes
### `orama sandbox ssh <node-number>`
Opens an interactive SSH session to a sandbox node (1-5).
```bash
orama sandbox ssh 1 # SSH into node 1 (genesis/ns1)
orama sandbox ssh 3 # SSH into node 3 (regular node)
```
## Architecture
### Floating IPs
Hetzner Floating IPs are persistent IPv4 addresses that can be reassigned between servers. They solve the DNS chicken-and-egg problem:
- Glue records at the registrar point to 2 Floating IPs (configured once)
- Each new sandbox assigns the Floating IPs to its nameserver nodes
- DNS works instantly — no propagation delay between clusters
### SSH Authentication
Sandbox uses a standalone ed25519 keypair at `~/.orama/sandbox_key`, separate from the production wallet-derived keys. The public key is uploaded to Hetzner during setup and injected into every server at creation time.
### Server Naming
Servers: `sbx-<name>-<N>` (e.g., `sbx-swift-falcon-1` through `sbx-swift-falcon-5`)
### State Files
Sandbox state is stored at `~/.orama/sandboxes/<name>.yaml`. This tracks server IDs, IPs, roles, and cluster status.
## Cost
| Resource | Cost | Qty | Total |
|----------|------|-----|-------|
| CX22 (2 vCPU, 4GB) | €0.006/hr | 5 | €0.03/hr |
| Floating IPv4 | €0.005/hr | 2 | €0.01/hr |
| **Total** | | | **~€0.04/hr** |
Servers are billed per hour. Floating IPs are billed as long as they exist (even unassigned). Destroy the sandbox when not in use to save on server costs.
## Troubleshooting
### "sandbox not configured"
Run `orama sandbox setup` first.
### "no binary archive found"
Run `orama build` to create the binary archive.
### "sandbox X is already active"
Only one sandbox can be active at a time. Destroy it first:
```bash
orama sandbox destroy --name <name>
```
### Server creation fails
Check:
- Hetzner API token is valid and has read/write permissions
- You haven't hit Hetzner's server limit (default: 10 per project)
- The selected location has CX22 capacity
### Genesis install fails
SSH into the node to debug:
```bash
orama sandbox ssh 1
journalctl -u orama-node -f
```
The sandbox will be left in "error" state. You can destroy and recreate it.
### DNS not resolving
1. Verify glue records are configured at your registrar
2. Check propagation: `dig NS sbx.dbrs.space @8.8.8.8`
3. Propagation can take 24-48 hours for new domains
### Orphaned servers
If `orama sandbox list` shows orphaned servers, delete them manually at [console.hetzner.cloud](https://console.hetzner.cloud). Sandbox servers are labeled `orama-sandbox=<name>` for easy identification.

View File

@ -0,0 +1,121 @@
package sandboxcmd
import (
"fmt"
"os"
"github.com/DeBrosOfficial/network/pkg/cli/sandbox"
"github.com/spf13/cobra"
)
// Cmd is the root command for sandbox operations.
var Cmd = &cobra.Command{
Use: "sandbox",
Short: "Manage ephemeral Hetzner Cloud clusters for testing",
Long: `Spin up temporary 5-node Orama clusters on Hetzner Cloud for development and testing.
Setup (one-time):
orama sandbox setup
Usage:
orama sandbox create [--name <name>] Create a new 5-node cluster
orama sandbox destroy [--name <name>] Tear down a cluster
orama sandbox list List active sandboxes
orama sandbox status [--name <name>] Show cluster health
orama sandbox rollout [--name <name>] Build + push + rolling upgrade
orama sandbox ssh <node-number> SSH into a sandbox node (1-5)`,
}
var setupCmd = &cobra.Command{
Use: "setup",
Short: "Interactive setup: Hetzner API key, domain, floating IPs, SSH key",
RunE: func(cmd *cobra.Command, args []string) error {
return sandbox.Setup()
},
}
var createCmd = &cobra.Command{
Use: "create",
Short: "Create a new 5-node sandbox cluster (~5 min)",
RunE: func(cmd *cobra.Command, args []string) error {
name, _ := cmd.Flags().GetString("name")
return sandbox.Create(name)
},
}
var destroyCmd = &cobra.Command{
Use: "destroy",
Short: "Destroy a sandbox cluster and release resources",
RunE: func(cmd *cobra.Command, args []string) error {
name, _ := cmd.Flags().GetString("name")
force, _ := cmd.Flags().GetBool("force")
return sandbox.Destroy(name, force)
},
}
var listCmd = &cobra.Command{
Use: "list",
Short: "List active sandbox clusters",
RunE: func(cmd *cobra.Command, args []string) error {
return sandbox.List()
},
}
var statusCmd = &cobra.Command{
Use: "status",
Short: "Show cluster health report",
RunE: func(cmd *cobra.Command, args []string) error {
name, _ := cmd.Flags().GetString("name")
return sandbox.Status(name)
},
}
var rolloutCmd = &cobra.Command{
Use: "rollout",
Short: "Build + push + rolling upgrade to sandbox cluster",
RunE: func(cmd *cobra.Command, args []string) error {
name, _ := cmd.Flags().GetString("name")
return sandbox.Rollout(name)
},
}
var sshCmd = &cobra.Command{
Use: "ssh <node-number>",
Short: "SSH into a sandbox node (1-5)",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
name, _ := cmd.Flags().GetString("name")
var nodeNum int
if _, err := fmt.Sscanf(args[0], "%d", &nodeNum); err != nil {
fmt.Fprintf(os.Stderr, "Invalid node number: %s (expected 1-5)\n", args[0])
os.Exit(1)
}
return sandbox.SSHInto(name, nodeNum)
},
}
func init() {
// create flags
createCmd.Flags().String("name", "", "Sandbox name (random if not specified)")
// destroy flags
destroyCmd.Flags().String("name", "", "Sandbox name (uses active if not specified)")
destroyCmd.Flags().Bool("force", false, "Skip confirmation")
// status flags
statusCmd.Flags().String("name", "", "Sandbox name (uses active if not specified)")
// rollout flags
rolloutCmd.Flags().String("name", "", "Sandbox name (uses active if not specified)")
// ssh flags
sshCmd.Flags().String("name", "", "Sandbox name (uses active if not specified)")
Cmd.AddCommand(setupCmd)
Cmd.AddCommand(createCmd)
Cmd.AddCommand(destroyCmd)
Cmd.AddCommand(listCmd)
Cmd.AddCommand(statusCmd)
Cmd.AddCommand(rolloutCmd)
Cmd.AddCommand(sshCmd)
}

153
pkg/cli/sandbox/config.go Normal file
View File

@ -0,0 +1,153 @@
package sandbox
import (
"fmt"
"os"
"path/filepath"
"gopkg.in/yaml.v3"
)
// Config holds sandbox configuration, stored at ~/.orama/sandbox.yaml.
type Config struct {
HetznerAPIToken string `yaml:"hetzner_api_token"`
Domain string `yaml:"domain"`
Location string `yaml:"location"` // Hetzner datacenter (default: fsn1)
ServerType string `yaml:"server_type"` // Hetzner server type (default: cx22)
FloatingIPs []FloatIP `yaml:"floating_ips"`
SSHKey SSHKeyConfig `yaml:"ssh_key"`
FirewallID int64 `yaml:"firewall_id,omitempty"` // Hetzner firewall resource ID
}
// FloatIP holds a Hetzner floating IP reference.
type FloatIP struct {
ID int64 `yaml:"id"`
IP string `yaml:"ip"`
}
// SSHKeyConfig holds SSH key paths and the Hetzner resource ID.
type SSHKeyConfig struct {
HetznerID int64 `yaml:"hetzner_id"`
PrivateKeyPath string `yaml:"private_key_path"`
PublicKeyPath string `yaml:"public_key_path"`
}
// configDir returns ~/.orama/, creating it if needed.
func configDir() (string, error) {
home, err := os.UserHomeDir()
if err != nil {
return "", fmt.Errorf("get home directory: %w", err)
}
dir := filepath.Join(home, ".orama")
if err := os.MkdirAll(dir, 0700); err != nil {
return "", fmt.Errorf("create config directory: %w", err)
}
return dir, nil
}
// configPath returns the full path to ~/.orama/sandbox.yaml.
func configPath() (string, error) {
dir, err := configDir()
if err != nil {
return "", err
}
return filepath.Join(dir, "sandbox.yaml"), nil
}
// LoadConfig reads the sandbox config from ~/.orama/sandbox.yaml.
// Returns an error if the file doesn't exist (user must run setup first).
func LoadConfig() (*Config, error) {
path, err := configPath()
if err != nil {
return nil, err
}
data, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return nil, fmt.Errorf("sandbox not configured, run: orama sandbox setup")
}
return nil, fmt.Errorf("read config: %w", err)
}
var cfg Config
if err := yaml.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("parse config %s: %w", path, err)
}
if err := cfg.validate(); err != nil {
return nil, fmt.Errorf("invalid config: %w", err)
}
cfg.Defaults()
return &cfg, nil
}
// SaveConfig writes the sandbox config to ~/.orama/sandbox.yaml.
func SaveConfig(cfg *Config) error {
path, err := configPath()
if err != nil {
return err
}
data, err := yaml.Marshal(cfg)
if err != nil {
return fmt.Errorf("marshal config: %w", err)
}
if err := os.WriteFile(path, data, 0600); err != nil {
return fmt.Errorf("write config: %w", err)
}
return nil
}
// validate checks that required fields are present.
func (c *Config) validate() error {
if c.HetznerAPIToken == "" {
return fmt.Errorf("hetzner_api_token is required")
}
if c.Domain == "" {
return fmt.Errorf("domain is required")
}
if len(c.FloatingIPs) < 2 {
return fmt.Errorf("2 floating IPs required, got %d", len(c.FloatingIPs))
}
if c.SSHKey.PrivateKeyPath == "" {
return fmt.Errorf("ssh_key.private_key_path is required")
}
return nil
}
// Defaults fills in default values for optional fields.
func (c *Config) Defaults() {
if c.Location == "" {
c.Location = "fsn1"
}
if c.ServerType == "" {
c.ServerType = "cx22"
}
}
// ExpandedPrivateKeyPath returns the absolute path to the SSH private key.
func (c *Config) ExpandedPrivateKeyPath() string {
return expandHome(c.SSHKey.PrivateKeyPath)
}
// ExpandedPublicKeyPath returns the absolute path to the SSH public key.
func (c *Config) ExpandedPublicKeyPath() string {
return expandHome(c.SSHKey.PublicKeyPath)
}
// expandHome replaces a leading ~ with the user's home directory.
func expandHome(path string) string {
if len(path) < 2 || path[:2] != "~/" {
return path
}
home, err := os.UserHomeDir()
if err != nil {
return path
}
return filepath.Join(home, path[2:])
}

554
pkg/cli/sandbox/create.go Normal file
View File

@ -0,0 +1,554 @@
package sandbox
import (
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
"github.com/DeBrosOfficial/network/pkg/cli/remotessh"
"github.com/DeBrosOfficial/network/pkg/inspector"
)
// Create orchestrates the creation of a new sandbox cluster.
func Create(name string) error {
cfg, err := LoadConfig()
if err != nil {
return err
}
// Check for existing active sandbox
active, err := FindActiveSandbox()
if err != nil {
return err
}
if active != nil {
return fmt.Errorf("sandbox %q is already active (status: %s)\nDestroy it first: orama sandbox destroy --name %s",
active.Name, active.Status, active.Name)
}
// Generate name if not provided
if name == "" {
name = GenerateName()
}
fmt.Printf("Creating sandbox %q (%s, %d nodes)\n\n", name, cfg.Domain, 5)
client := NewHetznerClient(cfg.HetznerAPIToken)
state := &SandboxState{
Name: name,
CreatedAt: time.Now().UTC(),
Domain: cfg.Domain,
Status: StatusCreating,
}
// Phase 1: Provision servers
fmt.Println("Phase 1: Provisioning servers...")
if err := phase1ProvisionServers(client, cfg, state); err != nil {
cleanupFailedCreate(client, state)
return fmt.Errorf("provision servers: %w", err)
}
SaveState(state)
// Phase 2: Assign floating IPs
fmt.Println("\nPhase 2: Assigning floating IPs...")
if err := phase2AssignFloatingIPs(client, cfg, state); err != nil {
return fmt.Errorf("assign floating IPs: %w", err)
}
SaveState(state)
// Phase 3: Upload binary archive
fmt.Println("\nPhase 3: Uploading binary archive...")
if err := phase3UploadArchive(cfg, state); err != nil {
return fmt.Errorf("upload archive: %w", err)
}
// Phase 4: Install genesis node
fmt.Println("\nPhase 4: Installing genesis node...")
tokens, err := phase4InstallGenesis(cfg, state)
if err != nil {
state.Status = StatusError
SaveState(state)
return fmt.Errorf("install genesis: %w", err)
}
// Phase 5: Join remaining nodes
fmt.Println("\nPhase 5: Joining remaining nodes...")
if err := phase5JoinNodes(cfg, state, tokens); err != nil {
state.Status = StatusError
SaveState(state)
return fmt.Errorf("join nodes: %w", err)
}
// Phase 6: Verify cluster
fmt.Println("\nPhase 6: Verifying cluster...")
phase6Verify(cfg, state)
state.Status = StatusRunning
SaveState(state)
printCreateSummary(cfg, state)
return nil
}
// phase1ProvisionServers creates 5 Hetzner servers in parallel.
func phase1ProvisionServers(client *HetznerClient, cfg *Config, state *SandboxState) error {
type serverResult struct {
index int
server *HetznerServer
err error
}
results := make(chan serverResult, 5)
for i := 0; i < 5; i++ {
go func(idx int) {
role := "node"
if idx < 2 {
role = "nameserver"
}
serverName := fmt.Sprintf("sbx-%s-%d", state.Name, idx+1)
labels := map[string]string{
"orama-sandbox": state.Name,
"orama-sandbox-role": role,
}
req := CreateServerRequest{
Name: serverName,
ServerType: cfg.ServerType,
Image: "ubuntu-24.04",
Location: cfg.Location,
SSHKeys: []int64{cfg.SSHKey.HetznerID},
Labels: labels,
}
if cfg.FirewallID > 0 {
req.Firewalls = []struct {
Firewall int64 `json:"firewall"`
}{{Firewall: cfg.FirewallID}}
}
srv, err := client.CreateServer(req)
results <- serverResult{index: idx, server: srv, err: err}
}(i)
}
servers := make([]ServerState, 5)
for i := 0; i < 5; i++ {
r := <-results
if r.err != nil {
return fmt.Errorf("server %d: %w", r.index+1, r.err)
}
fmt.Printf(" Created %s (ID: %d, initializing...)\n", r.server.Name, r.server.ID)
role := "node"
if r.index < 2 {
role = "nameserver"
}
servers[r.index] = ServerState{
ID: r.server.ID,
Name: r.server.Name,
Role: role,
}
}
// Wait for all servers to reach "running"
fmt.Print(" Waiting for servers to boot...")
for i := range servers {
srv, err := client.WaitForServer(servers[i].ID, 3*time.Minute)
if err != nil {
return fmt.Errorf("wait for %s: %w", servers[i].Name, err)
}
servers[i].IP = srv.PublicNet.IPv4.IP
fmt.Print(".")
}
fmt.Println(" OK")
// Assign floating IPs to nameserver entries
if len(cfg.FloatingIPs) >= 2 {
servers[0].FloatingIP = cfg.FloatingIPs[0].IP
servers[1].FloatingIP = cfg.FloatingIPs[1].IP
}
state.Servers = servers
for _, srv := range servers {
fmt.Printf(" %s: %s (%s)\n", srv.Name, srv.IP, srv.Role)
}
return nil
}
// phase2AssignFloatingIPs assigns floating IPs and configures loopback.
func phase2AssignFloatingIPs(client *HetznerClient, cfg *Config, state *SandboxState) error {
sshKeyPath := cfg.ExpandedPrivateKeyPath()
for i := 0; i < 2 && i < len(cfg.FloatingIPs) && i < len(state.Servers); i++ {
fip := cfg.FloatingIPs[i]
srv := state.Servers[i]
// Unassign if currently assigned elsewhere (ignore "not assigned" errors)
fmt.Printf(" Assigning %s to %s...\n", fip.IP, srv.Name)
if err := client.UnassignFloatingIP(fip.ID); err != nil {
// Log but continue — may fail if not currently assigned, which is fine
fmt.Printf(" Note: unassign %s: %v (continuing)\n", fip.IP, err)
}
if err := client.AssignFloatingIP(fip.ID, srv.ID); err != nil {
return fmt.Errorf("assign %s to %s: %w", fip.IP, srv.Name, err)
}
// Configure floating IP on the server's loopback interface
// Hetzner floating IPs require this: ip addr add <floating_ip>/32 dev lo
node := inspector.Node{
User: "root",
Host: srv.IP,
SSHKey: sshKeyPath,
}
// Wait for SSH to be ready on freshly booted servers
if err := waitForSSH(node, 2*time.Minute); err != nil {
return fmt.Errorf("SSH not ready on %s: %w", srv.Name, err)
}
cmd := fmt.Sprintf("ip addr add %s/32 dev lo 2>/dev/null || true", fip.IP)
if err := remotessh.RunSSHStreaming(node, cmd); err != nil {
return fmt.Errorf("configure loopback on %s: %w", srv.Name, err)
}
}
return nil
}
// waitForSSH polls until SSH is responsive on the node.
func waitForSSH(node inspector.Node, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
_, err := runSSHOutput(node, "echo ok")
if err == nil {
return nil
}
time.Sleep(3 * time.Second)
}
return fmt.Errorf("timeout after %s", timeout)
}
// phase3UploadArchive builds (if needed) and uploads the binary archive to all nodes.
func phase3UploadArchive(cfg *Config, state *SandboxState) error {
// Find existing archive
archivePath := findNewestArchive()
if archivePath == "" {
fmt.Println(" No binary archive found, run `orama build` first")
return fmt.Errorf("no binary archive found in /tmp/ (run `orama build` first)")
}
info, _ := os.Stat(archivePath)
fmt.Printf(" Archive: %s (%s)\n", filepath.Base(archivePath), formatBytes(info.Size()))
sshKeyPath := cfg.ExpandedPrivateKeyPath()
remotePath := "/tmp/" + filepath.Base(archivePath)
// Upload to all 5 nodes in parallel
var wg sync.WaitGroup
errs := make([]error, len(state.Servers))
for i, srv := range state.Servers {
wg.Add(1)
go func(idx int, srv ServerState) {
defer wg.Done()
node := inspector.Node{User: "root", Host: srv.IP, SSHKey: sshKeyPath}
if err := remotessh.UploadFile(node, archivePath, remotePath); err != nil {
errs[idx] = fmt.Errorf("upload to %s: %w", srv.Name, err)
return
}
// Extract + install CLI
extractCmd := fmt.Sprintf("mkdir -p /opt/orama && tar xzf %s -C /opt/orama && rm -f %s && cp /opt/orama/bin/orama /usr/local/bin/orama && chmod +x /usr/local/bin/orama",
remotePath, remotePath)
if err := remotessh.RunSSHStreaming(node, extractCmd); err != nil {
errs[idx] = fmt.Errorf("extract on %s: %w", srv.Name, err)
return
}
fmt.Printf(" Uploaded to %s\n", srv.Name)
}(i, srv)
}
wg.Wait()
for _, err := range errs {
if err != nil {
return err
}
}
return nil
}
// phase4InstallGenesis installs the genesis node and generates invite tokens.
func phase4InstallGenesis(cfg *Config, state *SandboxState) ([]string, error) {
genesis := state.GenesisServer()
sshKeyPath := cfg.ExpandedPrivateKeyPath()
node := inspector.Node{User: "root", Host: genesis.IP, SSHKey: sshKeyPath}
// Install genesis
installCmd := fmt.Sprintf("orama node install --vps-ip %s --domain %s --base-domain %s --nameserver --skip-checks",
genesis.IP, cfg.Domain, cfg.Domain)
fmt.Printf(" Installing on %s (%s)...\n", genesis.Name, genesis.IP)
if err := remotessh.RunSSHStreaming(node, installCmd); err != nil {
return nil, fmt.Errorf("install genesis: %w", err)
}
// Wait for RQLite leader
fmt.Print(" Waiting for RQLite leader...")
if err := waitForRQLiteHealth(node, 3*time.Minute); err != nil {
return nil, fmt.Errorf("genesis health: %w", err)
}
fmt.Println(" OK")
// Generate invite tokens (one per remaining node)
fmt.Print(" Generating invite tokens...")
remaining := len(state.Servers) - 1
tokens := make([]string, remaining)
for i := 0; i < remaining; i++ {
token, err := generateInviteToken(node)
if err != nil {
return nil, fmt.Errorf("generate invite token %d: %w", i+1, err)
}
tokens[i] = token
fmt.Print(".")
}
fmt.Println(" OK")
return tokens, nil
}
// phase5JoinNodes joins the remaining 4 nodes to the cluster (serial).
func phase5JoinNodes(cfg *Config, state *SandboxState, tokens []string) error {
genesisIP := state.GenesisServer().IP
sshKeyPath := cfg.ExpandedPrivateKeyPath()
for i := 1; i < len(state.Servers); i++ {
srv := state.Servers[i]
node := inspector.Node{User: "root", Host: srv.IP, SSHKey: sshKeyPath}
token := tokens[i-1]
var installCmd string
if srv.Role == "nameserver" {
installCmd = fmt.Sprintf("orama node install --join http://%s --token %s --vps-ip %s --domain %s --base-domain %s --nameserver --skip-checks",
genesisIP, token, srv.IP, cfg.Domain, cfg.Domain)
} else {
installCmd = fmt.Sprintf("orama node install --join http://%s --token %s --vps-ip %s --base-domain %s --skip-checks",
genesisIP, token, srv.IP, cfg.Domain)
}
fmt.Printf(" [%d/%d] Joining %s (%s, %s)...\n", i, len(state.Servers)-1, srv.Name, srv.IP, srv.Role)
if err := remotessh.RunSSHStreaming(node, installCmd); err != nil {
return fmt.Errorf("join %s: %w", srv.Name, err)
}
// Wait for node health before proceeding
fmt.Printf(" Waiting for %s health...", srv.Name)
if err := waitForRQLiteHealth(node, 3*time.Minute); err != nil {
fmt.Printf(" WARN: %v\n", err)
} else {
fmt.Println(" OK")
}
}
return nil
}
// phase6Verify runs a basic cluster health check.
func phase6Verify(cfg *Config, state *SandboxState) {
sshKeyPath := cfg.ExpandedPrivateKeyPath()
genesis := state.GenesisServer()
node := inspector.Node{User: "root", Host: genesis.IP, SSHKey: sshKeyPath}
// Check RQLite cluster
out, err := runSSHOutput(node, "curl -s http://localhost:5001/status | grep -o '\"state\":\"[^\"]*\"' | head -1")
if err == nil {
fmt.Printf(" RQLite: %s\n", strings.TrimSpace(out))
}
// Check DNS (if floating IPs configured, only with safe domain names)
if len(cfg.FloatingIPs) > 0 && isSafeDNSName(cfg.Domain) {
out, err = runSSHOutput(node, fmt.Sprintf("dig +short @%s test.%s 2>/dev/null || echo 'DNS not responding'",
cfg.FloatingIPs[0].IP, cfg.Domain))
if err == nil {
fmt.Printf(" DNS: %s\n", strings.TrimSpace(out))
}
}
}
// waitForRQLiteHealth polls RQLite until it reports Leader or Follower state.
func waitForRQLiteHealth(node inspector.Node, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
out, err := runSSHOutput(node, "curl -sf http://localhost:5001/status 2>/dev/null | grep -o '\"state\":\"[^\"]*\"'")
if err == nil {
result := strings.TrimSpace(out)
if strings.Contains(result, "Leader") || strings.Contains(result, "Follower") {
return nil
}
}
time.Sleep(5 * time.Second)
}
return fmt.Errorf("timeout waiting for RQLite health after %s", timeout)
}
// generateInviteToken runs `orama node invite` on the node and parses the token.
func generateInviteToken(node inspector.Node) (string, error) {
out, err := runSSHOutput(node, "orama node invite --expiry 1h 2>&1")
if err != nil {
return "", fmt.Errorf("invite command failed: %w", err)
}
// Parse token from output — the invite command outputs:
// "sudo orama install --join https://... --token <64-char-hex> --vps-ip ..."
// Look for the --token flag value first
fields := strings.Fields(out)
for i, field := range fields {
if field == "--token" && i+1 < len(fields) {
candidate := fields[i+1]
if len(candidate) == 64 && isHex(candidate) {
return candidate, nil
}
}
}
// Fallback: look for any standalone 64-char hex string
for _, word := range fields {
if len(word) == 64 && isHex(word) {
return word, nil
}
}
return "", fmt.Errorf("could not parse token from invite output:\n%s", out)
}
// isSafeDNSName returns true if the string is safe to use in shell commands.
func isSafeDNSName(s string) bool {
for _, c := range s {
if !((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '.' || c == '-') {
return false
}
}
return len(s) > 0
}
// isHex returns true if s contains only hex characters.
func isHex(s string) bool {
for _, c := range s {
if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F')) {
return false
}
}
return true
}
// runSSHOutput runs a command via SSH and returns stdout as a string.
func runSSHOutput(node inspector.Node, command string) (string, error) {
args := []string{
"ssh", "-n",
"-o", "StrictHostKeyChecking=accept-new",
"-o", "ConnectTimeout=10",
"-o", "BatchMode=yes",
"-i", node.SSHKey,
fmt.Sprintf("%s@%s", node.User, node.Host),
command,
}
out, err := execCommand(args[0], args[1:]...)
return string(out), err
}
// execCommand runs a command and returns its output.
func execCommand(name string, args ...string) ([]byte, error) {
return exec.Command(name, args...).Output()
}
// findNewestArchive finds the newest binary archive in /tmp/.
func findNewestArchive() string {
entries, err := os.ReadDir("/tmp")
if err != nil {
return ""
}
var best string
var bestMod int64
for _, entry := range entries {
name := entry.Name()
if strings.HasPrefix(name, "orama-") && strings.Contains(name, "-linux-") && strings.HasSuffix(name, ".tar.gz") {
info, err := entry.Info()
if err != nil {
continue
}
if info.ModTime().Unix() > bestMod {
best = filepath.Join("/tmp", name)
bestMod = info.ModTime().Unix()
}
}
}
return best
}
// formatBytes formats a byte count as human-readable.
func formatBytes(b int64) string {
const unit = 1024
if b < unit {
return fmt.Sprintf("%d B", b)
}
div, exp := int64(unit), 0
for n := b / unit; n >= unit; n /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %cB", float64(b)/float64(div), "KMGTPE"[exp])
}
// printCreateSummary prints the cluster summary after creation.
func printCreateSummary(cfg *Config, state *SandboxState) {
fmt.Printf("\nSandbox %q ready (%d nodes)\n", state.Name, len(state.Servers))
fmt.Println()
fmt.Println("Nameservers:")
for _, srv := range state.NameserverNodes() {
floating := ""
if srv.FloatingIP != "" {
floating = fmt.Sprintf(" (floating: %s)", srv.FloatingIP)
}
fmt.Printf(" %s: %s%s\n", srv.Name, srv.IP, floating)
}
fmt.Println("Nodes:")
for _, srv := range state.RegularNodes() {
fmt.Printf(" %s: %s\n", srv.Name, srv.IP)
}
fmt.Println()
fmt.Printf("Domain: %s\n", cfg.Domain)
fmt.Printf("Gateway: https://%s\n", cfg.Domain)
fmt.Println()
fmt.Println("SSH: orama sandbox ssh 1")
fmt.Println("Destroy: orama sandbox destroy")
}
// cleanupFailedCreate deletes any servers that were created during a failed provision.
func cleanupFailedCreate(client *HetznerClient, state *SandboxState) {
if len(state.Servers) == 0 {
return
}
fmt.Println("\nCleaning up failed creation...")
for _, srv := range state.Servers {
if srv.ID > 0 {
client.DeleteServer(srv.ID)
fmt.Printf(" Deleted %s\n", srv.Name)
}
}
DeleteState(state.Name)
}

122
pkg/cli/sandbox/destroy.go Normal file
View File

@ -0,0 +1,122 @@
package sandbox
import (
"bufio"
"fmt"
"os"
"strings"
"sync"
)
// Destroy tears down a sandbox cluster.
func Destroy(name string, force bool) error {
cfg, err := LoadConfig()
if err != nil {
return err
}
// Resolve sandbox name
state, err := resolveSandbox(name)
if err != nil {
return err
}
// Confirm destruction
if !force {
reader := bufio.NewReader(os.Stdin)
fmt.Printf("Destroy sandbox %q? This deletes %d servers. [y/N]: ", state.Name, len(state.Servers))
choice, _ := reader.ReadString('\n')
choice = strings.TrimSpace(strings.ToLower(choice))
if choice != "y" && choice != "yes" {
fmt.Println("Aborted.")
return nil
}
}
state.Status = StatusDestroying
SaveState(state) // best-effort status update
client := NewHetznerClient(cfg.HetznerAPIToken)
// Step 1: Unassign floating IPs from nameserver nodes
fmt.Println("Unassigning floating IPs...")
for _, srv := range state.NameserverNodes() {
if srv.FloatingIP == "" {
continue
}
// Find the floating IP ID from config
for _, fip := range cfg.FloatingIPs {
if fip.IP == srv.FloatingIP {
if err := client.UnassignFloatingIP(fip.ID); err != nil {
fmt.Fprintf(os.Stderr, " Warning: could not unassign floating IP %s: %v\n", fip.IP, err)
} else {
fmt.Printf(" Unassigned %s from %s\n", fip.IP, srv.Name)
}
break
}
}
}
// Step 2: Delete all servers in parallel
fmt.Printf("Deleting %d servers...\n", len(state.Servers))
var wg sync.WaitGroup
var mu sync.Mutex
var failed []string
for _, srv := range state.Servers {
wg.Add(1)
go func(srv ServerState) {
defer wg.Done()
if err := client.DeleteServer(srv.ID); err != nil {
// Treat 404 as already deleted (idempotent)
if strings.Contains(err.Error(), "404") || strings.Contains(err.Error(), "not found") {
fmt.Printf(" %s (ID %d): already deleted\n", srv.Name, srv.ID)
} else {
mu.Lock()
failed = append(failed, fmt.Sprintf("%s (ID %d): %v", srv.Name, srv.ID, err))
mu.Unlock()
fmt.Fprintf(os.Stderr, " Warning: failed to delete %s: %v\n", srv.Name, err)
}
} else {
fmt.Printf(" Deleted %s (ID %d)\n", srv.Name, srv.ID)
}
}(srv)
}
wg.Wait()
if len(failed) > 0 {
fmt.Fprintf(os.Stderr, "\nFailed to delete %d server(s):\n", len(failed))
for _, f := range failed {
fmt.Fprintf(os.Stderr, " %s\n", f)
}
fmt.Fprintf(os.Stderr, "\nManual cleanup: delete servers at https://console.hetzner.cloud\n")
state.Status = StatusError
SaveState(state)
return fmt.Errorf("failed to delete %d server(s)", len(failed))
}
// Step 3: Remove state file
if err := DeleteState(state.Name); err != nil {
return fmt.Errorf("delete state: %w", err)
}
fmt.Printf("\nSandbox %q destroyed (%d servers deleted)\n", state.Name, len(state.Servers))
return nil
}
// resolveSandbox finds a sandbox by name or returns the active one.
func resolveSandbox(name string) (*SandboxState, error) {
if name != "" {
return LoadState(name)
}
// Find the active sandbox
active, err := FindActiveSandbox()
if err != nil {
return nil, err
}
if active == nil {
return nil, fmt.Errorf("no active sandbox found, specify --name")
}
return active, nil
}

438
pkg/cli/sandbox/hetzner.go Normal file
View File

@ -0,0 +1,438 @@
package sandbox
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"strconv"
"time"
)
const hetznerBaseURL = "https://api.hetzner.cloud/v1"
// HetznerClient is a minimal Hetzner Cloud API client.
type HetznerClient struct {
token string
httpClient *http.Client
}
// NewHetznerClient creates a new Hetzner API client.
func NewHetznerClient(token string) *HetznerClient {
return &HetznerClient{
token: token,
httpClient: &http.Client{
Timeout: 30 * time.Second,
},
}
}
// --- Request helpers ---
func (c *HetznerClient) doRequest(method, path string, body interface{}) ([]byte, int, error) {
var bodyReader io.Reader
if body != nil {
data, err := json.Marshal(body)
if err != nil {
return nil, 0, fmt.Errorf("marshal request body: %w", err)
}
bodyReader = bytes.NewReader(data)
}
req, err := http.NewRequest(method, hetznerBaseURL+path, bodyReader)
if err != nil {
return nil, 0, fmt.Errorf("create request: %w", err)
}
req.Header.Set("Authorization", "Bearer "+c.token)
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, 0, fmt.Errorf("request %s %s: %w", method, path, err)
}
defer resp.Body.Close()
respBody, err := io.ReadAll(resp.Body)
if err != nil {
return nil, resp.StatusCode, fmt.Errorf("read response: %w", err)
}
return respBody, resp.StatusCode, nil
}
func (c *HetznerClient) get(path string) ([]byte, error) {
body, status, err := c.doRequest("GET", path, nil)
if err != nil {
return nil, err
}
if status < 200 || status >= 300 {
return nil, parseHetznerError(body, status)
}
return body, nil
}
func (c *HetznerClient) post(path string, payload interface{}) ([]byte, error) {
body, status, err := c.doRequest("POST", path, payload)
if err != nil {
return nil, err
}
if status < 200 || status >= 300 {
return nil, parseHetznerError(body, status)
}
return body, nil
}
func (c *HetznerClient) delete(path string) error {
_, status, err := c.doRequest("DELETE", path, nil)
if err != nil {
return err
}
if status < 200 || status >= 300 {
return fmt.Errorf("delete %s: HTTP %d", path, status)
}
return nil
}
// --- API types ---
// HetznerServer represents a Hetzner Cloud server.
type HetznerServer struct {
ID int64 `json:"id"`
Name string `json:"name"`
Status string `json:"status"` // initializing, running, off, ...
PublicNet HetznerPublicNet `json:"public_net"`
Labels map[string]string `json:"labels"`
ServerType struct {
Name string `json:"name"`
} `json:"server_type"`
}
// HetznerPublicNet holds public networking info for a server.
type HetznerPublicNet struct {
IPv4 struct {
IP string `json:"ip"`
} `json:"ipv4"`
}
// HetznerFloatingIP represents a Hetzner floating IP.
type HetznerFloatingIP struct {
ID int64 `json:"id"`
IP string `json:"ip"`
Server *int64 `json:"server"` // nil if unassigned
Labels map[string]string `json:"labels"`
Description string `json:"description"`
HomeLocation struct {
Name string `json:"name"`
} `json:"home_location"`
}
// HetznerSSHKey represents a Hetzner SSH key.
type HetznerSSHKey struct {
ID int64 `json:"id"`
Name string `json:"name"`
Fingerprint string `json:"fingerprint"`
PublicKey string `json:"public_key"`
}
// HetznerFirewall represents a Hetzner firewall.
type HetznerFirewall struct {
ID int64 `json:"id"`
Name string `json:"name"`
Rules []HetznerFWRule `json:"rules"`
Labels map[string]string `json:"labels"`
}
// HetznerFWRule represents a firewall rule.
type HetznerFWRule struct {
Direction string `json:"direction"`
Protocol string `json:"protocol"`
Port string `json:"port"`
SourceIPs []string `json:"source_ips"`
Description string `json:"description,omitempty"`
}
// HetznerError represents an API error response.
type HetznerError struct {
Error struct {
Code string `json:"code"`
Message string `json:"message"`
} `json:"error"`
}
func parseHetznerError(body []byte, status int) error {
var he HetznerError
if err := json.Unmarshal(body, &he); err == nil && he.Error.Message != "" {
return fmt.Errorf("hetzner API error (HTTP %d): %s — %s", status, he.Error.Code, he.Error.Message)
}
return fmt.Errorf("hetzner API error: HTTP %d", status)
}
// --- Server operations ---
// CreateServerRequest holds parameters for server creation.
type CreateServerRequest struct {
Name string `json:"name"`
ServerType string `json:"server_type"`
Image string `json:"image"`
Location string `json:"location"`
SSHKeys []int64 `json:"ssh_keys"`
Labels map[string]string `json:"labels"`
Firewalls []struct {
Firewall int64 `json:"firewall"`
} `json:"firewalls,omitempty"`
}
// CreateServer creates a new server and returns it.
func (c *HetznerClient) CreateServer(req CreateServerRequest) (*HetznerServer, error) {
body, err := c.post("/servers", req)
if err != nil {
return nil, fmt.Errorf("create server %q: %w", req.Name, err)
}
var resp struct {
Server HetznerServer `json:"server"`
}
if err := json.Unmarshal(body, &resp); err != nil {
return nil, fmt.Errorf("parse create server response: %w", err)
}
return &resp.Server, nil
}
// GetServer retrieves a server by ID.
func (c *HetznerClient) GetServer(id int64) (*HetznerServer, error) {
body, err := c.get("/servers/" + strconv.FormatInt(id, 10))
if err != nil {
return nil, fmt.Errorf("get server %d: %w", id, err)
}
var resp struct {
Server HetznerServer `json:"server"`
}
if err := json.Unmarshal(body, &resp); err != nil {
return nil, fmt.Errorf("parse server response: %w", err)
}
return &resp.Server, nil
}
// DeleteServer deletes a server by ID.
func (c *HetznerClient) DeleteServer(id int64) error {
return c.delete("/servers/" + strconv.FormatInt(id, 10))
}
// ListServersByLabel lists servers filtered by a label selector.
func (c *HetznerClient) ListServersByLabel(selector string) ([]HetznerServer, error) {
body, err := c.get("/servers?label_selector=" + selector)
if err != nil {
return nil, fmt.Errorf("list servers: %w", err)
}
var resp struct {
Servers []HetznerServer `json:"servers"`
}
if err := json.Unmarshal(body, &resp); err != nil {
return nil, fmt.Errorf("parse servers response: %w", err)
}
return resp.Servers, nil
}
// WaitForServer polls until the server reaches "running" status.
func (c *HetznerClient) WaitForServer(id int64, timeout time.Duration) (*HetznerServer, error) {
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
srv, err := c.GetServer(id)
if err != nil {
return nil, err
}
if srv.Status == "running" {
return srv, nil
}
time.Sleep(3 * time.Second)
}
return nil, fmt.Errorf("server %d did not reach running state within %s", id, timeout)
}
// --- Floating IP operations ---
// CreateFloatingIP creates a new floating IP.
func (c *HetznerClient) CreateFloatingIP(location, description string, labels map[string]string) (*HetznerFloatingIP, error) {
payload := map[string]interface{}{
"type": "ipv4",
"home_location": location,
"description": description,
"labels": labels,
}
body, err := c.post("/floating_ips", payload)
if err != nil {
return nil, fmt.Errorf("create floating IP: %w", err)
}
var resp struct {
FloatingIP HetznerFloatingIP `json:"floating_ip"`
}
if err := json.Unmarshal(body, &resp); err != nil {
return nil, fmt.Errorf("parse floating IP response: %w", err)
}
return &resp.FloatingIP, nil
}
// ListFloatingIPsByLabel lists floating IPs filtered by label.
func (c *HetznerClient) ListFloatingIPsByLabel(selector string) ([]HetznerFloatingIP, error) {
body, err := c.get("/floating_ips?label_selector=" + selector)
if err != nil {
return nil, fmt.Errorf("list floating IPs: %w", err)
}
var resp struct {
FloatingIPs []HetznerFloatingIP `json:"floating_ips"`
}
if err := json.Unmarshal(body, &resp); err != nil {
return nil, fmt.Errorf("parse floating IPs response: %w", err)
}
return resp.FloatingIPs, nil
}
// AssignFloatingIP assigns a floating IP to a server.
func (c *HetznerClient) AssignFloatingIP(floatingIPID, serverID int64) error {
payload := map[string]int64{"server": serverID}
_, err := c.post("/floating_ips/"+strconv.FormatInt(floatingIPID, 10)+"/actions/assign", payload)
if err != nil {
return fmt.Errorf("assign floating IP %d to server %d: %w", floatingIPID, serverID, err)
}
return nil
}
// UnassignFloatingIP removes a floating IP assignment.
func (c *HetznerClient) UnassignFloatingIP(floatingIPID int64) error {
_, err := c.post("/floating_ips/"+strconv.FormatInt(floatingIPID, 10)+"/actions/unassign", struct{}{})
if err != nil {
return fmt.Errorf("unassign floating IP %d: %w", floatingIPID, err)
}
return nil
}
// --- SSH Key operations ---
// UploadSSHKey uploads a public key to Hetzner.
func (c *HetznerClient) UploadSSHKey(name, publicKey string) (*HetznerSSHKey, error) {
payload := map[string]string{
"name": name,
"public_key": publicKey,
}
body, err := c.post("/ssh_keys", payload)
if err != nil {
return nil, fmt.Errorf("upload SSH key: %w", err)
}
var resp struct {
SSHKey HetznerSSHKey `json:"ssh_key"`
}
if err := json.Unmarshal(body, &resp); err != nil {
return nil, fmt.Errorf("parse SSH key response: %w", err)
}
return &resp.SSHKey, nil
}
// GetSSHKey retrieves an SSH key by ID.
func (c *HetznerClient) GetSSHKey(id int64) (*HetznerSSHKey, error) {
body, err := c.get("/ssh_keys/" + strconv.FormatInt(id, 10))
if err != nil {
return nil, fmt.Errorf("get SSH key %d: %w", id, err)
}
var resp struct {
SSHKey HetznerSSHKey `json:"ssh_key"`
}
if err := json.Unmarshal(body, &resp); err != nil {
return nil, fmt.Errorf("parse SSH key response: %w", err)
}
return &resp.SSHKey, nil
}
// --- Firewall operations ---
// CreateFirewall creates a firewall with the given rules.
func (c *HetznerClient) CreateFirewall(name string, rules []HetznerFWRule, labels map[string]string) (*HetznerFirewall, error) {
payload := map[string]interface{}{
"name": name,
"rules": rules,
"labels": labels,
}
body, err := c.post("/firewalls", payload)
if err != nil {
return nil, fmt.Errorf("create firewall: %w", err)
}
var resp struct {
Firewall HetznerFirewall `json:"firewall"`
}
if err := json.Unmarshal(body, &resp); err != nil {
return nil, fmt.Errorf("parse firewall response: %w", err)
}
return &resp.Firewall, nil
}
// ListFirewallsByLabel lists firewalls filtered by label.
func (c *HetznerClient) ListFirewallsByLabel(selector string) ([]HetznerFirewall, error) {
body, err := c.get("/firewalls?label_selector=" + selector)
if err != nil {
return nil, fmt.Errorf("list firewalls: %w", err)
}
var resp struct {
Firewalls []HetznerFirewall `json:"firewalls"`
}
if err := json.Unmarshal(body, &resp); err != nil {
return nil, fmt.Errorf("parse firewalls response: %w", err)
}
return resp.Firewalls, nil
}
// DeleteFirewall deletes a firewall by ID.
func (c *HetznerClient) DeleteFirewall(id int64) error {
return c.delete("/firewalls/" + strconv.FormatInt(id, 10))
}
// --- Validation ---
// ValidateToken checks if the API token is valid by making a simple request.
func (c *HetznerClient) ValidateToken() error {
_, err := c.get("/servers?per_page=1")
if err != nil {
return fmt.Errorf("invalid Hetzner API token: %w", err)
}
return nil
}
// --- Sandbox firewall rules ---
// SandboxFirewallRules returns the standard firewall rules for sandbox nodes.
func SandboxFirewallRules() []HetznerFWRule {
allIPv4 := []string{"0.0.0.0/0"}
allIPv6 := []string{"::/0"}
allIPs := append(allIPv4, allIPv6...)
return []HetznerFWRule{
{Direction: "in", Protocol: "tcp", Port: "22", SourceIPs: allIPs, Description: "SSH"},
{Direction: "in", Protocol: "tcp", Port: "53", SourceIPs: allIPs, Description: "DNS TCP"},
{Direction: "in", Protocol: "udp", Port: "53", SourceIPs: allIPs, Description: "DNS UDP"},
{Direction: "in", Protocol: "tcp", Port: "80", SourceIPs: allIPs, Description: "HTTP"},
{Direction: "in", Protocol: "tcp", Port: "443", SourceIPs: allIPs, Description: "HTTPS"},
{Direction: "in", Protocol: "udp", Port: "51820", SourceIPs: allIPs, Description: "WireGuard"},
}
}

View File

@ -0,0 +1,303 @@
package sandbox
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
)
func TestValidateToken_Success(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer test-token" {
t.Errorf("unexpected auth header: %s", r.Header.Get("Authorization"))
}
w.WriteHeader(200)
json.NewEncoder(w).Encode(map[string]interface{}{"servers": []interface{}{}})
}))
defer srv.Close()
client := newTestClient(srv, "test-token")
if err := client.ValidateToken(); err != nil {
t.Errorf("ValidateToken() error = %v, want nil", err)
}
}
func TestValidateToken_InvalidToken(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(401)
json.NewEncoder(w).Encode(map[string]interface{}{
"error": map[string]string{
"code": "unauthorized",
"message": "unable to authenticate",
},
})
}))
defer srv.Close()
client := newTestClient(srv, "bad-token")
if err := client.ValidateToken(); err == nil {
t.Error("ValidateToken() expected error for invalid token")
}
}
func TestCreateServer(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" || r.URL.Path != "/v1/servers" {
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
}
var req CreateServerRequest
json.NewDecoder(r.Body).Decode(&req)
if req.Name != "sbx-test-1" {
t.Errorf("unexpected server name: %s", req.Name)
}
if req.ServerType != "cx22" {
t.Errorf("unexpected server type: %s", req.ServerType)
}
w.WriteHeader(201)
json.NewEncoder(w).Encode(map[string]interface{}{
"server": map[string]interface{}{
"id": 12345,
"name": req.Name,
"status": "initializing",
"public_net": map[string]interface{}{
"ipv4": map[string]string{"ip": "1.2.3.4"},
},
"labels": req.Labels,
"server_type": map[string]string{"name": "cx22"},
},
})
}))
defer srv.Close()
client := newTestClient(srv, "test-token")
server, err := client.CreateServer(CreateServerRequest{
Name: "sbx-test-1",
ServerType: "cx22",
Image: "ubuntu-24.04",
Location: "fsn1",
SSHKeys: []int64{1},
Labels: map[string]string{"orama-sandbox": "test"},
})
if err != nil {
t.Fatalf("CreateServer() error = %v", err)
}
if server.ID != 12345 {
t.Errorf("server ID = %d, want 12345", server.ID)
}
if server.Name != "sbx-test-1" {
t.Errorf("server name = %s, want sbx-test-1", server.Name)
}
if server.PublicNet.IPv4.IP != "1.2.3.4" {
t.Errorf("server IP = %s, want 1.2.3.4", server.PublicNet.IPv4.IP)
}
}
func TestDeleteServer(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != "DELETE" || r.URL.Path != "/v1/servers/12345" {
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
}
w.WriteHeader(200)
}))
defer srv.Close()
client := newTestClient(srv, "test-token")
if err := client.DeleteServer(12345); err != nil {
t.Errorf("DeleteServer() error = %v", err)
}
}
func TestListServersByLabel(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Get("label_selector") != "orama-sandbox=test" {
t.Errorf("unexpected label_selector: %s", r.URL.Query().Get("label_selector"))
}
w.WriteHeader(200)
json.NewEncoder(w).Encode(map[string]interface{}{
"servers": []map[string]interface{}{
{"id": 1, "name": "sbx-test-1", "status": "running", "public_net": map[string]interface{}{"ipv4": map[string]string{"ip": "1.1.1.1"}}, "server_type": map[string]string{"name": "cx22"}},
{"id": 2, "name": "sbx-test-2", "status": "running", "public_net": map[string]interface{}{"ipv4": map[string]string{"ip": "2.2.2.2"}}, "server_type": map[string]string{"name": "cx22"}},
},
})
}))
defer srv.Close()
client := newTestClient(srv, "test-token")
servers, err := client.ListServersByLabel("orama-sandbox=test")
if err != nil {
t.Fatalf("ListServersByLabel() error = %v", err)
}
if len(servers) != 2 {
t.Errorf("got %d servers, want 2", len(servers))
}
}
func TestWaitForServer_AlreadyRunning(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
json.NewEncoder(w).Encode(map[string]interface{}{
"server": map[string]interface{}{
"id": 1,
"name": "test",
"status": "running",
"public_net": map[string]interface{}{
"ipv4": map[string]string{"ip": "1.1.1.1"},
},
"server_type": map[string]string{"name": "cx22"},
},
})
}))
defer srv.Close()
client := newTestClient(srv, "test-token")
server, err := client.WaitForServer(1, 5*time.Second)
if err != nil {
t.Fatalf("WaitForServer() error = %v", err)
}
if server.Status != "running" {
t.Errorf("server status = %s, want running", server.Status)
}
}
func TestAssignFloatingIP(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" || r.URL.Path != "/v1/floating_ips/100/actions/assign" {
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
}
var body map[string]int64
json.NewDecoder(r.Body).Decode(&body)
if body["server"] != 200 {
t.Errorf("unexpected server ID: %d", body["server"])
}
w.WriteHeader(200)
json.NewEncoder(w).Encode(map[string]interface{}{"action": map[string]interface{}{"id": 1, "status": "running"}})
}))
defer srv.Close()
client := newTestClient(srv, "test-token")
if err := client.AssignFloatingIP(100, 200); err != nil {
t.Errorf("AssignFloatingIP() error = %v", err)
}
}
func TestUploadSSHKey(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" || r.URL.Path != "/v1/ssh_keys" {
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
}
w.WriteHeader(201)
json.NewEncoder(w).Encode(map[string]interface{}{
"ssh_key": map[string]interface{}{
"id": 42,
"name": "orama-sandbox",
"fingerprint": "aa:bb:cc:dd",
"public_key": "ssh-ed25519 AAAA...",
},
})
}))
defer srv.Close()
client := newTestClient(srv, "test-token")
key, err := client.UploadSSHKey("orama-sandbox", "ssh-ed25519 AAAA...")
if err != nil {
t.Fatalf("UploadSSHKey() error = %v", err)
}
if key.ID != 42 {
t.Errorf("key ID = %d, want 42", key.ID)
}
}
func TestCreateFirewall(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" || r.URL.Path != "/v1/firewalls" {
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
}
w.WriteHeader(201)
json.NewEncoder(w).Encode(map[string]interface{}{
"firewall": map[string]interface{}{
"id": 99,
"name": "orama-sandbox",
},
})
}))
defer srv.Close()
client := newTestClient(srv, "test-token")
fw, err := client.CreateFirewall("orama-sandbox", SandboxFirewallRules(), map[string]string{"orama-sandbox": "infra"})
if err != nil {
t.Fatalf("CreateFirewall() error = %v", err)
}
if fw.ID != 99 {
t.Errorf("firewall ID = %d, want 99", fw.ID)
}
}
func TestSandboxFirewallRules(t *testing.T) {
rules := SandboxFirewallRules()
if len(rules) != 6 {
t.Errorf("got %d rules, want 6", len(rules))
}
expectedPorts := map[string]bool{"22": false, "53": false, "80": false, "443": false, "51820": false}
for _, r := range rules {
expectedPorts[r.Port] = true
if r.Direction != "in" {
t.Errorf("rule %s direction = %s, want in", r.Port, r.Direction)
}
}
for port, seen := range expectedPorts {
if !seen {
t.Errorf("missing firewall rule for port %s", port)
}
}
}
func TestParseHetznerError(t *testing.T) {
body := `{"error":{"code":"uniqueness_error","message":"server name already used"}}`
err := parseHetznerError([]byte(body), 409)
if err == nil {
t.Fatal("expected error")
}
expected := "hetzner API error (HTTP 409): uniqueness_error — server name already used"
if err.Error() != expected {
t.Errorf("error = %q, want %q", err.Error(), expected)
}
}
// newTestClient creates a HetznerClient pointing at a test server.
func newTestClient(ts *httptest.Server, token string) *HetznerClient {
client := NewHetznerClient(token)
// Override the base URL by using a custom transport
client.httpClient = ts.Client()
// We need to override the base URL — wrap the transport
origTransport := client.httpClient.Transport
client.httpClient.Transport = &testTransport{
base: origTransport,
testURL: ts.URL,
}
return client
}
// testTransport rewrites requests to point at the test server.
type testTransport struct {
base http.RoundTripper
testURL string
}
func (t *testTransport) RoundTrip(req *http.Request) (*http.Response, error) {
// Rewrite the URL to point at the test server
req.URL.Scheme = "http"
req.URL.Host = t.testURL[len("http://"):]
if t.base != nil {
return t.base.RoundTrip(req)
}
return http.DefaultTransport.RoundTrip(req)
}

26
pkg/cli/sandbox/names.go Normal file
View File

@ -0,0 +1,26 @@
package sandbox
import (
"math/rand"
)
var adjectives = []string{
"swift", "bright", "calm", "dark", "eager",
"fair", "gold", "hazy", "iron", "jade",
"keen", "lush", "mild", "neat", "opal",
"pure", "raw", "sage", "teal", "warm",
}
var nouns = []string{
"falcon", "beacon", "cedar", "delta", "ember",
"frost", "grove", "haven", "ivory", "jewel",
"knot", "latch", "maple", "nexus", "orbit",
"prism", "reef", "spark", "tide", "vault",
}
// GenerateName produces a random adjective-noun name like "swift-falcon".
func GenerateName() string {
adj := adjectives[rand.Intn(len(adjectives))]
noun := nouns[rand.Intn(len(nouns))]
return adj + "-" + noun
}

137
pkg/cli/sandbox/rollout.go Normal file
View File

@ -0,0 +1,137 @@
package sandbox
import (
"fmt"
"os"
"path/filepath"
"time"
"github.com/DeBrosOfficial/network/pkg/cli/remotessh"
"github.com/DeBrosOfficial/network/pkg/inspector"
)
// Rollout builds, pushes, and performs a rolling upgrade on a sandbox cluster.
func Rollout(name string) error {
cfg, err := LoadConfig()
if err != nil {
return err
}
state, err := resolveSandbox(name)
if err != nil {
return err
}
sshKeyPath := cfg.ExpandedPrivateKeyPath()
fmt.Printf("Rolling out to sandbox %q (%d nodes)\n\n", state.Name, len(state.Servers))
// Step 1: Find or require binary archive
archivePath := findNewestArchive()
if archivePath == "" {
return fmt.Errorf("no binary archive found in /tmp/ (run `orama build` first)")
}
info, _ := os.Stat(archivePath)
fmt.Printf("Archive: %s (%s)\n\n", filepath.Base(archivePath), formatBytes(info.Size()))
// Step 2: Push archive to all nodes
fmt.Println("Pushing archive to all nodes...")
remotePath := "/tmp/" + filepath.Base(archivePath)
for i, srv := range state.Servers {
node := inspector.Node{User: "root", Host: srv.IP, SSHKey: sshKeyPath}
fmt.Printf(" [%d/%d] Uploading to %s...\n", i+1, len(state.Servers), srv.Name)
if err := remotessh.UploadFile(node, archivePath, remotePath); err != nil {
return fmt.Errorf("upload to %s: %w", srv.Name, err)
}
// Extract archive
extractCmd := fmt.Sprintf("mkdir -p /opt/orama && tar xzf %s -C /opt/orama && rm -f %s",
remotePath, remotePath)
if err := remotessh.RunSSHStreaming(node, extractCmd); err != nil {
return fmt.Errorf("extract on %s: %w", srv.Name, err)
}
}
// Step 3: Rolling upgrade — followers first, leader last
fmt.Println("\nRolling upgrade (followers first, leader last)...")
// Find the leader
leaderIdx := findLeaderIndex(state, sshKeyPath)
if leaderIdx < 0 {
fmt.Fprintf(os.Stderr, " Warning: could not detect RQLite leader, upgrading in order\n")
}
// Upgrade non-leaders first
for i, srv := range state.Servers {
if i == leaderIdx {
continue // skip leader, do it last
}
if err := upgradeNode(srv, sshKeyPath, i+1, len(state.Servers)); err != nil {
return err
}
// Wait between nodes
if i < len(state.Servers)-1 {
fmt.Printf(" Waiting 15s before next node...\n")
time.Sleep(15 * time.Second)
}
}
// Upgrade leader last
if leaderIdx >= 0 {
srv := state.Servers[leaderIdx]
if err := upgradeNode(srv, sshKeyPath, len(state.Servers), len(state.Servers)); err != nil {
return err
}
}
fmt.Printf("\nRollout complete for sandbox %q\n", state.Name)
return nil
}
// findLeaderIndex returns the index of the RQLite leader node, or -1 if unknown.
func findLeaderIndex(state *SandboxState, sshKeyPath string) int {
for i, srv := range state.Servers {
node := inspector.Node{User: "root", Host: srv.IP, SSHKey: sshKeyPath}
out, err := runSSHOutput(node, "curl -sf http://localhost:5001/status 2>/dev/null | grep -o '\"state\":\"[^\"]*\"'")
if err == nil && contains(out, "Leader") {
return i
}
}
return -1
}
// upgradeNode performs `orama node upgrade --restart` on a single node.
func upgradeNode(srv ServerState, sshKeyPath string, current, total int) error {
node := inspector.Node{User: "root", Host: srv.IP, SSHKey: sshKeyPath}
fmt.Printf(" [%d/%d] Upgrading %s (%s)...\n", current, total, srv.Name, srv.IP)
if err := remotessh.RunSSHStreaming(node, "orama node upgrade --restart"); err != nil {
return fmt.Errorf("upgrade %s: %w", srv.Name, err)
}
// Wait for health
fmt.Printf(" Checking health...")
if err := waitForRQLiteHealth(node, 2*time.Minute); err != nil {
fmt.Printf(" WARN: %v\n", err)
} else {
fmt.Println(" OK")
}
return nil
}
// contains checks if s contains substr.
func contains(s, substr string) bool {
return len(s) >= len(substr) && findSubstring(s, substr)
}
func findSubstring(s, substr string) bool {
for i := 0; i <= len(s)-len(substr); i++ {
if s[i:i+len(substr)] == substr {
return true
}
}
return false
}

319
pkg/cli/sandbox/setup.go Normal file
View File

@ -0,0 +1,319 @@
package sandbox
import (
"bufio"
"crypto/ed25519"
"crypto/rand"
"encoding/pem"
"fmt"
"os"
"os/exec"
"strings"
"golang.org/x/crypto/ssh"
)
// Setup runs the interactive sandbox setup wizard.
func Setup() error {
fmt.Println("Orama Sandbox Setup")
fmt.Println("====================")
fmt.Println()
reader := bufio.NewReader(os.Stdin)
// Step 1: Hetzner API token
fmt.Print("Hetzner Cloud API token: ")
token, err := reader.ReadString('\n')
if err != nil {
return fmt.Errorf("read token: %w", err)
}
token = strings.TrimSpace(token)
if token == "" {
return fmt.Errorf("API token is required")
}
fmt.Print(" Validating token... ")
client := NewHetznerClient(token)
if err := client.ValidateToken(); err != nil {
fmt.Println("FAILED")
return fmt.Errorf("invalid token: %w", err)
}
fmt.Println("OK")
fmt.Println()
// Step 2: Domain
fmt.Print("Sandbox domain (e.g., sbx.dbrs.space): ")
domain, err := reader.ReadString('\n')
if err != nil {
return fmt.Errorf("read domain: %w", err)
}
domain = strings.TrimSpace(domain)
if domain == "" {
return fmt.Errorf("domain is required")
}
cfg := &Config{
HetznerAPIToken: token,
Domain: domain,
}
cfg.Defaults()
// Step 3: Floating IPs
fmt.Println()
fmt.Println("Checking floating IPs...")
floatingIPs, err := setupFloatingIPs(client, cfg.Location)
if err != nil {
return err
}
cfg.FloatingIPs = floatingIPs
// Step 4: Firewall
fmt.Println()
fmt.Println("Checking firewall...")
fwID, err := setupFirewall(client)
if err != nil {
return err
}
cfg.FirewallID = fwID
// Step 5: SSH key
fmt.Println()
fmt.Println("Setting up SSH key...")
sshKeyConfig, err := setupSSHKey(client)
if err != nil {
return err
}
cfg.SSHKey = sshKeyConfig
// Step 6: Display DNS instructions
fmt.Println()
fmt.Println("DNS Configuration")
fmt.Println("-----------------")
fmt.Println("Configure the following at your domain registrar:")
fmt.Println()
fmt.Printf(" 1. Add glue records (Personal DNS Servers):\n")
fmt.Printf(" ns1.%s -> %s\n", domain, cfg.FloatingIPs[0].IP)
fmt.Printf(" ns2.%s -> %s\n", domain, cfg.FloatingIPs[1].IP)
fmt.Println()
fmt.Printf(" 2. Set custom nameservers for %s:\n", domain)
fmt.Printf(" ns1.%s\n", domain)
fmt.Printf(" ns2.%s\n", domain)
fmt.Println()
// Step 7: Verify DNS (optional)
fmt.Print("Verify DNS now? [y/N]: ")
verifyChoice, _ := reader.ReadString('\n')
verifyChoice = strings.TrimSpace(strings.ToLower(verifyChoice))
if verifyChoice == "y" || verifyChoice == "yes" {
verifyDNS(domain)
}
// Save config
if err := SaveConfig(cfg); err != nil {
return fmt.Errorf("save config: %w", err)
}
fmt.Println()
fmt.Println("Setup complete! Config saved to ~/.orama/sandbox.yaml")
fmt.Println()
fmt.Println("Next: orama sandbox create")
return nil
}
// setupFloatingIPs checks for existing floating IPs or creates new ones.
func setupFloatingIPs(client *HetznerClient, location string) ([]FloatIP, error) {
existing, err := client.ListFloatingIPsByLabel("orama-sandbox-dns=true")
if err != nil {
return nil, fmt.Errorf("list floating IPs: %w", err)
}
if len(existing) >= 2 {
fmt.Printf(" Found %d existing floating IPs:\n", len(existing))
result := make([]FloatIP, 2)
for i := 0; i < 2; i++ {
fmt.Printf(" ns%d: %s (ID: %d)\n", i+1, existing[i].IP, existing[i].ID)
result[i] = FloatIP{ID: existing[i].ID, IP: existing[i].IP}
}
return result, nil
}
// Need to create missing floating IPs
needed := 2 - len(existing)
fmt.Printf(" Need to create %d floating IP(s)...\n", needed)
reader := bufio.NewReader(os.Stdin)
fmt.Printf(" Create %d floating IP(s) in %s? (~$0.005/hr each) [Y/n]: ", needed, location)
choice, _ := reader.ReadString('\n')
choice = strings.TrimSpace(strings.ToLower(choice))
if choice == "n" || choice == "no" {
return nil, fmt.Errorf("floating IPs required, aborting setup")
}
result := make([]FloatIP, 0, 2)
for _, fip := range existing {
result = append(result, FloatIP{ID: fip.ID, IP: fip.IP})
}
for i := len(existing); i < 2; i++ {
desc := fmt.Sprintf("orama-sandbox-ns%d", i+1)
labels := map[string]string{"orama-sandbox-dns": "true"}
fip, err := client.CreateFloatingIP(location, desc, labels)
if err != nil {
return nil, fmt.Errorf("create floating IP %d: %w", i+1, err)
}
fmt.Printf(" Created ns%d: %s (ID: %d)\n", i+1, fip.IP, fip.ID)
result = append(result, FloatIP{ID: fip.ID, IP: fip.IP})
}
return result, nil
}
// setupFirewall ensures a sandbox firewall exists.
func setupFirewall(client *HetznerClient) (int64, error) {
existing, err := client.ListFirewallsByLabel("orama-sandbox=infra")
if err != nil {
return 0, fmt.Errorf("list firewalls: %w", err)
}
if len(existing) > 0 {
fmt.Printf(" Found existing firewall: %s (ID: %d)\n", existing[0].Name, existing[0].ID)
return existing[0].ID, nil
}
fmt.Print(" Creating sandbox firewall... ")
fw, err := client.CreateFirewall(
"orama-sandbox",
SandboxFirewallRules(),
map[string]string{"orama-sandbox": "infra"},
)
if err != nil {
fmt.Println("FAILED")
return 0, fmt.Errorf("create firewall: %w", err)
}
fmt.Printf("OK (ID: %d)\n", fw.ID)
return fw.ID, nil
}
// setupSSHKey generates an SSH keypair and uploads it to Hetzner.
func setupSSHKey(client *HetznerClient) (SSHKeyConfig, error) {
dir, err := configDir()
if err != nil {
return SSHKeyConfig{}, err
}
privPath := dir + "/sandbox_key"
pubPath := privPath + ".pub"
// Check for existing key
if _, err := os.Stat(privPath); err == nil {
fmt.Printf(" SSH key already exists: %s\n", privPath)
// Read public key and check if it's on Hetzner
pubData, err := os.ReadFile(pubPath)
if err != nil {
return SSHKeyConfig{}, fmt.Errorf("read public key: %w", err)
}
// Try to upload (will fail with uniqueness error if already exists)
key, err := client.UploadSSHKey("orama-sandbox", strings.TrimSpace(string(pubData)))
if err != nil {
// Key likely already exists on Hetzner — find it by listing
fmt.Printf(" SSH key may already be on Hetzner (upload: %v)\n", err)
fmt.Print(" Enter the Hetzner SSH key ID (or 0 to re-upload): ")
reader := bufio.NewReader(os.Stdin)
idStr, _ := reader.ReadString('\n')
idStr = strings.TrimSpace(idStr)
var hetznerID int64
fmt.Sscanf(idStr, "%d", &hetznerID)
if hetznerID == 0 {
return SSHKeyConfig{}, fmt.Errorf("could not resolve SSH key on Hetzner, try deleting and re-running setup")
}
return SSHKeyConfig{
HetznerID: hetznerID,
PrivateKeyPath: "~/.orama/sandbox_key",
PublicKeyPath: "~/.orama/sandbox_key.pub",
}, nil
}
fmt.Printf(" Uploaded to Hetzner (ID: %d)\n", key.ID)
return SSHKeyConfig{
HetznerID: key.ID,
PrivateKeyPath: "~/.orama/sandbox_key",
PublicKeyPath: "~/.orama/sandbox_key.pub",
}, nil
}
// Generate new ed25519 keypair
fmt.Print(" Generating ed25519 keypair... ")
pubKey, privKey, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
fmt.Println("FAILED")
return SSHKeyConfig{}, fmt.Errorf("generate key: %w", err)
}
// Marshal private key to OpenSSH format
pemBlock, err := ssh.MarshalPrivateKey(privKey, "")
if err != nil {
fmt.Println("FAILED")
return SSHKeyConfig{}, fmt.Errorf("marshal private key: %w", err)
}
privPEM := pem.EncodeToMemory(pemBlock)
if err := os.WriteFile(privPath, privPEM, 0600); err != nil {
fmt.Println("FAILED")
return SSHKeyConfig{}, fmt.Errorf("write private key: %w", err)
}
// Marshal public key to authorized_keys format
sshPubKey, err := ssh.NewPublicKey(pubKey)
if err != nil {
return SSHKeyConfig{}, fmt.Errorf("convert public key: %w", err)
}
pubStr := strings.TrimSpace(string(ssh.MarshalAuthorizedKey(sshPubKey)))
if err := os.WriteFile(pubPath, []byte(pubStr+"\n"), 0644); err != nil {
return SSHKeyConfig{}, fmt.Errorf("write public key: %w", err)
}
fmt.Println("OK")
// Upload to Hetzner
fmt.Print(" Uploading to Hetzner... ")
key, err := client.UploadSSHKey("orama-sandbox", pubStr)
if err != nil {
fmt.Println("FAILED")
return SSHKeyConfig{}, fmt.Errorf("upload SSH key: %w", err)
}
fmt.Printf("OK (ID: %d)\n", key.ID)
return SSHKeyConfig{
HetznerID: key.ID,
PrivateKeyPath: "~/.orama/sandbox_key",
PublicKeyPath: "~/.orama/sandbox_key.pub",
}, nil
}
// verifyDNS checks if the sandbox domain resolves.
func verifyDNS(domain string) {
fmt.Printf(" Checking NS records for %s...\n", domain)
out, err := exec.Command("dig", "+short", "NS", domain, "@8.8.8.8").Output()
if err != nil {
fmt.Printf(" Warning: dig failed: %v\n", err)
fmt.Println(" DNS verification skipped. You can verify later with:")
fmt.Printf(" dig NS %s @8.8.8.8\n", domain)
return
}
result := strings.TrimSpace(string(out))
if result == "" {
fmt.Println(" Warning: No NS records found yet.")
fmt.Println(" DNS propagation can take up to 48 hours.")
fmt.Println(" The sandbox will still work once DNS is configured.")
} else {
fmt.Printf(" NS records:\n")
for _, line := range strings.Split(result, "\n") {
fmt.Printf(" %s\n", line)
}
}
}

View File

@ -0,0 +1,56 @@
package sandbox
import (
"fmt"
"os"
"syscall"
)
// SSHInto opens an interactive SSH session to a sandbox node.
func SSHInto(name string, nodeNum int) error {
cfg, err := LoadConfig()
if err != nil {
return err
}
state, err := resolveSandbox(name)
if err != nil {
return err
}
if nodeNum < 1 || nodeNum > len(state.Servers) {
return fmt.Errorf("node number must be between 1 and %d", len(state.Servers))
}
srv := state.Servers[nodeNum-1]
sshKeyPath := cfg.ExpandedPrivateKeyPath()
fmt.Printf("Connecting to %s (%s, %s)...\n", srv.Name, srv.IP, srv.Role)
// Find ssh binary
sshBin, err := findSSHBinary()
if err != nil {
return err
}
// Replace current process with SSH
args := []string{
"ssh",
"-o", "StrictHostKeyChecking=accept-new",
"-i", sshKeyPath,
fmt.Sprintf("root@%s", srv.IP),
}
return syscall.Exec(sshBin, args, os.Environ())
}
// findSSHBinary locates the ssh binary in PATH.
func findSSHBinary() (string, error) {
paths := []string{"/usr/bin/ssh", "/usr/local/bin/ssh", "/opt/homebrew/bin/ssh"}
for _, p := range paths {
if _, err := os.Stat(p); err == nil {
return p, nil
}
}
return "", fmt.Errorf("ssh binary not found")
}

211
pkg/cli/sandbox/state.go Normal file
View File

@ -0,0 +1,211 @@
package sandbox
import (
"fmt"
"os"
"path/filepath"
"strings"
"time"
"github.com/DeBrosOfficial/network/pkg/inspector"
"gopkg.in/yaml.v3"
)
// SandboxStatus represents the lifecycle state of a sandbox.
type SandboxStatus string
const (
StatusCreating SandboxStatus = "creating"
StatusRunning SandboxStatus = "running"
StatusDestroying SandboxStatus = "destroying"
StatusError SandboxStatus = "error"
)
// SandboxState holds the full state of an active sandbox cluster.
type SandboxState struct {
Name string `yaml:"name"`
CreatedAt time.Time `yaml:"created_at"`
Domain string `yaml:"domain"`
Status SandboxStatus `yaml:"status"`
Servers []ServerState `yaml:"servers"`
}
// ServerState holds the state of a single server in the sandbox.
type ServerState struct {
ID int64 `yaml:"id"` // Hetzner server ID
Name string `yaml:"name"` // e.g., sbx-feature-webrtc-1
IP string `yaml:"ip"` // Public IPv4
Role string `yaml:"role"` // "nameserver" or "node"
FloatingIP string `yaml:"floating_ip,omitempty"` // Only for nameserver nodes
WgIP string `yaml:"wg_ip,omitempty"` // WireGuard IP (populated after install)
}
// sandboxesDir returns ~/.orama/sandboxes/, creating it if needed.
func sandboxesDir() (string, error) {
dir, err := configDir()
if err != nil {
return "", err
}
sbxDir := filepath.Join(dir, "sandboxes")
if err := os.MkdirAll(sbxDir, 0700); err != nil {
return "", fmt.Errorf("create sandboxes directory: %w", err)
}
return sbxDir, nil
}
// statePath returns the path for a sandbox's state file.
func statePath(name string) (string, error) {
dir, err := sandboxesDir()
if err != nil {
return "", err
}
return filepath.Join(dir, name+".yaml"), nil
}
// SaveState persists the sandbox state to disk.
func SaveState(state *SandboxState) error {
path, err := statePath(state.Name)
if err != nil {
return err
}
data, err := yaml.Marshal(state)
if err != nil {
return fmt.Errorf("marshal state: %w", err)
}
if err := os.WriteFile(path, data, 0600); err != nil {
return fmt.Errorf("write state: %w", err)
}
return nil
}
// LoadState reads a sandbox state from disk.
func LoadState(name string) (*SandboxState, error) {
path, err := statePath(name)
if err != nil {
return nil, err
}
data, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return nil, fmt.Errorf("sandbox %q not found", name)
}
return nil, fmt.Errorf("read state: %w", err)
}
var state SandboxState
if err := yaml.Unmarshal(data, &state); err != nil {
return nil, fmt.Errorf("parse state: %w", err)
}
return &state, nil
}
// DeleteState removes the sandbox state file.
func DeleteState(name string) error {
path, err := statePath(name)
if err != nil {
return err
}
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("delete state: %w", err)
}
return nil
}
// ListStates returns all sandbox states from disk.
func ListStates() ([]*SandboxState, error) {
dir, err := sandboxesDir()
if err != nil {
return nil, err
}
entries, err := os.ReadDir(dir)
if err != nil {
return nil, fmt.Errorf("read sandboxes directory: %w", err)
}
var states []*SandboxState
for _, entry := range entries {
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".yaml") {
continue
}
name := strings.TrimSuffix(entry.Name(), ".yaml")
state, err := LoadState(name)
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: could not load sandbox %q: %v\n", name, err)
continue
}
states = append(states, state)
}
return states, nil
}
// FindActiveSandbox returns the first sandbox in running or creating state.
// Returns nil if no active sandbox exists.
func FindActiveSandbox() (*SandboxState, error) {
states, err := ListStates()
if err != nil {
return nil, err
}
for _, s := range states {
if s.Status == StatusRunning || s.Status == StatusCreating {
return s, nil
}
}
return nil, nil
}
// ToNodes converts sandbox servers to inspector.Node structs for SSH operations.
// Sets SSHKey to the provided key path on each node.
func (s *SandboxState) ToNodes(sshKeyPath string) []inspector.Node {
nodes := make([]inspector.Node, len(s.Servers))
for i, srv := range s.Servers {
nodes[i] = inspector.Node{
Environment: "sandbox",
User: "root",
Host: srv.IP,
Role: srv.Role,
SSHKey: sshKeyPath,
}
}
return nodes
}
// NameserverNodes returns only the nameserver nodes.
func (s *SandboxState) NameserverNodes() []ServerState {
var ns []ServerState
for _, srv := range s.Servers {
if srv.Role == "nameserver" {
ns = append(ns, srv)
}
}
return ns
}
// RegularNodes returns only the non-nameserver nodes.
func (s *SandboxState) RegularNodes() []ServerState {
var nodes []ServerState
for _, srv := range s.Servers {
if srv.Role == "node" {
nodes = append(nodes, srv)
}
}
return nodes
}
// GenesisServer returns the first server (genesis node).
func (s *SandboxState) GenesisServer() ServerState {
if len(s.Servers) == 0 {
return ServerState{}
}
return s.Servers[0]
}

View File

@ -0,0 +1,214 @@
package sandbox
import (
"os"
"path/filepath"
"testing"
"time"
)
func TestSaveAndLoadState(t *testing.T) {
// Use temp dir for test
tmpDir := t.TempDir()
origHome := os.Getenv("HOME")
os.Setenv("HOME", tmpDir)
defer os.Setenv("HOME", origHome)
state := &SandboxState{
Name: "test-sandbox",
CreatedAt: time.Date(2026, 2, 25, 10, 0, 0, 0, time.UTC),
Domain: "test.example.com",
Status: StatusRunning,
Servers: []ServerState{
{ID: 1, Name: "sbx-test-1", IP: "1.1.1.1", Role: "nameserver", FloatingIP: "10.0.0.1", WgIP: "10.0.0.1"},
{ID: 2, Name: "sbx-test-2", IP: "2.2.2.2", Role: "nameserver", FloatingIP: "10.0.0.2", WgIP: "10.0.0.2"},
{ID: 3, Name: "sbx-test-3", IP: "3.3.3.3", Role: "node", WgIP: "10.0.0.3"},
{ID: 4, Name: "sbx-test-4", IP: "4.4.4.4", Role: "node", WgIP: "10.0.0.4"},
{ID: 5, Name: "sbx-test-5", IP: "5.5.5.5", Role: "node", WgIP: "10.0.0.5"},
},
}
if err := SaveState(state); err != nil {
t.Fatalf("SaveState() error = %v", err)
}
// Verify file exists
expected := filepath.Join(tmpDir, ".orama", "sandboxes", "test-sandbox.yaml")
if _, err := os.Stat(expected); err != nil {
t.Fatalf("state file not created at %s: %v", expected, err)
}
// Load back
loaded, err := LoadState("test-sandbox")
if err != nil {
t.Fatalf("LoadState() error = %v", err)
}
if loaded.Name != "test-sandbox" {
t.Errorf("name = %s, want test-sandbox", loaded.Name)
}
if loaded.Domain != "test.example.com" {
t.Errorf("domain = %s, want test.example.com", loaded.Domain)
}
if loaded.Status != StatusRunning {
t.Errorf("status = %s, want running", loaded.Status)
}
if len(loaded.Servers) != 5 {
t.Errorf("servers = %d, want 5", len(loaded.Servers))
}
}
func TestLoadState_NotFound(t *testing.T) {
tmpDir := t.TempDir()
origHome := os.Getenv("HOME")
os.Setenv("HOME", tmpDir)
defer os.Setenv("HOME", origHome)
_, err := LoadState("nonexistent")
if err == nil {
t.Error("LoadState() expected error for nonexistent sandbox")
}
}
func TestDeleteState(t *testing.T) {
tmpDir := t.TempDir()
origHome := os.Getenv("HOME")
os.Setenv("HOME", tmpDir)
defer os.Setenv("HOME", origHome)
state := &SandboxState{
Name: "to-delete",
Status: StatusRunning,
}
if err := SaveState(state); err != nil {
t.Fatalf("SaveState() error = %v", err)
}
if err := DeleteState("to-delete"); err != nil {
t.Fatalf("DeleteState() error = %v", err)
}
_, err := LoadState("to-delete")
if err == nil {
t.Error("LoadState() should fail after DeleteState()")
}
}
func TestListStates(t *testing.T) {
tmpDir := t.TempDir()
origHome := os.Getenv("HOME")
os.Setenv("HOME", tmpDir)
defer os.Setenv("HOME", origHome)
// Create 2 sandboxes
for _, name := range []string{"sandbox-a", "sandbox-b"} {
if err := SaveState(&SandboxState{Name: name, Status: StatusRunning}); err != nil {
t.Fatalf("SaveState(%s) error = %v", name, err)
}
}
states, err := ListStates()
if err != nil {
t.Fatalf("ListStates() error = %v", err)
}
if len(states) != 2 {
t.Errorf("ListStates() returned %d, want 2", len(states))
}
}
func TestFindActiveSandbox(t *testing.T) {
tmpDir := t.TempDir()
origHome := os.Getenv("HOME")
os.Setenv("HOME", tmpDir)
defer os.Setenv("HOME", origHome)
// No sandboxes
active, err := FindActiveSandbox()
if err != nil {
t.Fatalf("FindActiveSandbox() error = %v", err)
}
if active != nil {
t.Error("expected nil when no sandboxes exist")
}
// Add one running sandbox
if err := SaveState(&SandboxState{Name: "active-one", Status: StatusRunning}); err != nil {
t.Fatal(err)
}
if err := SaveState(&SandboxState{Name: "errored-one", Status: StatusError}); err != nil {
t.Fatal(err)
}
active, err = FindActiveSandbox()
if err != nil {
t.Fatalf("FindActiveSandbox() error = %v", err)
}
if active == nil || active.Name != "active-one" {
t.Errorf("FindActiveSandbox() = %v, want active-one", active)
}
}
func TestToNodes(t *testing.T) {
state := &SandboxState{
Servers: []ServerState{
{IP: "1.1.1.1", Role: "nameserver"},
{IP: "2.2.2.2", Role: "node"},
},
}
nodes := state.ToNodes("/tmp/key")
if len(nodes) != 2 {
t.Fatalf("ToNodes() returned %d nodes, want 2", len(nodes))
}
if nodes[0].Host != "1.1.1.1" {
t.Errorf("node[0].Host = %s, want 1.1.1.1", nodes[0].Host)
}
if nodes[0].User != "root" {
t.Errorf("node[0].User = %s, want root", nodes[0].User)
}
if nodes[0].SSHKey != "/tmp/key" {
t.Errorf("node[0].SSHKey = %s, want /tmp/key", nodes[0].SSHKey)
}
if nodes[0].Environment != "sandbox" {
t.Errorf("node[0].Environment = %s, want sandbox", nodes[0].Environment)
}
}
func TestNameserverAndRegularNodes(t *testing.T) {
state := &SandboxState{
Servers: []ServerState{
{Role: "nameserver"},
{Role: "nameserver"},
{Role: "node"},
{Role: "node"},
{Role: "node"},
},
}
ns := state.NameserverNodes()
if len(ns) != 2 {
t.Errorf("NameserverNodes() = %d, want 2", len(ns))
}
regular := state.RegularNodes()
if len(regular) != 3 {
t.Errorf("RegularNodes() = %d, want 3", len(regular))
}
}
func TestGenesisServer(t *testing.T) {
state := &SandboxState{
Servers: []ServerState{
{Name: "first"},
{Name: "second"},
},
}
if state.GenesisServer().Name != "first" {
t.Errorf("GenesisServer().Name = %s, want first", state.GenesisServer().Name)
}
empty := &SandboxState{}
if empty.GenesisServer().Name != "" {
t.Error("GenesisServer() on empty state should return zero value")
}
}

160
pkg/cli/sandbox/status.go Normal file
View File

@ -0,0 +1,160 @@
package sandbox
import (
"encoding/json"
"fmt"
"strings"
"github.com/DeBrosOfficial/network/pkg/inspector"
)
// List prints all sandbox clusters.
func List() error {
states, err := ListStates()
if err != nil {
return err
}
if len(states) == 0 {
fmt.Println("No sandboxes found.")
fmt.Println("Create one: orama sandbox create")
return nil
}
fmt.Printf("%-20s %-10s %-5s %-25s %s\n", "NAME", "STATUS", "NODES", "CREATED", "DOMAIN")
for _, s := range states {
fmt.Printf("%-20s %-10s %-5d %-25s %s\n",
s.Name, s.Status, len(s.Servers), s.CreatedAt.Format("2006-01-02 15:04"), s.Domain)
}
// Check for orphaned servers on Hetzner
cfg, err := LoadConfig()
if err != nil {
return nil // Config not set up, skip orphan check
}
client := NewHetznerClient(cfg.HetznerAPIToken)
hetznerServers, err := client.ListServersByLabel("orama-sandbox")
if err != nil {
return nil // API error, skip orphan check
}
// Build set of known server IDs
known := make(map[int64]bool)
for _, s := range states {
for _, srv := range s.Servers {
known[srv.ID] = true
}
}
var orphans []string
for _, srv := range hetznerServers {
if !known[srv.ID] {
orphans = append(orphans, fmt.Sprintf("%s (ID: %d, IP: %s)", srv.Name, srv.ID, srv.PublicNet.IPv4.IP))
}
}
if len(orphans) > 0 {
fmt.Printf("\nWarning: %d orphaned server(s) on Hetzner (no state file):\n", len(orphans))
for _, o := range orphans {
fmt.Printf(" %s\n", o)
}
fmt.Println("Delete manually at https://console.hetzner.cloud")
}
return nil
}
// Status prints the health report for a sandbox cluster.
func Status(name string) error {
cfg, err := LoadConfig()
if err != nil {
return err
}
state, err := resolveSandbox(name)
if err != nil {
return err
}
sshKeyPath := cfg.ExpandedPrivateKeyPath()
fmt.Printf("Sandbox: %s (status: %s)\n\n", state.Name, state.Status)
for _, srv := range state.Servers {
node := inspector.Node{User: "root", Host: srv.IP, SSHKey: sshKeyPath}
fmt.Printf("%s (%s) — %s\n", srv.Name, srv.IP, srv.Role)
// Get node report
out, err := runSSHOutput(node, "orama node report --json 2>/dev/null")
if err != nil {
fmt.Printf(" Status: UNREACHABLE (%v)\n", err)
fmt.Println()
continue
}
printNodeReport(out)
fmt.Println()
}
// Cluster summary
fmt.Println("Cluster Summary")
fmt.Println("---------------")
genesis := state.GenesisServer()
genesisNode := inspector.Node{User: "root", Host: genesis.IP, SSHKey: sshKeyPath}
out, err := runSSHOutput(genesisNode, "curl -sf http://localhost:5001/status 2>/dev/null")
if err != nil {
fmt.Println(" RQLite: UNREACHABLE")
} else {
var status map[string]interface{}
if err := json.Unmarshal([]byte(out), &status); err == nil {
if store, ok := status["store"].(map[string]interface{}); ok {
if raft, ok := store["raft"].(map[string]interface{}); ok {
fmt.Printf(" RQLite state: %v\n", raft["state"])
fmt.Printf(" Commit index: %v\n", raft["commit_index"])
if nodes, ok := raft["nodes"].([]interface{}); ok {
fmt.Printf(" Nodes: %d\n", len(nodes))
}
}
}
}
}
return nil
}
// printNodeReport parses and prints a node report JSON.
func printNodeReport(jsonStr string) {
var report map[string]interface{}
if err := json.Unmarshal([]byte(jsonStr), &report); err != nil {
fmt.Printf(" Report: (parse error)\n")
return
}
// Print key fields
if services, ok := report["services"].(map[string]interface{}); ok {
var active, inactive []string
for name, info := range services {
if svc, ok := info.(map[string]interface{}); ok {
if state, ok := svc["active"].(bool); ok && state {
active = append(active, name)
} else {
inactive = append(inactive, name)
}
}
}
if len(active) > 0 {
fmt.Printf(" Active: %s\n", strings.Join(active, ", "))
}
if len(inactive) > 0 {
fmt.Printf(" Inactive: %s\n", strings.Join(inactive, ", "))
}
}
if rqlite, ok := report["rqlite"].(map[string]interface{}); ok {
if state, ok := rqlite["state"].(string); ok {
fmt.Printf(" RQLite: %s\n", state)
}
}
}