# Upgrades

Orama nodes must be kept up to date with the latest releases to maintain compatibility with the network and benefit from performance improvements and security patches. This guide covers the upgrade process, rolling upgrades, and health verification.

## Upgrade Command

The `orama node upgrade` command handles downloading, extracting, and optionally restarting services. It supports both local (on-VPS) and remote (from your machine) execution.

### Local Upgrade (on VPS)

Run directly on a VPS node with root privileges:

```bash
# Upgrade binary only (does not restart services)
sudo orama node upgrade

# Upgrade and automatically restart services
sudo orama node upgrade --restart

# Reconfigure all settings during upgrade
sudo orama node upgrade --force

# Skip minimum resource checks (RAM/CPU)
sudo orama node upgrade --skip-checks
```

### Remote Rolling Upgrade

Run from your local machine to upgrade all nodes in an environment sequentially:

```bash
# Upgrade all nodes in an environment (rolling, one at a time)
orama node upgrade --env testnet

# Upgrade a single node only
orama node upgrade --env testnet --node 1.2.3.4

# Custom delay between nodes (default: 30 seconds)
orama node upgrade --env testnet --delay 60

# Upgrade with automatic restart
orama node upgrade --env testnet --restart
```

### All Upgrade Flags

| Flag | Default | Description |
|------|---------|-------------|
| `--force` | `false` | Reconfigure all settings during upgrade |
| `--restart` | `false` | Automatically restart services after upgrade |
| `--skip-checks` | `false` | Skip minimum resource checks (RAM/CPU) |
| `--env` | | Target environment for remote rolling upgrade (`devnet`, `testnet`, `mainnet`) |
| `--node` | | Upgrade a single node IP only (requires `--env`) |
| `--delay` | `30` | Seconds to wait between nodes during rolling upgrade |
| `--nameserver` | `false` | Make this node a nameserver |
| `--anyone-relay` | `false` | Run as Anyone relay operator |
| `--anyone-exit` | `false` | Run as exit relay (requires `--anyone-relay`) |
| `--anyone-client` | `false` | Install Anyone as client-only (SOCKS5 proxy) |
| `--anyone-migrate` | `false` | Migrate existing Anyone installation |
| `--anyone-nickname` | | Relay nickname (1-19 alphanumeric chars) |
| `--anyone-contact` | | Contact info (email or @telegram) |
| `--anyone-wallet` | | Ethereum wallet address for rewards |
| `--anyone-orport` | `9001` | ORPort for relay |
| `--anyone-family` | | Comma-separated fingerprints of other relays you operate |
| `--anyone-bandwidth` | `30` | Limit relay to N% of VPS bandwidth (0=unlimited) |
| `--anyone-accounting` | `0` | Monthly data cap in GB (0=unlimited) |

---

## Rolling Upgrade Process

Orama uses a rolling upgrade strategy — nodes are upgraded one at a time to maintain cluster availability. RQLite requires a majority of voter nodes (quorum) to function, so upgrading multiple nodes simultaneously risks losing quorum and causing a cluster outage.

### Upgrade Order

1. **Upgrade followers first** — Start with RQLite follower nodes
2. **Upgrade the leader last** — This triggers a leader election, which is normal
3. **Never upgrade more than one node at a time** — Wait for full verification between each

### Per-Node Process

1. Verify cluster health before starting
2. Download the new binary to the node
3. Stop services in reverse dependency order (Gateway → Olric → RQLite → WireGuard)
4. Replace the binary with the new version
5. Start services in correct dependency order (WireGuard → RQLite → Olric → Gateway)
6. Verify node health after restart
7. Verify cluster health before proceeding to the next node

---

## Health Verification

After upgrading each node, always verify health before moving to the next:

```bash
# Check the upgraded node
orama monitor report --env <environment> --node <node-ip>

# Check the full cluster
orama monitor report --env <environment>
```

What to verify:

| Check | Expected State |
|-------|---------------|
| All services | `active (running)` |
| RQLite state | `Leader` or `Follower` |
| Raft log index | In sync with other nodes |
| WireGuard peers | All connected, recent handshakes |

Do not proceed to the next node until all checks pass. If a node shows issues after upgrade, investigate before continuing.

---

## Rollout Command (Recommended)

The `orama rollout` command pushes a pre-built archive to all your nodes and performs a rolling upgrade. This is the preferred method for deploying code changes.

```bash
# Build the archive first
orama build

# Roll out to all your nodes (followers first, leader last)
orama rollout --env devnet

# Custom delay between nodes
orama rollout --env devnet --delay 60
```

| Flag | Default | Description |
|------|---------|-------------|
| `--env` | active env | Target environment |
| `--delay` | `30` | Seconds between nodes during rolling upgrade |

The rollout command discovers your nodes via wallet-based ownership (same nodes shown by `orama nodes`). It falls back to `nodes.conf` if the network API is unavailable.

### Legacy Rollout

The `orama node rollout` command is still available and uses `nodes.conf` directly:

```bash
orama node rollout --env testnet
orama node rollout --env testnet --no-build --yes --delay 60
```

---

## Push Command (Recommended)

The `orama push` command uploads the binary archive to nodes without restarting services. Useful for pre-staging a binary before a maintenance window. Uses RootWallet for SSH authentication.

```bash
# Push to a single node
orama push --ip 1.2.3.4

# Push to all nodes in an environment
orama push --env devnet

# Fan out server-to-server (faster for many nodes)
orama push --env devnet --fanout

# Push with a specific SSH user
orama push --ip 1.2.3.4 --user ubuntu
```

Use `--fanout` to upload to the first node, then distribute server-to-server via SSH agent forwarding. Much faster when pushing to many nodes.

| Flag | Default | Description |
|------|---------|-------------|
| `--ip` | | Push to a single node by IP |
| `--env` | active env | Push to all nodes in environment |
| `--user` | `root` | SSH user |
| `--fanout` | `false` | Fan out from first node server-to-server (faster) |

### Legacy Push

The `orama node push` command is still available and uses `nodes.conf`:

```bash
orama node push --env devnet
orama node push --env testnet --node 1.2.3.4
```

| Flag | Default | Description |
|------|---------|-------------|
| `--env` | *(required)* | Target environment |
| `--node` | | Push to a single node IP only |
| `--direct` | `false` | Upload directly to each node |

---

## Version Compatibility

The Orama network requires all nodes in an environment to run compatible versions. When a new version is released:

- Operators receive a notification with the target upgrade window
- All nodes should be upgraded within the specified window
- Nodes running incompatible versions may be temporarily excluded from the mesh

Use `orama monitor alerts --env <env>` to check for binary version mismatch warnings across the cluster.
