orama/docs/vault/OPERATOR_GUIDE.md

Orama Vault -- Operator Guide

Monitoring

Health Endpoint

The simplest way to check if a guardian is running:

curl -s http://127.0.0.1:7500/v1/vault/health | jq .

Expected response:

{
  "status": "degraded",
  "version": "0.1.0",
  "shares": 0,
  "peers": 0,
  "data_dir_ok": true
}

Status meanings:

• "unhealthy" -- the data directory is not accessible. Investigate immediately.
• "degraded" -- data directory OK, but zero alive peers. This is the expected status in v0.1.0: RQLite node discovery is a stub that returns an empty node list, so the peer count is always zero.
• "ok" -- data directory OK and at least one alive peer (requires the Phase 2 discovery integration).

If this endpoint does not respond, the guardian process is not running or the port is blocked. Check systemd status first.

Status Endpoint

Provides runtime configuration:

curl -s http://127.0.0.1:7500/v1/vault/status | jq .

Expected response:

{
  "status": "ok",
  "version": "0.1.0",
  "data_dir": "/opt/orama/.orama/data/vault",
  "client_port": 7500,
  "peer_port": 7501
}

Guardians Endpoint

Lists known guardian nodes in the cluster:

curl -s http://127.0.0.1:7500/v1/vault/guardians | jq .

The handler lists all alive nodes from the guardian's node list. In v0.1.0 the RQLite discovery stub returns an empty node list, so the response is:

{"guardians":[],"threshold":3,"total":0}

Full cluster listing requires RQLite integration (Phase 2).

Systemd Journal

The guardian logs to stderr using Zig's structured logging, which is captured by the systemd journal:

# View recent logs
sudo journalctl -u orama-vault -n 50 --no-pager

# Follow live logs
sudo journalctl -u orama-vault -f

# View logs since last boot
sudo journalctl -u orama-vault -b

# View error-level logs only
sudo journalctl -u orama-vault -p err

Log messages include:

• vault-guardian v0.1.0 starting -- startup confirmation
• config: <path> -- config file path
• listening on <addr>:<port> (client) -- client listener bound
• listening on <addr>:<port> (peer) -- configured peer port. Note (v0.1.0): this line is logged, but no peer listener is actually started -- nothing binds port 7501 at runtime.
• data directory: <path> -- data directory path
• guardian ready -- starting HTTP server -- initialization complete
• stored share for identity <hex> (<n> bytes, version <v>) -- successful push
• served share for identity <hex> (<n> bytes, v<v>) -- successful pull
• rejected rollback for <hex>: version <v> <= current <v> -- anti-rollback rejection
• accept error: <err> -- TCP accept failure (non-fatal, retried)
• connection error: <err> -- individual connection handling error
• failed to write share for <hex>: <err> -- disk write failure

Service Status

sudo systemctl status orama-vault

Check for:

• Active: active (running) -- service is up
• Main PID: <pid> -- process ID
• Memory and CPU usage in the status output

---

Troubleshooting

Port Already In Use

Symptom: Guardian fails to start with failed to bind 0.0.0.0:7500: error.AddressInUse

Diagnosis:

# Find what's using the port
sudo ss -tlnp | grep 7500

Resolution:

• If another vault-guardian is running: sudo systemctl stop orama-vault first.
• If another service is using port 7500: change the vault port with --port <other>.
• If the port is in TIME_WAIT state from a recent restart: wait 30-60 seconds. The guardian sets SO_REUSEADDR which should handle most cases.

Data Directory Permissions

Symptom: failed to create data directory <path>: error.AccessDenied

Diagnosis:

ls -la /opt/orama/.orama/data/vault/

Resolution:

sudo chown -R orama:orama /opt/orama/.orama/data/vault
sudo chmod 700 /opt/orama/.orama/data/vault

The systemd service uses ProtectSystem=strict with ReadWritePaths=/opt/orama/.orama/data/vault, so the data directory must be under this exact path or a CLI override must be used.

RQLite Connectivity

Symptom: Health endpoint reports "status":"degraded" with "peers":0; guardians endpoint returns an empty list.

Diagnosis:

# Check if RQLite is running
sudo systemctl status orama-*-rqlite

# Test RQLite endpoint
curl -s http://127.0.0.1:4001/status | jq .store.raft.state

Resolution:

• In v0.1.0 this is not fixable by configuration: RQLite node discovery is a stub that "succeeds" with an empty node list, so the guardian always runs with zero known peers regardless of RQLite health. (Because the stub does not return an error, the failed to fetch node list from RQLite, running in single-node mode fallback log never appears.)
• The guardian is fully functional for local push/pull in this mode.
• Real RQLite discovery is Phase 2; the rqlite_url config value is currently unused by the fetch.

Share Write Failures

Symptom: Push returns 500 Internal Server Error, logs show failed to write share for <hex>: <err>

Diagnosis:

# Check disk space
df -h /opt/orama/.orama/data/vault

# Check inode usage
df -i /opt/orama/.orama/data/vault

# Check directory permissions
ls -la /opt/orama/.orama/data/vault/shares/

Resolution:

• If disk is full: free space or expand the partition.
• If inodes are exhausted (unlikely but possible with millions of users): clean up orphaned temp files.
• If permissions are wrong: fix ownership as shown above.

Anti-Rollback Rejections

Symptom: Push returns 409 Conflict with "version must be greater than current stored version"

This is normal behavior -- the client tried to push an older version of a share. (409 rather than 400 lets clients distinguish a stale version from a malformed request.) Common causes:

• Client retry after a network timeout (the first push actually succeeded).
• Client software bug sending stale version numbers.

Diagnosis:

# Check current stored version (and threshold) for an identity
cat /opt/orama/.orama/data/vault/shares/<identity_hex>/meta.json

Resolution: The client must send a version number strictly greater than the stored value. This is not a guardian bug.

Guardian Crash Loop

Symptom: systemctl status shows rapid restarts.

Diagnosis:

# View recent crash logs
sudo journalctl -u orama-vault -n 100 --no-pager | tail -50

# Check for OOM kills
sudo journalctl -k | grep -i "oom\|kill"

Resolution:

• If OOM killed: the 512 MiB memory limit may be too low. Check if share data has grown unexpectedly.
• If config parse error: check the config file syntax (or remove it to use defaults).
• If bind error: another process is using the port.

---

Manual Operations

Check Stored Shares

List all identities with stored shares:

ls /opt/orama/.orama/data/vault/shares/

Check a specific identity's share:

# View version and threshold metadata
cat /opt/orama/.orama/data/vault/shares/<identity_hex>/meta.json

# View share size
ls -la /opt/orama/.orama/data/vault/shares/<identity_hex>/share.bin

# View checksum
xxd /opt/orama/.orama/data/vault/shares/<identity_hex>/checksum.bin

Count Total Shares

ls -d /opt/orama/.orama/data/vault/shares/*/ 2>/dev/null | wc -l

Verify Share Integrity Manually

The guardian verifies HMAC integrity on every read. To manually check if a share file has been corrupted:

# If you know the integrity key, you can compute HMAC externally:
# (The integrity key is internal to the guardian and not stored on disk in the current version)

# Check file exists and is non-empty
test -s /opt/orama/.orama/data/vault/shares/<identity>/share.bin && echo "OK" || echo "MISSING/EMPTY"
test -s /opt/orama/.orama/data/vault/shares/<identity>/checksum.bin && echo "OK" || echo "MISSING/EMPTY"

Test Push/Pull

Push and pull are authenticated: both handlers require an X-Session-Token header (obtained via the challenge/session endpoints) and an Ed25519 ownership proof (pubkey + signature fields in the body, where identity = SHA-256(pubkey)). A plain curl without these returns 401 Unauthorized -- which makes it a useful smoke test that auth is enforced:

# Verify auth is enforced (expected: 401 with "session token required")
curl -s -X POST http://127.0.0.1:7500/v1/vault/push \
  -H "Content-Type: application/json" \
  -d '{
    "identity": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "share": "dGVzdCBzaGFyZSBkYXRh",
    "version": 1
  }'

A full push/pull test requires a signing client (curl alone cannot produce Ed25519 signatures). The flow is:

1. Generate an Ed25519 keypair; the identity is SHA-256(pubkey) as 64 lowercase hex chars.
2. POST /v1/vault/auth/challenge with {"identity":"<hex>"} -- returns {"nonce":..., "created_ns":..., "tag":...}.
3. POST /v1/vault/auth/session echoing those fields plus the identity -- returns {"identity":..., "expiry_ns":..., "tag":...}. The session token is <identity_hex>:<expiry_ns>:<tag_hex>.
4. Push: POST /v1/vault/push with header X-Session-Token: <token> and body fields identity, share (base64), version, pubkey (64 hex), signature (128 hex) -- the signature is Ed25519 over the ASCII message vault-push-v1:<identity_hex>:<version>. Expected: {"status":"stored"}.
5. Pull: POST /v1/vault/pull with the same header and body fields identity, pubkey, signature, timestamp (Unix seconds, within 120s of the guardian's clock) -- the signature is over vault-pull-v1:<identity_hex>:<timestamp>. Expected: {"share":"<base64>","version":<n>,"threshold":<k>}.

In practice, use the Orama gateway's vault client (or the SDK) for end-to-end testing rather than hand-rolling signatures.

Delete a Share (Emergency)

Warning: Deleting a share is destructive and cannot be undone. Only do this if you understand the implications for the user's recovery capability.

# Remove a specific identity's share directory
rm -rf /opt/orama/.orama/data/vault/shares/<identity_hex>

---

Disaster Recovery

Failure Scenarios

Scenario	Impact	Recovery
1 node dies	No impact. K-1 other nodes can still reconstruct.	Replace node, shares will be re-pushed by clients.
K-1 nodes die	No impact. K remaining nodes can still reconstruct.	Replace nodes, reshare when quorum recovers.
N-K nodes die	No impact. K+1 or more nodes survive.	Replace nodes.
N-K+1 nodes die	CRITICAL. Only K-1 nodes remain. Cannot reconstruct.	Data loss for users who only have vault-based recovery. Users with mnemonic (Path A) are unaffected.
All nodes die	TOTAL LOSS. All shares destroyed.	Users must recover via mnemonic (Path A). Vault recovery (Path B) is permanently lost.
Data directory corrupted on 1 node	HMAC integrity check fails on read. Node returns errors for affected shares.	Delete corrupted share directory. Repair protocol will re-distribute once implemented.

Key Insight

The system can tolerate losing up to N - K nodes without any data loss. With default thresholds:

Cluster Size	K	Max Node Loss
5 nodes	3	2 nodes
14 nodes	4	10 nodes
50 nodes	16	34 nodes
100 nodes	33	67 nodes

Backup Strategy

The data directory can be backed up with standard tools:

# Rsync backup
rsync -av /opt/orama/.orama/data/vault/ /backup/vault/

# Tarball backup
tar czf vault-backup-$(date +%Y%m%d).tar.gz -C /opt/orama/.orama/data/vault .

However, backups are generally unnecessary because:

1. Every node stores a share, so the cluster itself is the redundancy.
2. Shares are re-pushed by clients on updates.
3. The proactive re-sharing protocol (when fully wired) will redistribute shares automatically.

Backups are only useful if you fear simultaneous catastrophic failure of many nodes.

---

Capacity Planning

Per-Share Storage

Each user's share directory contains:

File	Typical Size	Description
share.bin	~1 KB	Encrypted share data (same size as original secret)
meta.json	~40 bytes	Version counter and threshold, e.g. {"version":1,"threshold":3}
checksum.bin	32 bytes	HMAC-SHA256 checksum
wrapped_dek1.bin / wrapped_dek2.bin	~60 bytes each	KEK-wrapped DEKs (when present)
Directory entry	~4 KB	Filesystem overhead (depends on filesystem)

Total per user per node: ~5 KB (including filesystem overhead).

Cluster-Wide Storage

With all-node replication, each user has one share on every node:

Users	Per Node	Per Cluster (14 nodes)
1,000	~5 MB	~70 MB
10,000	~50 MB	~700 MB
100,000	~500 MB	~7 GB
1,000,000	~5 GB	~70 GB

At 1 million users, each node stores approximately 5 GB. This is well within the capability of any modern VPS.

Memory Usage

The guardian uses minimal memory:

• Static binary: ~2-5 MB RSS.
• Per-connection: ~128 KB (64 KB read buffer + 64 KB write buffer).
• Single-threaded: only one connection is active at a time (MVP).
• No in-memory caching of shares: every read/write goes to disk.

The systemd MemoryMax=512M limit is generous for the current architecture. Actual usage is typically under 20 MB.

Inode Usage

Each user creates one directory and 2-3 files. At 1 million users:

• ~4 million inodes.
• Most Linux filesystems default to millions of inodes (ext4: 1 inode per 16 KB).
• A 100 GB partition with ext4 defaults has ~6.5 million inodes.

This is unlikely to be a bottleneck but is worth monitoring on small partitions.

---

Cluster Scaling

Note (v0.1.0): This section describes the multi-guardian design. At runtime today, RQLite discovery returns an empty node list, the peer listener is never started, and the repair/re-sharing protocol is not wired in -- each guardian effectively operates single-node. The mechanics below take effect with the Phase 2 integration.

Adding Nodes

When a new node joins the Orama network:

1. The vault guardian starts and registers itself via RQLite.
2. Other guardians detect the new node via the discovery module (join event).
3. The new node initially has zero shares.
4. Shares are populated in two ways:
   • Client push: When clients push new versions, they include the new node.
   • Repair protocol: The re-sharing protocol redistributes shares to include the new node (Phase 2).

The threshold K is recomputed based on the alive count: K = max(3, floor(N/3)).

Removing Nodes

When a node leaves (graceful shutdown or failure):

1. Other guardians detect the departure via missed heartbeats (suspect at 15s, dead at 60s).
2. The departed node's shares are lost.
3. If N - K nodes remain alive, no data is lost.
4. If the departure drops below the safety threshold (K+1 alive), the repair protocol triggers re-sharing to adjust.

Threshold Adjustment

The threshold is dynamic and automatic:

K = max(3, floor(alive_count / 3))

• Adding nodes generally does not change K until the cluster grows significantly.
• Removing nodes may reduce K if the alive count drops enough.
• K never drops below 3, ensuring a minimum collusion resistance.

Write Quorum

Write quorum requires supermajority acknowledgment, with a floor that guarantees a successful write persists more shares than a read requires (W > K):

W = min(N, max(K + 1, ceil(2/3 * N)))

(The plain ceil(2N/3) formula was unsafe: at N=3 it gave W=2 < K, allowing "successful" writes that were silently unrecoverable.)

Alive	Write Quorum (W)
1	1
2	2
3	3
4	3
5	4
14	10
100	67

A push succeeds only if W guardians acknowledge storage. This ensures consistency even with some nodes being slow or temporarily unreachable.

Note (v0.1.0): Write quorum is computed but not enforced in the current single-node push handler. Multi-guardian fan-out is Phase 2.

---

Operational Checklist

Pre-Deploy

• Binary built with -Doptimize=ReleaseSafe for the correct target.
• Data directory exists with correct ownership and permissions.
• systemd service file installed and enabled.
• Firewall rules allow port 7500 (client) and 7501 (peer, WireGuard only). Note: in v0.1.0 nothing listens on 7501 (the peer listener is not started); the rule is forward-compatibility for Phase 2.
• WireGuard is up and peers are connected.

Post-Deploy

• Health endpoint responds: curl http://127.0.0.1:7500/v1/vault/health (expect "status":"degraded" in v0.1.0 -- see Health Endpoint above)
• Status endpoint shows correct config: curl http://127.0.0.1:7500/v1/vault/status
• No error-level log messages: sudo journalctl -u orama-vault -p err -n 10
• Test push/pull cycle works (see "Test Push/Pull" section above).

Periodic Checks

• Health endpoint responds on all nodes.
• Share count is consistent across nodes (same identities on each).
• Disk usage is within expected bounds.
• No repeated error messages in the journal.
• systemd reports active (running) with uptime matching expectations.