network/docs/ipfs-cluster-setup.md

IPFS Cluster Setup Guide

This guide explains how IPFS Cluster is configured to run on every DeBros Network node.

Overview

Each DeBros Network node runs its own IPFS Cluster peer, enabling distributed pinning and replication across the network. The cluster uses CRDT consensus for automatic peer discovery.

Architecture

• IPFS (Kubo): Runs on each node, handles content storage and retrieval
• IPFS Cluster: Runs on each node, manages pinning and replication
• Cluster Consensus: Uses CRDT (instead of Raft) for simpler multi-node setup

Automatic Setup

When you run network-cli setup, the following happens automatically:

1. IPFS (Kubo) and IPFS Cluster are installed
2. IPFS repository is initialized for each node
3. IPFS Cluster service.json config is generated
4. Systemd services are created and started:
   • debros-ipfs - IPFS daemon
   • debros-ipfs-cluster - IPFS Cluster service
   • debros-node - DeBros Network node (depends on cluster)
   • debros-gateway - HTTP Gateway (depends on node)

Configuration

Node Configs

Each node config (~/.debros/bootstrap.yaml, ~/.debros/node.yaml, etc.) includes:

database:
  ipfs:
    cluster_api_url: "http://localhost:9094" # Local cluster API
    api_url: "http://localhost:5001" # Local IPFS API
    replication_factor: 3 # Desired replication

Cluster Service Config

Cluster service configs are stored at:

• Bootstrap: ~/.debros/bootstrap/ipfs-cluster/service.json
• Nodes: ~/.debros/node/ipfs-cluster/service.json

Key settings:

• Consensus: CRDT (automatic peer discovery)
• API Listen: 0.0.0.0:9094 (REST API)
• Cluster Listen: 0.0.0.0:9096 (peer-to-peer)
• Secret: Shared cluster secret stored at ~/.debros/cluster-secret

Verification

Check Cluster Peers

From any node, verify all cluster peers are connected:

sudo -u debros ipfs-cluster-ctl --host http://localhost:9094 peers ls

You should see all cluster peers listed (bootstrap, node1, node2, etc.).

Check IPFS Daemon

Verify IPFS is running:

sudo -u debros ipfs daemon --repo-dir=~/.debros/bootstrap/ipfs/repo
# Or for regular nodes:
sudo -u debros ipfs daemon --repo-dir=~/.debros/node/ipfs/repo

Check Service Status

network-cli service status all

Should show:

• debros-ipfs - running
• debros-ipfs-cluster - running
• debros-node - running
• debros-gateway - running

Troubleshooting

Cluster Peers Not Connecting

If peers aren't discovering each other:

1. Check firewall: Ensure ports 9096 (cluster swarm) and 9094 (cluster API) are open
2. Verify secret: All nodes must use the same cluster secret from ~/.debros/cluster-secret
3. Check logs: journalctl -u debros-ipfs-cluster -f

Not Enough Peers Error

If you see "not enough peers to allocate CID" errors:

• The cluster needs at least replication_factor peers running
• Check that all nodes have debros-ipfs-cluster service running
• Verify with ipfs-cluster-ctl peers ls

IPFS Not Starting

If IPFS daemon fails to start:

1. Check IPFS repo exists: ls -la ~/.debros/bootstrap/ipfs/repo/
2. Check permissions: chown -R debros:debros ~/.debros/bootstrap/ipfs/
3. Check logs: journalctl -u debros-ipfs -f

Manual Setup (If Needed)

If automatic setup didn't work, you can manually initialize:

1. Initialize IPFS

sudo -u debros ipfs init --profile=server --repo-dir=~/.debros/bootstrap/ipfs/repo
sudo -u debros ipfs config --json Addresses.API '["/ip4/localhost/tcp/5001"]' --repo-dir=~/.debros/bootstrap/ipfs/repo

2. Initialize Cluster

# Generate or get cluster secret
CLUSTER_SECRET=$(cat ~/.debros/cluster-secret)

# Initialize cluster (will create service.json)
sudo -u debros ipfs-cluster-service init --consensus crdt

3. Start Services

systemctl start debros-ipfs
systemctl start debros-ipfs-cluster
systemctl start debros-node
systemctl start debros-gateway

Ports

• 4001: IPFS swarm (LibP2P)
• 5001: IPFS HTTP API
• 8080: IPFS Gateway (optional)
• 9094: IPFS Cluster REST API
• 9096: IPFS Cluster swarm (LibP2P)

Replication Factor

The default replication factor is 3, meaning content is pinned to 3 cluster peers. This requires at least 3 nodes running cluster peers.

To change replication factor, edit node configs:

database:
  ipfs:
    replication_factor: 1 # For single-node development

Security Notes

• Cluster secret is stored at ~/.debros/cluster-secret (mode 0600)
• Cluster API (port 9094) should be firewalled in production
• IPFS API (port 5001) should only be accessible locally