diff --git a/README.md b/README.md index f0826fd..7917871 100644 --- a/README.md +++ b/README.md @@ -10,14 +10,8 @@ A robust, decentralized peer-to-peer network built in Go, providing distributed - [Architecture Overview](#architecture-overview) - [System Requirements](#system-requirements) - [Quick Start](#quick-start) -- [Deployment & Installation](#deployment--installation) -- [Configuration](#configuration) -- [CLI Usage](#cli-usage) -- [HTTP Gateway](#http-gateway) -- [Development](#development) -- [Database Client (Go ORM-like)](#database-client-go-orm-like) -- [Troubleshooting](#troubleshooting) -- [License](#license) + + --- @@ -25,7 +19,7 @@ A robust, decentralized peer-to-peer network built in Go, providing distributed - **Distributed SQL Database:** RQLite-backed, Raft-consensus, ACID transactions, automatic failover. - **Pub/Sub Messaging:** Topic-based, real-time, namespaced, automatic cleanup. -- **Peer Discovery & Management:** Nodes discover peers, bootstrap support, health monitoring. +- **Peer Discovery & Management:** Nodes discover peers, health monitoring. - **Application Isolation:** Namespace-based multi-tenancy, per-app config. - **Secure by Default:** Noise/TLS transport, peer identity, systemd hardening. - **Simple Client API:** Lightweight Go client for apps and CLI tools. @@ -76,6 +70,7 @@ A robust, decentralized peer-to-peer network built in Go, providing distributed - **RQLite:** 8.x (distributed SQLite) - **Git:** For source management - **Make:** For build automation (recommended) +- **ANyONe Client:** For onion type routing ### Hardware @@ -87,6 +82,7 @@ A robust, decentralized peer-to-peer network built in Go, providing distributed - **4001:** LibP2P P2P communication - **5001:** RQLite HTTP API - **7001:** RQLite Raft consensus +- **9050:** ANyONe Client ### Filesystem Permissions @@ -200,6 +196,7 @@ This starts: - Node 2 (P2P: 4002, RQLite HTTP: 5002, Raft: 7002) - Node 3 (P2P: 4003, RQLite HTTP: 5003, Raft: 7003) - Gateway (HTTP: 6001) +- ANyONe Client (9050) Logs stream to terminal. Press **Ctrl+C** to stop all processes. @@ -214,116 +211,6 @@ Logs stream to terminal. Press **Ctrl+C** to stop all processes. --- -## Deployment & Installation - -### Automated Production Install - -Run the install script for a secure, production-ready setup: - -```bash -curl -sSL https://github.com/DeBrosOfficial/network/raw/main/scripts/install-debros-network.sh | sudo bash -``` - -**What the Script Does:** - -- Detects OS, installs Go, RQLite, dependencies -- Creates `debros` system user, secure directory structure -- Generates LibP2P identity keys -- Clones source, builds binaries -- Sets up systemd service (`debros-node`) -- Configures firewall (UFW) for required ports -- Generates YAML config in `/opt/debros/configs/node.yaml` - -**Directory Structure:** - -``` -/opt/debros/ -├── bin/ # Binaries -├── configs/ # YAML configs -├── keys/ # Identity keys -├── data/ # RQLite DB, storage -├── logs/ # Node logs -├── src/ # Source code -``` - -**Service Management:** - -```bash -sudo systemctl status debros-node -sudo systemctl start debros-node -sudo systemctl stop debros-node -sudo systemctl restart debros-node -sudo journalctl -u debros-node.service -f -``` - ---- - -## Configuration - -### Configuration Files Location - -All configuration files are stored in `~/.debros/` for both local development and production deployments: - -- `~/.debros/node.yaml` - Node configuration -- `~/.debros/node.yaml` - Bootstrap node configuration -- `~/.debros/gateway.yaml` - Gateway configuration - -The system will **only** load config from `~/.debros/` and will error if required config files are missing. - -### Generating Configuration Files - -Use the `network-cli config init` command to generate configuration files: - -### Generate Complete Stack (Recommended) - -```bash -# Generate bootstrap, node2, node3, and gateway configs in one command -./bin/network-cli config init - -# Force regenerate (overwrites existing configs) -./bin/network-cli config init --force -``` - -This is the **recommended way** to get started with a local development network. - -### Generate Individual Configs (Advanced) - -For custom setups or production deployments, you can generate individual configs: - -#### Generate a Single Node Config - -```bash -# Generate basic node config with bootstrap peers -./bin/network-cli config init --type node --bootstrap-peers "/ip4/127.0.0.1/tcp/4001/p2p/QmXxx" - -# With custom ports -./bin/network-cli config init --type node --name node2.yaml \ - --listen-port 4002 --rqlite-http-port 5002 --rqlite-raft-port 7002 \ - --join localhost:5001 --bootstrap-peers "/ip4/127.0.0.1/tcp/4001/p2p/QmXxx" - -# Force overwrite existing config -./bin/network-cli config init --type node --force -``` - -#### Generate a Bootstrap Node Config - -```bash -# Generate bootstrap node (no join address required) -./bin/network-cli config init --type bootstrap - -# With custom ports -./bin/network-cli config init --type bootstrap --listen-port 4001 --rqlite-http-port 5001 --rqlite-raft-port 7001 -``` - -#### Generate a Gateway Config - -```bash -# Generate gateway config -./bin/network-cli config init --type gateway - -# With bootstrap peers -./bin/network-cli config init --type gateway --bootstrap-peers "/ip4/127.0.0.1/tcp/4001/p2p/QmXxx" -``` ### Running the Network @@ -337,16 +224,16 @@ Or start individual components (in separate terminals): ```bash # Terminal 1 - Bootstrap node -go run ./cmd/node --config bootstrap.yaml +go run-node # Terminal 2 - Node 2 -go run ./cmd/node --config node2.yaml +go run-node2 # Terminal 3 - Node 3 -go run ./cmd/node --config node3.yaml +go run-node3 # Terminal 4 - Gateway -go run ./cmd/gateway --config gateway.yaml +go run-gateway ``` ### Running Multiple Nodes on the Same Machine @@ -397,172 +284,6 @@ If ports are wrong, regenerate the config with `--force`: --join localhost:5001 --bootstrap-peers '' --force ``` -### Validating Configuration - -DeBros Network performs strict validation of all configuration files at startup. This ensures invalid configurations are caught immediately rather than causing silent failures later. - -#### Validation Features - -- **Strict YAML Parsing:** Unknown configuration keys are rejected with helpful error messages -- **Format Validation:** Multiaddrs, ports, durations, and other formats are validated for correctness -- **Cross-Field Validation:** Configuration constraints (e.g., bootstrap nodes don't join clusters) are enforced -- **Aggregated Error Reporting:** All validation errors are reported together, not one-by-one - -#### Common Validation Errors - -**Missing or Invalid `node.type`** -``` -node.type: must be one of [bootstrap node]; got "invalid" -``` -Solution: Set `type: "bootstrap"` or `type: "node"` - -**Invalid Bootstrap Peer Format** -``` -discovery.bootstrap_peers[0]: invalid multiaddr; expected /ip{4,6}/.../tcp//p2p/ -discovery.bootstrap_peers[0]: missing /p2p/ component -``` -Solution: Use full multiaddr format: `/ip4/127.0.0.1/tcp/4001/p2p/12D3KooW...` - -**Port Conflicts** -``` -database.rqlite_raft_port: must differ from database.rqlite_port (5001) -``` -Solution: Use different ports for HTTP and Raft (e.g., 5001 and 7001) - -**RQLite Join Address Issues (Nodes)** -``` -database.rqlite_join_address: required for node type (non-bootstrap) -database.rqlite_join_address: invalid format; expected host:port -``` -Solution: Non-bootstrap nodes must specify where to join the cluster. Use Raft port: `127.0.0.1:7001` - -**Bootstrap Nodes Cannot Join** -``` -database.rqlite_join_address: must be empty for bootstrap type -``` -Solution: Bootstrap nodes should have `rqlite_join_address: ""` - -**Invalid Listen Addresses** -``` -node.listen_addresses[0]: invalid TCP port 99999; port must be between 1 and 65535 -``` -Solution: Use valid ports [1-65535], e.g., `/ip4/0.0.0.0/tcp/4001` - -**Unknown Configuration Keys** -``` -invalid config: yaml: unmarshal errors: - line 42: field migrations_path not found in type config.DatabaseConfig -``` -Solution: Remove unsupported keys. Supported keys are documented in the YAML Reference section above. - ---- - -## CLI Usage - -### Authentication Commands - -```bash -./bin/network-cli auth login # Authenticate with wallet -./bin/network-cli auth whoami # Show current authentication status -./bin/network-cli auth status # Show detailed authentication info -./bin/network-cli auth logout # Clear stored credentials -``` - -### Network Operations - -```bash -./bin/network-cli health # Check network health -./bin/network-cli status # Get network status -./bin/network-cli peers # List connected peers -``` - -### Database Operations - -```bash -./bin/network-cli query "SELECT * FROM table" # Execute SQL -./bin/network-cli query "CREATE TABLE users (id INTEGER)" # DDL operations -``` - -### Pub/Sub Messaging - -```bash -./bin/network-cli pubsub publish # Send message -./bin/network-cli pubsub subscribe [duration] # Listen for messages -./bin/network-cli pubsub topics # List active topics -``` - -### CLI Options - -```bash ---format json # Output in JSON format ---timeout 30s # Set operation timeout ---bootstrap # Override bootstrap peer ---production # Use production bootstrap peers -``` - -### Database Operations (Gateway REST) - -```http -POST /v1/rqlite/exec # Body: {"sql": "INSERT/UPDATE/DELETE/DDL ...", "args": [...]} -POST /v1/rqlite/find # Body: {"table":"...", "criteria":{"col":val,...}, "options":{...}} -POST /v1/rqlite/find-one # Body: same as /find, returns a single row (404 if not found) -POST /v1/rqlite/select # Body: {"table":"...", "select":[...], "where":[...], "joins":[...], "order_by":[...], "limit":N, "offset":N, "one":false} -POST /v1/rqlite/transaction # Body: {"ops":[{"kind":"exec|query","sql":"...","args":[...]}], "return_results": true} -POST /v1/rqlite/query # Body: {"sql": "SELECT ...", "args": [..]} (legacy-friendly SELECT) -GET /v1/rqlite/schema # Returns tables/views + create SQL -POST /v1/rqlite/create-table # Body: {"schema": "CREATE TABLE ..."} -POST /v1/rqlite/drop-table # Body: {"table": "table_name"} -``` - -Common workflows: - -```bash -# Exec (INSERT/UPDATE/DELETE/DDL) -curl -X POST "$GW/v1/rqlite/exec" \ - -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \ - -d '{"sql":"INSERT INTO users(name,email) VALUES(?,?)","args":["Alice","alice@example.com"]}' - -# Find (criteria + options) -curl -X POST "$GW/v1/rqlite/find" \ - -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \ - -d '{ - "table":"users", - "criteria":{"active":true}, - "options":{"select":["id","email"],"order_by":["created_at DESC"],"limit":25} - }' - -# Select (fluent builder via JSON) -curl -X POST "$GW/v1/rqlite/select" \ - -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \ - -d '{ - "table":"orders o", - "select":["o.id","o.total","u.email AS user_email"], - "joins":[{"kind":"INNER","table":"users u","on":"u.id = o.user_id"}], - "where":[{"conj":"AND","expr":"o.total > ?","args":[100]}], - "order_by":["o.created_at DESC"], - "limit":10 - }' - -# Transaction (atomic batch) -curl -X POST "$GW/v1/rqlite/transaction" \ - -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \ - -d '{ - "return_results": true, - "ops": [ - {"kind":"exec","sql":"INSERT INTO users(email) VALUES(?)","args":["bob@example.com"]}, - {"kind":"query","sql":"SELECT last_insert_rowid() AS id","args":[]} - ] - }' - -# Schema -curl "$GW/v1/rqlite/schema" -H "Authorization: Bearer $API_KEY" - -# DDL helpers -curl -X POST "$GW/v1/rqlite/create-table" -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \ - -d '{"schema":"CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT, email TEXT)"}' -curl -X POST "$GW/v1/rqlite/drop-table" -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \ - -d '{"table":"users"}' -``` ### Authentication @@ -623,344 +344,4 @@ export DEBROS_GATEWAY_URL="http://localhost:6001" --- -## HTTP Gateway - -The DeBros Network includes a powerful HTTP/WebSocket gateway that provides a modern REST API and WebSocket interface over the P2P network, featuring an enhanced authentication system with multi-wallet support. - -### Quick Start - -```bash -make run-gateway -# Or manually: -go run ./cmd/gateway -``` - -### Configuration - -The gateway can be configured via configs/gateway.yaml and environment variables (env override YAML): - -```bash -# Basic Configuration -export GATEWAY_ADDR="0.0.0.0:6001" -export GATEWAY_NAMESPACE="my-app" -export GATEWAY_BOOTSTRAP_PEERS="/ip4/127.0.0.1/tcp/4001/p2p/YOUR_PEER_ID" - -# Authentication Configuration -export GATEWAY_REQUIRE_AUTH=true -export GATEWAY_API_KEYS="key1:namespace1,key2:namespace2" -``` - -### Enhanced Authentication System - -The gateway features a significantly improved authentication system with the following capabilities: - -#### Key Features - -- **Automatic Authentication:** No manual auth commands required - authentication happens automatically when needed -- **Multi-Wallet Support:** Seamlessly manage multiple wallet credentials with automatic switching -- **Persistent Sessions:** Wallet credentials are automatically saved and restored -- **Enhanced User Experience:** Streamlined authentication flow with better error handling - -#### Authentication Methods - -**Wallet-Based Authentication (Ethereum EIP-191)** - -- Uses `personal_sign` for secure wallet verification -- Supports multiple wallets with automatic detection -- Addresses are case-insensitive with normalized signature handling - -**JWT Tokens** - -- Issued by the gateway with configurable expiration -- JWKS endpoints available at `/v1/auth/jwks` and `/.well-known/jwks.json` -- Automatic refresh capability - -**API Keys** - -- Support for pre-configured API keys via `Authorization: Bearer ` or `X-API-Key` headers -- Optional namespace mapping for multi-tenant applications - -### API Endpoints - -#### Health & Status - -```http -GET /health # Basic health check -GET /v1/health # Detailed health status -GET /v1/status # Network status -GET /v1/version # Version information -``` - -#### Authentication (Public Endpoints) - -```http -POST /v1/auth/challenge # Generate wallet challenge -POST /v1/auth/verify # Verify wallet signature -POST /v1/auth/register # Register new wallet -POST /v1/auth/refresh # Refresh JWT token -POST /v1/auth/logout # Clear authentication -GET /v1/auth/whoami # Current auth status -POST /v1/auth/api-key # Generate API key (authenticated) -``` - -#### RQLite HTTP ORM Gateway (/v1/db) - -The gateway now exposes a full HTTP interface over the Go ORM-like client (see `pkg/rqlite/gateway.go`) so you can build SDKs in any language. - -- Base path: `/v1/db` -- Endpoints: - - `POST /v1/rqlite/exec` — Execute write/DDL SQL; returns `{ rows_affected, last_insert_id }` - - `POST /v1/rqlite/find` — Map-based criteria; returns `{ items: [...], count: N }` - - `POST /v1/rqlite/find-one` — Single row; 404 if not found - - `POST /v1/rqlite/select` — Fluent SELECT via JSON (joins, where, order, group, limit, offset) - - `POST /v1/rqlite/transaction` — Atomic batch of exec/query ops, optional per-op results - - `POST /v1/rqlite/query` — Arbitrary SELECT (legacy-friendly), returns `items` - - `GET /v1/rqlite/schema` — List user tables/views + create SQL - - `POST /v1/rqlite/create-table` — Convenience for DDL - - `POST /v1/rqlite/drop-table` — Safe drop (identifier validated) - -Payload examples are shown in the [Database Operations (Gateway REST)](#database-operations-gateway-rest) section. - -#### Network Operations - -```http -GET /v1/network/status # Network status -GET /v1/network/peers # Connected peers -POST /v1/network/connect # Connect to peer -POST /v1/network/disconnect # Disconnect from peer -``` - -#### Pub/Sub Messaging - -**WebSocket Interface** - -```http -GET /v1/pubsub/ws?topic= # WebSocket connection for real-time messaging -``` - -**REST Interface** - -```http -POST /v1/pubsub/publish # Publish message to topic -GET /v1/pubsub/topics # List active topics -``` - ---- - -## SDK Authoring Guide - -### Base concepts - -- OpenAPI: a machine-readable spec is available at `openapi/gateway.yaml` for SDK code generation. -- **Auth**: send `X-API-Key: ` or `Authorization: Bearer ` with every request. -- **Versioning**: all endpoints are under `/v1/`. -- **Responses**: mutations return `{status:"ok"}`; queries/lists return JSON; errors return `{ "error": "message" }` with proper HTTP status. - -### Key HTTP endpoints for SDKs - -- **Database** - - Exec: `POST /v1/rqlite/exec` `{sql, args?}` → `{rows_affected,last_insert_id}` - - Find: `POST /v1/rqlite/find` `{table, criteria, options?}` → `{items,count}` - - FindOne: `POST /v1/rqlite/find-one` `{table, criteria, options?}` → single object or 404 - - Select: `POST /v1/rqlite/select` `{table, select?, joins?, where?, order_by?, group_by?, limit?, offset?, one?}` - - Transaction: `POST /v1/rqlite/transaction` `{ops:[{kind,sql,args?}], return_results?}` - - Query: `POST /v1/rqlite/query` `{sql, args?}` → `{items,count}` - - Schema: `GET /v1/rqlite/schema` - - Create Table: `POST /v1/rqlite/create-table` `{schema}` - - Drop Table: `POST /v1/rqlite/drop-table` `{table}` -- **PubSub** - - WS Subscribe: `GET /v1/pubsub/ws?topic=` - - Publish: `POST /v1/pubsub/publish` `{topic, data_base64}` → `{status:"ok"}` - - Topics: `GET /v1/pubsub/topics` → `{topics:[...]}` - ---- - -## Troubleshooting - -### Configuration & Permissions - -**Error: "Failed to create/access config directory"** - -This happens when DeBros cannot access or create `~/.debros/` directory. - -**Causes:** -1. Home directory is not writable -2. Home directory doesn't exist -3. Filesystem is read-only (sandboxed/containerized environment) -4. Permission denied (running with wrong user/umask) - -**Solutions:** - -```bash -# Check home directory exists and is writable -ls -ld ~ -touch ~/test-write && rm ~/test-write - -# Check umask (should be 0022 or 0002) -umask - -# If umask is too restrictive, change it -umask 0022 - -# Check disk space -df -h ~ - -# For containerized environments, ensure /home/ is mounted with write permissions -docker run -v /home:/home --user $(id -u):$(id -g) debros-network -``` - -**Error: "Config file not found at ~/.debros/node.yaml"** - -The node requires a config file to exist before starting. - -**Solution:** - -Generate config files first: - -```bash -# Build CLI -make build - -# Generate configs -./bin/network-cli config init --type bootstrap -./bin/network-cli config init --type node --bootstrap-peers '' -./bin/network-cli config init --type gateway -``` - -### Node Startup Issues - -**Error: "node.data_dir: parent directory not writable"** - -The data directory parent is not accessible. - -**Solution:** - -Ensure `~/.debros` is writable and has at least 10GB free space: - -```bash -# Check permissions -ls -ld ~/.debros - -# Check available space -df -h ~/.debros - -# Recreate if corrupted -rm -rf ~/.debros -./bin/network-cli config init --type bootstrap -``` - -**Error: "failed to create data directory"** - -The node cannot create its data directory in `~/.debros`. - -**Causes:** -1. `~/.debros` is not writable -2. Parent directory path in config uses `~` which isn't expanded properly -3. Disk is full - -**Solutions:** - -```bash -# Check ~/.debros exists and is writable -mkdir -p ~/.debros -ls -ld ~/.debros - -# Verify data_dir in config uses ~ (e.g., ~/.debros/node) -cat ~/.debros/node.yaml | grep data_dir - -# Check disk space -df -h ~ - -# Ensure user owns ~/.debros -chown -R $(whoami) ~/.debros - -# Retry node startup -make run-node -``` - -**Error: "stat ~/.debros: no such file or directory"** - -**Port Already in Use** - -If you get "address already in use" errors: - -```bash -# Find processes using ports -lsof -i :4001 # P2P port -lsof -i :5001 # RQLite HTTP -lsof -i :7001 # RQLite Raft - -# Kill if needed -kill -9 - -# Or use different ports in config -./bin/network-cli config init --type node --listen-port 4002 --rqlite-http-port 5002 --rqlite-raft-port 7002 -``` - -### Common Configuration Errors - -**Error: "discovery.bootstrap_peers: required for node type"** - -Nodes (non-bootstrap) must specify bootstrap peers to discover the network. - -**Solution:** - -Generate node config with bootstrap peers: - -```bash -./bin/network-cli config init --type node --bootstrap-peers '/ip4/127.0.0.1/tcp/4001/p2p/12D3KooW...' -``` - -**Error: "database.rqlite_join_address: required for node type"** - -Non-bootstrap nodes must specify which node to join in the Raft cluster. - -**Solution:** - -Generate config with join address: - -```bash -./bin/network-cli config init --type node --join localhost:5001 -``` - -**Error: "database.rqlite_raft_port: must differ from database.rqlite_port"** - -HTTP and Raft ports cannot be the same. - -**Solution:** - -Use different ports (RQLite HTTP and Raft must be on different ports): - -```bash -./bin/network-cli config init --type node \ - --rqlite-http-port 5001 \ - --rqlite-raft-port 7001 -``` - -### Peer Discovery Issues - -If nodes can't find each other: - -1. **Verify bootstrap node is running:** - ```bash - ./bin/network-cli health - ./bin/network-cli peers - ``` - -2. **Check bootstrap peer multiaddr is correct:** - ```bash - cat ~/.debros/bootstrap/peer.info # On bootstrap node - # Should match value in other nodes' discovery.bootstrap_peers - ``` - -3. **Ensure all nodes have same bootstrap peers in config** - -4. **Check firewall/network:** - ```bash - # Verify P2P port is open - nc -zv 127.0.0.1 4001 - ``` - ---- - -## License \ No newline at end of file +## License