# DeBros Network - Distributed P2P Database System A robust, decentralized peer-to-peer network built in Go, providing distributed SQL database, key-value storage, pub/sub messaging, and resilient peer management. Designed for applications needing reliable, scalable, and secure data sharing without centralized infrastructure. --- ## Table of Contents - [Features](#features) - [Architecture Overview](#architecture-overview) - [System Requirements](#system-requirements) - [Quick Start](#quick-start) - [Deployment & Installation](#deployment--installation) - [Configuration](#configuration) - [CLI Usage](#cli-usage) - [HTTP Gateway](#http-gateway) - [Development](#development) - [Troubleshooting](#troubleshooting) - [License](#license) --- ## Features - **Distributed SQL Database:** RQLite-backed, Raft-consensus, ACID transactions, automatic failover. - **Key-Value Storage:** Namespaced, replicated, CRUD operations, prefix queries. - **Pub/Sub Messaging:** Topic-based, real-time, namespaced, automatic cleanup. - **Peer Discovery & Management:** Nodes discover peers, bootstrap support, health monitoring. - **Application Isolation:** Namespace-based multi-tenancy, per-app config. - **Secure by Default:** Noise/TLS transport, peer identity, systemd hardening. - **Simple Client API:** Lightweight Go client for apps and CLI tools. --- ## Architecture Overview ``` ┌─────────────────────────────────────────────────────────────┐ │ DeBros Network Cluster │ ├─────────────────────────────────────────────────────────────┤ │ Application Layer │ │ ┌─────────────┐ ┌─────────────┐ ┌────────────────────────┐ │ │ │ Anchat │ │ Custom App │ │ CLI Tools │ │ │ └─────────────┘ └─────────────┘ └────────────────────────┘ │ ├─────────────────────────────────────────────────────────────┤ │ Client API │ │ ┌─────────────┐ ┌─────────────┐ ┌────────────────────────┐ │ │ │ Database │ │ Storage │ │ PubSub │ │ │ │ Client │ │ Client │ │ Client │ │ │ └─────────────┘ └─────────────┘ └────────────────────────┘ │ ├─────────────────────────────────────────────────────────────┤ │ Network Node Layer │ │ ┌─────────────┐ ┌─────────────┐ ┌────────────────────────┐ │ │ │ Discovery │ │ PubSub │ │ Database │ │ │ │ Manager │ │ Manager │ │ (RQLite) │ │ │ └─────────────┘ └─────────────┘ └────────────────────────┘ │ ├─────────────────────────────────────────────────────────────┤ │ Transport Layer │ │ ┌─────────────┐ ┌─────────────┐ ┌────────────────────────┐ │ │ │ LibP2P │ │ Noise/TLS │ │ RQLite │ │ │ │ Host │ │ Encryption │ │ Database │ │ │ └─────────────┘ └─────────────┘ └────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ ``` - **Node:** Full P2P participant, runs services, handles peer discovery, database, storage, pubsub. - **Client:** Lightweight, connects only to bootstrap peers, consumes services, no peer discovery. --- ## System Requirements ### Software - **Go:** 1.21+ (recommended) - **RQLite:** 8.x (distributed SQLite) - **Git:** For source management - **Make:** For build automation (recommended) ### Hardware - **Minimum:** 2 CPU cores, 4GB RAM, 10GB disk, stable internet - **Recommended:** 4+ cores, 8GB+ RAM, 50GB+ SSD, low-latency network ### Network Ports - **4001:** LibP2P P2P communication - **5001:** RQLite HTTP API - **7001:** RQLite Raft consensus --- ## Quick Start ### 1. Clone and Setup ```bash git clone https://git.debros.io/DeBros/network.git cd network ``` ### 2. Build All Executables ```bash make build ``` ### 3. Start a Bootstrap Node ```bash make run-node # Or manually: go run ./cmd/node --config configs/bootstrap.yaml ``` ### 4. Start Additional Nodes ```bash make run-node2 # Or manually: go run ./cmd/node --config configs/node.yaml ``` ### 5. Test with CLI ```bash ./bin/network-cli health ./bin/network-cli peers ./bin/network-cli storage put test-key "Hello Network" ./bin/network-cli storage get test-key ./bin/network-cli pubsub publish notifications "Hello World" ./bin/network-cli pubsub subscribe notifications 10s ``` --- ## Deployment & Installation ### Automated Production Install Run the install script for a secure, production-ready setup: ```bash curl -sSL https://git.debros.io/DeBros/network/raw/branch/main/scripts/install-debros-network.sh | sudo bash ``` **What the Script Does:** - Detects OS, installs Go, RQLite, dependencies - Creates `debros` system user, secure directory structure - Generates LibP2P identity keys - Clones source, builds binaries - Sets up systemd service (`debros-node`) - Configures firewall (UFW) for required ports - Generates YAML config in `/opt/debros/configs/node.yaml` **Directory Structure:** ``` /opt/debros/ ├── bin/ # Binaries ├── configs/ # YAML configs ├── keys/ # Identity keys ├── data/ # RQLite DB, storage ├── logs/ # Node logs ├── src/ # Source code ``` **Service Management:** ```bash sudo systemctl status debros-node sudo systemctl start debros-node sudo systemctl stop debros-node sudo systemctl restart debros-node sudo journalctl -u debros-node.service -f ``` --- ## Configuration ### Example Configuration Files #### `configs/bootstrap.yaml` ```yaml node: id: "" listen_addresses: - "/ip4/0.0.0.0/tcp/4001" data_dir: "./data/bootstrap" max_connections: 100 disable_anonrc: true database: data_dir: "./data/db" replication_factor: 3 shard_count: 16 max_database_size: 1073741824 backup_interval: 24h rqlite_port: 5001 rqlite_raft_port: 7001 rqlite_join_address: "" # Bootstrap node does not join discovery: bootstrap_peers: [] discovery_interval: 15s bootstrap_port: 4001 http_adv_address: "127.0.0.1" raft_adv_address: "" security: enable_tls: false private_key_file: "" certificate_file: "" auth_enabled: false logging: level: "info" format: "console" output_file: "" ``` #### `configs/node.yaml` ```yaml node: id: "node2" listen_addresses: - "/ip4/0.0.0.0/tcp/4002" data_dir: "./data/node2" max_connections: 50 disable_anonrc: true database: data_dir: "./data/db" replication_factor: 3 shard_count: 16 max_database_size: 1073741824 backup_interval: 24h rqlite_port: 5002 rqlite_raft_port: 7002 rqlite_join_address: "http://127.0.0.1:5001" discovery: bootstrap_peers: - "/ip4/127.0.0.1/tcp/4001/p2p/" discovery_interval: 15s bootstrap_port: 4002 http_adv_address: "127.0.0.1" raft_adv_address: "" security: enable_tls: false private_key_file: "" certificate_file: "" auth_enabled: false logging: level: "info" format: "console" output_file: "" ``` ### Flags & Environment Variables - **Flags**: Override config at startup (`--data`, `--p2p-port`, `--rqlite-http-port`, etc.) - **Env Vars**: Override config and flags (`NODE_ID`, `RQLITE_PORT`, `BOOTSTRAP_PEERS`, etc.) - **Precedence**: Flags > Env Vars > YAML > Defaults ### Bootstrap & Database Endpoints - **Bootstrap peers**: Set in config or via `BOOTSTRAP_PEERS` env var. - **Database endpoints**: Set in config or via `RQLITE_NODES` env var. - **Development mode**: Use `NETWORK_DEV_LOCAL=1` for localhost defaults. --- ## CLI Usage ### Network Operations ```bash ./bin/network-cli health # Check network health ./bin/network-cli status # Get network status ./bin/network-cli peers # List connected peers ``` ### Storage Operations ```bash ./bin/network-cli storage put # Store data ./bin/network-cli storage get # Retrieve data ./bin/network-cli storage list [prefix] # List keys ``` ### Database Operations ```bash ./bin/network-cli query "SELECT * FROM table" # Execute SQL ./bin/network-cli query "CREATE TABLE users (id INTEGER)" # DDL operations ``` ### Pub/Sub Messaging ```bash ./bin/network-cli pubsub publish # Send message ./bin/network-cli pubsub subscribe [duration] # Listen for messages ./bin/network-cli pubsub topics # List active topics ``` ### CLI Options ```bash --format json # Output in JSON format --timeout 30s # Set operation timeout --bootstrap # Override bootstrap peer --production # Use production bootstrap peers --disable-anonrc # Disable anonymous routing (Tor/SOCKS5) ``` --- ## Development ### Project Structure ``` network/ ├── cmd/ │ ├── node/ # Network node executable │ └── cli/ # Command-line interface ├── pkg/ │ ├── client/ # Client library │ ├── node/ # Node implementation │ ├── database/ # RQLite integration │ ├── storage/ # Storage service │ ├── pubsub/ # Pub/Sub messaging │ ├── config/ # Centralized config │ └── discovery/ # Peer discovery (node only) ├── scripts/ # Install, test scripts ├── configs/ # YAML configs ├── bin/ # Built executables ``` ### Build & Test ```bash make build # Build all executables make test # Run unit tests make clean # Clean build artifacts ``` ### Local Multi-Node Testing ```bash scripts/test-multinode.sh ``` --- ## Troubleshooting ### Common Issues #### Bootstrap Connection Failed - **Symptoms:** `Failed to connect to bootstrap peer` - **Solutions:** Check node is running, firewall settings, peer ID validity. #### Database Operations Timeout - **Symptoms:** `Query timeout` or `No RQLite connection available` - **Solutions:** Ensure RQLite ports are open, leader election completed, cluster join config correct. #### Message Delivery Failures - **Symptoms:** Messages not received by subscribers - **Solutions:** Verify topic names, active subscriptions, network connectivity. #### High Memory Usage - **Symptoms:** Memory usage grows continuously - **Solutions:** Unsubscribe when done, monitor connection pool, review message retention. ### Debugging & Health Checks ```bash export LOG_LEVEL=debug ./bin/network-cli health ./bin/network-cli peers ./bin/network-cli query "SELECT 1" ./bin/network-cli pubsub publish test "hello" ./bin/network-cli pubsub subscribe test 10s ``` ### Service Logs ```bash sudo journalctl -u debros-node.service --since "1 hour ago" ``` --- ## License Distributed under the MIT License. See [LICENSE](LICENSE) for details. --- ## Further Reading - [DeBros Network Documentation](https://docs.debros.io) - [RQLite Documentation](https://github.com/rqlite/rqlite) - [LibP2P Documentation](https://libp2p.io) --- _This README reflects the latest architecture, configuration, and operational practices for the DeBros Network. For questions or contributions, please open an issue or pull request._