DeBros Network - Distributed P2P Database System

A robust, decentralized peer-to-peer network built in Go, providing distributed SQL database, key-value storage, pub/sub messaging, and resilient peer management. Designed for applications needing reliable, scalable, and secure data sharing without centralized infrastructure.

---

Table of Contents

• Features
• Architecture Overview
• System Requirements
• Quick Start
• Deployment & Installation
• Configuration
• CLI Usage
• Development
• Troubleshooting
• License

---

Features

• Distributed SQL Database: RQLite-backed, Raft-consensus, ACID transactions, automatic failover.
• Key-Value Storage: Namespaced, replicated, CRUD operations, prefix queries.
• Pub/Sub Messaging: Topic-based, real-time, namespaced, automatic cleanup.
• Peer Discovery & Management: Nodes discover peers, bootstrap support, health monitoring.
• Application Isolation: Namespace-based multi-tenancy, per-app config.
• Secure by Default: Noise/TLS transport, peer identity, systemd hardening.
• Simple Client API: Lightweight Go client for apps and CLI tools.

---

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                  DeBros Network Cluster                    │
├─────────────────────────────────────────────────────────────┤
│                   Application Layer                        │
│ ┌─────────────┐ ┌─────────────┐ ┌────────────────────────┐ │
│ │   Anchat    │ │ Custom App  │ │      CLI Tools        │ │
│ └─────────────┘ └─────────────┘ └────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│                      Client API                            │
│ ┌─────────────┐ ┌─────────────┐ ┌────────────────────────┐ │
│ │  Database   │ │   Storage   │ │        PubSub         │ │
│ │   Client    │ │   Client    │ │        Client         │ │
│ └─────────────┘ └─────────────┘ └────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│                    Network Node Layer                      │
│ ┌─────────────┐ ┌─────────────┐ ┌────────────────────────┐ │
│ │ Discovery   │ │   PubSub    │ │      Database         │ │
│ │  Manager    │ │   Manager   │ │    (RQLite)          │ │
│ └─────────────┘ └─────────────┘ └────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│                  Transport Layer                           │
│ ┌─────────────┐ ┌─────────────┐ ┌────────────────────────┐ │
│ │   LibP2P    │ │   Noise/TLS │ │      RQLite           │ │
│ │   Host      │ │  Encryption │ │    Database           │ │
│ └─────────────┘ └─────────────┘ └────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

• Node: Full P2P participant, runs services, handles peer discovery, database, storage, pubsub.
• Client: Lightweight, connects only to bootstrap peers, consumes services, no peer discovery.

---

System Requirements

Software

• Go: 1.21+ (recommended)
• RQLite: 8.x (distributed SQLite)
• Git: For source management
• Make: For build automation (recommended)

Hardware

• Minimum: 2 CPU cores, 4GB RAM, 10GB disk, stable internet
• Recommended: 4+ cores, 8GB+ RAM, 50GB+ SSD, low-latency network

Network Ports

• 4001: LibP2P P2P communication
• 5001: RQLite HTTP API
• 7001: RQLite Raft consensus

---

Quick Start

1. Clone and Setup

git clone https://git.debros.io/DeBros/network.git
cd network

2. Build All Executables

make build

3. Start a Bootstrap Node

make run-node
# Or manually:
go run ./cmd/node --config configs/bootstrap.yaml

4. Start Additional Nodes

make run-node2
# Or manually:
go run ./cmd/node --config configs/node.yaml

5. Test with CLI

./bin/network-cli health
./bin/network-cli peers
./bin/network-cli storage put test-key "Hello Network"
./bin/network-cli storage get test-key
./bin/network-cli pubsub publish notifications "Hello World"
./bin/network-cli pubsub subscribe notifications 10s

---

Deployment & Installation

Automated Production Install

Run the install script for a secure, production-ready setup:

curl -sSL https://git.debros.io/DeBros/network/raw/branch/main/scripts/install-debros-network.sh | sudo bash

What the Script Does:

• Detects OS, installs Go, RQLite, dependencies
• Creates debros system user, secure directory structure
• Generates LibP2P identity keys
• Clones source, builds binaries
• Sets up systemd service (debros-node)
• Configures firewall (UFW) for required ports
• Generates YAML config in /opt/debros/configs/node.yaml

Directory Structure:

/opt/debros/
├── bin/           # Binaries
├── configs/       # YAML configs
├── keys/          # Identity keys
├── data/          # RQLite DB, storage
├── logs/          # Node logs
├── src/           # Source code

Service Management:

sudo systemctl status debros-node
sudo systemctl start debros-node
sudo systemctl stop debros-node
sudo systemctl restart debros-node
sudo journalctl -u debros-node.service -f

---

Configuration

Example Configuration Files

configs/bootstrap.yaml

node:
  id: ""
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4001"
  data_dir: "./data/bootstrap"
  max_connections: 100
  disable_anonrc: true

database:
  data_dir: "./data/db"
  replication_factor: 3
  shard_count: 16
  max_database_size: 1073741824
  backup_interval: 24h
  rqlite_port: 5001
  rqlite_raft_port: 7001
  rqlite_join_address: "" # Bootstrap node does not join

discovery:
  bootstrap_peers: []
  discovery_interval: 15s
  bootstrap_port: 4001
  http_adv_address: "127.0.0.1"
  raft_adv_address: ""

security:
  enable_tls: false
  private_key_file: ""
  certificate_file: ""
  auth_enabled: false

logging:
  level: "info"
  format: "console"
  output_file: ""

configs/node.yaml

node:
  id: "node2"
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4002"
  data_dir: "./data/node2"
  max_connections: 50
  disable_anonrc: true

database:
  data_dir: "./data/db"
  replication_factor: 3
  shard_count: 16
  max_database_size: 1073741824
  backup_interval: 24h
  rqlite_port: 5002
  rqlite_raft_port: 7002
  rqlite_join_address: "http://127.0.0.1:5001"

discovery:
  bootstrap_peers:
    - "/ip4/127.0.0.1/tcp/4001/p2p/<YOUR_BOOTSTRAP_PEER_ID>"
  discovery_interval: 15s
  bootstrap_port: 4002
  http_adv_address: "127.0.0.1"
  raft_adv_address: ""

security:
  enable_tls: false
  private_key_file: ""
  certificate_file: ""
  auth_enabled: false

logging:
  level: "info"
  format: "console"
  output_file: ""

Flags & Environment Variables

• Flags: Override config at startup (--data, --p2p-port, --rqlite-http-port, etc.)
• Env Vars: Override config and flags (NODE_ID, RQLITE_PORT, BOOTSTRAP_PEERS, etc.)
• Precedence: Flags > Env Vars > YAML > Defaults

Bootstrap & Database Endpoints

• Bootstrap peers: Set in config or via BOOTSTRAP_PEERS env var.
• Database endpoints: Set in config or via RQLITE_NODES env var.
• Development mode: Use NETWORK_DEV_LOCAL=1 for localhost defaults.

---

CLI Usage

Network Operations

./bin/network-cli health                    # Check network health
./bin/network-cli status                    # Get network status
./bin/network-cli peers                     # List connected peers

Storage Operations

./bin/network-cli storage put <key> <value> # Store data
./bin/network-cli storage get <key>         # Retrieve data
./bin/network-cli storage list [prefix]     # List keys

Database Operations

./bin/network-cli query "SELECT * FROM table"              # Execute SQL
./bin/network-cli query "CREATE TABLE users (id INTEGER)"  # DDL operations

Pub/Sub Messaging

./bin/network-cli pubsub publish <topic> <message>     # Send message
./bin/network-cli pubsub subscribe <topic> [duration]  # Listen for messages
./bin/network-cli pubsub topics                        # List active topics

CLI Options

--format json                 # Output in JSON format
--timeout 30s                 # Set operation timeout
--bootstrap <multiaddr>       # Override bootstrap peer
--production                  # Use production bootstrap peers
--disable-anonrc              # Disable anonymous routing (Tor/SOCKS5)

---

Development

Project Structure

network/
├── cmd/
│   ├── node/         # Network node executable
│   └── cli/          # Command-line interface
├── pkg/
│   ├── client/       # Client library
│   ├── node/         # Node implementation
│   ├── database/     # RQLite integration
│   ├── storage/      # Storage service
│   ├── pubsub/       # Pub/Sub messaging
│   ├── config/       # Centralized config
│   └── discovery/    # Peer discovery (node only)
├── scripts/          # Install, test scripts
├── configs/          # YAML configs
├── bin/              # Built executables

Build & Test

make build           # Build all executables
make test            # Run unit tests
make clean           # Clean build artifacts

Local Multi-Node Testing

scripts/test-multinode.sh

---

Troubleshooting

Common Issues

Bootstrap Connection Failed

• Symptoms: Failed to connect to bootstrap peer
• Solutions: Check node is running, firewall settings, peer ID validity.

Database Operations Timeout

• Symptoms: Query timeout or No RQLite connection available
• Solutions: Ensure RQLite ports are open, leader election completed, cluster join config correct.

Message Delivery Failures

• Symptoms: Messages not received by subscribers
• Solutions: Verify topic names, active subscriptions, network connectivity.

High Memory Usage

• Symptoms: Memory usage grows continuously
• Solutions: Unsubscribe when done, monitor connection pool, review message retention.

Debugging & Health Checks

export LOG_LEVEL=debug
./bin/network-cli health
./bin/network-cli peers
./bin/network-cli query "SELECT 1"
./bin/network-cli pubsub publish test "hello"
./bin/network-cli pubsub subscribe test 10s

Service Logs

sudo journalctl -u debros-node.service --since "1 hour ago"

---

License

Distributed under the MIT License. See LICENSE for details.

---

Further Reading

• DeBros Network Documentation
• RQLite Documentation
• LibP2P Documentation

---

This README reflects the latest architecture, configuration, and operational practices for the DeBros Network. For questions or contributions, please open an issue or pull request.