DeBros/network
4
0
Fork 0
mirror of https://github.com/DeBrosOfficial/network.git synced 2025-10-06 08:39:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
network/README.md

19 KiB
Raw Blame History

Network - Distributed P2P Database System v0.12.5-beta

A distributed peer-to-peer network built with Go and LibP2P, providing decentralized database capabilities with RQLite consensus and replication.

Table of Contents

Features

  • Peer-to-Peer Networking: Built on LibP2P for robust P2P communication
  • Distributed Database: RQLite-based distributed SQLite with Raft consensus
  • Automatic Peer Discovery: Nodes help new peers join the network
  • CLI Tool: Command-line interface for network operations and testing
  • Client Library: Simple Go API for applications to interact with the network
  • Application Isolation: Namespaced storage and messaging per application

System Requirements

Software Dependencies

  • Go: Version 1.21 or later
  • RQLite: Distributed SQLite database
  • Git: For cloning the repository
  • Make: For build automation (optional but recommended)

Installation

macOS

# Install Homebrew if you don't have it
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install dependencies
brew install go rqlite git make

# Verify installation
go version    # Should show Go 1.21+
rqlited --version

Ubuntu/Debian

# Install Go (latest version)
sudo rm -rf /usr/local/go
wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

# Install RQLite
wget https://github.com/rqlite/rqlite/releases/download/v8.43.0/rqlite-v8.43.0-linux-amd64.tar.gz
tar -xzf rqlite-v8.43.0-linux-amd64.tar.gz
sudo mv rqlite-v8.43.0-linux-amd64/rqlited /usr/local/bin/

# Install other dependencies
sudo apt update
sudo apt install git make

# Verify installation
go version
rqlited --version

Windows

# Install Go from https://golang.org/dl/
# Install Git from https://git-scm.com/download/win
# Install RQLite from https://github.com/rqlite/rqlite/releases

# Or use Chocolatey
choco install golang git make
# Download RQLite manually from releases page

Hardware Requirements

Minimum:

  • CPU: 2 cores
  • RAM: 4GB
  • Storage: 10GB free space
  • Network: Stable internet connection

Recommended:

  • CPU: 4+ cores
  • RAM: 8GB+
  • Storage: 50GB+ SSD
  • Network: Low-latency internet connection

Network Ports

The system uses these ports by default:

  • 4001: LibP2P communication
  • 5001: RQLite HTTP API
  • 7001: RQLite Raft consensus

Ensure these ports are available or configure firewall rules accordingly.

Quick Start

1. Clone and Setup Environment

# Clone the repository
git clone https://git.debros.io/DeBros/network.git
cd network

2. Generate Bootstrap Identity (Development Only)

For development, you need to generate a consistent bootstrap peer identity:

# Generate bootstrap peer identity
go run scripts/generate-bootstrap-identity.go

# This will create data/bootstrap/identity.key and show the peer ID (and multiaddr)
# Save the printed peer ID to use with the -bootstrap flag

Important: After generating the bootstrap identity, copy the printed multiaddr or peer ID for use with the -bootstrap flag when starting regular nodes.

3. Build the Project

# Build all network executables
make build

4. Start the Network

Terminal 1 - Bootstrap Node:

# Start an explicit bootstrap node (LibP2P 4001, RQLite 5001/7001)
go run cmd/node/main.go -role bootstrap -data ./data/bootstrap

Terminal 2 - Regular Node:

# Replace <BOOTSTRAP_PEER_ID> with the ID printed by the identity generator
go run cmd/node/main.go \
  -role node \
  -id node2 \
  -data ./data/node2 \
  -bootstrap /ip4/127.0.0.1/tcp/4001/p2p/<BOOTSTRAP_PEER_ID> \
  -rqlite-http-port 5002 \
  -rqlite-raft-port 7002

Terminal 3 - Another Node (optional):

go run cmd/node/main.go \
  -role node \
  -id node3 \
  -data ./data/node3 \
  -bootstrap /ip4/127.0.0.1/tcp/4001/p2p/<BOOTSTRAP_PEER_ID> \
  -rqlite-http-port 5003 \
  -rqlite-raft-port 7003

5. Test with CLI

# Check current bootstrap configuration
make show-bootstrap

# Check network health
./bin/cli health

# Test storage operations
./bin/cli storage put test-key "Hello Network"
./bin/cli storage get test-key

# List connected peers
./bin/cli peers

Deployment

Production Installation Script

For production deployments on Linux servers, we provide an automated installation script that handles all dependencies, configuration, and service setup.

One-Command Installation

# Download and run the installation script
curl -sSL https://git.debros.io/DeBros/network/raw/branch/main/scripts/install-debros-network.sh | sudo bash

What the Script Does

  1. System Setup:

    • Detects OS (Ubuntu/Debian/CentOS/RHEL/Fedora)
    • Installs Go 1.21+ with architecture detection
    • Installs system dependencies (git, make, build tools)
    • Checks port availability (4001, 5001, 7001)

  2. Configuration Wizard:

    • Solana wallet address for node operator rewards
    • Installation directory (default: /opt/debros)
    • Automatic firewall configuration (UFW)

  3. Secure Installation:

    • Creates dedicated debros system user
    • Sets up secure directory structure with proper permissions
    • Generates LibP2P identity keys with secure storage
    • Clones source code and builds binaries

  4. Service Management:

    • Creates systemd service with security hardening
    • Enables automatic startup and restart on failure
    • Configures structured logging to systemd journal

Directory Structure

The script creates a production-ready directory structure:

/opt/debros/
├── bin/                    # Compiled binaries
│   ├── bootstrap          # Bootstrap node executable
│   ├── node              # Node executable
│   └── cli               # CLI tools
├── configs/               # Configuration files
│   └── node.yaml         # Node configuration
├── keys/                  # Identity keys (secure 700 permissions)
│   └── node/
│       └── identity.key
├── data/                  # Runtime data
│   └── node/
│       ├── rqlite/       # RQLite database files
│       └── storage/      # P2P storage data
├── logs/                  # Application logs
│   └── node.log
└── src/                   # Source code (for updates)

Node Setup

The installation script sets up a network node:

  • Runs on ports: 4001 (P2P), 5001 (RQLite), 7001 (Raft)
  • Participates in DHT for peer discovery and data replication
  • Can be deployed on any server or VPS

For setup, please run these commands with adequate permissions:

  • Ensure you have elevated privileges or run as a user with the necessary permissions for server setup.
  • Follow the installation steps correctly to ensure a smooth deployment.

Service Management

After installation, manage your node with these commands:

# Check service status
sudo systemctl status debros-node

# Start/stop/restart service
sudo systemctl start debros-node
sudo systemctl stop debros-node
sudo systemctl restart debros-node

# View real-time logs
sudo journalctl -u debros-node.service -f

# Enable/disable auto-start
sudo systemctl enable debros-node
sudo systemctl disable debros-node

# Use CLI tools
/opt/debros/bin/cli health
/opt/debros/bin/cli peers
/opt/debros/bin/cli storage put key value

Configuration Files

The script generates YAML configuration files:

Node Configuration (/opt/debros/configs/node.yaml):

node:
  data_dir: "/opt/debros/data/node"
  key_file: "/opt/debros/keys/node/identity.key"
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4001"
  solana_wallet: "YOUR_WALLET_ADDRESS"

database:
  rqlite_port: 5001
  rqlite_raft_port: 7001

logging:
  level: "info"
  file: "/opt/debros/logs/node.log"

Security Features

The installation script implements production security best practices:

  • Dedicated User: Runs as debros system user (not root)
  • File Permissions: Key files have 600 permissions, directories have proper ownership
  • Systemd Security: Service runs with NoNewPrivileges, PrivateTmp, ProtectSystem=strict
  • Firewall: Automatic UFW configuration for required ports
  • Network Isolation: Proper port management to avoid conflicts

Network Discovery

  • Network Peers: Hardcoded in the application for automatic connection
  • DHT Discovery: Nodes automatically join Kademlia DHT for peer discovery
  • Peer Exchange: Connected nodes share information about other peers
  • No Manual Configuration: Nodes connect automatically without user intervention

Updates and Maintenance

# Update to latest version (re-run the installation script)
curl -sSL https://git.debros.io/DeBros/network/raw/branch/main/scripts/install-debros-network.sh | bash

# Manual source update
cd /opt/debros/src
sudo -u debros git pull
sudo -u debros make build
sudo cp bin/* /opt/debros/bin/
sudo systemctl restart debros-node

# Backup configuration and keys
sudo cp -r /opt/debros/configs /backup/
sudo cp -r /opt/debros/keys /backup/

Monitoring and Troubleshooting

# Check if ports are open
sudo netstat -tuln | grep -E "(4001|5001|7001)"

# Check service logs
sudo journalctl -u debros-node.service --since "1 hour ago"

# Check network connectivity
/opt/debros/bin/cli health
/opt/debros/bin/cli peers

# Check disk usage
du -sh /opt/debros/data/*

# Process information
ps aux | grep debros

For more advanced configuration options and development setup, see the sections below.

Configuration

Bootstrap and Ports (via flags)

  • Bootstrap node: -role bootstrap
  • Regular node: -role node -bootstrap <multiaddr>
  • RQLite ports: -rqlite-http-port (default 5001), -rqlite-raft-port (default 7001)

Examples are shown in Quick Start above for local multi-node on a single machine.

CLI Commands

The CLI can still accept --bootstrap <multiaddr> to override discovery when needed.

Network Operations

./bin/cli health                    # Check network health
./bin/cli status                    # Get network status
./bin/cli peers                     # List connected peers

Storage Operations

./bin/cli storage put <key> <value> # Store data
./bin/cli storage get <key>         # Retrieve data
./bin/cli storage list [prefix]     # List keys

Database Operations

./bin/cli query "SELECT * FROM table"              # Execute SQL
./bin/cli query "CREATE TABLE users (id INTEGER)"  # DDL operations

Pub/Sub Messaging

./bin/cli pubsub publish <topic> <message>     # Send message
./bin/cli pubsub subscribe <topic> [duration]  # Listen for messages
./bin/cli pubsub topics                        # List active topics

CLI Options

--format json                 # Output in JSON format
--timeout 30s                # Set operation timeout
--bootstrap <multiaddr>      # Override bootstrap peer

Development

Project Structure

network/
├── cmd/
│   ├── bootstrap/          # Bootstrap node
│   ├── node/              # Regular network node
│   └── cli/               # Command-line interface
├── pkg/
│   ├── client/            # Client library
│   ├── node/              # Node implementation
│   ├── database/          # RQLite integration
│   ├── storage/           # Storage service
│   ├── constants/         # Bootstrap configuration
│   └── config/            # System configuration
├── scripts/             # Helper scripts (install, security, tests)
├── scripts/             # Helper scripts (install, security, tests)
├── bin/                  # Built executables

Building and Testing

# Build all network executables
make build

# Show current bootstrap configuration
make show-bootstrap

# Run node (auto-detects bootstrap vs regular based on .env)
make run-node

# Clean data directories
make clean

# Run tests
go test ./...

# Full development workflow
make dev

Development Workflow

  1. Initial Setup:

    # Copy environment templates
cp .env.example .env

# Generate consistent bootstrap identity
go run scripts/generate-bootstrap-identity.go

# Update .env files with the generated peer ID

  2. Build Everything:

    make build        # Build network components

  3. Start Development Cluster:

    # Terminal 1: Bootstrap node (auto-detected)
make run-node

# Terminal 2: Regular node (auto-connects via .env)
make run-node

# Terminal 3: Test with CLI
./bin/cli health
./bin/cli peers

Environment Setup

  1. Install Dependencies:

    # macOS
brew install go rqlite git make

# Ubuntu/Debian
sudo apt install golang-go git make
# Install RQLite from https://github.com/rqlite/rqlite/releases

  2. Verify Installation:

    go version      # Should be 1.21+
rqlited --version
make --version

  3. Configure Environment:

    # Setup .env files
cp .env.example .env

# Generate bootstrap identity
go run scripts/generate-bootstrap-identity.go

# Update .env files with generated peer ID

Configuration System

The network uses a dual configuration system:

  1. Environment Variables (.env files): Primary configuration method
  2. Hardcoded Constants: Fallback when .env files are not found

Bootstrap Configuration Priority:

  1. Command line flags (if provided)
  2. Environment variables from .env files
  3. Hardcoded constants in pkg/constants/bootstrap.go
  4. Auto-discovery from running bootstrap nodes

This ensures the network can start even without configuration files, while allowing easy customization for different environments.

Client Library Usage

package main

import (
    "context"
    "log"
    "network/pkg/client"
)

func main() {
    // Create client (bootstrap peer discovered automatically)
    config := client.DefaultClientConfig("my-app")
    networkClient, err := client.NewClient(config)
    if err != nil {
        log.Fatal(err)
    }

    // Connect to network
    if err := networkClient.Connect(); err != nil {
        log.Fatal(err)
    }
    defer networkClient.Disconnect()

    // Use storage
    ctx := context.Background()
    storage := networkClient.Storage()

    err = storage.Put(ctx, "user:123", []byte("user data"))
    if err != nil {
        log.Fatal(err)
    }

    data, err := storage.Get(ctx, "user:123")
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Retrieved: %s", string(data))
}

Troubleshooting

Common Issues

Bootstrap peer not found / Peer ID mismatch:

  • Generate a new bootstrap identity: go run scripts/generate-bootstrap-identity.go
  • Update .env with the new peer ID
  • Restart the bootstrap node: make run-node
  • Check configuration: make show-bootstrap

Nodes can't connect:

  • Verify .env files have the correct bootstrap peer ID
  • Check that the bootstrap node is running: ps aux | grep node
  • Verify firewall settings and port availability (4001, 5001, 7001)
  • Try restarting with clean data: make clean && make run-node

Storage operations fail:

  • Ensure at least one node is running and connected
  • Check network health: ./bin/cli health
  • Verify RQLite is properly installed: rqlited --version
  • Check for port conflicts: netstat -an | grep -E "(4001|5001|7001)"

Debug Commands

# Check current configuration
make show-bootstrap
cat .env

# Check running processes
ps aux | grep -E "(bootstrap|node|rqlite)"

# Check port usage
netstat -an | grep -E "(4001|5001|7001)"

# Check bootstrap peer info
cat data/bootstrap/peer.info

# Clean and restart everything
make clean
make run-node      # In one terminal (auto-detects as bootstrap)
make run-node      # In another terminal (runs as regular node)

Environment-specific Issues

Development Environment:

  • Always run go run scripts/generate-bootstrap-identity.go first
  • Update .env files with the generated peer ID
  • Use make run-node - the system auto-detects if it should run as bootstrap

Production Environment:

  • Use stable, external bootstrap peer addresses
  • Configure multiple bootstrap peers for redundancy
  • Set ENVIRONMENT=production in .env files

Configuration Validation

# Test bootstrap configuration loading
go run -c 'package main; import "fmt"; import "network/pkg/constants"; func main() { fmt.Printf("Bootstrap peers: %v\n", constants.GetBootstrapPeers()) }'

# Verify .env file syntax
grep -E "^[A-Z_]+=.*" .env

Logs and Data

  • Node logs: Console output from each running process
  • Data directories: ./data/bootstrap/, ./data/node/, etc.
  • RQLite data: ./data/<node>/rqlite/
  • Peer info: ./data/<node>/peer.info
  • Bootstrap identity: ./data/bootstrap/identity.key
  • Environment config: ./.env

License

MIT License - see LICENSE file for details.