anonpenguin a6aa516d74 Initial commit

2025-08-03 16:24:04 +03:00

23 KiB

Raw Blame History

Network - Distributed P2P Database System

A distributed peer-to-peer network built with Go and LibP2P, providing decentralized database capabilities with RQLite consensus and replication.

Features

Peer-to-Peer Networking: Built on LibP2P for robust P2P communication
Distributed Database: RQLite-based distributed SQLite with Raft consensus
Automatic Peer Discovery: Bootstrap nodes help new peers join the network
CLI Tool: Command-line interface for network operations and testing
Client Library: Simple Go API for applications to interact with the network
Application Isolation: Namespaced storage and messaging per application

System Requirements

Software Dependencies

Go: Version 1.21 or later
RQLite: Distributed SQLite database
Git: For cloning the repository
Make: For build automation (optional but recommended)

Installation

macOS

# Install Homebrew if you don't have it
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install dependencies
brew install go rqlite git make

# Verify installation
go version    # Should show Go 1.21+
rqlited --version

Ubuntu/Debian

# Install Go (latest version)
sudo rm -rf /usr/local/go
wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

# Install RQLite
wget https://github.com/rqlite/rqlite/releases/download/v8.43.0/rqlite-v8.43.0-linux-amd64.tar.gz
tar -xzf rqlite-v8.43.0-linux-amd64.tar.gz
sudo mv rqlite-v8.43.0-linux-amd64/rqlited /usr/local/bin/

# Install other dependencies
sudo apt update
sudo apt install git make

# Verify installation
go version
rqlited --version

Windows

# Install Go from https://golang.org/dl/
# Install Git from https://git-scm.com/download/win
# Install RQLite from https://github.com/rqlite/rqlite/releases

# Or use Chocolatey
choco install golang git make
# Download RQLite manually from releases page

Hardware Requirements

Minimum:

CPU: 2 cores
RAM: 4GB
Storage: 10GB free space
Network: Stable internet connection

Recommended:

CPU: 4+ cores
RAM: 8GB+
Storage: 50GB+ SSD
Network: Low-latency internet connection

Network Ports

The system uses these ports by default:

4001-4003: LibP2P communication
5001-5003: RQLite HTTP API
7001-7003: RQLite Raft consensus

Ensure these ports are available or configure firewall rules accordingly.

Quick Start

1. Clone and Setup Environment

# Clone the repository
git clone https://git.debros.io/DeBros/network-cluster.git
cd network-cluster

# Copy environment configuration
cp .env.example .env

2. Generate Bootstrap Identity (Development Only)

For development, you need to generate a consistent bootstrap peer identity:

# Generate bootstrap peer identity
go run scripts/generate-bootstrap-identity.go

# This will create data/bootstrap/identity.key and show the peer ID
# Copy the peer ID and update your .env files

Important: After generating the bootstrap identity, update both .env files:

# Update main .env file
nano .env
# Update BOOTSTRAP_PEERS with the generated peer ID

# Update Anchat .env file
nano anchat/.env
# Update BOOTSTRAP_PEERS with the same peer ID

3. Build the Project

# Build all network executables
make build

# Build Anchat application
cd anchat
make build
cd ..

# Or build everything at once
make build && make build-anchat

4. Start the Network

Terminal 1 - Bootstrap Node:

make run-bootstrap
# This starts the bootstrap node on port 4001

Terminal 2 - Regular Node:

make run-node
# This automatically connects to bootstrap peers from .env
# No need to specify bootstrap manually anymore!

Terminal 3 - Another Node (optional):

# For additional nodes, use different ports
go run cmd/node/main.go -data ./data/node2 -port 4003

5. Test with CLI

# Check current bootstrap configuration
make show-bootstrap

# Check network health
./bin/cli health

# Test storage operations
./bin/cli storage put test-key "Hello Network"
./bin/cli storage get test-key

# List connected peers
./bin/cli peers

6. Test Anchat Messaging

# Terminal 1 - First user
cd anchat
./bin/anchat

# Terminal 2 - Second user
cd anchat
./bin/anchat

Deployment

Production Installation Script

For production deployments on Linux servers, we provide an automated installation script that handles all dependencies, configuration, and service setup.

One-Command Installation

# Download and run the installation script
curl -sSL https://raw.githubusercontent.com/DeBrosOfficial/debros-network/main/scripts/install-debros-network.sh | bash

What the Script Does

System Setup:
- Detects OS (Ubuntu/Debian/CentOS/RHEL/Fedora)
- Installs Go 1.21+ with architecture detection
- Installs system dependencies (git, make, build tools)
- Checks port availability (4001-4003, 5001-5003, 7001-7003)
Configuration Wizard:
- Node type selection (bootstrap vs regular node)
- Solana wallet address for node operator rewards
- Installation directory (default: /opt/debros)
- Automatic firewall configuration (UFW)
Secure Installation:
- Creates dedicated debros system user
- Sets up secure directory structure with proper permissions
- Generates LibP2P identity keys with secure storage
- Clones source code and builds binaries
Service Management:
- Creates systemd service with security hardening
- Enables automatic startup and restart on failure
- Configures structured logging to systemd journal

Directory Structure

The script creates a production-ready directory structure:

/opt/debros/
├── bin/                    # Compiled binaries
│   ├── bootstrap          # Bootstrap node executable
│   ├── node              # Regular node executable
│   └── cli               # CLI tools
├── configs/               # Configuration files
│   ├── bootstrap.yaml    # Bootstrap node config
│   └── node.yaml         # Regular node config
├── keys/                  # Identity keys (secure 700 permissions)
│   ├── bootstrap/
│   │   └── identity.key
│   └── node/
│       └── identity.key
├── data/                  # Runtime data
│   ├── bootstrap/
│   │   ├── rqlite/       # RQLite database files
│   │   └── storage/      # P2P storage data
│   └── node/
│       ├── rqlite/
│       └── storage/
├── logs/                  # Application logs
│   ├── bootstrap.log
│   └── node.log
└── src/                   # Source code (for updates)

Node Types

Bootstrap Node:

Network entry point that other nodes connect to
Runs on ports: 4001 (P2P), 5001 (RQLite), 7001 (Raft)
Should be deployed on stable, publicly accessible servers
Acts as initial seed for peer discovery

Regular Node:

Connects to bootstrap peers automatically (hardcoded in code)
Runs on ports: 4002 (P2P), 5002 (RQLite), 7002 (Raft)
Participates in DHT for peer discovery and data replication
Can be deployed on any server or VPS

Service Management

After installation, manage your node with these commands:

# Check service status
sudo systemctl status debros-bootstrap  # or debros-node

# Start/stop/restart service
sudo systemctl start debros-bootstrap
sudo systemctl stop debros-bootstrap
sudo systemctl restart debros-bootstrap

# View real-time logs
sudo journalctl -u debros-bootstrap.service -f

# Enable/disable auto-start
sudo systemctl enable debros-bootstrap
sudo systemctl disable debros-bootstrap

# Use CLI tools
/opt/debros/bin/cli health
/opt/debros/bin/cli peers
/opt/debros/bin/cli storage put key value

Configuration Files

The script generates YAML configuration files:

Bootstrap Node (/opt/debros/configs/bootstrap.yaml):

node:
  data_dir: "/opt/debros/data/bootstrap"
  key_file: "/opt/debros/keys/bootstrap/identity.key"
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4001"
  solana_wallet: "YOUR_WALLET_ADDRESS"

database:
  rqlite_port: 5001
  rqlite_raft_port: 7001

logging:
  level: "info"
  file: "/opt/debros/logs/bootstrap.log"

Regular Node (/opt/debros/configs/node.yaml):

node:
  data_dir: "/opt/debros/data/node"
  key_file: "/opt/debros/keys/node/identity.key"
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4002"
  solana_wallet: "YOUR_WALLET_ADDRESS"

database:
  rqlite_port: 5002
  rqlite_raft_port: 7002

logging:
  level: "info"
  file: "/opt/debros/logs/node.log"

Security Features

The installation script implements production security best practices:

Dedicated User: Runs as debros system user (not root)
File Permissions: Key files have 600 permissions, directories have proper ownership
Systemd Security: Service runs with NoNewPrivileges, PrivateTmp, ProtectSystem=strict
Firewall: Automatic UFW configuration for required ports
Network Isolation: Each node type uses different ports to avoid conflicts

Network Discovery

Bootstrap Peers: Hardcoded in the application for automatic connection
DHT Discovery: Nodes automatically join Kademlia DHT for peer discovery
Peer Exchange: Connected nodes share information about other peers
No Manual Configuration: Regular nodes connect automatically without user intervention

Updates and Maintenance

# Update to latest version (re-run the installation script)
curl -sSL https://raw.githubusercontent.com/DeBrosOfficial/debros-network/main/scripts/install-debros-network.sh | bash

# Manual source update
cd /opt/debros/src
sudo -u debros git pull
sudo -u debros make build
sudo cp bin/* /opt/debros/bin/
sudo systemctl restart debros-bootstrap  # or debros-node

# Backup configuration and keys
sudo cp -r /opt/debros/configs /backup/
sudo cp -r /opt/debros/keys /backup/

Monitoring and Troubleshooting

# Check if ports are open
sudo netstat -tuln | grep -E "(4001|4002|5001|5002|7001|7002)"

# Check service logs
sudo journalctl -u debros-bootstrap.service --since "1 hour ago"

# Check network connectivity
/opt/debros/bin/cli health
/opt/debros/bin/cli peers

# Check disk usage
du -sh /opt/debros/data/*

# Process information
ps aux | grep debros

For more advanced configuration options and development setup, see the sections below.

Environment Configuration

Bootstrap Peers Configuration

The network uses .env files to configure bootstrap peers automatically. This eliminates the need to manually specify bootstrap peer addresses when starting nodes.

Setup for Development

Copy example configuration:

cp .env.example .env
cp anchat/.env.example anchat/.env

Generate bootstrap identity:

go run scripts/generate-bootstrap-identity.go

Update .env files with the generated peer ID:

# Main network .env
BOOTSTRAP_PEERS=/ip4/127.0.0.1/tcp/4001/p2p/YOUR_GENERATED_PEER_ID

# Anchat .env
BOOTSTRAP_PEERS=/ip4/127.0.0.1/tcp/4001/p2p/YOUR_GENERATED_PEER_ID

Configuration Files

Main Network (.env):

# Bootstrap Node Configuration for Development
BOOTSTRAP_PEERS=/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWN3AQHuxAzXfu98tiFYw7W3N2SyDwdxDRANXJp3ktVf8j
BOOTSTRAP_PORT=4001
ENVIRONMENT=development

Anchat Application (anchat/.env):

# Anchat Bootstrap Configuration
BOOTSTRAP_PEERS=/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWN3AQHuxAzXfu98tiFYw7W3N2SyDwdxDRANXJp3ktVf8j
BOOTSTRAP_PORT=4001
ENVIRONMENT=development
ANCHAT_LOG_LEVEL=info
ANCHAT_DATABASE_NAME=anchattestingdb1

Multiple Bootstrap Peers

For production or redundancy, you can specify multiple bootstrap peers:

BOOTSTRAP_PEERS=/ip4/bootstrap1.example.com/tcp/4001/p2p/12D3KooWPeer1,/ip4/bootstrap2.example.com/tcp/4001/p2p/12D3KooWPeer2,/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWLocalPeer

Checking Configuration

# View current bootstrap configuration
make show-bootstrap

# Check which .env file is being used
cat .env
cat anchat/.env

CLI Commands

The CLI and nodes now automatically load bootstrap peers from .env files - no manual configuration needed!

Network Operations

./bin/cli health                    # Check network health
./bin/cli status                    # Get network status
./bin/cli peers                     # List connected peers

Storage Operations

./bin/cli storage put <key> <value> # Store data
./bin/cli storage get <key>         # Retrieve data
./bin/cli storage list [prefix]     # List keys

Database Operations

./bin/cli query "SELECT * FROM table"              # Execute SQL
./bin/cli query "CREATE TABLE users (id INTEGER)"  # DDL operations

Pub/Sub Messaging

./bin/cli pubsub publish <topic> <message>     # Send message
./bin/cli pubsub subscribe <topic> [duration]  # Listen for messages
./bin/cli pubsub topics                        # List active topics

CLI Options

--format json                 # Output in JSON format
--timeout 30s                # Set operation timeout
--bootstrap <multiaddr>      # Override bootstrap peer

Development

Project Structure

network-cluster/
├── cmd/
│   ├── bootstrap/          # Bootstrap node
│   ├── node/              # Regular network node
│   └── cli/               # Command-line interface
├── pkg/
│   ├── client/            # Client library
│   ├── node/              # Node implementation
│   ├── database/          # RQLite integration
│   ├── storage/           # Storage service
│   ├── constants/         # Bootstrap configuration
│   └── config/            # System configuration
├── anchat/                # Anchat messaging application
│   ├── cmd/cli/          # Anchat CLI
│   ├── pkg/
│   │   ├── chat/         # Chat functionality
│   │   ├── crypto/       # Encryption
│   │   └── constants/    # Anchat bootstrap config
│   ├── .env              # Anchat environment config
│   └── .env.example      # Anchat config template
├── scripts/
│   └── generate-bootstrap-identity.go  # Bootstrap ID generator
├── .env                   # Main environment config
├── .env.example          # Main config template
├── bin/                  # Built executables
├── data/                 # Runtime data directories
└── Makefile             # Build and run commands

Building and Testing

# Build all network executables
make build

# Build Anchat application
cd anchat && make build && cd ..
# or
make build-anchat

# Show current bootstrap configuration
make show-bootstrap

# Run bootstrap node (uses .env automatically)
make run-bootstrap

# Run regular node (uses .env automatically - no bootstrap flag needed!)
make run-node

# Clean data directories
make clean

# Run tests
go test ./...

# Full development workflow
make dev

Development Workflow

Initial Setup:

# Copy environment templates
cp .env.example .env
cp anchat/.env.example anchat/.env

# Generate consistent bootstrap identity
go run scripts/generate-bootstrap-identity.go

# Update both .env files with the generated peer ID

Build Everything:

make build        # Build network components
make build-anchat # Build Anchat application

Start Development Cluster:

# Terminal 1: Bootstrap node
make run-bootstrap

# Terminal 2: Regular node (auto-connects via .env)
make run-node

# Terminal 3: Test with CLI
./bin/cli health
./bin/cli peers

# Terminal 4 & 5: Test Anchat
cd anchat && ./bin/anchat

Environment Setup

Install Dependencies:

# macOS
brew install go rqlite git make

# Ubuntu/Debian
sudo apt install golang-go git make
# Install RQLite from https://github.com/rqlite/rqlite/releases

Verify Installation:

go version      # Should be 1.21+
rqlited --version
make --version

Configure Environment:

# Setup .env files
cp .env.example .env
cp anchat/.env.example anchat/.env

# Generate bootstrap identity
go run scripts/generate-bootstrap-identity.go

# Update .env files with generated peer ID

Configuration System

The network uses a dual configuration system:

Environment Variables (.env files): Primary configuration method
Hardcoded Constants: Fallback when .env files are not found

Bootstrap Configuration Priority:

Command line flags (if provided)
Environment variables from .env files
Hardcoded constants in pkg/constants/bootstrap.go
Auto-discovery from running bootstrap nodes

This ensures the network can start even without configuration files, while allowing easy customization for different environments.

Client Library Usage

package main

import (
    "context"
    "log"
    "network/pkg/client"
)

func main() {
    // Create client (bootstrap peer discovered automatically)
    config := client.DefaultClientConfig("my-app")
    networkClient, err := client.NewClient(config)
    if err != nil {
        log.Fatal(err)
    }

    // Connect to network
    if err := networkClient.Connect(); err != nil {
        log.Fatal(err)
    }
    defer networkClient.Disconnect()

    // Use storage
    ctx := context.Background()
    storage := networkClient.Storage()

    err = storage.Put(ctx, "user:123", []byte("user data"))
    if err != nil {
        log.Fatal(err)
    }

    data, err := storage.Get(ctx, "user:123")
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Retrieved: %s", string(data))
}

Anchat - Decentralized Messaging Application

Anchat is a demonstration application built on the network that provides decentralized, encrypted messaging capabilities.

Features

Decentralized Messaging: No central servers, messages flow through the P2P network
Wallet-based Authentication: Connect using Solana wallet addresses
Encrypted Communications: End-to-end encryption for private messages
Room-based Chat: Create and join chat rooms
Network Auto-discovery: Automatically finds and connects to other Anchat users

Quick Start with Anchat

Setup Environment:

# Ensure main network is configured
cp .env.example .env
cp anchat/.env.example anchat/.env

# Generate bootstrap identity and update .env files
go run scripts/generate-bootstrap-identity.go

Build Anchat:
```
cd anchat
make build
```

Start Network Infrastructure:

# Terminal 1: Bootstrap node
make run-bootstrap

# Terminal 2: Regular node (optional but recommended)
make run-node

Start Anchat Clients:

# Terminal 3: First user
cd anchat
./bin/anchat

# Terminal 4: Second user
cd anchat
./bin/anchat

Anchat Commands

# Room management
/list                    # List available rooms
/join <room>            # Join a room
/leave                  # Leave current room
/create <room> [desc]   # Create a new room

# Messaging
<message>               # Send message to current room
/msg <user> <message>   # Send private message
/me <action>           # Send action message

# User management
/users                  # List users in current room
/nick <username>       # Change username
/who                   # Show your user info

# System
/help                  # Show all commands
/debug                 # Show debug information
/quit                  # Exit Anchat

Anchat Configuration

Anchat uses its own bootstrap configuration in anchat/.env:

# Anchat-specific environment variables
BOOTSTRAP_PEERS=/ip4/127.0.0.1/tcp/4001/p2p/YOUR_BOOTSTRAP_PEER_ID
BOOTSTRAP_PORT=4001
ENVIRONMENT=development
ANCHAT_LOG_LEVEL=info
ANCHAT_DATABASE_NAME=anchattestingdb1

The Anchat application also includes hardcoded fallback bootstrap peers in anchat/pkg/constants/bootstrap.go for reliability.

Troubleshooting

Common Issues

Bootstrap peer not found / Peer ID mismatch:

Generate a new bootstrap identity: go run scripts/generate-bootstrap-identity.go
Update both .env and anchat/.env with the new peer ID
Restart the bootstrap node: make run-bootstrap
Check configuration: make show-bootstrap

Nodes can't connect:

Verify .env files have the correct bootstrap peer ID
Check that the bootstrap node is running: ps aux | grep bootstrap
Verify firewall settings and port availability (4001, 5001, 7001)
Try restarting with clean data: make clean && make run-bootstrap

Storage operations fail:

Ensure at least one node is running and connected
Check network health: ./bin/cli health
Verify RQLite is properly installed: rqlited --version
Check for port conflicts: netstat -an | grep -E "(4001|5001|7001)"

Anchat clients can't discover each other:

Ensure both clients use the same bootstrap peer ID in anchat/.env
Verify the bootstrap node is running
Check that both clients successfully connect to bootstrap
Look for "peer id mismatch" errors in the logs

Debug Commands

# Check current configuration
make show-bootstrap
cat .env
cat anchat/.env

# Check running processes
ps aux | grep -E "(bootstrap|node|rqlite)"

# Check port usage
netstat -an | grep -E "(4001|4002|4003|5001|5002|5003|7001|7002|7003)"

# Check bootstrap peer info
cat data/bootstrap/peer.info

# Clean and restart everything
make clean
make run-bootstrap  # In one terminal
make run-node      # In another terminal

Environment-specific Issues

Development Environment:

Always run go run scripts/generate-bootstrap-identity.go first
Update .env files with the generated peer ID
Use make run-node instead of manual bootstrap specification

Production Environment:

Use stable, external bootstrap peer addresses
Configure multiple bootstrap peers for redundancy
Set ENVIRONMENT=production in .env files

Configuration Validation

# Test bootstrap configuration loading
go run -c 'package main; import "fmt"; import "network/pkg/constants"; func main() { fmt.Printf("Bootstrap peers: %v\n", constants.GetBootstrapPeers()) }'

# Verify .env file syntax
grep -E "^[A-Z_]+=.*" .env
grep -E "^[A-Z_]+=.*" anchat/.env

Logs and Data

Node logs: Console output from each running process
Data directories: ./data/bootstrap/, ./data/node/, etc.
RQLite data: ./data/<node>/rqlite/
Peer info: ./data/<node>/peer.info
Bootstrap identity: ./data/bootstrap/identity.key
Environment config: ./.env, ./anchat/.env

License

MIT License - see LICENSE file for details.

23 KiB Raw Blame History

Network - Distributed P2P Database System

Features

System Requirements

Software Dependencies

Installation

macOS

Ubuntu/Debian

Windows

Hardware Requirements

Network Ports

Quick Start

1. Clone and Setup Environment

2. Generate Bootstrap Identity (Development Only)

3. Build the Project

4. Start the Network

5. Test with CLI

6. Test Anchat Messaging

Deployment

Production Installation Script

One-Command Installation

What the Script Does

Directory Structure

Node Types

Service Management

Configuration Files

Security Features

Network Discovery

Updates and Maintenance

Monitoring and Troubleshooting

Environment Configuration

Bootstrap Peers Configuration

Setup for Development

Configuration Files

Multiple Bootstrap Peers

Checking Configuration

CLI Commands

Network Operations

Storage Operations

Database Operations

Pub/Sub Messaging

CLI Options

Development

Project Structure

Building and Testing

Development Workflow

Environment Setup

Configuration System

Bootstrap Configuration Priority:

Client Library Usage

Anchat - Decentralized Messaging Application

Features

Quick Start with Anchat

Anchat Commands

Anchat Configuration

Troubleshooting

Common Issues

Debug Commands

Environment-specific Issues

Configuration Validation

Logs and Data

License

23 KiB

Raw Blame History