DeBros/network
4
0
Fork 0
mirror of https://github.com/DeBrosOfficial/network.git synced 2025-10-06 06:39:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
network/network.md

410 KiB
Raw Blame History

Wiki Documentation for https://git.debros.io/DeBros/network

Generated on: 2025-09-01 16:38:50

Table of Contents

Project Overview

Related Pages

Related topics: Architecture Overview

Relevant source files

The following files were used as context for generating this wiki page:

Project Overview

The DeBros Network Cluster is a decentralized peer-to-peer (P2P) system developed in Go, designed for resilient, distributed data management and communication. It provides core functionalities such as distributed database operations, key-value storage, and pub/sub messaging. The system distinguishes between full network nodes, which participate in core services and discovery, and lightweight clients, which connect via bootstrap peers to consume services. This architecture aims for reliable, scalable, and secure data sharing without relying on centralized infrastructure. Sources: AI_CONTEXT.md:9-13, README.md:14-16

Architecture Overview

The DeBros Network Cluster features a modular and robust architecture that supports both full nodes and lightweight clients. Full nodes are active P2P participants, running core services and handling peer discovery, database, storage, and pub/sub. Clients are designed to be lightweight, connecting only to bootstrap peers to consume services without participating in peer discovery. The system layers include an Application Layer, Client API, and Transport Layer. Sources: AI_CONTEXT.md:18-20, README.md:27-38

Architectural Layers

The architecture can be visualized as a layered system:

graph TD
    A[Application Layer] --> B[Client API]
    B --> C[Transport Layer]

    subgraph Application Layer
        Anchat
        CustomApp[Custom App]
        CLITools[CLI Tools]
    end

    subgraph Client API
        DatabaseClient[Database Client]
        StorageClient[Storage Client]
        PubSubClient[PubSub Client]
    end

    subgraph Transport Layer
        LibP2PHost[LibP2P Host]
        NoiseTLS[Noise/TLS Encryption]
        RQLiteDB[RQLite Database]
    end

Sources: AI_CONTEXT.md:22-38, README.md:27-38

Node vs Client Roles

The DeBros Network defines clear roles for nodes and clients:

  • Node: A full P2P participant that runs core services, manages peer discovery, and provides distributed database, storage, and pub/sub functionalities.
  • Client: A lightweight component that connects to bootstrap peers to consume services. Clients do not participate in peer discovery. Sources: AI_CONTEXT.md:40-43, README.md:39-42

Key Features

The DeBros Network offers several key features for distributed applications:

  • Distributed SQL Database: Utilizes RQLite, providing Raft-consensus, ACID transactions, and automatic failover.
  • Key-Value Storage: Offers namespaced, replicated, and CRUD operations with support for prefix queries.
  • Pub/Sub Messaging: Provides topic-based, real-time, and namespaced messaging with automatic cleanup.
  • Peer Discovery & Management: Includes mechanisms for nodes to discover peers, bootstrap support, and health monitoring.
  • Application Isolation: Achieved through namespace-based multi-tenancy, allowing per-application configuration.
  • Secure by Default: Incorporates Noise/TLS transport encryption, peer identity, and systemd hardening.
  • Simple Client API: A lightweight Go client is available for building applications and CLI tools. Sources: README.md:20-25

System Requirements

To set up and run the DeBros Network, specific software and hardware prerequisites are necessary, along with defined network ports for communication. Sources: README.md:44

Software Requirements

Requirement Version/Description
Go 1.21+ (recommended)
RQLite 8.x (distributed SQLite)
Git For source management
Make For build automation (recommended)
Sources: AI_CONTEXT.md:47-50, README.md:46-49, CONTRIBUTING.md:6-8

Hardware Requirements

Requirement Minimum Recommended
CPU 2 cores 4+ cores
RAM 4GB 8GB+
Disk 10GB 50GB+ SSD
Network Stable internet Low-latency network
Sources: README.md:51-54

Network Ports

Port Purpose
4001 LibP2P P2P communication
5001 RQLite HTTP API
7001 RQLite Raft consensus
Sources: README.md:56-59

Build and Development

The project provides Makefile targets for building, testing, and linting the executables. Sources: CONTRIBUTING.md:20-22, README.md:64

Build Commands

make build        # Build all executables
make test         # Run tests
make run-node     # Start node (auto-detects bootstrap vs regular)
make run-gateway  # Start HTTP gateway (env overrides supported)
make fmt vet      # Format/Vet code (also available as make lint)

Sources: AI_CONTEXT.md:126-129, CONTRIBUTING.md:20-21, cmd/node/main.go:107-109

Project Structure

The project repository is organized into several key directories:

network/
├── cmd/             # Command-line executables (node, cli, gateway)
├── pkg/             # Go packages (client, node, database, storage, pubsub, config, discovery, auth, anyoneproxy, logging)
├── scripts/         # Install, test scripts
├── configs/         # YAML configurations
├── bin/             # Built executables
├── examples/        # Example usage code
├── openapi/         # OpenAPI specification for the gateway
├── CONTRIBUTING.md  # Contribution guidelines
├── README.md        # Project overview
├── AI_CONTEXT.md    # AI context documentation
├── CODE_OF_CONDUCT.md # Code of Conduct

Sources: README.md:154-162, AI_CONTEXT.md:10-11, CONTRIBUTING.md:1-2, cmd/node/main.go:1-12, cmd/cli/main.go:1-12, examples/basic_usage.go:1-12, openapi/gateway.yaml:1-12

Client API

The DeBros Network provides a Go client library (pkg/client) for interacting with the distributed system. This client allows applications to perform database operations, key-value storage, pub/sub messaging, and retrieve network information. Sources: AI_CONTEXT.md:132, examples/basic_usage.go:10-12

Client Creation

A client is initialized with a configuration, including bootstrap peers.

import "git.debros.io/DeBros/network/pkg/client"

config := client.DefaultClientConfig("my-app")
config.BootstrapPeers = []string{"/ip4/127.0.0.1/tcp/4001/p2p/{PEER_ID}"}
client, err := client.NewClient(config)
err = client.Connect()
defer client.Disconnect()

Sources: AI_CONTEXT.md:133-139, examples/basic_usage.go:14-25

Database Operations

The client's Database() interface allows for SQL operations.

result, err := client.Database().Query(ctx, "SELECT * FROM users")
err := client.Database().CreateTable(ctx, "CREATE TABLE ...")

Sources: AI_CONTEXT.md:141-142, examples/basic_usage.go:34-58

Storage Operations

The Storage() interface supports key-value operations.

err := client.Storage().Put(ctx, "key", []byte("value"))
data, err := client.Storage().Get(ctx, "key")

Sources: AI_CONTEXT.md:144-145, examples/basic_usage.go:60-88

PubSub Operations

The PubSub() interface enables topic-based messaging.

err := client.PubSub().Subscribe(ctx, "topic", handler)
err := client.PubSub().Publish(ctx, "topic", []byte("msg"))

Sources: AI_CONTEXT.md:147-148, examples/basic_usage.go:90-117

Network Information

The Network() interface provides access to status and peer information.

status, err := client.Network().GetStatus(ctx)
peers, err := client.Network().GetPeers(ctx)

Sources: AI_CONTEXT.md:150-151, examples/basic_usage.go:119-127

HTTP Gateway

The DeBros Network includes an HTTP/WebSocket gateway that exposes a REST API and WebSocket interface over the P2P network. This gateway features an enhanced authentication system with multi-wallet support. Sources: README.md:144-146

Gateway Endpoints

The gateway provides various endpoints for interacting with the network's services:

Service Method Path Description Request Body (Example) Response Body (Example)
Auth POST /v1/auth/challenge Get authentication challenge { "nonce": "..." }
POST /v1/auth/verify Verify wallet signature { "wallet": "0x...", "nonce": "...", "signature": "0x..." } { "token": "...", "expires_at": "..." }
POST /v1/auth/refresh Refresh JWT token { "token": "...", "expires_at": "..." }
POST /v1/auth/api-key Generate API key { "key": "...", "namespace": "..." }
Storage POST /v1/storage/put Store data by key { "key": "user:123", "value": "base64encodedvalue" } { "status": "ok" }
POST /v1/storage/get Retrieve data by key { "key": "user:123" } { "value": "base64encodedvalue" }
GET /v1/storage/exists Check if key exists { "exists": true }
GET /v1/storage/list List keys with optional prefix { "keys": ["key1", "key2"] }
Database POST /v1/db/create-table Create table with SQL DDL { "schema": "CREATE TABLE users (id INTEGER PRIMARY KEY)" } { "status": "ok" }
POST /v1/db/query Execute single SQL query { "sql": "SELECT * FROM users", "args": [] } { "columns": ["id", "name"], "rows": [[1, "Alice"]], "count": 1 }
POST /v1/db/transaction Apply multiple statements atomically { "statements": ["INSERT ...", "UPDATE ..."] } { "status": "ok" }
PubSub GET /v1/pubsub/ws?topic=<topic> WebSocket connection for real-time messaging (Binary frames)
POST /v1/pubsub/publish Publish message to topic { "topic": "chat", "data_base64": "base64encodedmessage" } { "status": "ok" }
Sources: AI_CONTEXT.md:200-264, README.md:120-137, openapi/gateway.yaml:1-100

Authentication System

The gateway's authentication system supports wallet-based authentication (Ethereum EIP-191) and API keys. It handles automatic authentication, multi-wallet management, and persistent sessions. Sources: README.md:147-152, AI_CONTEXT.md:270-272

sequenceDiagram
    participant C as Client
    participant GW as Gateway
    participant N as Network Node
    participant DB as RQLite Database
    
    C->>GW: POST /v1/auth/challenge
    activate GW
    GW-->>C: Nonce (challenge)
    deactivate GW

    Note over C,GW: Client signs nonce with wallet
    C->>GW: POST /v1/auth/verify (wallet, nonce, signature)
    activate GW
    GW->>N: VerifySignature(wallet, nonce, signature)
    activate N
    N-->>GW: Signature Valid
    deactivate N
    GW->>DB: StoreAuthSession(wallet, token)
    activate DB
    DB-->>GW: Session Stored
    deactivate DB
    GW-->>C: JWT Token
    deactivate GW

    C->>GW: POST /v1/storage/put (Auth: Bearer Token)
    activate GW
    GW->>GW: Validate JWT Token
    GW->>N: PutStorage(key, value)
    activate N
    N-->>GW: Data Stored
    deactivate N
    GW-->>C: {status:"ok"}
    deactivate GW

Sources: AI_CONTEXT.md:273-286, README.md:104-118

Troubleshooting

The project documentation provides common issues and their solutions, along with debugging and health check commands. Sources: AI_CONTEXT.md:154, README.md:164

Common Issues and Solutions

Symptom Possible Cause Solution
Bootstrap Connection Failed Node not running, firewall, invalid peer ID Check node status, firewall settings, peer ID validity.
Database Operations Timeout RQLite issues, leader election, join config Ensure RQLite ports open, leader election complete, correct join config.
Message Delivery Failures Incorrect topic, no active subscriptions Verify topic names, active subscriptions, network connectivity.
High Memory Usage Unsubscribed topics, connection pool size Unsubscribe when done, monitor connection pool, review message retention.
Authentication Failed Invalid signature, expired token, wallet issues Check signature format, nonce, wallet address case, refresh token.
Gateway Connection Refused Gateway not running, CORS, authentication headers Verify gateway status, CORS config, proper authentication headers.
Database Migration Issues SQL syntax, version conflicts, transaction rollbacks Check SQL syntax, migration file naming, review migration logs.
Sources: AI_CONTEXT.md:155-177, README.md:166-187

Debugging & Health Checks

CLI and curl commands are available for diagnostics.

export LOG_LEVEL=debug
./bin/network-cli health
./bin/network-cli peers
./bin/network-cli query "SELECT 1"
./bin/network-cli pubsub publish test "hello"
./bin/network-cli pubsub subscribe test 10s

# Test authentication flow
./bin/network-cli storage put test-key test-value

# Gateway health checks
curl http://localhost:8080/health
curl http://localhost:8080/v1/status

Sources: AI_CONTEXT.md:179-191, README.md:189-200, cmd/cli/main.go:107-124

Conclusion

The DeBros Network Cluster provides a comprehensive and decentralized solution for distributed data management and communication. Its modular architecture, clear node/client roles, and robust feature set for SQL database, key-value storage, and pub/sub messaging make it suitable for resilient applications. The project emphasizes security, ease of use through its client API and HTTP gateway, and provides detailed documentation for setup, development, and troubleshooting. Sources: AI_CONTEXT.md:9-13, README.md:14-16, AI_CONTEXT.md:291-294

System Requirements

Related Pages

Related topics: Quick Start Guide

Relevant source files

System Requirements

This document outlines the essential software and hardware prerequisites, along with network configurations, necessary for the successful deployment and operation of the DeBros Network Cluster. Adhering to these requirements ensures optimal performance, stability, and security for both full network nodes and lightweight clients. The DeBros Network is designed as a robust, decentralized peer-to-peer (P2P) system providing distributed database operations, key-value storage, and pub/sub messaging. Sources: AI_CONTEXT.md:19-22

Software Requirements

The DeBros Network requires specific software versions to build and run correctly. These include programming language runtimes, database systems, and build automation tools.

Go Language

The project is built in Go and requires a specific version for compilation and execution. Sources: CONTRIBUTING.md:7, README.md:31, AI_CONTEXT.md:197

Requirement Minimum Version Recommended Version
Go 1.21+ 1.22+ (1.23 recommended)
Sources: CONTRIBUTING.md:7, README.md:31

RQLite Database

RQLite is used as the distributed SQLite database within the network. Sources: README.md:32, AI_CONTEXT.md:198

Requirement Version Notes
RQLite 8.x Distributed SQLite
Sources: README.md:32

Other Tools

Additional tools are recommended for source management and build automation. Sources: README.md:33-34

Requirement Purpose Notes
Git Source management Required for cloning the repository
Make Build automation Recommended
Sources: README.md:33-34

Hardware Requirements

Hardware specifications are provided for both minimum operational capacity and recommended performance.

Minimum Hardware

Sources: README.md:38

Component Specification
CPU Cores 2
RAM 4GB
Disk 10GB
Network Stable internet
Sources: README.md:38

Recommended Hardware

Sources: README.md:39

Component Specification
CPU Cores 4+
RAM 8GB+
Disk 50GB+ SSD
Network Low-latency
Sources: README.md:39

Network Port Requirements

The DeBros Network utilizes specific ports for peer-to-peer communication, database API access, and consensus mechanisms. These ports must be open and accessible for proper network function. Sources: README.md:42-44

Port Protocol Purpose Default Value
4001 TCP LibP2P P2P communication 4001
5001 TCP RQLite HTTP API 5001
7001 TCP RQLite Raft consensus 7001
Sources: README.md:42-44, cmd/node/main.go:60-61

A visual representation of the network ports and their functions:

graph TD
    A[DeBros Network Node] --> B(LibP2P P2P Communication: Port 4001);
    A --> C(RQLite HTTP API: Port 5001);
    A --> D(RQLite Raft Consensus: Port 7001);

Sources: README.md:42-44

Configuration Considerations

The system can be configured using YAML files, command-line flags, and environment variables. These configurations can influence the network's behavior, including port assignments and data directories. Sources: README.md:144-147, cmd/node/main.go:56-66

Node Configuration

Example configuration for a node, showing listen addresses, data directories, and RQLite ports. Sources: README.md:120-141

node:
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4001" # Configurable P2P port
  data_dir: "./data/bootstrap"
database:
  rqlite_port: 5001 # Configurable RQLite HTTP API port
  rqlite_raft_port: 7001 # Configurable RQLite Raft port

Sources: README.md:120-141

Configuration Precedence

Settings are applied based on a specific precedence order:

  1. Command-line Flags
  2. Environment Variables
  3. YAML Configuration Files
  4. Default Values Sources: README.md:144-147
graph TD
    A[Command-line Flags] --> B(Highest Precedence);
    B --> C(Environment Variables);
    C --> D(YAML Configuration);
    D --> E(Default Values);
    E --> F(Lowest Precedence);

Sources: README.md:144-147

Installation and Deployment

The project provides an automated installation script for production environments, which handles dependency installation, directory setup, and service configuration. Sources: README.md:67-97

Automated Production Install

The install-debros-network.sh script automates the setup process. Sources: README.md:67-68

curl -sSL https://git.debros.io/DeBros/network/raw/branch/main/scripts/install-debros-network.sh | sudo bash

Sources: README.md:67-68

This script performs several actions:

  • Detects OS, installs Go, RQLite, and other dependencies.
  • Creates a debros system user and a secure directory structure.
  • Generates LibP2P identity keys.
  • Clones the source code and builds binaries.
  • Sets up a systemd service (debros-node).
  • Configures firewall (UFW) for required ports.
  • Generates YAML config in /opt/debros/configs/node.yaml. Sources: README.md:70-79

The typical directory structure after installation is: Sources: README.md:81-88

/opt/debros/
├── bin/           # Binaries
├── configs/       # YAML configs
├── keys/          # Identity keys
├── data/          # RQLite DB, storage
├── logs/          # Node logs
├── src/           # Source code

Sources: README.md:81-88

Conclusion

Meeting the specified system requirements for software, hardware, and network configuration is crucial for deploying and operating the DeBros Network Cluster effectively. These requirements ensure the underlying infrastructure can support the distributed database, key-value storage, and pub/sub messaging features, facilitating a stable and performant decentralized environment.

Quick Start Guide

Related Pages

Related topics: CLI Overview, Deployment & Installation

Timeout while calling OpenRouter API (120s). The service may be slow or unreachable.

Deployment & Installation

Related Pages

Related topics: Quick Start Guide

Relevant source files

Deployment & Installation

The DeBros Network is a decentralized peer-to-peer (P2P) system designed for resilient, distributed data management and communication. This document outlines the procedures and considerations for deploying and installing the DeBros Network components, including full network nodes and the HTTP Gateway. The installation process covers prerequisites, automated and manual setup, and directory structures. Sources: README.md:1-2, AI_CONTEXT.md:1-2

System Requirements

Successful deployment of the DeBros Network requires specific software and hardware configurations, along with open network ports for communication. Sources: README.md:65-65, AI_CONTEXT.md:29-29

Software Requirements

The following software components are necessary for building and running the DeBros Network:

Hardware Requirements

Hardware recommendations vary based on the intended role and load:

  • Minimum: 2 CPU cores, 4GB RAM, 10GB disk, stable internet connection. Sources: README.md:74
  • Recommended: 4+ CPU cores, 8GB+ RAM, 50GB+ SSD, low-latency network. Sources: README.md:75

Network Ports

The DeBros Network components utilize specific ports for inter-node communication and API access:

These ports can be overridden via command-line flags or configuration files. Sources: cmd/node/main.go:133-135

Automated Production Installation

For a secure and production-ready setup, an automated installation script is provided. This script handles all necessary steps from OS detection to service setup. Sources: README.md:105-105

curl -sSL https://git.debros.io/DeBros/network/raw/branch/main/scripts/install-debros-network.sh | sudo bash

Sources: README.md:107-107

The installation script performs the following actions:

  • Detects the operating system and installs Go, RQLite, and other dependencies. Sources: README.md:110
  • Creates a debros system user and establishes a secure directory structure under /opt/debros/. Sources: README.md:111
  • Generates LibP2P identity keys for the node. Sources: README.md:112
  • Clones the source code and builds the necessary binaries. Sources: README.md:113
  • Sets up a systemd service named debros-node for automatic startup and management. Sources: README.md:114
  • Configures the Uncomplicated Firewall (UFW) to open required ports. Sources: README.md:115
  • Generates a YAML configuration file at /opt/debros/configs/node.yaml. Sources: README.md:116

Installed Directory Structure

The automated installation creates a standardized directory structure for the DeBros Network components:

/opt/debros/
├── bin/           # Binaries
├── configs/       # YAML configs
├── keys/          # Identity keys
├── data/          # RQLite DB, storage
├── logs/          # Node logs
└── src/           # Source code

Sources: README.md:119-125

Service Management

The debros-node systemd service can be managed using standard systemctl commands:

  • sudo systemctl status debros-node: Check the service status. Sources: README.md:128
  • sudo systemctl start debros-node: Start the service. Sources: README.md:129
  • sudo systemctl stop debros-node: Stop the service. Sources: README.md:130
  • sudo systemctl restart debros-node: Restart the service. Sources: README.md:131
  • sudo journalctl -u debros-node.service -f: View real-time logs for the node service. Sources: README.md:132

Manual Quick Start

For development or manual deployment, the DeBros Network can be set up by cloning the repository, building executables, and starting nodes. Sources: README.md:83-83

1. Clone and Setup

git clone https://git.debros.io/DeBros/network.git
cd network

Sources: README.md:86-87

2. Build All Executables

The make build command compiles all necessary binaries. Sources: README.md:90

make build

Sources: README.md:90

3. Start a Bootstrap Node

A bootstrap node is essential for peer discovery and network initialization. It can be started using make run-node or manually. Sources: README.md:93-94

make run-node
# Or manually:
go run ./cmd/node --config configs/bootstrap.yaml

Sources: README.md:94-96

The cmd/node/main.go file is responsible for parsing command-line flags, loading configurations, and starting the node. It saves the peer ID and multiaddress to a peer.info file in the node's data directory. Sources: cmd/node/main.go:167-174

// cmd/node/main.go
func startNode(ctx context.Context, cfg *config.Config, port int) error {
	// ... (node creation and start) ...

	// Save the peer ID to a file for CLI access (especially useful for bootstrap)
	peerID := n.GetPeerID()
	peerInfoFile := filepath.Join(cfg.Node.DataDir, "peer.info")
	peerMultiaddr := fmt.Sprintf("/ip4/0.0.0.0/tcp/%d/p2p/%s", port, peerID)

	if err := os.WriteFile(peerInfoFile, []byte(peerMultiaddr), 0644); err != nil {
		logger.Error("Failed to save peer info: %v", zap.Error(err))
	} else {
		logger.Info("Peer info saved to: %s", zap.String("path", peerInfoFile))
		logger.Info("Bootstrap multiaddr: %s", zap.String("path", peerMultiaddr))
	}
	// ...
	return n.Stop()
}

Sources: cmd/node/main.go:167-183

4. Start Additional Nodes

Additional nodes can be started, typically configured to join an existing bootstrap node. Sources: README.md:99

make run-node2
# Or manually:
go run ./cmd/node --config configs/node.yaml

Sources: README.md:99-101

The configs/node.yaml specifies the rqlite_join_address and bootstrap_peers to connect to the network. Sources: README.md:167-168, README.md:172-174

Node Startup Flow

The cmd/node/main.go entry point defines the node's startup process:

graph TD
    A[Start Node Application] --> B{Parse Flags & Config Path?};
    B -- Yes --> C[Load Config from YAML];
    B -- No --> D[Use Default Config];
    C --> E[Apply Flag Overrides to Config];
    D --> E;
    E --> F[Initialize Logger];
    F --> G[Create Node Instance];
    G --> H[Start Node Services];
    H --> I[Save Peer Info to File];
    I --> J[Wait for Context Cancellation];
    J --> K[Stop Node Services];
    K --> L[Exit];

Sources: cmd/node/main.go:42-230

Configuration

The DeBros Network components are highly configurable through YAML files, command-line flags, and environment variables. Sources: README.md:143-143

Configuration Precedence

Configuration values are applied in a specific order of precedence:

  1. Flags: Command-line flags override all other settings.
  2. Environment Variables: Environment variables override YAML configurations.
  3. YAML: Configuration loaded from YAML files.
  4. Defaults: Built-in default values. Sources: README.md:183-186

This precedence allows for flexible and dynamic configuration management. The load_args_into_config function in cmd/node/main.go demonstrates how command-line arguments override configuration values. Sources: cmd/node/main.go:186-218

// cmd/node/main.go
func load_args_into_config(cfg *config.Config, p2pPort, rqlHTTP, rqlRaft *int, rqlJoinAddr *string, advAddr *string, dataDir *string) {
	logger := setup_logger(logging.ComponentNode)

	if *rqlHTTP != 5001 {
		cfg.Database.RQLitePort = *rqlHTTP
		logger.ComponentInfo(logging.ComponentNode, "Overriding RQLite HTTP port", zap.Int("port", *rqlHTTP))
	}
	// ... similar overrides for RQLite Raft port, P2P port, RQLite join address, advertise address, and data directory
}

Sources: cmd/node/main.go:186-218

Example Configuration Files

The project provides example YAML configuration files for different node roles.

configs/bootstrap.yaml

This configuration is for a bootstrap node, which typically does not join another RQLite cluster and acts as an initial peer for others. Sources: README.md:149-164

node:
  id: ""
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4001"
  data_dir: "./data/bootstrap"
  max_connections: 100
  disable_anonrc: true

database:
  data_dir: "./data/db"
  replication_factor: 3
  shard_count: 16
  max_database_size: 1073741824
  backup_interval: 24h
  rqlite_port: 5001
  rqlite_raft_port: 7001
  rqlite_join_address: "" # Bootstrap node does not join

discovery:
  bootstrap_peers: []
  discovery_interval: 15s
  bootstrap_port: 4001
  http_adv_address: "127.0.0.1"
  raft_adv_address: ""

security:
  enable_tls: false
  private_key_file: ""
  certificate_file: ""

logging:
  level: "info"
  format: "console"
  output_file: ""

Sources: README.md:150-181

configs/node.yaml

This configuration is for a regular network node, designed to join an existing RQLite cluster and connect to bootstrap peers. Sources: README.md:166-181

node:
  id: "node2"
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4002"
  data_dir: "./data/node2"
  max_connections: 50
  disable_anonrc: true

database:
  data_dir: "./data/db"
  replication_factor: 3
  shard_count: 16
  max_database_size: 1073741824
  backup_interval: 24h
  rqlite_port: 5002
  rqlite_raft_port: 7002
  rqlite_join_address: "http://127.0.0.1:5001"

discovery:
  bootstrap_peers:
    - "/ip4/127.0.0.1/tcp/4001/p2p/<YOUR_BOOTSTRAP_PEER_ID>"
  discovery_interval: 15s
  bootstrap_port: 4002
  http_adv_address: "127.0.0.1"
  raft_adv_address: ""

security:
  enable_tls: false
  private_key_file: ""
  certificate_file: ""

logging:
  level: "info"
  format: "console"
  output_file: ""

Sources: README.md:167-198

HTTP Gateway Deployment

The DeBros Network includes an HTTP/WebSocket gateway that provides a modern REST API and WebSocket interface over the P2P network. This gateway features an enhanced authentication system with multi-wallet support. Sources: README.md:465-465

Quick Start Gateway

The gateway can be started using make run-gateway or manually. Sources: README.md:468-469

make run-gateway
# Or manually:
go run ./cmd/gateway

Sources: README.md:468-470

Gateway Configuration

The gateway's behavior can be configured via environment variables:

Environment Variable Description Example Value
GATEWAY_ADDR Address and port for the gateway to listen on. 0.0.0.0:8080
GATEWAY_NAMESPACE Namespace for the gateway operations. my-app
GATEWAY_BOOTSTRAP_PEERS Comma-separated list of bootstrap peers for the gateway to connect to. /ip4/127.0.0.1/tcp/4001/p2p/YOUR_PEER_ID
GATEWAY_REQUIRE_AUTH Boolean to enable/disable authentication for gateway endpoints. true
GATEWAY_API_KEYS Comma-separated list of API keys and their associated namespaces. key1:namespace1,key2:namespace2
Sources: README.md:473-477

Client SDK Usage

The pkg/client module provides a Go client library for interacting with the DeBros Network. This client can be used to perform database, storage, and pub/sub operations. Sources: AI_CONTEXT.md:120-120, examples/basic_usage.go:11-11

Client Creation

A client is initialized with a configuration, which can include bootstrap peers and an API key. Sources: AI_CONTEXT.md:121-125

import "git.debros.io/DeBros/network/pkg/client"

config := client.DefaultClientConfig("my-app")
config.BootstrapPeers = []string{"/ip4/127.0.0.1/tcp/4001/p2p/{PEER_ID}"}
client, err := client.NewClient(config)
err = client.Connect()
defer client.Disconnect()

Sources: AI_CONTEXT.md:121-126

The e2e/client_e2e_test.go also demonstrates client configuration, including setting an API key and bootstrap peers from environment variables. Sources: e2e/client_e2e_test.go:30-47

Client Connection Flow

The client connection process involves creating a client instance and then connecting to the network:

sequenceDiagram
    participant App as Application
    participant Client as NetworkClient
    participant Node as DeBros Node
    participant Gateway as HTTP Gateway

    App->>Client: NewClient(config)
    activate Client
    Client-->>App: Client Instance
    deactivate Client

    App->>Client: Connect()
    activate Client
    Client->>Node: Establish P2P Connection (LibP2P)
    activate Node
    Node-->>Client: Connection Acknowledged
    deactivate Node
    Client->>Gateway: Establish HTTP/WS Connection (if configured)
    activate Gateway
    Gateway-->>Client: Connection Acknowledged
    deactivate Gateway
    Client-->>App: Connection Success/Error
    deactivate Client

Sources: examples/basic_usage.go:18-21, e2e/client_e2e_test.go:50-51

Conclusion

The DeBros Network offers flexible deployment options, ranging from a single-command automated production setup to manual configuration for development and testing. Key components like the node and HTTP gateway are configurable via YAML, flags, and environment variables, providing fine-grained control over their operation. The provided client SDK facilitates application integration, ensuring robust interaction with the distributed network services.

Architecture Overview

Related Pages

Related topics: Project Structure, Node vs. Client Roles

Relevant source files

Architecture Overview

The DeBros Network Cluster is a decentralized peer-to-peer (P2P) system developed in Go, designed for distributed database operations, key-value storage, pub/sub messaging, and peer management. It aims to provide resilient, distributed data management and communication by distinguishing between full network nodes and lightweight clients. The architecture emphasizes modularity and robustness, supporting both full nodes that run core services and participate in discovery, and lightweight clients that connect via bootstrap peers. Sources: AI_CONTEXT.md:8-11, README.md:1-4

Core Components

The DeBros Network Cluster comprises several key components that work together to provide its distributed functionalities. These include the Node, Client API, LibP2P Host, Noise/TLS Encryption, and RQLite Database. Sources: README.md:36-49

Component Diagram

The following diagram illustrates the high-level architecture and interaction between the main components of the DeBros Network Cluster. Sources: README.md:36-49, AI_CONTEXT.md:16-30

graph TD
    A[Application Layer] --> B{Client API}
    B --> C[Database Client]
    B --> D[Storage Client]
    B --> E[PubSub Client]
    C --> F[LibP2P Host]
    D --> F
    E --> F
    F --> G[Noise/TLS Encryption]
    G --> H[RQLite Database]

Sources: README.md:36-49, AI_CONTEXT.md:16-30

Node vs. Client Roles

The system differentiates between two primary roles:

  • Node: A full P2P participant that runs core services, handles peer discovery, and provides distributed database, storage, and pub/sub functionalities. Nodes are active members of the network.
  • Client: A lightweight component that connects to the network via bootstrap peers. Clients consume services provided by nodes but do not participate in peer discovery or run core network services themselves. Sources: AI_CONTEXT.md:46-48, README.md:51-54

Key Services

The DeBros Network provides several core services:

  • Distributed SQL Database: Utilizes RQLite for a distributed SQLite database with Raft consensus, ACID transactions, and automatic failover.
  • Key-Value Storage: Offers namespaced, replicated key-value storage with CRUD operations and prefix queries.
  • Pub/Sub Messaging: Provides topic-based, real-time, namespaced messaging with automatic cleanup.
  • Peer Discovery & Management: Enables nodes to discover peers, supports bootstrapping, and includes health monitoring.
  • Application Isolation: Achieves multi-tenancy through namespace-based isolation and per-application configuration.
  • Security: Defaults to Noise/TLS transport encryption, peer identity, and systemd hardening.
  • Client API: Offers a lightweight Go client for application integration and CLI tools. Sources: README.md:19-30, AI_CONTEXT.md:8-11

Network Protocol and Data Flow

The network relies on LibP2P for its underlying P2P communication, enhanced with Noise/TLS encryption for secure transport. Data flows between clients and nodes, and among nodes, through a structured set of APIs.

Client-Node Interaction Flow

Clients interact with the network services (Database, Storage, PubSub) through a unified client API. This API abstracts the underlying P2P communication. Sources: examples/basic_usage.go:19-24, AI_CONTEXT.md:16-30

sequenceDiagram
    participant App as Application
    participant C as Client API
    participant N as Network Node
    participant DB as RQLite Database
    participant S as Storage Service
    participant P as PubSub Service

    App->>C: Initialize Client(config)
    activate C
    C->>N: Connect()
    activate N
    N-->>C: Connection Established
    deactivate N
    App->>C: Call Database Operation (e.g., Query)
    C->>N: Request (e.g., SQL Query)
    activate N
    N->>DB: Forward Query
    activate DB
    DB-->>N: Query Result
    deactivate DB
    N-->>C: Response (e.g., Query Result)
    deactivate N
    App->>C: Call Storage Operation (e.g., Put)
    C->>N: Request (e.g., Put Key/Value)
    activate N
    N->>S: Store Data
    activate S
    S-->>N: Storage Confirmation
    deactivate S
    N-->>C: Response (e.g., Status OK)
    deactivate N
    App->>C: Call PubSub Operation (e.g., Subscribe)
    C->>N: Subscribe to Topic
    activate N
    N->>P: Register Subscriber
    activate P
    P-->>N: Subscription Confirmation
    deactivate P
    N-->>C: Subscription Acknowledged
    deactivate N
    N->>P: Publish Message (from another node/client)
    activate P
    P-->>N: Message to Subscribers
    deactivate P
    N->>C: Deliver Message
    C->>App: Message Callback

Sources: examples/basic_usage.go:19-24, AI_CONTEXT.md:16-30, examples/sdk-typescript/src/client.ts:90-94

Network Ports

The DeBros Network utilizes specific ports for its operations:

Port Purpose
4001 LibP2P P2P communication
5001 RQLite HTTP API
7001 RQLite Raft consensus
Sources: README.md:63-66

Configuration System

The system uses a flexible configuration approach, allowing settings to be defined via YAML files, command-line flags, and environment variables. Sources: AI_CONTEXT.md:58-60, cmd/node/main.go:41-43

Configuration Precedence

Settings are applied in a specific order of precedence:

  1. Command-line Flags (highest)
  2. Environment Variables
  3. YAML Configuration File
  4. Default Values (lowest) Sources: AI_CONTEXT.md:126

Example Configuration

Node configurations typically define network addresses, data directories, RQLite settings, and discovery parameters. Sources: AI_CONTEXT.md:74-124

node:
  id: "node2"
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4002"
  data_dir: "./data/node2"
  max_connections: 50
  disable_anonrc: true

database:
  data_dir: "./data/db"
  replication_factor: 3
  shard_count: 16
  max_database_size: 1073741824
  backup_interval: 24h
  rqlite_port: 5002
  rqlite_raft_port: 7002
  rqlite_join_address: "http://127.0.0.1:5001"

discovery:
  bootstrap_peers:
    - "/ip4/127.0.0.1/tcp/4001/p2p/<YOUR_BOOTSTRAP_PEER_ID>"
  discovery_interval: 15s
  bootstrap_port: 4002
  http_adv_address: "127.0.0.1"
  raft_adv_address: ""

security:
  enable_tls: false
  private_key_file: ""
  certificate_file: ""

logging:
  level: "info"
  format: "console"
  output_file: ""

Sources: AI_CONTEXT.md:99-124

Gateway Service

The HTTP Gateway acts as a centralized access point for clients to interact with the DeBros Network services (Storage, Database, PubSub) over standard HTTP/WebSocket protocols. It enforces namespace ownership and provides an enhanced authentication system. Sources: AI_CONTEXT.md:247-249, AI_CONTEXT.md:251-255

Gateway Endpoints

The gateway exposes various REST and WebSocket endpoints:

Health/Version

Authentication (Enhanced Multi-Wallet System)

  • POST /v1/auth/challenge
  • POST /v1/auth/verify
  • POST /v1/auth/register
  • POST /v1/auth/refresh
  • POST /v1/auth/logout
  • GET /v1/auth/whoami
  • POST /v1/auth/api-key Sources: AI_CONTEXT.md:278-284

Storage Operations

  • POST /v1/storage/get
  • POST /v1/storage/put
  • POST /v1/storage/delete
  • GET /v1/storage/list?prefix=...
  • GET /v1/storage/exists?key=... Sources: AI_CONTEXT.md:286-290

Database Operations

  • POST /v1/db/create-table
  • POST /v1/db/drop-table
  • POST /v1/db/query
  • POST /v1/db/transaction
  • GET /v1/db/schema Sources: AI_CONTEXT.md:309-313

Gateway Interaction Flow

sequenceDiagram
    participant C as Client (Web/App)
    participant GW as HTTP Gateway
    participant N as Network Node
    participant S as Storage Service
    participant DB as Database Service
    participant P as PubSub Service

    C->>GW: HTTP Request (e.g., POST /v1/storage/put)
    activate GW
    GW->>GW: Authenticate & Authorize
    alt Authenticated & Authorized
        GW->>N: Forward Request (internal P2P)
        activate N
        N->>S: Process Storage Request
        activate S
        S-->>N: Storage Result
        deactivate S
        N-->>GW: Response
        deactivate N
        GW-->>C: HTTP Response (e.g., 200 OK)
    else Not Authenticated / Authorized
        GW-->>C: HTTP 401/403 Error
    end
    deactivate GW

    C->>GW: WebSocket Connect (GET /v1/pubsub/ws)
    activate GW
    GW->>GW: Authenticate & Authorize WS
    GW->>N: Establish PubSub Stream
    activate N
    N->>P: Register WS Subscriber
    activate P
    P-->>N: Confirmation
    deactivate P
    N-->>GW: WS Stream Ready
    deactivate N
    GW-->>C: WebSocket Connection Established
    deactivate GW

    N->>P: New Message Published (from another node/client)
    activate P
    P-->>N: Deliver Message
    deactivate P
    N->>GW: Forward Message to WS
    activate GW
    GW-->>C: WebSocket Message (binary frame)
    deactivate GW

Sources: AI_CONTEXT.md:247-255, AI_CONTEXT.md:309-313, AI_CONTEXT.md:292-294

Database Migration System

The DeBros Network includes a robust database migration system designed to manage schema changes reliably. It ensures that database updates are applied consistently and safely across the cluster. Sources: AI_CONTEXT.md:144-147

Key Features

  • Atomicity: Each migration runs within a transaction.
  • Version Control: Tracks migration versions to prevent duplicate execution.
  • Error Recovery: Provides comprehensive error handling and logging.
  • SQL File Support: Processes standard .sql migration files. Sources: AI_CONTEXT.md:148-154

Migration File Structure

Migrations are organized sequentially in a migrations/ directory. Sources: AI_CONTEXT.md:156-161

migrations/
├── 001_initial_schema.sql
├── 002_add_auth_tables.sql
├── 003_add_indexes.sql
└── ...

Sources: AI_CONTEXT.md:156-161

Migration Process

Migrations are automatically processed during gateway startup. For programmatic migrations, SQL DDL statements can be sent via the /v1/db/transaction endpoint. Sources: AI_CONTEXT.md:163-164, AI_CONTEXT.md:316

// Example: Database migration via HTTP transaction (from e2e test)
// This demonstrates changing a column type using the recreate pattern
// Sources: [e2e/gateway_e2e_test.go:198-208]()
txBody := fmt.Sprintf(`{"statements":["CREATE TABLE %s_new (id INTEGER PRIMARY KEY, name TEXT, age INTEGER)", "INSERT INTO %s_new (id, name, age) SELECT id, name, age FROM %s", "DROP TABLE %s", "ALTER TABLE %s_new RENAME TO %s"]}`, users, users, users, users, users, users)
req, _ = http.NewRequest(http.MethodPost, base+"/v1/db/transaction", strings.NewReader(txBody))
req.Header = authHeader(key)
resp, err = httpClient().Do(req)
if err != nil { t.Fatalf("mig tx do: %v", err) }
resp.Body.Close()
if resp.StatusCode != http.StatusOK { t.Fatalf("mig tx status: %d", resp.StatusCode) }

Sources: e2e/gateway_e2e_test.go:198-208

Conclusion

The DeBros Network Cluster provides a robust, decentralized platform for distributed data management and communication. Its modular architecture, clear separation of node and client roles, and comprehensive API design facilitate resilient and scalable applications. The system's emphasis on secure communication, flexible configuration, and a reliable database migration system underpins its utility for building decentralized applications.

Project Structure

Related Pages

Related topics: Architecture Overview

Relevant source files

Project Structure

The DeBros Network Cluster is a decentralized peer-to-peer (P2P) system built in Go, designed for resilient, distributed data management and communication. Its project structure is modular, separating core functionalities into distinct packages and organizing executables for nodes and command-line interfaces. This organization facilitates development, testing, and deployment, ensuring a clear separation of concerns between different components of the network. The architecture supports both full network nodes and lightweight clients, enabling various application types to interact with the network. Sources: AI_CONTEXT.md:16-17, README.md:13-14

Top-Level Directories

The project's root directory contains several key folders that organize its source code, build artifacts, configurations, and scripts. This structure supports a streamlined development workflow from building to deployment. Sources: AI_CONTEXT.md:275-283, README.md:120-128

graph TD
    A[network/] --> B[cmd/]
    A --> C[pkg/]
    A --> D[scripts/]
    A --> E[configs/]
    A --> F[bin/]
    A --> G[anyoneproxy/]
    A --> H[data/]
    A --> I[docs/]
    A --> J[examples/]
    A --> K[openapi/]
    A --> L[tests/]

Sources: AI_CONTEXT.md:275-283

cmd/

This directory holds the main executable applications of the project. Each subdirectory within cmd/ represents a distinct application. Sources: AI_CONTEXT.md:276, README.md:121

  • node/: Contains the source code for the DeBros Network node executable. This is the core P2P participant that runs all network services. Sources: AI_CONTEXT.md:277, README.md:122
  • cli/: Contains the source code for the command-line interface tool, network-cli, used to interact with the network. Sources: AI_CONTEXT.md:278, README.md:123

pkg/

The pkg/ directory contains the reusable Go packages that encapsulate the core logic and services of the DeBros Network. This modular design promotes code reusability and maintainability. Sources: AI_CONTEXT.md:279, README.md:124

  • client/: Provides the lightweight P2P client library for applications and CLI tools to connect to the network. It offers interfaces for database, storage, pub/sub, and network information operations. Sources: AI_CONTEXT.md:290-293, README.md:52
  • node/: Implements the full P2P node functionality, including peer discovery, service registration, connection monitoring, and data replication. Sources: AI_CONTEXT.md:295-298
  • database/: Handles the integration with RQLite, providing distributed SQLite capabilities, including SQL queries, transactions, schema management, and a robust migration system. Sources: AI_CONTEXT.md:304-308
  • storage/: Implements the distributed key-value storage system, supporting namespace-isolated CRUD operations and prefix queries. Sources: AI_CONTEXT.md:310
  • pubsub/: Manages the topic-based, real-time messaging system with automatic subscription management and namespace isolation. Sources: AI_CONTEXT.md:312
  • config/: Centralized package for managing configuration via YAML files, environment variables, and CLI flags. Sources: AI_CONTEXT.md:300-302
  • discovery/: Handles peer discovery mechanisms, primarily for full nodes, using peerstore and peer exchange. Sources: AI_CONTEXT.md:314-315
  • anyoneproxy/: Optional SOCKS5 proxy support. Sources: AI_CONTEXT.md:275
  • auth/: Manages authentication mechanisms, including wallet verification, JWT, and API key support. Sources: AI_CONTEXT.md:144-145

scripts/

This directory contains various shell scripts for building, testing, and deploying the network. Sources: AI_CONTEXT.md:280, README.md:125

  • install-debros-network.sh: An automated script for production installation, handling OS detection, dependency installation, system user creation, binary building, systemd service setup, and firewall configuration. Sources: README.md:80-92
  • test-multinode.sh: Script for local multi-node testing. Sources: README.md:139

configs/

This directory stores YAML configuration files for different network components, such as node.yaml and bootstrap.yaml. These files define settings for nodes, databases, discovery, and security. Sources: AI_CONTEXT.md:281, README.md:126

bin/

This directory is where the compiled executable binaries are placed after a build operation. Sources: AI_CONTEXT.md:282, README.md:127

data/

This directory is used for runtime data, including identity keys, RQLite database files, and logs. Sources: AI_CONTEXT.md:283, README.md:96

examples/

Contains example applications demonstrating how to use the DeBros Network client, such as basic_usage.go and the anchat/ decentralized chat app. Sources: AI_CONTEXT.md:261, examples/basic_usage.go:1-6

openapi/

This directory contains the OpenAPI specification for the gateway's REST endpoints, enabling machine-readable contract definitions for Storage, Database, and PubSub services. Sources: AI_CONTEXT.md:175-176

Core Components and Their Interactions

The DeBros Network is composed of several interconnected core components that work together to provide distributed data and messaging services.

graph TD
    A[Application Layer] --> B(Client API)
    B --> C{Network Client}
    C --> D[Database Client]
    C --> E[Storage Client]
    C --> F[PubSub Client]
    C --> G[Network Info Client]

    D --> H[RQLite Database]
    E --> I[Distributed KV Storage]
    F --> J[Topic-based Pub/Sub]

    H -- Raft Consensus --> K[Node]
    I -- Replication --> K
    J -- Messaging --> K

    K --> L[Peer Discovery]
    K --> M[Service Registration]
    K --> N[Connection Monitoring]

    O[Configuration System] --> K
    O --> C

    P[HTTP Gateway] --> Q[Client API]
    Q --> R[Auth System]
    Q --> S[Namespace Enforcement]
    Q --> T[Transport Security]

    K -- P2P Communication --> K

Sources: AI_CONTEXT.md:24-34, README.md:27-37, AI_CONTEXT.md:288-315, AI_CONTEXT.md:317-327

Node vs. Client Roles

The system distinguishes between full network nodes and lightweight clients, each with specific responsibilities and capabilities. Sources: AI_CONTEXT.md:18, AI_CONTEXT.md:330-331

Node (pkg/node/)

A full P2P participant that runs core services like RQLite, storage, and pub/sub. Nodes are responsible for peer discovery and network management. Sources: AI_CONTEXT.md:295-298, AI_CONTEXT.md:333-335

Client (pkg/client/)

A lightweight P2P participant primarily for applications and CLI tools. Clients connect only to bootstrap peers and do not participate in peer discovery. They consume services provided by the network. Sources: AI_CONTEXT.md:290-293, AI_CONTEXT.md:337-339

Configuration System (pkg/config/)

The configuration system is centralized, primarily using YAML files (configs/node.yaml). Settings can be overridden by CLI flags and environment variables, following a specific precedence: CLI flags > Environment variables > YAML > Code defaults. Sources: AI_CONTEXT.md:300-302, AI_CONTEXT.md:317-327, README.md:182-184

An example of configuration parameters:

Parameter Description Source File
data_dir Directory for node data (identity, database, logs). AI_CONTEXT.md:322
listen_addresses LibP2P listen addresses for the node. README.md:156
rqlite_port HTTP API port for RQLite. README.md:162
rqlite_raft_port Raft consensus port for RQLite. README.md:163
rqlite_join_address RQLite address to join an existing cluster. README.md:164
bootstrap_peers List of multiaddresses for bootstrap peers. README.md:167
logging.level Log level (e.g., "info", "debug"). README.md:173
Sources: AI_CONTEXT.md:322-327, README.md:156-174

HTTP Gateway

The HTTP Gateway provides a RESTful interface for interacting with the DeBros Network. It acts as a bridge, offering an API for Storage, Database, and PubSub operations over the network client with strict namespace enforcement. It includes an enhanced authentication system supporting multi-wallet, JWT, and API keys. Sources: AI_CONTEXT.md:118-125

graph TD
    A[External Client] --> B(HTTP Gateway)
    B --> C[Auth System]
    B --> D[Namespace Enforcement]
    B --> E[Network Client]

    E --> F[Database Service]
    E --> G[Storage Service]
    E --> H[PubSub Service]

    C --> I[Wallet Verification]
    C --> J[JWT Management]
    C --> K[API Key Mapping]

    D --> F
    D --> G
    D --> H

Sources: AI_CONTEXT.md:118-125

Key HTTP endpoints provided by the gateway:

Category Endpoint Description Method Source File
Health/Version /health, /v1/health Returns gateway health status. GET AI_CONTEXT.md:195
/v1/status Returns network status. GET AI_CONTEXT.md:196
/v1/version Returns version, commit, build time, uptime. GET AI_CONTEXT.md:197
Auth /v1/auth/challenge Generates a wallet challenge (nonce). POST AI_CONTEXT.md:202
/v1/auth/verify Verifies wallet signature. POST AI_CONTEXT.md:203
/v1/auth/api-key Generates API keys for authenticated users. POST AI_CONTEXT.md:207
Storage /v1/storage/put Stores data by key. POST AI_CONTEXT.md:209
/v1/storage/get Retrieves data by key. POST AI_CONTEXT.md:209
/v1/storage/list Lists keys with an optional prefix. GET AI_CONTEXT.md:210
Database /v1/db/create-table Creates a table with SQL DDL. POST AI_CONTEXT.md:164
/v1/db/query Executes a single SQL query. POST AI_CONTEXT.md:166
/v1/db/transaction Applies multiple SQL statements atomically. POST AI_CONTEXT.md:167
PubSub /v1/pubsub/ws WebSocket connection for real-time messaging. GET AI_CONTEXT.md:216
/v1/pubsub/publish Publishes a message to a topic. POST AI_CONTEXT.md:218
Sources: AI_CONTEXT.md:164-173, AI_CONTEXT.md:194-220

Build and Development Workflow

The project provides a straightforward build and development workflow using make commands. Sources: CONTRIBUTING.md:21-22, AI_CONTEXT.md:88-91

make build        # Build all executables
make test         # Run tests
make run-node     # Start node (auto-detects bootstrap vs regular)
make run-gateway  # Start HTTP gateway (env overrides supported)

Sources: AI_CONTEXT.md:89-92

For local development, make run-node can be used. Node settings are configured via YAML files, and the CLI tool is used for network operations and testing. Sources: AI_CONTEXT.md:94-96

Conclusion

The DeBros Network project structure is designed for clarity, modularity, and scalability. By separating concerns into distinct cmd/ for executables and pkg/ for reusable libraries, it promotes efficient development and maintenance. The clear distinction between node and client roles, coupled with a robust configuration system and an HTTP Gateway, provides a comprehensive framework for building decentralized applications. Sources: AI_CONTEXT.md:16-18, AI_CONTEXT.md:275-283

Node vs. Client Roles

Related Pages

Related topics: Network Client, Node Component

Relevant source files

Node vs. Client Roles

The DeBros Network Cluster is a decentralized peer-to-peer (P2P) system designed for resilient, distributed data management and communication. Within this architecture, a clear distinction is made between full network nodes and lightweight clients, each serving specific roles and interacting with the network in different capacities. This separation allows for both robust network participation and efficient application integration. Sources: AI_CONTEXT.md:16-19, README.md:12-14

Core Architectural Distinction

The DeBros Network employs a modular and robust architecture that differentiates between full nodes and lightweight clients. Full nodes run core services and participate in network discovery, while lightweight clients connect to the network via bootstrap peers to consume services. Sources: AI_CONTEXT.md:23-25, AI_CONTEXT.md:155-156

Node Role

A Node is a full participant in the DeBros Network. It runs the complete suite of network services, including the distributed SQL database (RQLite), key-value storage, and Pub/Sub messaging. Nodes are responsible for peer discovery, managing network topology, participating in consensus and data replication, and overseeing service lifecycle and monitoring. Sources: AI_CONTEXT.md:158-163, AI_CONTEXT.md:104-107

Key characteristics and responsibilities of a Node:

  • Runs full network services: This includes RQLite, storage, and Pub/Sub. Sources: AI_CONTEXT.md:159
  • Handles peer discovery and network topology: Nodes actively discover and manage connections with other peers in the network. Sources: AI_CONTEXT.md:106, AI_CONTEXT.md:160
  • Participates in consensus and replication: Essential for maintaining data consistency and availability across the distributed database and storage systems. Sources: AI_CONTEXT.md:161
  • Manages service lifecycle and monitoring: Ensures the proper operation and health of its integrated services. Sources: AI_CONTEXT.md:162
  • Configuration: Node configuration is primarily managed via YAML files (configs/node.yaml), with overrides possible via CLI flags and environment variables. Sources: AI_CONTEXT.md:139-140

Node Startup Process

The cmd/node/main.go file outlines the startup sequence for a DeBros Network node. It involves parsing command-line flags, loading configuration, setting up logging, and initializing the node instance.

graph TD
    A[main()] --> B{Parse Command-Line Flags};
    B --> C{Load Configuration};
    C --> D{Apply Overrides};
    D --> E{Initialize Logger};
    E --> F[Create Node Context];
    F --> G[Create NewNode(cfg)];
    G --> H{Start Node};
    H --> I[Save Peer ID to File];
    I --> J[Wait for Context Cancellation];
    J --> K[Stop Node];

Sources: cmd/node/main.go:41-117, cmd/node/main.go:197-234

The startNode function in cmd/node/main.go creates and starts the node, also saving its peer ID to a file for external access. Sources: cmd/node/main.go:167-194

Client Role

A Client is a lightweight participant in the DeBros Network. Unlike nodes, clients do not run full network services, nor do they participate in peer discovery or DHT. Instead, clients connect only to known bootstrap peers and consume network services (Database, Storage, Pub/Sub, NetworkInfo) via a dedicated API. Sources: AI_CONTEXT.md:165-170, AI_CONTEXT.md:99-102

Key characteristics and responsibilities of a Client:

  • Lightweight participant: Does not run its own services. Sources: AI_CONTEXT.md:166
  • Connects only to known bootstrap peers: Relies on pre-configured bootstrap nodes to access the network. Sources: AI_CONTEXT.md:167
  • No peer discovery or DHT: Does not actively discover other peers or maintain a distributed hash table. Sources: AI_CONTEXT.md:168
  • Consumes network services via API: Provides interfaces for Database, Storage, Pub/Sub, and NetworkInfo operations. Sources: AI_CONTEXT.md:169
  • Used by CLI and application integrations: The pkg/client/ package provides the necessary API for applications and command-line tools to interact with the network. Sources: AI_CONTEXT.md:170, AI_CONTEXT.md:99

Client Usage Example

The examples/basic_usage.go file demonstrates how an application client interacts with the DeBros Network.

package main

import (
	"context"
	"log"
	"time"

	"git.debros.io/DeBros/network/pkg/client"
)

func main() {
	// Create client configuration
	config := client.DefaultClientConfig("example_app")
	config.BootstrapPeers = []string{
		"/ip4/127.0.0.1/tcp/4001/p2p/QmBootstrap1",
	}

	// Create network client
	networkClient, err := client.NewClient(config)
	if err != nil {
		log.Fatalf("Failed to create network client: %v", err)
	}

	// Connect to network
	if err := networkClient.Connect(); err != nil {
		log.Fatalf("Failed to connect to network: %v", err)
	}
	defer networkClient.Disconnect()

	log.Printf("Connected to network successfully!")

	// Example: Database operations
	demonstrateDatabase(networkClient)

	// Example: Storage operations
	demonstrateStorage(networkClient)

	// Example: Pub/Sub messaging
	demonstratePubSub(networkClient)

	// Example: Network information
	demonstrateNetworkInfo(networkClient)

	log.Printf("Example completed successfully!")
}

Sources: examples/basic_usage.go:5-64

Comparison of Node and Client Roles

The following table summarizes the key differences between Nodes and Clients in the DeBros Network Cluster:

Feature Node (pkg/node/) Client (pkg/client/)
Role Full P2P participant Lightweight P2P participant
Services Run RQLite, Storage, Pub/Sub None
Peer Discovery Yes (via peerstore and peer exchange) No (connects only to bootstrap peers)
Network Management Handles network topology, consensus, replication Consumes services via API
Data Replication Yes No
Use Case Core network infrastructure, service provision Application integration, CLI tools
Configuration Source configs/node.yaml (YAML, CLI flags, Env vars) ClientConfig (explicit, Env vars, Library defaults)

Sources: AI_CONTEXT.md:99-107, AI_CONTEXT.md:158-170, AI_CONTEXT.md:139-148

Network Interaction and Data Flow

Both nodes and clients interact with the network, but their mechanisms and responsibilities differ significantly.

Node Network Interaction

Nodes establish connections, discover peers, and register their services. They participate in the core data flow for database operations (via Raft consensus), storage replication, and Pub/Sub message routing. Sources: AI_CONTEXT.md:175-185

sequenceDiagram
    participant N as Node
    participant B as Bootstrap Peer
    participant O as Other Node
    participant RQL as RQLite Leader
    participant RQLC as RQLite Cluster
    participant S as Storage System
    participant PS as PubSub System

    N->>B: Connect (initial)
    N->>O: Discover Peers (peer exchange)
    activate N
    N->>N: Register Services (internal)
    N->>RQL: Database Operations (SQL queries)
    RQL->>RQLC: Raft Consensus / Replication
    N->>S: Storage Operations (KV)
    N->>PS: PubSub Messaging (Publish/Subscribe)
    deactivate N

Sources: AI_CONTEXT.md:175-185

Client Network Interaction

Clients connect to bootstrap peers and then use the provided API interfaces to perform operations. They do not directly participate in the underlying consensus or replication mechanisms; instead, they send requests to the nodes which then handle the distributed logic. Sources: AI_CONTEXT.md:176, AI_CONTEXT.md:181-185

sequenceDiagram
    participant C as Client
    participant B as Bootstrap Peer
    participant N as Node
    participant DB as Database Service
    participant ST as Storage Service
    participant PB as PubSub Service

    C->>B: Connect (initial)
    activate C
    C->>N: Database Query (via Client API)
    N->>DB: Process Query
    DB-->>N: Query Result
    N-->>C: Query Result
    C->>N: Storage Put (via Client API)
    N->>ST: Store Data
    ST-->>N: Confirmation
    N-->>C: Confirmation
    C->>N: PubSub Publish (via Client API)
    N->>PB: Route Message
    PB-->>N: Confirmation
    N-->>C: Confirmation
    deactivate C

Sources: AI_CONTEXT.md:181-185, examples/basic_usage.go:66-180

Conclusion

The distinction between Nodes and Clients is fundamental to the DeBros Network's design, enabling a flexible and scalable distributed system. Nodes form the backbone, providing core services and ensuring data integrity and availability through distributed protocols. Clients, on the other hand, offer a lightweight interface for applications and users to interact with the network's capabilities without the overhead of running full services. This architectural separation optimizes resource utilization and simplifies application development on the DeBros Network. Sources: AI_CONTEXT.md:16-19, AI_CONTEXT.md:23-25

Network Protocol & Data Flow

Related Pages

Related topics: Architecture Overview

Relevant source files

Network Protocol & Data Flow

The DeBros Network Cluster is a decentralized peer-to-peer (P2P) system designed for resilient, distributed data management and communication. Its core functionality revolves around a robust network protocol and efficient data flow mechanisms, enabling distributed database operations, key-value storage, and pub/sub messaging. The system distinguishes between full network nodes and lightweight clients, each interacting with the network in a defined manner to ensure scalability and performance. Sources: AI_CONTEXT.md:43-46, AI_CONTEXT.md:53-56

Architecture Overview

The DeBros Network Cluster employs a modular architecture that supports both full nodes and lightweight clients. Full nodes run core services and participate in peer discovery, while clients connect to the network via bootstrap peers without running their own services. Sources: AI_CONTEXT.md:53-56, AI_CONTEXT.md:167-170

The overall architecture can be visualized as follows:

graph TD
    A[Application Layer] --> B(Client API)
    B --> C(Transport Layer)
    C --> D(LibP2P Host)
    C --> E(Noise/TLS Encryption)
    C --> F(RQLite Database)

    subgraph Application Layer
        Anchat(Anchat)
        CustomApp(Custom App)
        CLITools(CLI Tools)
    end

    subgraph Client API
        DatabaseClient(Database Client)
        StorageClient(Storage Client)
        PubSubClient(PubSub Client)
    end

    subgraph Transport Layer
        LibP2P(LibP2P Host)
        NoiseTLS(Noise/TLS Encryption)
        RQLite(RQLite Database)
    end

    Anchat --> DatabaseClient
    CustomApp --> StorageClient
    CLITools --> PubSubClient

    DatabaseClient --> LibP2P
    StorageClient --> NoiseTLS
    PubSubClient --> RQLite

Sources: AI_CONTEXT.md:60-75, README.md:52-67

Node vs Client Roles

The DeBros Network defines distinct roles for nodes and clients, impacting their network participation and data flow. Sources: AI_CONTEXT.md:165

Node Role

A full node runs all core network services. It actively participates in peer discovery, consensus, and data replication. Nodes manage the lifecycle of services like RQLite, storage, and pub/sub. Sources: AI_CONTEXT.md:167-170

graph TD
    A[Node] --> B{Runs Full Network Services}
    B --> C(RQLite)
    B --> D(Storage)
    B --> E(PubSub)
    A --> F{Handles Peer Discovery}
    A --> G{Participates in Consensus & Replication}
    A --> H{Manages Service Lifecycle & Monitoring}

Sources: AI_CONTEXT.md:167-170

Client Role

A client is a lightweight participant that does not run core network services. Instead, it connects to known bootstrap peers and consumes network services via APIs for database, storage, pub/sub, and network information. Clients do not perform peer discovery or participate in DHT. Sources: AI_CONTEXT.md:172-176

graph TD
    A[Client] --> B{Connects to Bootstrap Peers Only}
    A --> C{Consumes Network Services via API}
    C --> D(Database Client)
    C --> E(Storage Client)
    C --> F(PubSub Client)
    C --> G(NetworkInfo Client)
    A --> H{No Peer Discovery or DHT}

Sources: AI_CONTEXT.md:172-176

Connection Establishment

The process of establishing connections differs between nodes and clients.

Node Connection

Nodes connect to bootstrap peers, discover additional peers, and register their services. Sources: AI_CONTEXT.md:180

Client Connection

Clients connect exclusively to bootstrap peers. Sources: AI_CONTEXT.md:181

The examples/basic_usage.go file demonstrates client connection:

// examples/basic_usage.go:14-22
func main() {
	// Create client configuration
	config := client.DefaultClientConfig("example_app")
	config.BootstrapPeers = []string{
		"/ip4/127.0.0.1/tcp/4001/p2p/QmBootstrap1",
	}

	// Create network client
	networkClient, err := client.NewClient(config)
	if err != nil {
		log.Fatalf("Failed to create network client: %v", err)
	}

	// Connect to network
	if err := networkClient.Connect(); err != nil {
		log.Fatalf("Failed to connect to network: %v", err)
	}
	defer networkClient.Disconnect()

	log.Printf("Connected to network successfully!")
}

Message Types

The network protocol handles various message types to facilitate distributed operations. Sources: AI_CONTEXT.md:183

Message Type Description
Control Node status, heartbeats, topology updates
Database SQL queries, transactions, schema operations
Storage Key-value operations, replication
PubSub Topic subscriptions, published messages
Sources: AI_CONTEXT.md:184-187

Security Model

The DeBros Network incorporates several security features to ensure secure communication and data integrity. Sources: AI_CONTEXT.md:189

  • Transport: Noise/TLS encryption for all connections.
  • Authentication: Peer identity verification.
  • Isolation: Namespace-based access control. Sources: AI_CONTEXT.md:190-192

Data Flow

Data flows through the network differently based on the type of operation.

Database Data Flow

sequenceDiagram
    participant Client
    participant DatabaseClient
    participant RQLiteLeader as RQLite Leader
    participant RaftConsensus as Raft Consensus
    participant AllNodes as All Nodes

    Client->>DatabaseClient: SQL Query/Transaction
    activate DatabaseClient
    DatabaseClient->>RQLiteLeader: Forward Query
    deactivate DatabaseClient
    activate RQLiteLeader
    RQLiteLeader->>RaftConsensus: Propose Change
    activate RaftConsensus
    RaftConsensus-->>RQLiteLeader: Commit Result
    deactivate RaftConsensus
    RQLiteLeader->>AllNodes: Replicate State
    deactivate RQLiteLeader
    RQLiteLeader-->>DatabaseClient: Query Result
    DatabaseClient-->>Client: Query Result

Sources: AI_CONTEXT.md:194

Storage Data Flow

sequenceDiagram
    participant Client
    participant StorageClient
    participant Node
    participant Replication

    Client->>StorageClient: Put/Get Key-Value
    activate StorageClient
    StorageClient->>Node: Request Storage Operation
    deactivate StorageClient
    activate Node
    Node->>Replication: Replicate Data
    activate Replication
    Replication-->>Node: Replication Status
    deactivate Replication
    Node-->>StorageClient: Operation Result
    deactivate Node
    StorageClient-->>Client: Operation Result

Sources: AI_CONTEXT.md:195

Pub/Sub Data Flow

sequenceDiagram
    participant Client
    participant PubSubClient
    participant Node
    participant TopicRouter
    participant Subscribers

    Client->>PubSubClient: Publish Message
    activate PubSubClient
    PubSubClient->>Node: Forward Message
    deactivate PubSubClient
    activate Node
    Node->>TopicRouter: Route Message to Topic
    activate TopicRouter
    TopicRouter->>Subscribers: Deliver Message
    deactivate TopicRouter
    Node-->>PubSubClient: Publish Acknowledgment
    deactivate Node
    PubSubClient-->>Client: Publish Acknowledgment

Sources: AI_CONTEXT.md:196

Network Ports

The DeBros Network utilizes specific network ports for its various components:

Port Protocol Description Source
4001 TCP LibP2P P2P communication AI_CONTEXT.md:92, README.md:83
5001 TCP RQLite HTTP API AI_CONTEXT.md:93, README.md:84
7001 TCP RQLite Raft consensus AI_CONTEXT.md:94, README.md:85

These ports are configurable, and can be overridden via command-line flags or environment variables. For instance, the cmd/node/main.go file shows that p2p-port, rqlite-http-port, and rqlite-raft-port flags can be used. Sources: cmd/node/main.go:50-52, AI_CONTEXT.md:155-156

Conclusion

The network protocol and data flow mechanisms are fundamental to the DeBros Network Cluster's operation, enabling its distributed capabilities. The clear distinction between node and client roles, coupled with robust connection establishment, varied message types, and a strong security model, ensures a resilient and scalable P2P system. The use of LibP2P for communication and RQLite for distributed database operations forms the backbone of its distributed data management and communication capabilities. Sources: AI_CONTEXT.md:43-46, AI_CONTEXT.md:53-56, AI_CONTEXT.md:167-176

Network Client

Related Pages

Related topics: Node vs. Client Roles, CLI Overview

Error with OpenRouter API: argument of type 'NoneType' is not iterable

Please check that you have set the OPENROUTER_API_KEY environment variable with a valid API key.

Node Component

Related Pages

Related topics: Node vs. Client Roles, Discovery Component

Relevant source files

Node Component

The Node component in the DeBros Network is a full peer-to-peer (P2P) participant responsible for running core services such as the distributed SQL database (RQLite), key-value storage, and pub/sub messaging. It handles peer discovery, service registration, connection monitoring, and data replication. Nodes are designed to be robust and decentralized, forming the backbone of the DeBros Network Cluster. Unlike clients, nodes participate in peer discovery and contribute to the network's distributed nature. Sources: README.md:163, AI_CONTEXT.md:20-22, AI_CONTEXT.md:120-123

Node Architecture and Core Responsibilities

A DeBros Network node integrates several key services to provide its distributed functionalities. These include the transport layer (LibP2P, Noise/TLS encryption), the RQLite Database, and various application-level services like storage and pub/sub. The node's primary responsibilities encompass maintaining network connectivity, managing data persistence and replication, and facilitating real-time communication across the P2P network. Sources: README.md:154-160, AI_CONTEXT.md:120-123

Key Components of a Node

The Node component is composed of several modules working in concert:

  • LibP2P Host: Manages P2P communication and peer connections.
  • Noise/TLS Encryption: Ensures secure transport layer communication.
  • RQLite Database: Provides distributed SQL database capabilities with Raft consensus.
  • Storage Service: Offers distributed key-value storage.
  • Pub/Sub Service: Enables real-time topic-based messaging.
  • Discovery: Manages peer discovery and network topology. Sources: README.md:154-160, AI_CONTEXT.md:120-123, AI_CONTEXT.md:125-149

The interaction between these components can be visualized as follows:

graph TD
    subgraph DeBros Network Node
        A[LibP2P Host] --> B(Noise/TLS Encryption)
        B --> C[RQLite Database]
        B --> D[Storage Service]
        B --> E[Pub/Sub Service]
        A --> F[Discovery]
        F --> A
        C --> G[Data Replication]
        D --> G
        E --> G
    end

Sources: README.md:154-160, AI_CONTEXT.md:125-149

Node Configuration

Nodes are configured primarily through YAML files, with command-line flags and environment variables providing overrides. This layered approach allows for flexible deployment and management. Sources: README.md:255-256, AI_CONTEXT.md:151-152

Configuration Precedence

The order of precedence for configuration values is:

  1. Command-line flags
  2. Environment variables
  3. YAML configuration files
  4. Code defaults Sources: README.md:255-256, AI_CONTEXT.md:151-152

Example Configuration Parameters

The configs/node.yaml file defines various parameters for a network node. Sources: README.md:275-307

Parameter Type Description Default (Flags) Source
node.id string Unique identifier for the node. "" README.md:277
node.listen_addresses []string Multiaddrs for P2P listening (e.g., /ip4/0.0.0.0/tcp/4002). /ip4/0.0.0.0/tcp/4001 README.md:278, cmd/node/main.go:90
node.data_dir string Directory for node-specific data (e.g., LibP2P keys, RQLite DB). ./data/node README.md:280, cmd/node/main.go:182
database.rqlite_port int HTTP API port for RQLite. 5001 README.md:288, cmd/node/main.go:91
database.rqlite_raft_port int Raft consensus port for RQLite. 7001 README.md:289, cmd/node/main.go:92
database.rqlite_join_address string Address of an existing RQLite node to join the cluster. "" README.md:290
discovery.bootstrap_peers []string List of multiaddrs for bootstrap peers to connect to. [] README.md:293
security.enable_tls bool Enables TLS encryption for communications. false README.md:299
logging.level string Logging verbosity level (e.g., "info", "debug"). "info" README.md:303

Loading and Applying Configuration

The cmd/node/main.go file illustrates how configuration is loaded and applied. Sources: cmd/node/main.go:76-105, cmd/node/main.go:167-183

// cmd/node/main.go
func parse_and_return_network_flags() (configPath *string, dataDir, nodeID *string, p2pPort, rqlHTTP, rqlRaft *int, disableAnon *bool, rqlJoinAddr *string, advAddr *string, help *bool) {
	// ... flag parsing ...
	flag.Parse()

	if *configPath != "" {
		cfg, err := LoadConfigFromYAML(*configPath) // [cmd/node/main.go:82]()
		// ... error handling ...
		// Instead of returning flag values, return config values
		// ... port parsing from ListenAddresses ...
		return configPath,
			&cfg.Node.DataDir,
			&cfg.Node.ID,
			&p2pPortVal, // Derived from config
			&cfg.Database.RQLitePort,
			&cfg.Database.RQLiteRaftPort,
			&cfg.Node.DisableAnonRC,
			&cfg.Database.RQLiteJoinAddress,
			&cfg.Discovery.HttpAdvAddress,
			help
	}
	return
}

// load_args_into_config applies command line argument overrides to the config
func load_args_into_config(cfg *config.Config, p2pPort, rqlHTTP, rqlRaft *int, rqlJoinAddr *string, advAddr *string, dataDir *string) {
	// Apply RQLite HTTP port override
	if *rqlHTTP != 5001 { // [cmd/node/main.go:169]()
		cfg.Database.RQLitePort = *rqlHTTP
	}
	// ... similar logic for other overrides ...
	if *dataDir != "" { // [cmd/node/main.go:181]()
		cfg.Node.DataDir = *dataDir
	}
}

Sources: cmd/node/main.go:76-105, cmd/node/main.go:167-183

Node Lifecycle and Operation

The node's lifecycle involves initialization, starting services, handling graceful shutdowns, and persisting peer information. Sources: cmd/node/main.go:122-159

Starting a Node

A node is initialized with a configuration and then started. During startup, it creates and starts its internal services (LibP2P host, RQLite, etc.). The node's Peer ID and multiaddr are saved to a peer.info file for external access, particularly useful for bootstrap nodes. Sources: cmd/node/main.go:122-159

sequenceDiagram
    participant Main as cmd/node/main.go
    participant Node as pkg/node/Node
    participant Config as pkg/config/Config

    Main->>Main: parse_and_return_network_flags()
    Main->>Config: DefaultConfig()
    Main->>Main: load_args_into_config(cfg, ...)
    Main->>Node: NewNode(cfg)
    activate Node
    Node-->>Main: n, err
    deactivate Node
    Main->>Node: Start(ctx)
    activate Node
    Node->>Node: Initialize internal services
    Node->>Node: Start LibP2P Host
    Node->>Node: Start RQLite Database
    Node->>Node: Start Storage Service
    Node->>Node: Start Pub/Sub Service
    Node-->>Main: err
    deactivate Node
    Main->>Main: GetPeerID()
    Main->>Main: Save peer.info file
    Main->>Main: Wait for context cancellation
    Main->>Node: Stop()
    activate Node
    Node->>Node: Shut down services
    Node-->>Main: err
    deactivate Node

Sources: cmd/node/main.go:122-159

Data Directory Structure

When a node runs, it creates a structured data directory to store various runtime artifacts. Sources: README.md:363-368

/opt/debros/
├── bin/           # Binaries
├── configs/       # YAML configs
├── keys/          # Identity keys
├── data/          # RQLite DB, storage
├── logs/          # Node logs
├── src/           # Source code (if installed via script)

Sources: README.md:363-368

Graceful Shutdown

Nodes are designed to shut down gracefully upon receiving termination signals (e.g., SIGINT, SIGTERM). This ensures that all running services are properly closed and any persistent data is saved. Sources: cmd/node/main.go:207-217

Node vs. Client Roles

It is crucial to distinguish between a full Node and a lightweight Client in the DeBros Network. Sources: README.md:162-165, AI_CONTEXT.md:161-168

Node (pkg/node/)

  • Role: Full P2P participant.
  • Services: Runs core services (RQLite, storage, pubsub).
  • Discovery: Handles peer discovery, service registration, and connection monitoring.
  • Data: Persists data, participates in replication and consensus. Sources: README.md:163, AI_CONTEXT.md:161-164

Client (pkg/client/)

  • Role: Lightweight P2P participant for applications and CLI tools.
  • Services: Consumes services provided by nodes.
  • Discovery: Connects only to bootstrap peers; does not perform peer discovery.
  • Data: Does not persist core network data; consumes data from nodes. Sources: README.md:164, AI_CONTEXT.md:165-168

An example of a client connecting to a network node:

// examples/basic_usage.go
func main() {
	// Create client configuration
	config := client.DefaultClientConfig("example_app") // [examples/basic_usage.go:14]()
	config.BootstrapPeers = []string{
		"/ip4/127.0.0.1/tcp/4001/p2p/QmBootstrap1", // [examples/basic_usage.go:16]()
	}

	// Create network client
	networkClient, err := client.NewClient(config) // [examples/basic_usage.go:19]()
	if err != nil {
		log.Fatalf("Failed to create network client: %v", err)
	}

	// Connect to network
	if err := networkClient.Connect(); err != nil { // [examples/basic_usage.go:25]()
		log.Fatalf("Failed to connect to network: %v", err)
	}
	defer networkClient.Disconnect() // [examples/basic_usage.go:28]()

	log.Printf("Connected to network successfully!")
	// ... further operations ...
}

Sources: examples/basic_usage.go:11-31

Conclusion

The Node component is fundamental to the DeBros Network, embodying its decentralized and distributed nature. By hosting core services and participating in peer management, nodes enable the network's capabilities for distributed SQL databases, key-value storage, and pub/sub messaging. Proper configuration and understanding of the node's lifecycle are essential for deploying and managing a robust DeBros Network Cluster. Sources: README.md:163, AI_CONTEXT.md:20-22

Configuration System

Related Pages

Related topics: Quick Start Guide, Gateway Configuration

Relevant source files

The following files were used as context for generating this wiki page:

Configuration System

The DeBros Network Cluster utilizes a flexible and hierarchical configuration system to manage the operational parameters of its nodes and services. This system allows for defining settings through YAML files, environment variables, and command-line flags, with a clear precedence order ensuring that runtime parameters can override static configurations. This approach facilitates easy deployment, testing, and customization for various network roles, such as bootstrap nodes or regular peer nodes. Sources: README.md:37-38, AI_CONTEXT.md:21-22

Configuration Sources and Precedence

The DeBros Network configuration can be defined from multiple sources, which are applied in a specific order of precedence. This hierarchy ensures that more dynamic or specific settings can override broader or default ones. Sources: README.md:95-97, AI_CONTEXT.md:35-37

The order of precedence, from highest to lowest, is:

  1. Command-line Flags: These provide the highest level of override, allowing specific parameters to be set directly when the application is launched.
  2. Environment Variables: Environment variables can override values specified in configuration files.
  3. YAML Configuration Files: These files serve as the primary source for defining the majority of the application's settings.
  4. Default Values: Hardcoded default values are used if no other configuration source provides a specific parameter. Sources: README.md:95-97, AI_CONTEXT.md:35-37
graph TD
    A[Command-line Flags] --> B[Environment Variables]
    B --> C[YAML Config Files]
    C --> D[Default Values]
    style A fill:#f9f,stroke:#333,stroke-width:2px
    style D fill:#ccc,stroke:#333,stroke-width:1px
    classDef highest fill:#f0f,stroke:#333,stroke-width:2px;
    classDef lowest fill:#ccc,stroke:#333,stroke-width:1px;
    class A highest;
    class D lowest;

Sources: README.md:95-97, AI_CONTEXT.md:35-37

YAML Configuration Files

YAML files are the recommended method for structuring and managing complex configurations. The project provides example YAML files for different node roles, such as bootstrap.yaml and node.yaml. Sources: README.md:58-59, AI_CONTEXT.md:27-28

configs/bootstrap.yaml

This configuration is tailored for a bootstrap node, which typically has a fixed identity and does not join another RQLite cluster. Sources: configs/bootstrap.yaml:1-2

node:
  id: ""
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4001"
  data_dir: "./data/bootstrap"
  max_connections: 100
  disable_anonrc: true

database:
  data_dir: "./data/db"
  replication_factor: 3
  shard_count: 16
  max_database_size: 1073741824
  backup_interval: 24h
  rqlite_port: 5001
  rqlite_raft_port: 7001
  rqlite_join_address: "" # Bootstrap node does not join

discovery:
  bootstrap_peers: []
  discovery_interval: 15s
  bootstrap_port: 4001
  http_adv_address: "127.0.0.1"
  raft_adv_address: ""

security:
  enable_tls: false
  private_key_file: ""
  certificate_file: ""

logging:
  level: "info"
  format: "console"
  output_file: ""

Sources: configs/bootstrap.yaml:3-40

configs/node.yaml

This configuration is for a regular network node that joins an existing RQLite cluster, typically via a bootstrap node. Sources: configs/node.yaml:1-2

node:
  id: "node2"
  listen_addresses:
    - "/ip4/0.0.0.0/tcp/4002"
  data_dir: "./data/node2"
  max_connections: 50
  disable_anonrc: true

database:
  data_dir: "./data/db"
  replication_factor: 3
  shard_count: 16
  max_database_size: 1073741824
  backup_interval: 24h
  rqlite_port: 5002
  rqlite_raft_port: 7002
  rqlite_join_address: "http://127.0.0.1:5001"

discovery:
  bootstrap_peers:
    - "/ip4/127.0.0.1/tcp/4001/p2p/<YOUR_BOOTSTRAP_PEER_ID>"
  discovery_interval: 15s
  bootstrap_port: 4002
  http_adv_address: "127.0.0.1"
  raft_adv_address: ""

security:
  enable_tls: false
  private_key_file: ""
  certificate_file: ""

logging:
  level: "info"
  format: "console"
  output_file: ""

Sources: configs/node.yaml:3-40

Node Configuration Parameters

The primary node configuration parameters are structured under node, database, discovery, security, and logging sections. Sources: configs/bootstrap.yaml:3-40, configs/node.yaml:3-40

Section Parameter Description Default Value (Bootstrap) Default Value (Node) Source File
node id Unique identifier for the node. "" "node2" configs/bootstrap.yaml:4, configs/node.yaml:4
listen_addresses Multiaddrs for LibP2P to listen on. ["/ip4/0.0.0.0/tcp/4001"] ["/ip4/0.0.0.0/tcp/4002"] configs/bootstrap.yaml:5-6, configs/node.yaml:5-6
data_dir Directory for node-specific data. "./data/bootstrap" "./data/node2" configs/bootstrap.yaml:7, configs/node.yaml:7
max_connections Maximum number of P2P connections. 100 50 configs/bootstrap.yaml:8, configs/node.yaml:8
disable_anonrc Disables Anyone proxy routing. true true configs/bootstrap.yaml:9, configs/node.yaml:9
database data_dir Directory for RQLite database files. "./data/db" "./data/db" configs/bootstrap.yaml:12, configs/node.yaml:12
replication_factor Number of replicas for database shards. 3 3 configs/bootstrap.yaml:13, configs/node.yaml:13
shard_count Number of database shards. 16 16 configs/bootstrap.yaml:14, configs/node.yaml:14
max_database_size Maximum database size in bytes. 1GB 1GB configs/bootstrap.yaml:15, configs/node.yaml:15
backup_interval Interval for database backups. 24h 24h configs/bootstrap.yaml:16, configs/node.yaml:16
rqlite_port RQLite HTTP API port. 5001 5002 configs/bootstrap.yaml:17, configs/node.yaml:17
rqlite_raft_port RQLite Raft consensus port. 7001 7002 configs/bootstrap.yaml:18, configs/node.yaml:18
rqlite_join_address Address of an RQLite node to join. "" "http://127.0.0.1:5001" configs/bootstrap.yaml:19, configs/node.yaml:19
discovery bootstrap_peers List of multiaddrs for bootstrap peers. [] ["/ip4/127.0.0.1/tcp/4001/p2p/<YOUR_BOOTSTRAP_PEER_ID>"] configs/bootstrap.yaml:22, configs/node.yaml:22-23
discovery_interval Interval for peer discovery. 15s 15s configs/bootstrap.yaml:23, configs/node.yaml:24
bootstrap_port Bootstrap port for discovery. 4001 4002 configs/bootstrap.yaml:24, configs/node.yaml:25
http_adv_address Advertised HTTP address. "127.0.0.1" "127.0.0.1" configs/bootstrap.yaml:25, configs/node.yaml:26
raft_adv_address Advertised Raft address. "" "" configs/bootstrap.yaml:26, configs/node.yaml:27
security enable_tls Enables TLS for communication. false false configs/bootstrap.yaml:29, configs/node.yaml:30
private_key_file Path to TLS private key file. "" "" configs/bootstrap.yaml:30, configs/node.yaml:31
certificate_file Path to TLS certificate file. "" "" configs/bootstrap.yaml:31, configs/node.yaml:32
logging level Logging level (e.g., "info", "debug"). "info" "info" configs/bootstrap.yaml:34, configs/node.yaml:35
format Logging output format (e.g., "console", "json"). "console" "console" configs/bootstrap.yaml:35, configs/node.yaml:36
output_file Path to log output file. "" "" configs/bootstrap.yaml:36, configs/node.yaml:37

Node Configuration Loading and Overrides

The cmd/node/main.go file handles the loading of configuration, applying overrides from command-line flags. Sources: cmd/node/main.go:37-128

sequenceDiagram
    participant Main
    participant Logger
    participant FlagParser
    participant ConfigLoader
    participant Config
    participant Node

    Main->>Logger: setup_logger(ComponentNode)
    Main->>FlagParser: parse_and_return_network_flags()
    activate FlagParser
    FlagParser->>FlagParser: Parse CLI flags
    alt configPath is provided
        FlagParser->>ConfigLoader: LoadConfigFromYAML(configPath)
        activate ConfigLoader
        ConfigLoader-->>FlagParser: cfg
        deactivate ConfigLoader
        FlagParser-->>Main: cfg values
    else no configPath
        FlagParser-->>Main: flag values
    end
    deactivate FlagParser

    Main->>Main: disable_anon_proxy(disableAnon)
    Main->>Main: check_if_should_open_help(help)
    Main->>Main: select_data_dir(dataDir, nodeID)

    Main->>Config: DefaultConfig()
    Main->>Main: load_args_into_config(cfg, ...)
    activate Main
    Main->>Config: Apply overrides (p2pPort, rqlHTTP, rqlRaft, rqlJoinAddr, advAddr, dataDir)
    deactivate Main

    Main->>Node: NewNode(cfg)
    activate Node
    Node-->>Main: n
    deactivate Node
    Main->>Node: Start(ctx)
    activate Node
    Node-->>Main: err (if any)
    deactivate Node
    Main->>Main: Handle graceful shutdown (<-ctx.Done())
    Main->>Node: Stop()

Sources: cmd/node/main.go:37-128

The LoadConfigFromYAML function reads and unmarshals the YAML content into a config.Config struct. Sources: cmd/node/main.go:107-117

// LoadConfigFromYAML loads a config from a YAML file
func LoadConfigFromYAML(path string) (*config.Config, error) {
	data, err := ioutil.ReadFile(path)
	if err != nil {
		return nil, fmt.Errorf("failed to read config file: %w", err)
	}
	var cfg config.Config
	if err := yaml.Unmarshal(data, &cfg); err != nil {
		return nil, fmt.Errorf("failed to unmarshal YAML: %w", err)
	}
	return &cfg, nil
}

Sources: cmd/node/main.go:109-117

Command-line arguments are parsed and then explicitly loaded into the config.Config struct, overriding any values from the YAML file or defaults. For example, the P2P listen address is constructed using the --p2p-port flag. Sources: cmd/node/main.go:203-236

// load_args_into_config applies command line argument overrides to the config
func load_args_into_config(cfg *config.Config, p2pPort, rqlHTTP, rqlRaft *int, rqlJoinAddr *string, advAddr *string, dataDir *string) {
	logger := setup_logger(logging.ComponentNode)

	// Apply RQLite HTTP port override
	if *rqlHTTP != 5001 {
		cfg.Database.RQLitePort = *rqlHTTP
		logger.ComponentInfo(logging.ComponentNode, "Overriding RQLite HTTP port", zap.Int("port", *rqlHTTP))
	}

	// Apply RQLite Raft port override
	if *rqlRaft != 7001 {
		cfg.Database.RQLiteRaftPort = *rqlRaft
		logger.ComponentInfo(logging.ComponentNode, "Overriding RQLite Raft port", zap.Int("port", *rqlRaft))
	}

	// Apply P2P port override
	if *p2pPort != 4001 {
		cfg.Node.ListenAddresses = []string{
			fmt.Sprintf("/ip4/0.0.0.0/tcp/%d", *p2pPort),
		}
		logger.ComponentInfo(logging.ComponentNode, "Overriding P2P port", zap.Int("port", *p2pPort))
	}

	// Apply RQLite join address
	if *rqlJoinAddr != "" {
		cfg.Database.RQLiteJoinAddress = *rqlJoinAddr
		logger.ComponentInfo(logging.ComponentNode, "Setting RQLite join address", zap.String("address", *rqlJoinAddr))
	}

	if *advAddr != "" {
		cfg.Discovery.HttpAdvAddress = fmt.Sprintf("%s:%d", *advAddr, *rqlHTTP)
		cfg.Discovery.RaftAdvAddress = fmt.Sprintf("%s:%d", *advAddr, *rqlRaft)
	}

	if *dataDir != "" {
		cfg.Node.DataDir = *dataDir
	}
}

Sources: cmd/node/main.go:203-236

Gateway Configuration

The HTTP Gateway component also has its own configuration, primarily handling its listen address, client namespace, and bootstrap peers. This configuration prioritizes command-line flags over environment variables, then defaults. Sources: cmd/gateway/config.go:1-2

// parseGatewayConfig parses flags and environment variables into GatewayConfig.
// Priority: flags > env > defaults.
func parseGatewayConfig(logger *logging.ColoredLogger) *gateway.Config {
	addr := flag.String("addr", getEnvDefault("GATEWAY_ADDR", ":8080"), "HTTP listen address (e.g., :8080)")
	ns := flag.String("namespace", getEnvDefault("GATEWAY_NAMESPACE", "default"), "Client namespace for scoping resources")
	peers := flag.String("bootstrap-peers", getEnvDefault("GATEWAY_BOOTSTRAP_PEERS", ""), "Comma-separated bootstrap peers for network client")

	// Do not call flag.Parse() elsewhere to avoid double-parsing
	flag.Parse()

	var bootstrap []string
	if p := strings.TrimSpace(*peers); p != "" {
		parts := strings.Split(p, ",")
		for _, part := range parts {
			val := strings.TrimSpace(part)
			if val != "" {
				bootstrap = append(bootstrap, val)
			}
		}
	}

	logger.ComponentInfo(logging.ComponentGeneral, "Loaded gateway configuration",
		zap.String("addr", *addr),
		zap.String("namespace", *ns),
		zap.Int("bootstrap_peer_count", len(bootstrap)),
	)

	return &gateway.Config{
		ListenAddr:      *addr,
		ClientNamespace: *ns,
		BootstrapPeers:  bootstrap,
	}
}

Sources: cmd/gateway/config.go:21-52

Client Configuration

The pkg/client package provides a DefaultClientConfig function, which sets up a basic configuration for a network client, including a default namespace and a placeholder for bootstrap peers. This configuration can then be customized programmatically. Sources: examples/basic_usage.go:12-16

// Create client configuration
config := client.DefaultClientConfig("example_app")
config.BootstrapPeers = []string{
	"/ip4/127.0.0.1/tcp/4001/p2p/QmBootstrap1",
}

Sources: examples/basic_usage.go:12-16

Conclusion

The DeBros Network's configuration system is designed for flexibility and ease of management, supporting YAML files for structured definitions, environment variables for deployment-specific adjustments, and command-line flags for immediate overrides. This layered approach ensures that nodes and services can be configured precisely for their roles within the decentralized network, from initial setup to ongoing operations. Sources: README.md:95-97, AI_CONTEXT.md:35-37

Database Layer

Related Pages

Related topics: CLI Database Operations, Gateway API Endpoints

Relevant source files

Database Layer

The Database Layer in the DeBros Network Cluster provides distributed SQL database capabilities, primarily backed by RQLite. This layer ensures resilient, scalable, and secure data management through features like Raft consensus, ACID transactions, and automatic failover. It is a core component enabling applications to store and retrieve structured data across the decentralized network. The database functionality is accessible via a client API for direct integration and through an HTTP Gateway for broader accessibility. Sources: README.md:1-3, AI_CONTEXT.md:38-41

Architecture and Components

The DeBros Network's database layer leverages RQLite, which is a distributed SQLite database. RQLite uses the Raft consensus protocol to ensure data consistency and fault tolerance across the network. Each full node in the DeBros Network can host a part of this distributed database, with replication and sharding configured for robustness and scalability. Sources: README.md:23, AI_CONTEXT.md:38-41, AI_CONTEXT.md:118-120

The key components involved in the database layer include:

  • RQLite: The underlying distributed SQLite database.
  • DeBros Network Node: Full P2P participants that run RQLite instances and handle peer discovery.
  • Client API: A Go client library (pkg/client) for direct programmatic interaction.
  • HTTP Gateway: Provides RESTful endpoints for database operations, acting as a facade for clients that cannot use the Go API directly.
  • CLI Tools: Command-line interface for interacting with the database, primarily for queries.

Data Flow for Database Operations

Database operations initiated by a client or CLI tool are routed through either the Client API or the HTTP Gateway to the DeBros Network nodes. These nodes then interact with their local RQLite instances, which coordinate using Raft to ensure the operation is applied consistently across the distributed database.

graph TD
    A[Client/CLI] -->|1. Request DB Operation| B{Client API / HTTP Gateway}
    B -->|2. Forward Request| C[DeBros Network Node]
    C -->|3. Execute on RQLite| D[RQLite Instance]
    D -->|4. Raft Consensus| E[Other RQLite Instances]
    E -->|5. Acknowledge/Replicate| D
    D -->|6. Return Result| C
    C -->|7. Return Result| B
    B -->|8. Respond to Client| A

Sources: AI_CONTEXT.md:118-120, pkg/gateway/storage_handlers.go:11-17

Database Operations via Client API

The pkg/client package provides a Go interface for interacting with the distributed database. This API allows applications to perform standard SQL operations like creating tables, executing queries, and running transactions. Sources: AI_CONTEXT.md:67-70, examples/basic_usage.go:50-78

Key Functions

Function Name Description Source File
Query(ctx, sql, args...) Executes a SQL query or DML statement and returns results. AI_CONTEXT.md:67, examples/basic_usage.go:66
CreateTable(ctx, schema) Creates a new table using a SQL DDL schema. AI_CONTEXT.md:68, examples/basic_usage.go:58
Transaction(ctx, statements) Executes multiple SQL statements atomically within a transaction. AI_CONTEXT.md:223, e2e/client_e2e_test.go:61
GetSchema(ctx) Retrieves the schema of the database. pkg/gateway/storage_handlers.go:42
DropTable(ctx, table) Drops an existing table. AI_CONTEXT.md:220
Sources: AI_CONTEXT.md:67-70, AI_CONTEXT.md:220-223, examples/basic_usage.go:50-78, pkg/gateway/storage_handlers.go:42

Example Go Client Usage

// examples/basic_usage.go
func demonstrateDatabase(client client.NetworkClient) {
	ctx := context.Background()
	db := client.Database()

	log.Printf("=== Database Operations ===")

	// Create a table
	schema := `
		CREATE TABLE IF NOT EXISTS messages (
			id INTEGER PRIMARY KEY,
			content TEXT NOT NULL,
			timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
		)
	`
	if err := db.CreateTable(ctx, schema); err != nil {
		log.Printf("Error creating table: %v", err)
		return
	}
	log.Printf("Table created successfully")

	// Insert some data
	insertSQL := "INSERT INTO messages (content) VALUES (?)"
	result, err := db.Query(ctx, insertSQL, "Hello, distributed world!")
	if err != nil {
		log.Printf("Error inserting data: %v", err)
		return
	}
	log.Printf("Data inserted, result: %+v", result)

	// Query data
	selectSQL := "SELECT * FROM messages"
	result, err = db.Query(ctx, selectSQL)
	if err != nil {
		log.Printf("Error querying data: %v", err)
		return
	}
	log.Printf("Query result: %+v", result)
}

Sources: examples/basic_usage.go:50-78

HTTP Gateway Database Endpoints

The HTTP Gateway provides RESTful endpoints for database operations, allowing clients in various languages to interact with the DeBros Network's database layer. All endpoints are versioned under /v1/db/ and enforce authentication and namespace ownership. Sources: AI_CONTEXT.md:219-232, README.md:104-111

HTTP Endpoints Summary

Method Path Description Request Body (JSON) Response Body (JSON) Source File
POST /v1/db/create-table Creates a table with SQL DDL. {"schema":"CREATE TABLE ..."} {"status":"ok"} AI_CONTEXT.md:220, pkg/gateway/storage_handlers.go:52
POST /v1/db/drop-table Drops a table. {"table":"table_name"} {"status":"ok"} AI_CONTEXT.md:221
POST /v1/db/query Executes a single SQL query. {"sql":"SELECT...", "args":[]} {"columns":[], "rows":[], "count":0} AI_CONTEXT.md:222, pkg/gateway/storage_handlers.go:14
POST /v1/db/transaction Applies multiple statements atomically. {"statements":["stmt1", "stmt2"]} {"status":"ok"} AI_CONTEXT.md:223, pkg/gateway/storage_handlers.go:27
GET /v1/db/schema Returns the database schema. None Schema JSON AI_CONTEXT.md:224, pkg/gateway/storage_handlers.go:45
Sources: AI_CONTEXT.md:219-224, README.md:104-111, pkg/gateway/storage_handlers.go:11-54

Gateway Request Handling Flow for Queries

sequenceDiagram
    participant C as Client
    participant GW as HTTP Gateway
    participant NC as NetworkClient
    participant DB as DatabaseService
    participant RQL as RQLite

    C->>GW: POST /v1/db/query {sql, args}
    activate GW
    GW->>GW: Decode Request Body
    alt Client not initialized
        GW--x C: 503 Service Unavailable
    end
    GW->>NC: Query(ctx, sql, args)
    activate NC
    NC->>DB: Query(ctx, sql, args)
    activate DB
    DB->>RQL: Execute SQL
    activate RQL
    RQL-->>DB: Query Result
    deactivate RQL
    DB-->>NC: Query Result
    deactivate DB
    NC-->>GW: Query Result
    deactivate NC
    GW->>GW: Encode Response
    GW-->>C: 200 OK {columns, rows, count}
    deactivate GW

Sources: pkg/gateway/storage_handlers.go:11-20

Database Migrations

Database migrations are managed by applying SQL DDL statements to the distributed database. The system supports standard SQL migration files (e.g., .sql extension in a migrations/ directory). Each migration runs in a transaction to ensure atomicity and data integrity. Sources: AI_CONTEXT.md:44-53, AI_CONTEXT.md:228-229, README.md:113-116

Migration File Structure

migrations/
├── 001_initial_schema.sql     # Initial database setup
├── 002_add_auth_tables.sql    # Authentication tables
├── 003_add_indexes.sql        # Performance indexes
└── ...                        # Additional migrations

Sources: AI_CONTEXT.md:55-60

Performing Migrations

Migrations are typically performed by sending a POST request to the /v1/db/transaction endpoint with an array of SQL statements. This ensures that all changes for a migration are applied atomically. Sources: AI_CONTEXT.md:223, AI_CONTEXT.md:228-229, README.md:116

// e2e/gateway_e2e_test.go
// Example of a migration transaction
stmts := []string{
    fmt.Sprintf("ALTER TABLE %s ADD COLUMN age INTEGER", users),
    fmt.Sprintf("UPDATE %s SET age = 0 WHERE age IS NULL", users),
}
migBody := fmt.Sprintf(`{"statements":%s}`, toJSON(stmts))
req, _ = http.NewRequest(http.MethodPost, base+"/v1/db/transaction", strings.NewReader(migBody))
req.Header = authHeader(key)
resp, err = httpClient().Do(req)

Sources: e2e/gateway_e2e_test.go:25-33

Troubleshooting Migrations

Common issues include SQL syntax errors, improper statement termination, and version conflicts. Debugging involves checking SQL syntax, ensuring sequential order of migration files, and reviewing migration logs. Sources: AI_CONTEXT.md:104-108, README.md:175-178

Configuration

The database layer's behavior can be configured via YAML files, flags, or environment variables. Key configuration parameters include data directories, replication factor, sharding, and RQLite specific ports. Sources: AI_CONTEXT.md:139-140, README.md:226-240

Example Configuration (configs/bootstrap.yaml)

database:
  data_dir: "./data/db"
  replication_factor: 3
  shard_count: 16
  max_database_size: 1073741824
  backup_interval: 24h
  rqlite_port: 5001
  rqlite_raft_port: 7001
  rqlite_join_address: "" # Bootstrap node does not join

Sources: README.md:231-240

Parameter Type Description Default Value Source File
data_dir string Directory for RQLite database files. ./data/db README.md:232
replication_factor int Number of copies of data maintained across nodes. 3 README.md:233
shard_count int Number of database shards. 16 README.md:234
max_database_size int Maximum size of the database in bytes. 1GB README.md:235
backup_interval string Interval for automatic database backups. 24h README.md:236
rqlite_port int HTTP API port for RQLite. 5001 README.md:237
rqlite_raft_port int Raft consensus port for RQLite. 7001 README.md:238
rqlite_join_address string Address of an existing RQLite node to join the cluster. "" README.md:239
Sources: README.md:231-240

Conclusion

The Database Layer of the DeBros Network, powered by RQLite and Raft consensus, forms the backbone for distributed data storage. It offers robust SQL capabilities through both a native Go client API and a flexible HTTP Gateway, catering to diverse application needs. The system's design emphasizes atomicity, data integrity, and ease of use, with built-in mechanisms for migrations and comprehensive configuration options to support scalable and resilient decentralized applications.

Storage System

Related Pages

Related topics: CLI Storage Operations, Gateway API Endpoints

Relevant source files

Storage System

The DeBros Network's Storage System provides a decentralized, key-value store designed for resilient and scalable data management. It offers basic CRUD (Create, Read, Update, Delete) operations, along with prefix-based listing and existence checks. The storage is namespaced, ensuring application isolation and multi-tenancy. This system is a core component of the DeBros Network Cluster, enabling distributed data persistence alongside the distributed SQL database and Pub/Sub messaging. Sources: README.md:22-23, AI_CONTEXT.md:20-21

Architecture and Components

The Storage System is implemented as a service within the DeBros Network's full nodes. Clients interact with this service through a dedicated API, either directly via the Go client library or indirectly via the HTTP Gateway. The underlying storage mechanism is replicated and ensures data availability across the network. Namespace enforcement is a key aspect, automatically prefixing operations to isolate data per application. Sources: AI_CONTEXT.md:124-125, README.md:22-23

Data Flow for Storage Operations

The following diagram illustrates the typical data flow for a storage operation, such as putting a key-value pair.

graph TD
    A[Client Application] -->|1. Call Storage API (e.g., Put)| B[DeBros Network Client Library];
    B -->|2. Network Request| C[DeBros Network Node/Gateway];
    C -->|3. Authenticate & Authorize| D{Authentication/Authorization Middleware};
    D -->|4. Enforce Namespace| E[Storage Service];
    E -->|5. Store Data (Replicated)| F[Underlying Distributed Storage];
    F -->|6. Acknowledge Storage| E;
    E -->|7. Return Success| C;
    C -->|8. Respond to Client| B;
    B -->|9. Return Result| A;

Sources: AI_CONTEXT.md:124-125, README.md:22-23

Key-Value Operations

The Storage System supports standard key-value operations, including Put, Get, Delete, List, and Exists. These operations are accessible via the client API and the HTTP Gateway. Sources: README.md:23

API Endpoints (HTTP Gateway)

The HTTP Gateway exposes RESTful endpoints for interacting with the Storage System. All operations are under the /v1/storage path. Sources: AI_CONTEXT.md:166-169, README.md:104-108

Endpoint Method Description Request Body/Query Response Source
/v1/storage/put POST Stores data associated with a key. body: raw bytes (key in query param ?key=<k>) or {"key":"<key>", "value":"<base64_encoded_value>"} {status:"ok"} AI_CONTEXT.md:166, README.md:104
/v1/storage/get GET Retrieves data for a given key. query: ?key=<k> raw bytes (may be base64-encoded by some backends) AI_CONTEXT.md:167, README.md:105
/v1/storage/delete POST Deletes data for a given key. {"key":"<key>"} {status:"ok"} AI_CONTEXT.md:168, README.md:108
/v1/storage/list GET Lists keys matching an optional prefix. query: ?prefix=<p> {"keys":["<key1>", "..."]} AI_CONTEXT.md:169, README.md:107
/v1/storage/exists GET Checks if a key exists. query: ?key=<k> {"exists":boolean} AI_CONTEXT.md:169, README.md:106

Client Library Usage (Go)

The pkg/client package provides a Go client for programmatic interaction with the Storage System. Sources: examples/basic_usage.go:19-21

func demonstrateStorage(client client.NetworkClient) {
	ctx := context.Background()
	storage := client.Storage()

	log.Printf("=== Storage Operations ===")

	// Store some data
	key := "user:123"
	value := []byte(`{"name": "Alice", "age": 30}`)

	if err := storage.Put(ctx, key, value); err != nil {
		log.Printf("Error storing data: %v", err)
		return
	}
	log.Printf("Data stored successfully")

	// Retrieve data
	retrieved, err := storage.Get(ctx, key)
	if err != nil {
		log.Printf("Error retrieving data: %v", err)
		return
	}
	log.Printf("Retrieved data: %s", string(retrieved))

	// Check if key exists
	exists, err := storage.Exists(ctx, key)
	if err != nil {
		log.Printf("Error checking existence: %v", err)
		return
	}
	log.Printf("Key exists: %v", exists)

	// List keys
	keys, err := storage.List(ctx, "user:", 10) // The 'limit' parameter (10) is not reflected in the docs
	if err != nil {
		log.Printf("Error listing keys: %v", err)
		return
	}
	log.Printf("Keys: %v", keys)
}

Sources: examples/basic_usage.go:70-109

Client Library Usage (TypeScript SDK)

A TypeScript SDK client is available for web and Node.js applications, mirroring the Go client's functionality. Sources: examples/sdk-typescript/src/client.ts:25-63

// Storage
async put(key: string, value: Uint8Array | string): Promise<void> {
  const body = typeof value === 'string' ? new TextEncoder().encode(value) : value;
  const r = await this.http(`${this.baseUrl}/v1/storage/put?key=${encodeURIComponent(key)}`, {
    method: 'POST', headers: { 'X-API-Key': this.apiKey }, body
  });
  if (!r.ok) throw new Error(`put failed: ${r.status}`);
}

async get(key: string): Promise<Uint8Array> {
  const r = await this.http(`${this.baseUrl}/v1/storage/get?key=${encodeURIComponent(key)}`, {
    headers: { 'X-API-Key': this.apiKey }
  });
  if (!r.ok) throw new Error(`get failed: ${r.status}`);
  const buf = new Uint8Array(await r.arrayBuffer());
  return buf;
}

async exists(key: string): Promise<boolean> {
  const r = await this.http(`${this.baseUrl}/v1/storage/exists?key=${encodeURIComponent(key)}`, {
    headers: this.headers(false)
  });
  if (!r.ok) throw new Error(`exists failed: ${r.status}`);
  const j = await r.json();
  return !!j.exists;
}

async list(prefix = ""): Promise<string[]> {
  const r = await this.http(`${this.baseUrl}/v1/storage/list?prefix=${encodeURIComponent(prefix)}`, {
    headers: this.headers(false)
  });
  if (!r.ok) throw new Error(`list failed: ${r.status}`);
  const j = await r.json();
  return j.keys || [];
}

async delete(key: string): Promise<void> {
  const r = await this.http(`${this.baseUrl}/v1/storage/delete`, {
    method: 'POST', headers: this.headers(), body: JSON.stringify({ key })
  });
  if (!r.ok) throw new Error(`delete failed: ${r.status}`);
}

Sources: examples/sdk-typescript/src/client.ts:25-63

CLI Usage

The network-cli tool provides command-line access to storage operations. Sources: cmd/cli/main.go:60-61

# Store data
network-cli storage put test-key "Hello Network"
# Retrieve data
network-cli storage get test-key

Sources: README.md:65-66

Security and Isolation

The Storage System incorporates several security features:

  • Namespace Enforcement: All operations are automatically prefixed with a namespace, ensuring data isolation between different applications or tenants. This is enforced by middleware for specific routes. Sources: README.md:139-140, AI_CONTEXT.md:124-125
  • Authentication Middleware: Flexible authentication mechanisms, including JWT and API keys, are used to secure access to storage endpoints. Sources: README.md:142
  • Transport Security: All network communications leverage Noise/TLS encryption for secure data transfer. Sources: README.md:141

Conclusion

The DeBros Network's Storage System offers a robust, decentralized, and secure key-value store. Its namespaced design, comprehensive API, and integration with the network's security features make it suitable for distributed applications requiring resilient data persistence and isolation.

Pub/Sub System

Related Pages

Related topics: CLI Pub/Sub Operations, Gateway API Endpoints

Relevant source files

Pub/Sub System

The DeBros Network Cluster incorporates a robust Pub/Sub (Publish/Subscribe) messaging system designed for real-time, topic-based communication. This system enables different components or applications within the network to exchange messages asynchronously without direct coupling, facilitating resilient and scalable data flow. It supports both WebSocket and REST interfaces for publishing and subscribing to topics. The Pub/Sub system is namespaced, ensuring application isolation and preventing topic name collisions. Sources: README.md:15-16, AI_CONTEXT.md:88-89

Architecture and Components

The Pub/Sub system is an integral part of the DeBros Network's client API, allowing applications to interact with it through a lightweight Go client or an HTTP Gateway. Messages are delivered over the underlying LibP2P network. Sources: README.md:46-55, AI_CONTEXT.md:57-66

graph TD
    A[Application] -->|Client API| B(DeBros Network Client)
    B --> C[PubSub Service]
    C --> D[LibP2P Host]
    D --> E[Noise/TLS Encryption]
    E --> F[Network Peers]
    F --> G[Other Applications]
    G -->|Client API| H(DeBros Network Client)
    H --> I[PubSub Service]
    I --> J[LibP2P Host]
    J --> K[Noise/TLS Encryption]

Sources: README.md:46-55

Key Features

The Pub/Sub system offers several key features: Sources: README.md:15-16

Feature Description
Topic-based Messages are categorized and delivered based on specific topics.
Real-time Designed for low-latency message delivery.
Namespaced Topics are prefixed internally with a namespace (ns::<namespace>::...) for multi-tenancy and isolation.
Automatic Cleanup Implies management of subscriptions and topics.
WebSocket Support Provides a persistent connection for real-time message streaming.
REST Interface Allows publishing messages and listing topics via HTTP endpoints.
Sources: README.md:15-16, AI_CONTEXT.md:176-177, AI_CONTEXT.md:195-197, AI_CONTEXT.md:214-216, AI_CONTEXT.md:220-222

Data Flow and Operations

The Pub/Sub system supports both publishing messages to topics and subscribing to receive messages from topics. These operations can be performed via the Go client library or through the HTTP Gateway. Sources: AI_CONTEXT.md:176-177, AI_CONTEXT.md:195-197, AI_CONTEXT.md:214-216, AI_CONTEXT.md:220-222

Publishing Messages

Messages can be published using either the Go client or the HTTP Gateway's REST endpoint.

Go Client Publishing

Applications using the Go client can publish messages by calling the Publish method on the PubSub client. Sources: examples/basic_usage.go:134-137

err := client.PubSub().Publish(ctx, "topic", []byte("msg"))

Sources: AI_CONTEXT.md:27

HTTP Gateway REST Publishing

The Gateway provides a POST /v1/pubsub/publish endpoint for publishing messages. The message data must be base64 encoded. Sources: AI_CONTEXT.md:195-197, pkg/gateway/pubsub_handlers.go:61-64

POST /v1/pubsub/publish → body {topic, data_base64} → {status:"ok"}

Sources: AI_CONTEXT.md:195-197

sequenceDiagram
    participant App as Application
    participant GW as HTTP Gateway
    participant PS as PubSub Service
    participant Net as DeBros Network

    App->>GW: POST /v1/pubsub/publish {topic, data_base64}
    activate GW
    GW->>PS: PubSub().Publish(ctx, topic, data)
    activate PS
    PS->>Net: Send message to network
    deactivate PS
    GW-->>App: HTTP 200 OK {status:"ok"}
    deactivate GW

Sources: pkg/gateway/pubsub_handlers.go:61-91

Subscribing to Messages

Subscriptions are primarily handled via the Go client or through a WebSocket connection to the HTTP Gateway.

Go Client Subscription

The Go client allows applications to subscribe to topics by providing a handler function that processes incoming messages. Sources: examples/basic_usage.go:125-132

handler := func(topic string, data []byte) error {
    log.Printf("Received message on topic '%s': %s", topic, string(data))
    return nil
}
if err := pubsub.Subscribe(ctx, topic, handler); err != nil {
    log.Printf("Error subscribing: %v", err)
    return
}

Sources: examples/basic_usage.go:125-132

HTTP Gateway WebSocket Subscription

The Gateway offers a WebSocket endpoint GET /v1/pubsub/ws?topic=<topic> for real-time message reception. Clients connect to this endpoint, and the server sends messages as binary frames. Client text/binary frames sent over the WebSocket are published to the same namespaced topic. Sources: AI_CONTEXT.md:191-193, pkg/gateway/pubsub_handlers.go:15-32, examples/sdk-typescript/src/client.ts:83-87

GET /v1/pubsub/ws?topic=<topic>

Sources: AI_CONTEXT.md:191-193

sequenceDiagram
    participant Client as WebSocket Client
    participant GW as HTTP Gateway
    participant PS as PubSub Service
    participant Net as DeBros Network

    Client->>GW: GET /v1/pubsub/ws?topic=<topic>
    activate GW
    GW->>PS: PubSub().Subscribe(ctx, topic, handler)
    activate PS
    PS->>Net: Register subscription
    deactivate PS
    GW-->>Client: WebSocket connection established
    loop Message Flow
        Net->>PS: Message received for topic
        activate PS
        PS->>GW: Forward message to handler
        deactivate PS
        GW-->>Client: WebSocket binary frame (message)
    end
    Client->>GW: WebSocket text/binary frame (publish)
    activate GW
    GW->>PS: PubSub().Publish(ctx, topic, data)
    activate PS
    PS->>Net: Send message to network
    deactivate PS

Sources: pkg/gateway/pubsub_handlers.go:15-32

Listing Topics

Both the Go client and the HTTP Gateway provide functionality to list active topics. The Gateway's GET /v1/pubsub/topics endpoint returns topics trimmed to the caller's namespace. Sources: AI_CONTEXT.md:198-199, pkg/gateway/pubsub_handlers.go:94-110

GET /v1/pubsub/topics → {topics:["<topic>", ...]}

Sources: AI_CONTEXT.md:198-199

Namespace Enforcement

The Pub/Sub system strictly enforces namespaces. All topics are internally prefixed with ns::<namespace>::... to ensure isolation between different applications or tenants. This means that a client operating within a specific namespace will only interact with topics belonging to that namespace. The resolveNamespaceFromRequest function extracts the namespace from the request context, typically set by an authentication middleware. Sources: AI_CONTEXT.md:176-177, pkg/gateway/pubsub_handlers.go:112-124

func namespacePrefix(ns string) string {
	return "ns::" + ns + "::"
}

func namespacedTopic(ns, topic string) string {
	return namespacePrefix(ns) + topic
}

Sources: pkg/gateway/pubsub_handlers.go:126-133

Troubleshooting Pub/Sub

Common issues related to message delivery failures include incorrect topic names, inactive subscriptions, or general network connectivity problems. Sources: AI_CONTEXT.md:38-39, AI_CONTEXT.md:209-210

Symptom Solution
Messages not received Verify topic names, active subscriptions, and network connectivity.
High Memory Usage Unsubscribe from topics when no longer needed; monitor connection pool size.
Sources: AI_CONTEXT.md:38-39, AI_CONTEXT.md:209-210

Conclusion

The DeBros Network's Pub/Sub system provides a fundamental capability for real-time, decentralized communication. Its topic-based, namespaced design, coupled with flexible API access via Go clients and HTTP Gateway endpoints (including WebSockets), makes it suitable for building responsive and isolated distributed applications. Sources: README.md:15-16, AI_CONTEXT.md:88-89

Discovery Component

Related Pages

Related topics: Node Component

Error with OpenRouter API: argument of type 'NoneType' is not iterable

Please check that you have set the OPENROUTER_API_KEY environment variable with a valid API key.

CLI Overview

Related Pages

Related topics: Quick Start Guide

Timeout while calling OpenRouter API (120s). The service may be slow or unreachable.

CLI Network Operations

Related Pages

Related topics: CLI Overview

Relevant source files

The following files were used as context for generating this wiki page:

CLI Network Operations

The Command Line Interface (CLI) for the DeBros Network provides a direct and efficient way to interact with the decentralized P2P system. It allows users to perform various operations, including network health checks, peer management, distributed SQL queries, key-value storage operations, and Pub/Sub messaging. The CLI is designed to be a lightweight client, connecting to bootstrap peers to consume network services without participating in peer discovery or running full node services. It supports automatic wallet authentication for operations requiring credentials, enhancing user experience. Sources: README.md:23-24, cmd/cli/main.go:37-40, AI_CONTEXT.md:65-67

Architecture and Components

The CLI is implemented as a standalone executable (network-cli) that communicates with the DeBros Network. It leverages the pkg/client library to establish connections and interact with the network's core services. Global flags allow for configuration of bootstrap peers, output format, operation timeouts, and the disabling of anonymous routing. Sources: cmd/cli/main.go:37-40, cmd/cli/main.go:46-51, cmd/cli/main.go:94-107

The CLI's operational flow involves parsing commands and arguments, creating a network client instance, executing the requested operation, and then formatting and printing the results.

graph TD
    A[User] --> B[Execute network-cli command]
    B --> C{Parse Command & Global Flags}
    C --> D[Initialize Client Configuration]
    D --> E[Create Network Client]
    E --> F{Connect to Network}
    F -- On Success --> G{Perform Requested Operation}
    G -- Query/Storage/PubSub --> H[Interact with Network Services]
    H -- On Success --> I[Process & Print Results]
    F -- On Failure --> J[Handle Error & Exit]
    G -- On Failure --> J

Sources: cmd/cli/main.go:53-91, cmd/cli/main.go:94-107

Key Features and Commands

The network-cli supports a range of commands for interacting with the DeBros Network.

General Commands

Command Description
version Displays the CLI version, commit hash, and build date.
health Checks the overall health status of the connected network.
peers Lists the peers currently connected to the network.
status Retrieves detailed status information about the network.
connect Attempts to connect to a specific peer address.
peer-id Displays the local peer ID.
help Shows the command-line help message, listing available commands and global flags.
Sources: cmd/cli/main.go:61-91

Database Operations

The CLI allows direct interaction with the distributed SQL database, backed by RQLite. The query command is used to execute SQL statements. Operations requiring authentication, such as query, will trigger an automatic authentication flow if not already authenticated. Sources: README.md:46-47, cmd/cli/main.go:179-199

./bin/network-cli query "CREATE TABLE users (id INTEGER)"
./bin/network-cli query "SELECT name FROM users WHERE id = ?"

Sources: README.md:46-47, AI_CONTEXT.md:364

Storage Operations

Key-value storage operations are supported for managing data within the network. The storage command provides sub-commands for put, get, and list operations. Sources: cmd/cli/main.go:201-204

Operation Usage Description
put network-cli storage put <key> <value> Stores a key-value pair.
get network-cli storage get <key> Retrieves the value associated with a key.
list network-cli storage list [prefix] [limit] Lists keys, optionally filtered by a prefix.
Sources: cmd/cli/main.go:201-204

Example:

./bin/network-cli storage put test-key "Hello Network"
./bin/network-cli storage get test-key

Sources: README.md:48-49

Pub/Sub Messaging

The CLI facilitates real-time messaging through Pub/Sub functionality. The pubsub command supports publish, subscribe, and topics operations. Sources: cmd/cli/main.go:206-209

Operation Usage Description
publish network-cli pubsub publish <topic> <message> Publishes a message to a specified topic.
subscribe network-cli pubsub subscribe <topic> [duration] Subscribes to a topic to receive messages.
topics network-cli pubsub topics Lists all currently active Pub/Sub topics.
Sources: cmd/cli/main.go:206-209

Example:

./bin/network-cli pubsub publish notifications "Hello World"
./bin/network-cli pubsub subscribe notifications 10s

Sources: README.md:50-51

CLI Options

Global flags can be used to modify the behavior of CLI commands. These flags are parsed early in the execution flow.

| Flag | Description
A[Client] --> B[Connect to Network] B --> C[Establish Connection to Bootstrap Peers] C --> D[Query] D[Query] -- SQL Query --> E[RQLite Database] E[RQLite Database] -- Query Result --> D A[Client] --> F[Storage] F[Storage] -- Put/Get/List --> G[Key-Value Storage Service] G[Key-Value Storage Service] -- Data --> F A[Client] --> H[Pub/Sub] H[Pub/Sub] -- Publish/Subscribe --> I[Pub/Sub Messaging Service] I[Pub/Sub Messaging Service] -- Message --> H

Sources: [examples/basic_usage.go:19-24](), [examples/basic_usage.go:34-36](), [examples/basic_usage.go:62-63](), [examples/basic_usage.go:91-92](), [examples/basic_usage.go:127-128]()

## Authentication

The CLI incorporates an enhanced authentication system, particularly for operations requiring user credentials (e.g., storage, database, pubsub). This system features automatic wallet detection, multi-wallet management, and persistent sessions. When an authenticated operation is invoked, the CLI will:
1. Check for existing valid credentials.
2. Prompt for wallet authentication if necessary.
3. Handle signature verification.
4. Persist credentials for future use.
Sources: [README.md:299-315](), [AI_CONTEXT.md:278-294]()

This streamlines the authentication process, making it largely automatic for the user after the initial setup.

```mermaid
sequenceDiagram
    participant U as User
    participant C as CLI
    participant GW as Gateway
    participant W as Wallet

    U->>C: Execute authenticated command (e.g., storage put)
    activate C
    C->>C: Check for existing credentials
    alt No valid credentials
        C->>GW: Request challenge (POST /v1/auth/challenge)
        activate GW
        GW-->>C: Returns {nonce}
        deactivate GW
        C->>W: Prompt to sign nonce with wallet
        activate W
        W-->>C: Returns signature (r||s||v hex)
        deactivate W
        C->>GW: Verify signature (POST /v1/auth/verify)
        activate GW
        GW-->>C: Returns JWT token
        deactivate GW
        C->>C: Save credentials
    end
    C->>GW: Execute requested operation with JWT
    activate GW
    GW-->>C: Operation result
    deactivate GW
    C-->>U: Display result
    deactivate C

Sources: README.md:299-315, AI_CONTEXT.md:278-294

Development and Building

To build the network-cli executable, the make build command can be used, which compiles all executables in the project. The CLI's version information (version, commit, date) is populated at build time using -ldflags. Sources: CONTRIBUTING.md:20, cmd/cli/main.go:27-31, AI_CONTEXT.md:347

make build

Sources: CONTRIBUTING.md:20

Conclusion

The DeBros Network CLI serves as a crucial interface for users to interact with the decentralized network's core services. Its design emphasizes ease of use through streamlined commands and automatic authentication, while providing comprehensive access to distributed database, storage, and messaging capabilities.

CLI Storage Operations

Related Pages

Related topics: CLI Overview, Storage System

Relevant source files

CLI Storage Operations

The DeBros Network provides a command-line interface (CLI) for interacting with its various services, including distributed key-value storage. This wiki page details the CLI commands and underlying mechanisms for performing storage operations such as putting, getting, listing, and deleting data. These operations are essential for managing application-specific data within the decentralized network. The CLI interacts with the network via a client library, abstracting the complexities of peer-to-peer communication and authentication. Sources: README.md:20-21, cmd/cli/main.go:46-52, examples/basic_usage.go:12-16

Overview of Storage Commands

The network-cli storage command provides subcommands for common key-value storage operations. All storage operations require authentication, which the CLI handles automatically, including wallet-based authentication. Sources: cmd/cli/main.go:148-150, README.md:167-172

Available Storage Commands

Command Description Usage Example Sources
put Stores data associated with a key. network-cli storage put <key> <value> cmd/cli/main.go:275
get Retrieves data for a given key. network-cli storage get <key> cmd/cli/main.go:284
list Lists keys, optionally filtered by a prefix. network-cli storage list [prefix] [limit] cmd/cli/main.go:293
delete Deletes data associated with a key. network-cli storage delete <key> cmd/cli/main.go:302

Command Structure

graph TD
    A[network-cli] --> B{storage}
    B --> C[put]
    B --> D[get]
    B --> E[list]
    B --> F[delete]

Sources: cmd/cli/main.go:148-150

Data Flow for Storage Operations

CLI storage operations involve the network-cli executable, the underlying client library, and the DeBros Network's storage service. The CLI first ensures authentication, then constructs a request using the client library, which communicates with the network. Sources: cmd/cli/main.go:268-270, AI_CONTEXT.md:126-128

storage put Operation Flow

When a user executes network-cli storage put <key> <value>, the following sequence occurs:

sequenceDiagram
    participant U as User
    participant C as network-cli
    participant A as Authentication System
    participant CL as Client Library
    participant S as DeBros Network Storage Service

    U->>C: `storage put <key> <value>`
    C->>A: `ensureAuthenticated()`
    activate A
    A-->>C: Authentication Status (e.g., JWT)
    deactivate A
    C->>CL: `NewClient(config)`
    activate CL
    CL->>CL: Connect to Network
    C->>CL: `Storage().Put(ctx, key, value)`
    activate CL
    CL->>S: Put Request (key, value)
    activate S
    S-->>CL: Success/Error Response
    deactivate S
    CL-->>C: Result
    deactivate CL
    C->>U: Display Result

Sources: cmd/cli/main.go:268-281, examples/basic_usage.go:68-75

storage get Operation Flow

sequenceDiagram
    participant U as User
    participant C as network-cli
    participant A as Authentication System
    participant CL as Client Library
    participant S as DeBros Network Storage Service

    U->>C: `storage get <key>`
    C->>A: `ensureAuthenticated()`
    activate A
    A-->>C: Authentication Status
    deactivate A
    C->>CL: `NewClient(config)`
    activate CL
    CL->>CL: Connect to Network
    C->>CL: `Storage().Get(ctx, key)`
    activate CL
    CL->>S: Get Request (key)
    activate S
    S-->>CL: Data/Error Response
    deactivate S
    CL-->>C: Result
    deactivate CL
    C->>U: Display Data

Sources: cmd/cli/main.go:284-291, examples/basic_usage.go:77-84

Authentication for CLI Operations

The CLI features an enhanced authentication system that automatically handles wallet detection and multi-wallet support. When an operation requires authentication (such as storage, database, or pub/sub), the CLI will:

  1. Check for existing valid credentials.
  2. Prompt for wallet authentication if needed.
  3. Handle signature verification.
  4. Persist credentials for future use. Sources: README.md:167-172, AI_CONTEXT.md:155-160

This streamlined process means users do not need to manually execute separate authentication commands. The ensureAuthenticated() function is called before executing authenticated commands. Sources: cmd/cli/main.go:268, README.md:175-179

Configuration and Client Initialization

The network-cli tool initializes a client connection to the DeBros Network. This involves setting up configuration parameters such as the bootstrap peer address, timeout, and output format. Sources: cmd/cli/main.go:20-25, cmd/cli/main.go:104-109

Global Flags

The CLI supports global flags that can modify its behavior:

Flag Description Default Value Sources
-b, --bootstrap Specifies the bootstrap peer address. /ip4/127.0.0.1/tcp/4001 cmd/cli/main.go:20, cmd/cli/main.go:116-119
-f, --format Sets the output format (table or json). table cmd/cli/main.go:22, cmd/cli/main.go:120-123
-t, --timeout Sets the operation timeout duration. 30s cmd/cli/main.go:21, cmd/cli/main.go:124-128
--production Enables production mode settings. false cmd/cli/main.go:23, cmd/cli/main.go:129
--disable-anonrc Disables Anyone proxy routing. false cmd/cli/main.go:24, cmd/cli/main.go:130

Client Creation

The createClient() function in cmd/cli/main.go is responsible for setting up the client.NetworkClient instance. This function configures the client with the parsed global flags, including the bootstrap peer and timeout. Sources: cmd/cli/main.go:204-210

// cmd/cli/main.go
func createClient() (client.NetworkClient, error) {
	cfg := client.DefaultClientConfig("cli")
	cfg.BootstrapPeers = []string{bootstrapPeer}
	cfg.Timeout = timeout
	cfg.UseProduction = useProduction
	
	// Set the authentication provider for the client
	cfg.AuthProvider = auth.NewCLIAuthProvider() 

	return client.NewClient(cfg)
}

Sources: cmd/cli/main.go:204-210

Conclusion

The DeBros Network CLI provides a robust and user-friendly interface for interacting with the distributed key-value storage. Through commands like put, get, list, and delete, users can manage data efficiently within the decentralized environment. The automatic authentication system simplifies the user experience, while configurable options allow for tailored client behavior. The underlying client library handles the complexities of network communication, ensuring reliable and secure data operations.

CLI Database Operations

Related Pages

Related topics: CLI Overview, Database Layer

Relevant source files

CLI Database Operations

The DeBros Network provides a Command Line Interface (CLI) tool, network-cli, that allows users to interact with the distributed SQL database functionality of the network. This includes executing SQL queries, managing tables, and performing atomic transactions. The CLI acts as a lightweight client, connecting to bootstrap peers to access the network's services. Sources: README.md, cmd/cli/main.go, AI_CONTEXT.md

Overview of Database Interaction

The network-cli facilitates direct interaction with the RQLite-backed distributed SQL database. Operations such as CREATE TABLE, INSERT, SELECT, ALTER TABLE, and PRAGMA can be executed. The CLI handles the underlying network communication and authentication processes, providing a simplified interface for database management and data manipulation. Sources: README.md:38-44, cmd/cli/main.go:142-164, examples/basic_usage.go:50-78

Core Commands

The primary command for database interaction via the CLI is query. This command allows users to execute arbitrary SQL statements against the distributed database.

query Command

The network-cli query <sql> command sends an SQL statement to the network's database. It supports various SQL operations, including Data Definition Language (DDL) and Data Manipulation Language (DML). The CLI automatically handles authentication if required for the operation. Sources: cmd/cli/main.go:138-164, README.md:188

Usage Example

./bin/network-cli query "CREATE TABLE users (id INTEGER)"
./bin/network-cli query "INSERT INTO users (id) VALUES (1)"
./bin/network-cli query "SELECT * FROM users"

Sources: README.md:188

Query Execution Flow

When a query command is issued, the CLI performs several steps:

  1. Authentication Check: Ensures the user is authenticated. If not, it may prompt for wallet authentication. Sources: cmd/cli/main.go:143, README.md:209-218
  2. Client Creation: A network client is created to connect to the DeBros Network. Sources: cmd/cli/main.go:146-151
  3. Context with Timeout: A context with a defined timeout (default 30 seconds) is established for the operation. Sources: cmd/cli/main.go:153-154
  4. Database Query: The SQL statement is passed to the client's Database().Query() method. Sources: cmd/cli/main.go:156
  5. Result Handling: The result of the query is processed and printed, either in JSON or table format, based on global flags. Sources: cmd/cli/main.go:158-163
sequenceDiagram
    participant User
    participant CLI as network-cli
    participant Client as pkg/client
    participant NetworkNode as DeBros Network Node

    User->>CLI: Execute `network-cli query "SQL_STATEMENT"`
    activate CLI
    CLI->>CLI: Parse global flags (timeout, format)
    CLI->>CLI: Check authentication (ensureAuthenticated)
    activate CLI
    CLI-->>CLI: (Prompts for wallet if needed)
    deactivate CLI
    CLI->>Client: Create new client (NewClient)
    activate Client
    Client->>NetworkNode: Connect to network
    activate NetworkNode
    NetworkNode-->>Client: Connection established
    deactivate NetworkNode
    CLI->>Client: Call `Database().Query(ctx, SQL_STATEMENT)`
    activate Client
    Client->>NetworkNode: Send SQL query
    activate NetworkNode
    NetworkNode-->>Client: Return query result
    deactivate NetworkNode
    Client-->>CLI: Return query result
    deactivate Client
    CLI->>CLI: Format and print result (JSON/Table)
    CLI-->>User: Display query output
    deactivate CLI

Sources: cmd/cli/main.go:142-164, pkg/client (implied)

Database Operations via HTTP Gateway

While the network-cli query command provides a direct way to execute SQL, the DeBros Network also exposes database operations through its HTTP Gateway. This allows applications and scripts to interact with the database using standard HTTP requests. Sources: README.md:46-56, README.md:195-206, e2e/gateway_e2e_test.go:1-40

Key HTTP Endpoints for Database

Endpoint Method Description Request Body (JSON) Response (JSON) Source
/v1/db/create-table POST Creates a new table based on the provided schema. {"schema": "CREATE TABLE ..."} {"status":"ok"} README.md:40
/v1/db/drop-table POST Drops an existing table. {"table": "table_name"} {"status":"ok"} README.md:41
/v1/db/query POST Executes an SQL query. {"sql": "SELECT ...", "args": [...]?} {"columns", "rows", "count"} README.md:42
/v1/db/transaction POST Executes multiple SQL statements atomically. {"statements": ["SQL 1", "SQL 2", ...]} {"status":"ok"} README.md:43
/v1/db/schema GET Retrieves the current database schema. None Schema JSON README.md:44
Sources: README.md:40-44, README.md:195-206

Example: Create Table via Gateway

curl -X POST "$GW/v1/db/create-table" \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{"schema":"CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT)"}'

Sources: README.md:199-201

Example: Atomic Transaction via Gateway

The /v1/db/transaction endpoint is crucial for executing multiple SQL statements as a single, atomic operation, ensuring data integrity. Sources: README.md:43, README.md:203-207

curl -X POST "$GW/v1/db/transaction" \
  -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
  -d '{"statements":[
        "ALTER TABLE users ADD COLUMN email TEXT",
        "CREATE INDEX IF NOT EXISTS idx_users_email ON users(email)"
      ]}'

Sources: README.md:203-207

Database Migrations

The DeBros Network incorporates a robust migration system, primarily managed by the Gateway service, to handle schema evolution. Migrations are executed automatically during gateway startup. Sources: AI_CONTEXT.md:11-13

Migration System Features

  • Version Control: Tracks migration versions to prevent duplicate execution and ensure proper upgrade paths. Sources: AI_CONTEXT.md:14
  • Error Recovery: Comprehensive error handling with detailed logging for debugging. Sources: AI_CONTEXT.md:15
  • Transaction Safety: Each migration runs in a transaction to ensure atomicity and data integrity. Sources: AI_CONTEXT.md:16
  • SQL File Support: Supports standard SQL migration files with .sql extension in the migrations/ directory. Sources: AI_CONTEXT.md:17-18

Migration File Structure

Migration files are sequentially named SQL files located in the migrations/ directory. Sources: AI_CONTEXT.md:20-24

migrations/
├── 001_initial_schema.sql
├── 002_add_auth_tables.sql
├── 003_add_indexes.sql
└── ...

Sources: AI_CONTEXT.md:20-24

Common Migration Workflow

A common pattern for migrations, especially for changing column types or adding foreign keys, involves recreating tables. This is typically done as an atomic transaction. Sources: README.md:46-48

graph TD
    A[Start Migration] --> B{Migration System Initiated};
    B --> C{Read Migration Files};
    C --> D{For Each Migration File};
    D --> E{Begin Transaction};
    E --> F[Execute SQL Statements];
    F --> G{Commit Transaction};
    G --> H{Update Migration Version};
    H --> I{Migration Complete};
    D -- No More Files --> I;
    E -- Error --> J{Rollback Transaction};
    J --> K[Log Error];
    K --> I;

Sources: AI_CONTEXT.md:14-16 (implied flow)

Troubleshooting Database Operations

Common issues related to database operations include:

  • Database Operations Timeout: Symptoms like "Query timeout" or "No RQLite connection available". Solutions involve ensuring RQLite ports are open (5001 HTTP, 7001 Raft), leader election is complete, and cluster join configuration is correct. Sources: README.md:139-142, AI_CONTEXT.md:79-81
  • Database Migration Issues: Symptoms such as "Migration failed", "SQL syntax error", or "Version conflict". Solutions include checking SQL syntax, ensuring proper statement termination, verifying migration file naming and sequential order, and reviewing migration logs for transaction rollbacks. Sources: README.md:168-172, AI_CONTEXT.md:92-96

For debugging, setting LOG_LEVEL=debug and checking service logs (sudo journalctl -u debros-node.service -f) are recommended. Sources: README.md:175-176, AI_CONTEXT.md:98-99

Conclusion

The network-cli provides essential command-line access to the DeBros Network's distributed SQL database, enabling direct query execution and management. Complementing this, the HTTP Gateway offers a RESTful interface for programmatic database interactions, including robust migration capabilities. These tools collectively ensure flexible and reliable management of the network's decentralized data store. Sources: README.md, cmd/cli/main.go, AI_CONTEXT.md

CLI Pub/Sub Operations

Related Pages

Related topics: CLI Overview, Pub/Sub System

Relevant source files

The following files were used as context for generating this wiki page:

CLI Pub/Sub Operations

The DeBros Network provides a decentralized Pub/Sub messaging system that allows applications and services to publish messages to topics and subscribe to receive messages from those topics. This functionality is exposed via the command-line interface (CLI) tool, network-cli, enabling direct interaction with the Pub/Sub system for testing, debugging, and administrative tasks. The CLI Pub/Sub operations are designed to be simple and efficient, allowing users to quickly send and receive messages across the distributed network. Sources: README.md:1-11, AI_CONTEXT.md:1-3, cmd/cli/main.go:37-40

Pub/Sub Architecture Overview

The Pub/Sub system in DeBros Network is topic-based and supports real-time messaging. Messages are namespaced, and the system handles automatic cleanup of topics. Clients can subscribe to topics, and publishers can send messages to those topics. This enables many-to-many communication patterns. Sources: README.md:37-39, AI_CONTEXT.md:1-3

graph TD
    A[Publisher] -->|Publish Message| B(PubSub Service)
    B -->|Distribute Message| C[Subscriber 1]
    B -->|Distribute Message| D[Subscriber 2]
    C -->|Receive Message| E[Application]
    D -->|Receive Message| F[Application]

Sources: README.md:37-39

CLI Commands for Pub/Sub

The network-cli tool provides specific subcommands under pubsub to interact with the messaging system. These include publish for sending messages and subscribe for receiving them. Sources: cmd/cli/main.go:102-104

Publishing Messages

To publish a message to a topic, the publish subcommand is used. This command requires the topic name and the message content. The message content is typically provided as a string. Sources: cmd/cli/main.go:102-104, cmd/cli/main.go:303-314

./bin/network-cli pubsub publish <topic> <message>

Sources: README.md:73, AI_CONTEXT.md:71

The internal process for publishing involves the CLI client connecting to the network, authenticating if required, and then calling the Publish method on the PubSub client. The message data is converted to a byte array before being sent. Sources: cmd/cli/main.go:303-314, examples/basic_usage.go:124-128

sequenceDiagram
    participant CLI as network-cli
    participant Client as Client Library
    participant Network as DeBros Network
    participant PubSubService as Pub/Sub Service

    CLI->>Client: Call handlePubSub(publish, topic, message)
    activate Client
    Client->>Client: Ensure Authenticated
    Client->>Network: Connect()
    activate Network
    Client->>PubSubService: Publish(ctx, topic, []byte(message))
    activate PubSubService
    PubSubService-->>Network: Message Acknowledged
    deactivate PubSubService
    Network-->>Client: Publish Result
    deactivate Network
    Client-->>CLI: Success/Error
    deactivate Client

Sources: cmd/cli/main.go:303-314, cmd/cli/main.go:264-268, examples/basic_usage.go:124-128

Subscribing to Topics

Subscribing to a topic allows the CLI to receive messages published to that topic in real-time. The subscribe subcommand is used for this purpose, specifying the topic name and an optional duration for how long to listen. Sources: cmd/cli/main.go:102-104, cmd/cli/main.go:316-340

./bin/network-cli pubsub subscribe <topic> [duration]

Sources: README.md:74, AI_CONTEXT.md:72

When subscribing, the CLI establishes a connection and registers a handler function that processes incoming messages. Messages are received as binary frames over a WebSocket connection when using the HTTP Gateway. Sources: AI_CONTEXT.md:236-237, examples/basic_usage.go:115-121, examples/sdk-typescript/src/client.ts:63-66

sequenceDiagram
    participant CLI as network-cli
    participant Client as Client Library
    participant Network as DeBros Network
    participant PubSubService as Pub/Sub Service

    CLI->>Client: Call handlePubSub(subscribe, topic, duration)
    activate Client
    Client->>Client: Ensure Authenticated
    Client->>Network: Connect()
    activate Network
    Client->>PubSubService: Subscribe(ctx, topic, handler)
    activate PubSubService
    PubSubService-->>Network: Subscription Confirmed
    deactivate PubSubService
    Network-->>Client: Subscription Result
    deactivate Network
    loop Message Flow
        PubSubService->>PubSubService: Message Published
        PubSubService->>Client: Deliver Message (via handler)
        Client->>CLI: Display Message
    end
    Client->>Network: Disconnect (after duration/interrupt)
    deactivate Network
    Client-->>CLI: Subscription Ended
    deactivate Client

Sources: cmd/cli/main.go:316-340, cmd/cli/main.go:264-268, examples/basic_usage.go:115-121

Listing Topics

The CLI also supports listing active topics. This is useful for discovering available communication channels within the network. Sources: AI_CONTEXT.md:240, examples/basic_usage.go:133-138

./bin/network-cli pubsub topics

The command internally queries the Pub/Sub service to retrieve a list of currently active topics.

Troubleshooting Pub/Sub Operations

Common issues related to Pub/Sub messaging include message delivery failures. These can often be resolved by verifying topic names, ensuring subscription status, and checking overall network connectivity. High memory usage might occur if subscriptions are not properly managed; users should unsubscribe from topics when they are no longer needed. Sources: AI_CONTEXT.md:65-67, AI_CONTEXT.md:144-146

Symptom Solution Source
Messages not received Verify topic names, active subscriptions, network connectivity. AI_CONTEXT.md:66-67, AI_CONTEXT.md:144-146
High Memory Usage Unsubscribe when done, monitor connection pool, review message retention. AI_CONTEXT.md:67, AI_CONTEXT.md:146

Conclusion

The network-cli provides essential command-line tools for interacting with the DeBros Network's Pub/Sub messaging system. It allows users to easily publish and subscribe to topics, facilitating real-time communication within the decentralized network. These CLI operations are crucial for development, testing, and managing the messaging flow in a DeBros Network deployment. Sources: README.md:73-74, cmd/cli/main.go:102-104

CLI Authentication

Related Pages

Related topics: Gateway Authentication, Authentication Troubleshooting

Error with OpenRouter API: argument of type 'NoneType' is not iterable

Please check that you have set the OPENROUTER_API_KEY environment variable with a valid API key.

HTTP Gateway Overview

Related Pages

Related topics: Gateway API Endpoints

Relevant source files

The following files were used as context for generating this wiki page:

HTTP Gateway Overview

The DeBros Network includes a powerful HTTP/WebSocket gateway that provides a modern REST API and WebSocket interface over the P2P network. It acts as a bridge, enabling external applications and clients to interact with the decentralized services like distributed SQL database, key-value storage, and pub/sub messaging without directly engaging with the underlying P2P protocols. This gateway features an enhanced authentication system with multi-wallet support and namespace enforcement for resource isolation. Sources: README.md:129-132, AI_CONTEXT.md:11-12

Architecture and Components

The HTTP Gateway operates as a service that exposes the core functionalities of the DeBros Network via standard HTTP and WebSocket protocols. It integrates with the network's client library to communicate with the P2P nodes.

graph TD
    A[External Client/Application] -->|HTTP/WebSocket| B(HTTP Gateway)
    B -->|Client API| C[DeBros Network Client]
    C -->|P2P Communication| D[DeBros Network Node]
    D --> E{Distributed Services}
    E --> F[RQLite Database]
    E --> G[Key-Value Storage]
    E --> H[Pub/Sub Messaging]

Sources: README.md:129-132, AI_CONTEXT.md:11-12

Gateway Configuration

The gateway's behavior can be configured through environment variables or command-line flags. Flags take precedence over environment variables, which in turn override default settings. Sources: README.md:141-145, cmd/gateway/config.go:1-62

Configuration Option Environment Variable Command-line Flag Description Default Value
Listen Address GATEWAY_ADDR --addr HTTP listen address. :8080
Client Namespace GATEWAY_NAMESPACE --namespace Client namespace for scoping resources. default
Bootstrap Peers GATEWAY_BOOTSTRAP_PEERS --bootstrap-peers Comma-separated bootstrap peers for network client. ""
Require Authentication GATEWAY_REQUIRE_AUTH N/A Enables/disables authentication requirement. true
API Keys GATEWAY_API_KEYS N/A Comma-separated API keys and their namespaces. ""
Sources: README.md:141-145, cmd/gateway/config.go:1-62

Health and Status Endpoints

The gateway provides several endpoints to monitor its operational status and the underlying network client's health. Sources: pkg/gateway/status_handlers.go:1-74

graph TD
    A[Client] -->|GET /health| B{healthHandler}
    B --> C{server healthResponse}
    B --> D{client.Health()}
    C & D --> E[JSON Response]

    A -->|GET /v1/status| F{statusHandler}
    F --> G{client.Network().GetStatus()}
    G --> H[JSON Response]

    A -->|GET /v1/version| I{versionHandler}
    I --> J[JSON Response with Build Info]

Sources: pkg/gateway/status_handlers.go:1-74

Authentication System

The gateway features an enhanced authentication system with automatic wallet detection, multi-wallet support, and API key support. It uses JWT for authenticated sessions and supports Ethereum EIP-191 personal_sign for wallet verification. Sources: README.md:147-156, AI_CONTEXT.md:275-288

Authentication Flow (Wallet-Based)

sequenceDiagram
    participant C as Client
    participant GW as HTTP Gateway
    participant NW as DeBros Network

    C->>GW: POST /v1/auth/challenge
    activate GW
    GW->>GW: Generate Nonce (internal auth context)
    GW-->>C: {nonce}
    deactivate GW

    C->>C: Sign nonce with wallet (EIP-191)
    C->>GW: POST /v1/auth/verify {wallet, nonce, signature}
    activate GW
    GW->>GW: Verify signature (65-byte r||s||v hex)
    GW->>NW: Authenticate/Verify Wallet
    activate NW
    NW-->>GW: Verification Result
    deactivate NW
    GW->>GW: Mark nonce as used (on success)
    GW-->>C: JWT Token / {status:"ok"}
    deactivate GW

Sources: README.md:109-117, AI_CONTEXT.md:280-285

Authentication Methods

Method Description Usage
Wallet-Based Uses Ethereum EIP-191 personal_sign for secure verification. Supports automatic wallet switching and credential persistence. POST /v1/auth/challenge, POST /v1/auth/verify, POST /v1/auth/register
JWT Authentication Issued by the gateway upon successful wallet verification. JWKS available for public key retrieval. Authorization: Bearer <JWT>
API Key Support Static API keys can be configured and mapped to namespaces. Authorization: Bearer <key> or X-API-Key: <key>
Sources: README.md:151-156, AI_CONTEXT.md:275-288

API Endpoints

The gateway exposes a comprehensive set of REST and WebSocket endpoints for interacting with the DeBros Network's services. Sources: README.md:42-87, AI_CONTEXT.md:290-332, openapi/gateway.yaml:1-68

General Endpoints

Endpoint Method Description
/health, /v1/health GET Health check, returns gateway and client status.
/v1/status GET Aggregates server uptime and network status.
/v1/version GET Returns gateway build/runtime information.
/v1/auth/jwks, /.well-known/jwks.json GET JSON Web Key Set for JWT verification.
/v1/auth/challenge POST Generates a wallet challenge (nonce).
/v1/auth/verify POST Verifies a wallet signature against a nonce.
/v1/auth/register POST Wallet registration.
/v1/auth/refresh POST Refreshes JWT token.
/v1/auth/logout POST Clears authentication state.
/v1/auth/whoami GET Returns current authentication status.
/v1/auth/api-key POST Generates API keys for authenticated users.
Sources: README.md:42-87, AI_CONTEXT.md:290-332

Storage Operations

The storage endpoints allow for key-value data manipulation with namespace enforcement. Sources: README.md:52-56, AI_CONTEXT.md:310-313, openapi/gateway.yaml:104-124

Endpoint Method Description Request Body/Query Response
/v1/storage/put POST Stores data by key. {"key":"<key>","value":"<base64_encoded_value>"} or raw bytes {"status":"ok"}
/v1/storage/get POST Retrieves data by key. {"key":"<key>"} {"value":"<base64_encoded_value>"} or raw bytes
/v1/storage/delete POST Deletes data by key. {"key":"<key>"} {"status":"ok"}
/v1/storage/list GET Lists keys with an optional prefix. ?prefix=<prefix> {"keys":["key1", "key2"]}
/v1/storage/exists GET Checks if a key exists. ?key=<key> {"exists":true/false}
Sources: README.md:52-56, AI_CONTEXT.md:310-313, openapi/gateway.yaml:104-124

Database Operations

The database endpoints provide an HTTP interface for interacting with the distributed SQL database. Sources: README.md:45-50, AI_CONTEXT.md:326-332, openapi/gateway.yaml:69-103

Endpoint Method Description Request Body Response
/v1/db/create-table POST Creates a table using SQL DDL. {"schema":"CREATE TABLE users (id INTEGER PRIMARY KEY)"} {"status":"ok"}
/v1/db/drop-table POST Drops a table. {"table":"users"} {"status":"ok"}
/v1/db/query POST Executes a single SQL query. {"sql":"SELECT * FROM users WHERE id = ?", "args":[1]} {"columns":["id","name"],"rows":[[1,"Alice"]],"count":1}
/v1/db/transaction POST Applies multiple SQL statements atomically. {"statements":["INSERT ...", "UPDATE ..."]} {"status":"ok"}
/v1/db/schema GET Returns the database schema. N/A JSON schema
Sources: README.md:45-50, AI_CONTEXT.md:326-332, openapi/gateway.yaml:69-103

Pub/Sub Messaging

The gateway supports both WebSocket and REST interfaces for real-time messaging. Sources: README.md:59-62, AI_CONTEXT.md:315-324

Endpoint Method Description Request Body/Query Response
/v1/pubsub/ws GET Establishes a WebSocket connection for real-time messaging. ?topic=<topic> WebSocket frames
/v1/pubsub/publish POST Publishes a message to a topic. {"topic":"<topic>","data_base64":"<base64_encoded_data>"} {"status":"ok"}
/v1/pubsub/topics GET Lists active topics (trimmed to caller namespace). N/A {"topics":["topic1", "topic2"]}
Sources: README.md:59-62, AI_CONTEXT.md:315-324

SDK Authoring Guidelines

The gateway provides an OpenAPI specification (openapi/gateway.yaml) for SDK code generation. SDKs should adhere to the following base concepts: Sources: README.md:92-95, AI_CONTEXT.md:334-337, openapi/gateway.yaml:1-68

  • Authentication: Send X-API-Key: <key> or Authorization: Bearer <key|JWT> with every request.
  • Versioning: All endpoints are under /v1/.
  • Responses: Mutations return {"status":"ok"}; queries/lists return JSON; errors return {"error":"message"} with appropriate HTTP status. Sources: README.md:96-101, AI_CONTEXT.md:338-342

A minimal TypeScript SDK example demonstrates interaction with the gateway:

import { GatewayClient } from './src/client';

const c = new GatewayClient(process.env.GATEWAY_BASE_URL!, process.env.GATEWAY_API_KEY!);
await c.createTable('CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)');
await c.transaction([
  'INSERT INTO users (id,name) VALUES (1,\'Alice\')'
]);
const res = await c.query('SELECT name FROM users WHERE id = ?', [1]);
console.log(res.rows);

Sources: examples/sdk-typescript/README.md:7-15

Conclusion

The HTTP Gateway is a critical component of the DeBros Network, simplifying access to its distributed services for external applications. Its robust API, flexible authentication, and clear configuration options make it an essential interface for building decentralized applications on the DeBros platform.

Gateway Configuration

Related Pages

Related topics: Configuration System, HTTP Gateway Overview

Error with OpenRouter API: argument of type 'NoneType' is not iterable

Please check that you have set the OPENROUTER_API_KEY environment variable with a valid API key.

Gateway Authentication

Related Pages

Related topics: CLI Authentication, Authentication Troubleshooting

Error with OpenRouter API: argument of type 'NoneType' is not iterable

Please check that you have set the OPENROUTER_API_KEY environment variable with a valid API key.

Gateway API Endpoints

Related Pages

Related topics: SDK Authoring Guide

Relevant source files

Gateway API Endpoints

The DeBros Network includes an HTTP/WebSocket gateway that provides a modern REST API and WebSocket interface over the P2P network. This gateway acts as a bridge, allowing external applications and clients to interact with the decentralized network's functionalities, including distributed database operations, key-value storage, pub/sub messaging, and peer management. All API endpoints are versioned under /v1/. Responses for mutations typically return {"status":"ok"}, while queries and lists return JSON data. Errors are returned as {"error": "message"} with appropriate HTTP status codes. Sources: README.md:135-139, AI_CONTEXT.md:275-279

Core API Endpoint Categories

The gateway exposes several categories of API endpoints to manage different aspects of the DeBros Network. These categories include Health & Status, Authentication, Storage Operations, Network Operations, Pub/Sub Messaging, and Database Operations. Sources: README.md:140-159, AI_CONTEXT.md:280-299

Health & Status Endpoints

These endpoints provide information about the gateway's operational status, network health, and version details.

Endpoint Method Description
/health GET Basic health check.
/v1/health GET Detailed health status of gateway and client.
/v1/status GET Aggregated server uptime and network status.
/v1/version GET Gateway build/runtime information.

Sources: README.md:141-144, AI_CONTEXT.md:281-284, pkg/gateway/status_handlers.go:42-83, pkg/gateway/status_handlers.go:85-103, pkg/gateway/status_handlers.go:105-114

The healthHandler (for /health and /v1/health) provides a JSON response including the server's status, started time, uptime, and optionally the client's health status if initialized. The statusHandler (for /v1/status) provides server uptime and network status from the client. The versionHandler (for /v1/version) returns build details such as version, commit, build time, started time, and uptime.

graph TD
    A[Client Request] --> B{GET /health or /v1/health};
    B --> C{Gateway.healthHandler};
    C --> D{Gateway.client.Health()};
    D -- HealthStatus --> E{Construct JSON Response};
    E --> F[Client Receives Health Data];

    G[Client Request] --> H{GET /v1/status};
    H --> I{Gateway.statusHandler};
    I --> J{Gateway.client.Network().GetStatus()};
    J -- NetworkStatus --> K{Construct JSON Response};
    K --> L[Client Receives Status Data];

    M[Client Request] --> N{GET /v1/version};
    N --> O{Gateway.versionHandler};
    O --> P{Construct Version Info};
    P --> Q[Client Receives Version Data];

Sources: pkg/gateway/status_handlers.go:42-114

Authentication Endpoints

The gateway features an enhanced authentication system supporting wallet-based authentication (Ethereum EIP-191), JWT tokens, and API keys. These endpoints facilitate the authentication flow.

Endpoint Method Description
/v1/auth/challenge POST Generates a wallet challenge (public endpoint).
/v1/auth/verify POST Verifies a wallet signature (public endpoint).
/v1/auth/register POST Registers a new wallet (public endpoint).
/v1/auth/refresh POST Refreshes a JWT token (public endpoint).
/v1/auth/logout POST Clears authentication state.
/v1/auth/whoami GET Returns the current authentication status.
/v1/auth/api-key POST Generates API keys (requires authentication).
/v1/auth/jwks GET JWKS endpoint for JWT verification.
/.well-known/jwks.json GET Standard JWKS endpoint.

Sources: README.md:146-152, AI_CONTEXT.md:286-292, AI_CONTEXT.md:330-331

Authentication is handled by middleware, which checks for Authorization: Bearer <JWT>, Authorization: Bearer <API key>, or X-API-Key. Public paths bypass authentication. Sources: pkg/gateway/middleware.go:33-35, pkg/gateway/middleware.go:73-82

sequenceDiagram
    participant C as Client
    participant GW as Gateway
    participant DB as Database

    C->>GW: POST /v1/auth/challenge
    activate GW
    GW-->>C: {nonce}
    deactivate GW

    C->>C: Sign nonce with wallet
    C->>GW: POST /v1/auth/verify {wallet, nonce, signature}
    activate GW
    GW->>DB: Verify signature and nonce
    DB-->>GW: Verification Result
    GW-->>C: JWT Token
    deactivate GW

    C->>GW: Authenticated Request (e.g., POST /v1/storage/put)
    activate GW
    Note over GW: Auth Middleware validates JWT/API Key
    GW->>DB: Perform operation
    DB-->>GW: Operation Result
    GW-->>C: {status: "ok"} or data
    deactivate GW

Sources: README.md:104-114, pkg/gateway/middleware.go:84-129

Storage Operations

These endpoints allow clients to interact with the distributed key-value storage.

Endpoint Method Description
/v1/storage/get POST Retrieve data by key.
/v1/storage/put POST Store data with a key.
/v1/storage/delete POST Delete data by key.
/v1/storage/list GET List keys with an optional prefix.
/v1/storage/exists GET Check if a key exists.

Sources: README.md:154-158, AI_CONTEXT.md:294-298, README.md:118-125

The storagePutHandler expects a JSON body with key and value (base64 encoded). The storageGetHandler expects a JSON body with key. Namespace enforcement is applied to all storage operations. Sources: pkg/gateway/storage_handlers.go:10-17, pkg/gateway/storage_handlers.go:19-26

Network Operations

These endpoints provide information and control over the peer-to-peer network connections.

Endpoint Method Description
/v1/network/status GET Network status.
/v1/network/peers GET Connected peers.
/v1/network/connect POST Connect to a specified peer.
/v1/network/disconnect POST Disconnect from a peer.

Sources: README.md:160-163, AI_CONTEXT.md:300-303

Pub/Sub Messaging

The gateway supports real-time messaging through both WebSocket and REST interfaces.

WebSocket Interface

Endpoint Method Description
/v1/pubsub/ws?topic=<topic> GET WebSocket connection for real-time messaging.

Sources: README.md:166, AI_CONTEXT.md:306, README.md:128-132

Server sends messages as binary frames; 30s ping keepalive. Client text/binary frames are published to the same namespaced topic. Sources: AI_CONTEXT.md:347-349

REST Interface

Endpoint Method Description
/v1/pubsub/publish POST Publish message to a topic.
/v1/pubsub/topics GET List active topics.

Sources: README.md:169-170, AI_CONTEXT.md:309-310

The pubsubPublishHandler expects a JSON body with topic and data_base64. The pubsubTopicsHandler returns a list of topics. Namespace enforcement is applied. Sources: pkg/gateway/storage_handlers.go:49-55, pkg/gateway/storage_handlers.go:57-63

Database Operations

These endpoints provide an interface for distributed SQL database operations, including table creation, querying, and transactions.

Endpoint Method Description
/v1/db/create-table POST Create tables with SQL DDL.
/v1/db/drop-table POST Drop a table.
/v1/db/query POST Execute a single SQL query.
/v1/db/transaction POST Apply multiple statements atomically.
/v1/db/schema GET Return database schema (tables and columns).

Sources: README.md:14-18, AI_CONTEXT.md:313-317

All /v1/db/* routes enforce authentication and namespace ownership. The gateway uses an internal database context for validation and execution, bypassing circular authentication checks. Migrations can be performed by POSTing DDL statements to /v1/db/transaction. Sources: AI_CONTEXT.md:320-323

The dbQueryHandler processes SQL queries with optional arguments. The dbTransactionHandler executes multiple SQL statements within a single transaction. The dbSchemaHandler retrieves the database schema. The dbCreateTableHandler creates a new table using a provided schema. Sources: pkg/gateway/storage_handlers.go:30-40, pkg/gateway/storage_handlers.go:42-48, pkg/gateway/storage_handlers.go:50-56, pkg/gateway/storage_handlers.go:58-64

graph TD
    C[Client] --> A{POST /v1/db/query};
    A --> G[Gateway.dbQueryHandler];
    G --> H{g.client.Database().Query};
    H -- Result --> I[Return JSON];

    C --> B{POST /v1/db/transaction};
    B --> J[Gateway.dbTransactionHandler];
    J --> K{g.client.Database().Transaction};
    K -- Status --> L[Return {status:"ok"}];

    C --> D{GET /v1/db/schema};
    D --> M[Gateway.dbSchemaHandler];
    M --> N{g.client.Database().GetSchema};
    N -- Schema --> O[Return JSON];

    C --> E{POST /v1/db/create-table};
    E --> P[Gateway.dbCreateTableHandler];
    P --> Q{g.client.Database().CreateTable};
    Q -- Status --> R[Return {status:"ok"}];

Sources: pkg/gateway/storage_handlers.go:30-64

Application Management Endpoints

The gateway also provides endpoints for managing applications within a namespace, including listing, creating, fetching, updating, and deleting application records.

Endpoint Method Description
/v1/apps GET List applications within the current namespace.
/v1/apps POST Create a new application.
/v1/apps/{app_id} GET Fetch details of a specific application.
/v1/apps/{app_id} PUT Update an application's name or public key.
/v1/apps/{app_id} DELETE Delete a specific application.

Sources: pkg/gateway/apps_handlers.go:10-15

The appsHandler manages these operations, using the database client to interact with an apps table. Application IDs are generated using a random buffer encoded with base64.RawURLEncoding. Sources: pkg/gateway/apps_handlers.go:66-73

Configuration

The gateway can be configured using environment variables or command-line flags. The precedence for configuration is: Flags > Environment Variables > YAML > Defaults. Sources: AI_CONTEXT.md:465-468

Key configuration parameters include:

Parameter Environment Variable Flag Default Description
Listen Address GATEWAY_ADDR --addr :8080 HTTP listen address for the gateway.
Client Namespace GATEWAY_NAMESPACE --namespace default Namespace for scoping resources.
Bootstrap Peers GATEWAY_BOOTSTRAP_PEERS --bootstrap-peers "" Comma-separated list of bootstrap peers.

Sources: AI_CONTEXT.md:255-263, cmd/gateway/config.go:40-42

The parseGatewayConfig function in cmd/gateway/config.go handles parsing these flags and environment variables into the gateway.Config structure. Sources: cmd/gateway/config.go:38-66

Conclusion

The Gateway API Endpoints provide a comprehensive and structured interface for external clients to interact with the DeBros Network. By categorizing endpoints for health, authentication, storage, network, pub/sub, and database operations, the gateway ensures modularity and ease of use. The built-in authentication system, namespace enforcement, and clear error handling contribute to a robust and secure interaction layer for the decentralized network.

SDK Authoring Guide

Related Pages

Related topics: Gateway API Endpoints

Relevant source files

SDK Authoring Guide

This guide outlines the fundamental concepts and key HTTP endpoints for authoring Software Development Kits (SDKs) that interact with the DeBros Network. The DeBros Network provides distributed SQL database, key-value storage, and pub/sub messaging capabilities. SDKs enable developers to integrate these features into their applications. The API endpoints are versioned under /v1/ and support various authentication methods. Sources: README.md:52-53, AI_CONTEXT.md:213-214

Base Concepts for SDKs

SDKs should adhere to several core concepts to ensure proper interaction with the DeBros Network. These include understanding the OpenAPI specification, authentication mechanisms, API versioning, and standard response formats. Sources: README.md:55-60, AI_CONTEXT.md:216-221

OpenAPI Specification

A machine-readable OpenAPI specification is available at openapi/gateway.yaml. This file can be used for automated SDK code generation, ensuring consistency and accuracy in API interactions. Sources: README.md:56, AI_CONTEXT.md:217, openapi/gateway.yaml

Authentication

All requests to the DeBros Network API require authentication. SDKs must send either an X-API-Key header with the API key or an Authorization header with a Bearer token (API key or JWT) with every request. Sources: README.md:57, AI_CONTEXT.md:218

Versioning

All API endpoints are prefixed with /v1/, indicating the current version of the API. This ensures backward compatibility and clear API evolution. Sources: README.md:58, AI_CONTEXT.md:219

Response Formats

The API employs pragmatic response envelopes. Mutations typically return {status:"ok"} upon success, while queries and list operations return JSON data. Errors are indicated by an HTTP status code and a JSON body in the format { "error": "message" }. Sources: README.md:59-60, AI_CONTEXT.md:220-221

Key HTTP Endpoints for SDKs

The DeBros Network exposes various HTTP endpoints categorized by their functionality: Storage, Database, and Pub/Sub. SDKs will primarily interact with these endpoints. Sources: README.md:62-95, AI_CONTEXT.md:223-256

Storage Operations

Storage operations allow SDKs to interact with the key-value store. This includes putting, getting, checking existence, listing, and deleting data.

graph TD
    A[SDK Client] -->|POST /v1/storage/put| B{Store Data}
    A -->|GET /v1/storage/get| C{Retrieve Data}
    A -->|GET /v1/storage/exists| D{Check Existence}
    A -->|GET /v1/storage/list| E{List Keys}
    A -->|POST /v1/storage/delete| F{Delete Data}

Sources: README.md:63-70, AI_CONTEXT.md:224-231

Operation Method Path Request Body/Query Response Description
Put POST /v1/storage/put?key=<k> Raw bytes {status:"ok"} Stores data associated with a key.
Get GET /v1/storage/get?key=<k> None Raw bytes (may be base64-encoded) Retrieves data for a given key.
Exists GET /v1/storage/exists?key=<k> None {exists:boolean} Checks if a key exists.
List GET /v1/storage/list?prefix=<p> None {keys:[...]} Lists keys with an optional prefix.
Delete POST /v1/storage/delete {key} {status:"ok"} Deletes data associated with a key.
Sources: README.md:63-70, AI_CONTEXT.md:224-231, openapi/gateway.yaml:127-167

Database Operations

Database operations enable SDKs to manage and query the distributed SQL database. This includes creating/dropping tables, executing queries, performing transactions, and retrieving schema information.

graph TD
    A[SDK Client] -->|POST /v1/db/create-table| B{Create Table}
    A -->|POST /v1/db/drop-table| C{Drop Table}
    A -->|POST /v1/db/query| D{Execute Query}
    A -->|POST /v1/db/transaction| E{Execute Transaction}
    A -->|GET /v1/db/schema| F{Get Schema}

Sources: README.md:71-78, AI_CONTEXT.md:232-239

Operation Method Path Request Body/Query Response Description
Create Table POST /v1/db/create-table {schema} {status:"ok"} Creates a new table with the specified schema.
Drop Table POST /v1/db/drop-table {table} {status:"ok"} Drops an existing table.
Query POST /v1/db/query {sql, args?} {columns, rows, count} Executes a single SQL query.
Transaction POST /v1/db/transaction {statements:[...]} {status:"ok"} Executes multiple SQL statements atomically.
Schema GET /v1/db/schema None Schema JSON Retrieves the database schema.
Sources: README.md:71-78, AI_CONTEXT.md:232-239, openapi/gateway.yaml:169-247

Pub/Sub Messaging

Pub/Sub operations allow SDKs to engage in real-time messaging. This involves connecting via WebSockets, publishing messages, and listing active topics.

graph TD
    A[SDK Client] -->|GET /v1/pubsub/ws?topic=<topic>| B{WebSocket Connection}
    A -->|POST /v1/pubsub/publish| C{Publish Message}
    A -->|GET /v1/pubsub/topics| D{List Topics}

Sources: README.md:79-82, AI_CONTEXT.md:240-243

Operation Method Path Request Body/Query Response Description
WS Subscribe GET /v1/pubsub/ws?topic=<topic> None WebSocket connection Establishes a WebSocket for real-time messaging.
Publish POST /v1/pubsub/publish {topic, data_base64} {status:"ok"} Publishes a message to a specified topic.
Topics GET /v1/pubsub/topics None {topics:[...]} Lists all active topics.
Sources: README.md:79-82, AI_CONTEXT.md:240-243, openapi/gateway.yaml:249-291

Migrations

Database migrations, such as adding columns or changing table schemas, should be handled by sending DDL statements as part of a POST /v1/db/transaction call. This ensures atomicity for schema changes. For complex changes like adding foreign keys or changing types, a "recreate pattern" is suggested: create a new table, copy data, drop the old table, and rename the new one. Sources: README.md:97-100, AI_CONTEXT.md:258-261

graph TD
    A[Start Migration] --> B{ALTER TABLE users ADD COLUMN age INTEGER}
    B --> C{POST /v1/db/transaction}
    C --> D{Migration Complete}

    E[Start Complex Migration] --> F{CREATE TABLE _new}
    F --> G{Copy Data to _new}
    G --> H{DROP old Table}
    H --> I{RENAME _new to old name}
    I --> J{POST /v1/db/transaction}
    J --> K{Migration Complete}

Sources: README.md:97-100, AI_CONTEXT.md:258-261

Suggested SDK Surface

SDKs should provide a clear and intuitive API surface for developers. This typically involves client objects or functions for Database, Storage, and Pub/Sub operations. Sources: AI_CONTEXT.md:310-332

Database Client Methods

Method Description
createTable(schema: string): Promise<void> Creates a new table.
dropTable(name: string): Promise<void> Drops an existing table.
query<T>(sql: string, args?: any[]): Promise<{ rows: T[] }> Executes a SQL query and returns rows.
transaction(statements: string[]): Promise<void> Executes multiple SQL statements in a transaction.
schema(): Promise<any> Retrieves the database schema.
Sources: AI_CONTEXT.md:311-316

Storage Client Methods

Method Description
`put(key: string, value: Uint8Array string): Promise`
get(key: string): Promise<Uint8Array> Retrieves data for a key.
exists(key: string): Promise<boolean> Checks if a key exists.
list(prefix?: string): Promise<string[]> Lists keys with an optional prefix.
delete(key: string): Promise<void> Deletes data for a key.
Sources: AI_CONTEXT.md:317-322

Pub/Sub Client Methods

Method Description
subscribe(topic: string, handler: (msg: Uint8Array) => void): Promise<() => void> Subscribes to a topic with a message handler.
`publish(topic: string, data: Uint8Array string): Promise`
listTopics(): Promise<string[]> Lists active topics.
Sources: AI_CONTEXT.md:323-326

Reliability Guidelines

SDKs should incorporate mechanisms to enhance reliability, such as timeouts, retries, and handling WebSocket reconnections. Sources: AI_CONTEXT.md:328-332

Timeouts

Implement default timeouts (e.g., 10-30 seconds) for API calls, with the option for per-call overrides. Sources: AI_CONTEXT.md:329

Retries

Employ exponential backoff for retries on 429 Too Many Requests and 5xx server error responses. SDKs should also respect the Retry-After header if provided by the server. Sources: AI_CONTEXT.md:330

Idempotency

For transactional operations, consider client-side implementation of an Idempotency-Key header, although the gateway may currently ignore it. Sources: AI_CONTEXT.md:331

WebSocket

For Pub/Sub WebSocket connections, implement automatic reconnection with jitter to handle disconnections gracefully. Re-subscription to topics should occur after a successful reconnect. Sources: AI_CONTEXT.md:332

Example SDK Usage

The following Go example demonstrates how an SDK client would connect to the network and perform basic database, storage, and pub/sub operations. Sources: examples/basic_usage.go

package main

import (
	"context"
	"log"
	"time"

	"git.debros.io/DeBros/network/pkg/client"
)

func main() {
	config := client.DefaultClientConfig("example_app")
	config.BootstrapPeers = []string{
		"/ip4/127.0.0.1/tcp/4001/p2p/QmBootstrap1",
	}

	networkClient, err := client.NewClient(config)
	if err != nil {
		log.Fatalf("Failed to create network client: %v", err)
	}

	if err := networkClient.Connect(); err != nil {
		log.Fatalf("Failed to connect to network: %v", err)
	}
	defer networkClient.Disconnect()

	log.Printf("Connected to network successfully!")

	demonstrateDatabase(networkClient)
	demonstrateStorage(networkClient)
	demonstratePubSub(networkClient)
	demonstrateNetworkInfo(networkClient)

	log.Printf("Example completed successfully!")
}

func demonstrateDatabase(client client.NetworkClient) {
	ctx := context.Background()
	db := client.Database()
	log.Printf("=== Database Operations ===")
	schema := `
		CREATE TABLE IF NOT EXISTS messages (
			id INTEGER PRIMARY KEY,
			content TEXT NOT NULL,
			timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
		)
	`
	if err := db.CreateTable(ctx, schema); err != nil {
		log.Printf("Error creating table: %v", err)
		return
	}
	log.Printf("Table created successfully")
	insertSQL := "INSERT INTO messages (content) VALUES (?)"
	result, err := db.Query(ctx, insertSQL, "Hello, distributed world!")
	if err != nil {
		log.Printf("Error inserting data: %v", err)
		return
	}
	log.Printf("Data inserted, result: %+v", result)
	selectSQL := "SELECT * FROM messages"
	result, err = db.Query(ctx, selectSQL)
	if err != nil {
		log.Printf("Error querying data: %v", err)
		return
	}
	log.Printf("Query result: %+v", result)
}

func demonstrateStorage(client client.NetworkClient) {
	ctx := context.Background()
	storage := client.Storage()
	log.Printf("=== Storage Operations ===")
	key := "user:123"
	value := []byte(`{"name": "Alice", "age": 30}`)
	if err := storage.Put(ctx, key, value); err != nil {
		log.Printf("Error storing data: %v", err)
		return
	}
	log.Printf("Data stored successfully")
	retrieved, err := storage.Get(ctx, key)
	if err != nil {
		log.Printf("Error retrieving data: %v", err)
		return
	}
	log.Printf("Retrieved data: %s", string(retrieved))
	exists, err := storage.Exists(ctx, key)
	if err != nil {
		log.Printf("Error checking existence: %v", err)
		return
	}
	log.Printf("Key exists: %v", exists)
	keys, err := storage.List(ctx, "user:", 10)
	if err != nil {
		log.Printf("Error listing keys: %v", err)
		return
	}
	log.Printf("Keys: %v", keys)
}

func demonstratePubSub(client client.NetworkClient) {
	ctx := context.Background()
	pubsub := client.PubSub()
	log.Printf("=== Pub/Sub Operations ===")
	topic := "notifications"
	handler := func(topic string, data []byte) error {
		log.Printf("Received message on topic '%s': %s", topic, string(data))
		return nil
	}
	if err := pubsub.Subscribe(ctx, topic, handler); err != nil {
		log.Printf("Error subscribing: %v", err)
		return
	}
	log.Printf("Subscribed to topic: %s", topic)
	message := []byte("Hello from pub/sub!")
	if err := pubsub.Publish(ctx, topic, message); err != nil {
		log.Printf("Error publishing: %v", err)
		return
	}
	log.Printf("Message published")
	time.Sleep(time.Millisecond * 100)
	topics, err := pubsub.ListTopics(ctx)
	if err != nil {
		log.Printf("Error listing topics: %v", err)
		return
	}
	log.Printf("Subscribed topics: %v", topics)
}

func demonstrateNetworkInfo(client client.NetworkClient) {
	ctx := context.Background()
	network := client.Network()
	log.Printf("=== Network Information ===")
	status, err := network.GetStatus(ctx)
	if err != nil {
		log.Printf("Error getting status: %v", err)
		return
	}
	log.Printf("Network Status: %+v", status)
	peers, err := network.GetPeers(ctx)
	if err != nil {
		log.Printf("Error getting peers: %v", err)
		return
	}
	log.Printf("Connected Peers: %+v", peers)
}

Sources: examples/basic_usage.go:1-177

This SDK Authoring Guide provides the foundational knowledge for building robust and reliable client libraries for the DeBros Network. By adhering to these concepts and utilizing the defined HTTP endpoints, developers can effectively integrate distributed database, storage, and messaging capabilities into their applications.

Development Setup

Related Pages

Related topics: Build, Test, and Lint

Relevant source files

Development Setup

The DeBros Network is a robust, decentralized peer-to-peer system designed for distributed data management and communication. This wiki page outlines the necessary steps and tools for setting up a development environment, building the project, running tests, and interacting with the network locally. Understanding the development setup is crucial for contributing to the project and building applications on top of the network. Sources: README.md:1-4, AI_CONTEXT.md:15-19

Requirements

To set up a development environment for the DeBros Network, the following software components are required:

  • Go: Version 1.22+ (1.23 recommended) is necessary for building and running the Go-based components.
  • RQLite: Version 8.x, a distributed SQLite database, is used for data persistence and replication.
  • Git: Essential for cloning the repository and managing source code.
  • Make: Recommended for build automation and simplifying common development tasks.

Sources: CONTRIBUTING.md:7-10, README.md:95-98, AI_CONTEXT.md:129-132

Setup and Installation

The initial setup involves cloning the repository and installing dependencies.

git clone https://git.debros.io/DeBros/network.git
cd network
make deps

Sources: CONTRIBUTING.md:14-16, README.md:108-110

Project Structure

The DeBros Network project follows a modular structure to organize its various components.

network/
├── cmd/
│   ├── node/         # Network node executable
│   └── cli/          # Command-line interface
├── pkg/
│   ├── client/       # Client library
│   ├── node/         # Node implementation
│   ├── database/     # RQLite integration
│   ├── storage/      # Storage service
│   ├── pubsub/       # Pub/Sub messaging
│   ├── config/       # Centralized config
│   └── discovery/    # Peer discovery (node only)
├── scripts/          # Install, test scripts
├── configs/          # YAML configs
├── bin/              # Built executables

This structure separates executables, core functionalities (packages), and development utilities. Sources: README.md:465-478, AI_CONTEXT.md:424-437

Build and Test

The project uses make for common build and test operations.

Build Commands

make build           # Build all executables
make test            # Run unit tests
make clean           # Clean build artifacts

Sources: README.md:480-483, AI_CONTEXT.md:439-442

The make build command compiles the source code into executables, which are placed in the bin/ directory. Sources: CONTRIBUTING.md:20, README.md:478

Build Process Flow

The build process typically involves compiling Go modules and generating executables.

graph TD
    A[Start Build] --> B{make build};
    B --> C[Compile Go Modules];
    C --> D[Generate Binaries];
    D --> E[Place in bin/ Directory];
    E --> F[Build Complete];

Sources: CONTRIBUTING.md:20, README.md:478

Testing

Unit tests can be executed using make test.

make test

Sources: CONTRIBUTING.md:21, README.md:481

For local multi-node testing, a dedicated script is provided:

scripts/test-multinode.sh

Sources: README.md:486, AI_CONTEXT.md:445

Running Nodes and CLI

Starting a Bootstrap Node

A bootstrap node is essential for the network. It can be started manually or via a Makefile command.

make run-node
# Or manually:
go run ./cmd/node --config configs/bootstrap.yaml

Sources: README.md:118-121

The cmd/node/main.go file handles the node's startup, configuration loading, and flag parsing. Sources: cmd/node/main.go:1-20

Node Configuration Loading Flow

The node's configuration can be loaded from a YAML file, with flags and environment variables overriding settings based on precedence.

graph TD
    A[Start Node] --> B{Parse CLI Flags};
    B --> C{Load YAML Config};
    C --> D{Apply Env Vars};
    D --> E[Final Config];
    E --> F[Initialize Node];
    F --> G[Start Node Services];

Sources: README.md:183-186, cmd/node/main.go:64-96

Starting Additional Nodes

Once a bootstrap node is running, additional nodes can join the network.

make run-node2
# Or manually:
go run ./cmd/node --config configs/node.yaml

Sources: README.md:123-126

CLI Usage

The network-cli tool provides commands to interact with the running network.

./bin/network-cli health
./bin/network-cli peers
./bin/network-cli status
./bin/network-cli storage put test-key "Hello Network"
./bin/network-cli storage get test-key
./bin/network-cli pubsub publish notifications "Hello World"
./bin/network-cli pubsub subscribe notifications 10s

Sources: CONTRIBUTING.md:25-31, README.md:128-134

The cmd/cli/main.go file defines the command-line interface logic, including parsing global flags and handling various commands like health, peers, status, query, storage, and pubsub. Sources: cmd/cli/main.go:27-106

Client Configuration

The pkg/client library is used to programmatically interact with the network. An example client configuration is shown below.

config := client.DefaultClientConfig("example_app")
config.BootstrapPeers = []string{
    "/ip4/127.0.0.1/tcp/4001/p2p/QmBootstrap1",
}

Sources: examples/basic_usage.go:13-16

HTTP Gateway

The DeBros Network provides an HTTP Gateway for external applications to interact with its services.

Gateway Endpoints

The gateway exposes various endpoints for authentication, storage, network information, Pub/Sub, and database operations.

Category Method Path Description
Auth POST /v1/auth/challenge Generates wallet challenge
POST /v1/auth/verify Verifies wallet signature
POST /v1/auth/register Registers new wallet
POST /v1/auth/refresh Refreshes JWT token
POST /v1/auth/logout Clears authentication
GET /v1/auth/whoami Current authentication status
POST /v1/auth/api-key Generates API key (authenticated)
Storage POST /v1/storage/get Retrieves data
POST /v1/storage/put Stores data
POST /v1/storage/delete Deletes data
GET /v1/storage/list Lists keys with optional prefix
GET /v1/storage/exists Checks key existence
Network GET /v1/network/status Network status
GET /v1/network/peers Connected peers
POST /v1/network/connect Connects to peer
POST /v1/network/disconnect Disconnects from peer
Pub/Sub (WS) GET /v1/pubsub/ws?topic=<topic> WebSocket connection for real-time messaging
Pub/Sub (REST) POST /v1/pubsub/publish Publishes message to topic
GET /v1/pubsub/topics Lists active topics
Database POST /v1/db/create-table Creates tables with SQL DDL
POST /v1/db/drop-table Drops a table
POST /v1/db/query Executes single SQL query
POST /v1/db/transaction Applies multiple statements atomically
GET /v1/db/schema Returns tables and columns schema

Sources: README.md:374-428, AI_CONTEXT.md:73-108, AI_CONTEXT.md:762-811

TypeScript SDK Example

An example of interacting with the gateway using a TypeScript SDK client:

import { GatewayClient } from "../examples/sdk-typescript/src/client";

const client = new GatewayClient(process.env.GATEWAY_BASE_URL!, process.env.GATEWAY_API_KEY!);
await client.createTable("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)");
const res = await client.query("SELECT name FROM users WHERE id = ?", [1]);

Sources: README.md:337-341

The TypeScript client code demonstrates methods for database operations (createTable, query, schema), storage operations (put, get, exists, list, delete), and Pub/Sub subscription. Sources: examples/sdk-typescript/src/client.ts:5-71

Troubleshooting

Common issues and their solutions during development:

  • Bootstrap Connection Failed: Verify the node is running, firewall settings are correct, and the peer ID is valid.
  • Database Operations Timeout: Ensure RQLite ports are open, leader election is complete, and the cluster join configuration is correct.
  • Message Delivery Failures: Check topic names, active subscriptions, and network connectivity.
  • High Memory Usage: Unsubscribe from topics when no longer needed, monitor connection pools, and review message retention policies.
  • Authentication Issues:
    • Wallet signature format (65-byte r||s||v hex)
    • Nonce must match exactly during wallet verification
    • Wallet address case-insensitivity
    • Use refresh endpoint or re-authenticate for expired tokens
    • Clear credential cache (rm -rf ~/.debros/credentials) for multi-wallet conflicts.
  • Gateway Issues: Verify the gateway is running and accessible, check CORS configuration, ensure proper authentication headers, and review namespace configuration.
  • Database Migration Issues: Check SQL syntax, ensure proper statement termination, verify migration file naming and sequential order, and review migration logs for transaction rollbacks.

Sources: README.md:493-518, AI_CONTEXT.md:167-195, AI_CONTEXT.md:496-524

Debugging and Health Checks

Useful commands for debugging and health checks:

export LOG_LEVEL=debug
./bin/network-cli health
./bin/network-cli peers
./bin/network-cli query "SELECT 1"
./bin/network-cli pubsub publish test "hello"
./bin/network-cli pubsub subscribe test 10s

# Test authentication flow
./bin/network-cli storage put test-key test-value

# Gateway health checks
curl http://localhost:8080/health
curl http://localhost:8080/v1/status

Sources: README.md:520-534

Service Logs

To monitor service logs:

# Node service logs
sudo journalctl -u debros-node.service --since "1 hour ago"

# Gateway service logs (if running as service)
sudo journalctl -u debros-gateway.service --since "1 hour ago"

# Application logs
tail -f ./logs/gateway.log
tail -f ./logs/node.log

Sources: README.md:537-545

Conclusion

This guide provides a comprehensive overview of setting up a development environment for the DeBros Network, covering installation, building, testing, and interacting with the network components. By following these steps and utilizing the provided tools and documentation, developers can effectively contribute to the project and build decentralized applications.

Build, Test, and Lint

Related Pages

Related topics: Development Setup

Relevant source files

Build, Test, and Lint

The DeBros Network project emphasizes a structured approach to software development, ensuring code quality, reliability, and maintainability through defined build, test, and lint processes. These processes are crucial for both local development and continuous integration, helping developers produce robust, decentralized peer-to-peer applications. The project provides tools and guidelines to facilitate these activities, ensuring that contributions adhere to established standards before integration.

Prerequisites

To build, test, and lint the DeBros Network, specific software versions and tools are required. These ensure a consistent development environment and compatibility with project dependencies. Sources: CONTRIBUTING.md:6-8, README.md:29-32, AI_CONTEXT.md:209-212

Software Requirements

The following software components are necessary for development:

Software Recommended Version Purpose
Go 1.22+ (1.23 recommended) Primary programming language runtime
RQLite 8.x (optional for local runs) Distributed SQLite database dependency
Git Any Version control system
Make Any Build automation tool
Sources: CONTRIBUTING.md:6-8, README.md:29-32, AI_CONTEXT.md:209-212

Setup

Before performing any build, test, or lint operations, the project repository must be cloned and dependencies installed. Sources: CONTRIBUTING.md:11-13

git clone https://git.debros.io/DeBros/network.git
cd network
make deps

Sources: CONTRIBUTING.md:11-13

Build Process

The build process compiles the Go source code into executable binaries. The project structure includes a cmd/ directory for executables and a bin/ directory for compiled binaries. Sources: README.md:179-183, AI_CONTEXT.md:200-204

Build Commands

To build all executables, use the make build command. This command compiles the node and cli executables, placing them in the bin/ directory. Sources: CONTRIBUTING.md:16, README.md:60, README.md:186, AI_CONTEXT.md:215

make build

Sources: CONTRIBUTING.md:16, README.md:60, README.md:186, AI_CONTEXT.md:215

Build Flow

The main.go files within cmd/node and cmd/cli serve as entry points for their respective executables. The build process involves compiling these Go packages. Sources: cmd/node/main.go:1, cmd/cli/main.go:1

graph TD
    A[Start Build Process] --> B{Execute 'make build'};
    B --> C[Compile cmd/node/main.go];
    B --> D[Compile cmd/cli/main.go];
    C --> E[Generate bin/network-node];
    D --> F[Generate bin/network-cli];
    E --> G[Build Complete];
    F --> G;

Description: This diagram illustrates the high-level flow of the build process, showing how the make build command compiles the node and CLI executables.

Testing

Testing is a critical part of the development workflow to ensure the correctness and reliability of the DeBros Network. The project provides commands for running unit tests and multi-node integration tests.

Unit Tests

Unit tests are executed using the make test command. This command runs all defined unit tests within the project. Sources: CONTRIBUTING.md:17, README.md:187, AI_CONTEXT.md:216

make test

Sources: CONTRIBUTING.md:17, README.md:187, AI_CONTEXT.md:216

Developers are required to include tests for new functionality and ensure that make build test passes before submitting pull requests. Sources: CONTRIBUTING.md:27

Local Multi-Node Testing

For more comprehensive testing of network interactions and distributed behavior, a script is provided to set up and test a multi-node environment locally. Sources: README.md:190-191, AI_CONTEXT.md:219-220

scripts/test-multinode.sh

Sources: README.md:190-191, AI_CONTEXT.md:219-220

This script is crucial for validating features that rely on peer-to-peer communication, such as peer discovery, distributed database operations, and pub/sub messaging across multiple nodes. Sources: README.md:190-191, AI_CONTEXT.md:219-220

Testing Flow (Example: examples/basic_usage.go)

The examples/basic_usage.go file demonstrates how a client interacts with the network for database, storage, and pub/sub operations. This example can be used to manually verify core functionalities. Sources: examples/basic_usage.go:1

sequenceDiagram
    participant C as Client
    participant N as Network
    participant DB as Database Service
    participant S as Storage Service
    participant P as PubSub Service

    C->>N: Connect()
    activate N
    N-->>C: Connection Success
    deactivate N

    C->>DB: CreateTable()
    activate DB
    DB-->>C: Table Created
    deactivate DB

    C->>S: Put("key", "value")
    activate S
    S-->>C: Data Stored
    deactivate S

    C->>P: Subscribe("topic", handler)
    activate P
    P-->>C: Subscribed
    deactivate P

    C->>P: Publish("topic", "message")
    activate P
    P-->>C: Message Published
    deactivate P

    P->>C: Received message on topic 'topic': message

Description: This sequence diagram illustrates the basic interaction flow of a client with the DeBros Network for database, storage, and pub/sub operations, as demonstrated in examples/basic_usage.go. Sources: examples/basic_usage.go:30-140

Linting and Formatting

Code linting and formatting ensure that the codebase adheres to a consistent style and catches potential issues early in the development cycle.

Linting Commands

The make fmt vet or make lint commands are used for formatting and vetting the Go source code. Sources: CONTRIBUTING.md:18

make fmt vet
# or
make lint

Sources: CONTRIBUTING.md:18

These commands help maintain code quality and readability, which is important for collaborative development.

Cleaning Build Artifacts

To remove compiled binaries and other build artifacts, the make clean command is available. Sources: README.md:188, AI_CONTEXT.md:217

make clean

Sources: README.md:188, AI_CONTEXT.md:217

This command is useful for ensuring a clean build environment, especially when switching branches or resolving build issues.

Pull Request Guidelines

Before submitting a pull request, contributors must ensure that their changes comply with the project's quality standards. Sources: CONTRIBUTING.md:25-29

  1. Fork and Branch: Fork the repository and create a topic branch for your changes.
  2. Pass Tests: Ensure that make build test passes successfully. New functionality must include corresponding tests.
  3. Focus and Description: Keep pull requests focused on a single logical change and provide a clear description including motivation, approach, and testing details.
  4. Documentation: Update README.md or other documentation if behavior changes are introduced. Sources: CONTRIBUTING.md:25-29

Conclusion

The build, test, and lint processes are integral to the development lifecycle of the DeBros Network. By adhering to these guidelines and utilizing the provided make commands, developers can contribute high-quality, reliable code that integrates seamlessly into the decentralized P2P system. These practices collectively ensure the stability and robustness of the distributed SQL database, key-value storage, and pub/sub messaging functionalities.

Local Multi-Node Testing

Related Pages

Related topics: Build, Test, and Lint

Timeout while calling OpenRouter API (120s). The service may be slow or unreachable.

Contributing to DeBros Network

Related Pages

Related topics: Code of Conduct

Relevant source files

Contributing to DeBros Network

Contributing to the DeBros Network project involves understanding its development workflow, testing procedures, and community guidelines. This guide outlines the necessary steps and resources for setting up a development environment, building the project, running tests, and submitting contributions. The project is built in Go and utilizes RQLite for its distributed database capabilities. Sources: CONTRIBUTING.md, README.md

Requirements

To contribute to the DeBros Network, specific software versions are required for a consistent development environment. Sources: CONTRIBUTING.md:7-9, README.md:43-46

Software Requirements

Requirement Version Description
Go 1.22+ Recommended: 1.23. The primary language for the project. Sources: CONTRIBUTING.md:7, README.md:43
RQLite 8.x Distributed SQLite. Optional for local runs, as Makefile can start embedded nodes. Sources: CONTRIBUTING.md:8, README.md:44
Git N/A For source management. Sources: README.md:45
Make N/A For build automation. Optional, but recommended. Sources: CONTRIBUTING.md:9, README.md:46

Hardware Requirements

Requirement Minimum Recommended Description
CPU 2 cores 4+ cores Processing power for nodes. Sources: README.md:50
RAM 4GB 8GB+ Memory for running services. Sources: README.md:50
Disk 10GB 50GB+ SSD Storage for data and binaries. Sources: README.md:50
Network Stable internet Low-latency network Connectivity for P2P communication. Sources: README.md:50

Network Ports

The DeBros Network uses specific ports for its operations. These ports must be accessible for proper node functionality. Sources: README.md:53-56

Port Protocol Description
4001 TCP LibP2P P2P communication. Sources: README.md:54
5001 TCP RQLite HTTP API. Sources: README.md:55
7001 TCP RQLite Raft consensus. Sources: README.md:56

Setup and Development Workflow

Setting up the development environment involves cloning the repository and installing dependencies. The Makefile provides convenient commands for building, testing, and linting the project. Sources: CONTRIBUTING.md:12-20

Initial Setup

To begin, clone the repository and navigate into the project directory. Sources: CONTRIBUTING.md:12-14, README.md:61-62

git clone https://git.debros.io/DeBros/network.git
cd network
make deps

Sources: CONTRIBUTING.md:12-14

Build, Test, and Lint

The project uses make for common development tasks. Sources: CONTRIBUTING.md:16-18

Command Description
make build Builds all executables. Sources: CONTRIBUTING.md:16, README.md:65
make test Runs unit tests. Sources: CONTRIBUTING.md:17, CONTRIBUTING.md:31, README.md:71
make fmt vet Formats and vets the code. Equivalent to make lint. Sources: CONTRIBUTING.md:18
make lint Formats and vets the code. Sources: CONTRIBUTING.md:18
make clean Cleans build artifacts. Sources: README.md:72

Local Multi-Node Testing

A script is provided to facilitate testing with multiple nodes locally. Sources: README.md:75

scripts/test-multinode.sh

Sources: README.md:75

Starting Nodes

The Makefile includes targets for quickly starting bootstrap and additional nodes for local testing. Sources: README.md:68-69

make run-node     # Starts a bootstrap node or a regular node based on configuration. Sources: [README.md:68](), [AI_CONTEXT.md:275]()
make run-node2    # Starts an additional node. Sources: [README.md:70]()
make run-gateway  # Starts the HTTP gateway. Sources: [AI_CONTEXT.md:276]()

Alternatively, nodes can be started manually with specific configuration files. Sources: README.md:69, README.md:71

go run ./cmd/node --config configs/bootstrap.yaml
go run ./cmd/node --config configs/node.yaml

Sources: README.md:69, README.md:71

CLI Usage

The network-cli tool provides commands for interacting with the DeBros Network. Sources: CONTRIBUTING.md:22-24, README.md:79-82

./bin/network-cli health
./bin/network-cli peers
./bin/network-cli status
./bin/network-cli storage put test-key "Hello Network"
./bin/network-cli storage get test-key
./bin/network-cli pubsub publish notifications "Hello World"
./bin/network-cli pubsub subscribe notifications 10s

Sources: CONTRIBUTING.md:22-24, README.md:79-82

Architecture Overview

The DeBros Network is a robust, decentralized peer-to-peer system built in Go. It offers distributed SQL database capabilities, key-value storage, and pub/sub messaging. The architecture distinguishes between full nodes and lightweight clients. Sources: README.md:131-134, AI_CONTEXT.md:154-157

Components and Data Flow

The system's architecture can be visualized as layers, with applications interacting through a client API that communicates with the underlying transport layer and RQLite database. Sources: README.md:137-147

graph TD
    A[Application Layer] --> B(Client API)
    B --> C(Database Client)
    B --> D(Storage Client)
    B --> E(PubSub Client)
    C --> F(Transport Layer)
    D --> F
    E --> F
    F --> G(LibP2P Host)
    F --> H(Noise/TLS Encryption)
    F --> I(RQLite Database)

Sources: README.md:137-147

Node vs. Client Roles

  • Node: A full P2P participant. It runs services, handles peer discovery, manages the database, storage, and pub/sub. Sources: README.md:30
  • Client: A lightweight participant. It connects only to bootstrap peers and consumes services without participating in peer discovery. Sources: README.md:31

Pull Request Guidelines

To ensure smooth integration of contributions, specific guidelines must be followed when submitting pull requests. Sources: CONTRIBUTING.md:28-33

  1. Fork and Branch: Fork the repository and create a new topic branch for your changes. Sources: CONTRIBUTING.md:29
  2. Pass Tests: Ensure that make build test passes successfully. New functionality must include corresponding tests. Sources: CONTRIBUTING.md:30-31
  3. Focus and Description: Keep pull requests focused on a single logical change and provide a clear description including motivation, approach, and testing details. Sources: CONTRIBUTING.md:32
  4. Documentation: Update README.md and other relevant documentation for any changes in behavior. Sources: CONTRIBUTING.md:33

Code of Conduct

The DeBros Network project adopts the Contributor Covenant Code of Conduct. All participants are expected to adhere to its standards to foster an open and welcoming community. Sources: CODE_OF_CONDUCT.md:3-4

Our Pledge

Project members, contributors, and leaders pledge to make participation a harassment-free experience for everyone. Sources: CODE_OF_CONDUCT.md:7

Our Standards

Positive Behaviors Unacceptable Behaviors
Demonstrating empathy and kindness Harassment, trolling, or insulting comments
Being respectful of differing opinions Public or private harassment
Gracefully accepting constructive criticism Publishing others’ private information
Focusing on what is best for the community Other conduct reasonably considered inappropriate
Sources: CODE_OF_CONDUCT.md:11-20

Enforcement

Project maintainers are responsible for clarifying and enforcing standards. Instances of unacceptable behavior can be reported to security@debros.io. All complaints are reviewed promptly and fairly. This Code applies within all project spaces and when an individual officially represents the project. Sources: CODE_OF_CONDUCT.md:23-32

Versioning

The CLI reports its version using network-cli version. Releases are tagged (e.g., v0.18.0-beta) and published via GoReleaser. Sources: CONTRIBUTING.md:26-27

Conclusion

Contributing to the DeBros Network involves a clear set of guidelines for development, testing, and collaboration. By following these steps and adhering to the Code of Conduct, contributors can effectively help improve the network, ensuring its continued growth and stability as a robust, decentralized P2P system.

Code of Conduct

Related Pages

Related topics: Contributing to DeBros Network

Relevant source files

The following files were used as context for generating this wiki page:

Code of Conduct

The DeBros Network project adopts a Code of Conduct to establish standards for a welcoming and harassment-free community. This document outlines the principles, expected behaviors, and enforcement mechanisms for all participants. It is designed to ensure a positive environment for contributors and users of the distributed P2P database system. Sources: CODE_OF_CONDUCT.md:1-2

Purpose and Scope

The Code of Conduct serves to define acceptable and unacceptable behaviors within the DeBros Network project. It applies to all project spaces and to individuals officially representing the project. This includes, but is not limited to, interactions in repositories, issue trackers, communication channels, and events. Sources: CODE_OF_CONDUCT.md:3-4, CODE_OF_CONDUCT.md:27-28

Our Pledge and Standards

Project members, contributors, and leaders pledge to make participation a harassment-free experience. The Code of Conduct specifies both positive and unacceptable behaviors to guide community interactions.

Positive Behaviors

Examples of behavior that contributes to a positive environment include:

  • Demonstrating empathy and kindness.
  • Being respectful of differing opinions.
  • Gracefully accepting constructive criticism.
  • Focusing on what is best for the community. Sources: CODE_OF_CONDUCT.md:9-14

Unacceptable Behaviors

Examples of unacceptable behavior include:

  • Harassment, trolling, or insulting comments.
  • Public or private harassment.
  • Publishing others’ private information.
  • Other conduct which could reasonably be considered inappropriate. Sources: CODE_OF_CONDUCT.md:16-21

Enforcement Responsibilities and Process

Project maintainers are responsible for clarifying and enforcing the standards outlined in the Code of Conduct. They are authorized to take appropriate and fair corrective action.

Reporting Instances

Instances of abusive, harassing, or otherwise unacceptable behavior should be reported to the maintainers. All complaints will be reviewed and investigated promptly and fairly.

Contact for Reporting: security@debros.io Sources: CODE_OF_CONDUCT.md:23-25, CODE_OF_CONDUCT.md:30-31

Enforcement Flow

The enforcement process involves reporting, review, and investigation, leading to potential corrective action by maintainers.

graph TD
    A[Participant Observes Unacceptable Behavior] --> B{Report to Maintainers};
    B --> C[Maintainers Receive Report];
    C --> D[Review and Investigate Complaint];
    D --> E{Determine Corrective Action};
    E --> F[Implement Corrective Action];
    F --> G[Resolution];

Sources: CODE_OF_CONDUCT.md:23-25, CODE_OF_CONDUCT.md:30-31

Attribution

This Code of Conduct is adapted from the Contributor Covenant, version 2.1. This indicates a commitment to widely recognized open-source community standards. Sources: CODE_OF_CONDUCT.md:33-34

Conclusion

The Code of Conduct is a foundational document for the DeBros Network project, ensuring a respectful and inclusive environment for all contributors and users. By outlining clear expectations and providing an enforcement mechanism, it supports the project's goal of fostering an open and welcoming community for its distributed P2P database system. Sources: CODE_OF_CONDUCT.md:1-4

Common Issues

Related Pages

Related topics: Authentication Troubleshooting

Relevant source files

Common Issues

This document outlines common issues encountered when operating the DeBros Network, along with their symptoms and recommended solutions. Understanding these issues is crucial for effective troubleshooting and maintaining a healthy network. This guide covers problems related to network connectivity, database operations, messaging, memory usage, authentication, gateway functionality, and database migrations. Sources: README.md, AI_CONTEXT.md

Network Connectivity Issues

Network connectivity problems often manifest as nodes failing to connect to the network or peers.

Bootstrap Connection Failed

Symptoms:

Solutions:

  • Verify the bootstrap node is running and accessible.
  • Check firewall settings on both the node and the bootstrap server to ensure the LibP2P port (default 4001) is open. Sources: README.md:168, AI_CONTEXT.md:251
  • Confirm the validity of the bootstrap peer ID in the configuration. Sources: README.md:168

CLI Check:

./bin/network-cli peers

Sources: README.md:175, cmd/cli/main.go:120-136

Database Operations Issues

Problems with database operations typically involve queries timing out or an inability to connect to the RQLite database.

Database Operations Timeout

Symptoms:

Solutions:

CLI Check:

./bin/network-cli query "SELECT 1"

Sources: README.md:175, cmd/cli/main.go:149-179

Messaging Issues

Issues with the Pub/Sub system can lead to messages not being delivered to subscribers.

Message Delivery Failures

Symptoms:

Solutions:

CLI Check:

./bin/network-cli pubsub publish test "hello"
./bin/network-cli pubsub subscribe test 10s

Sources: README.md:175, cmd/cli/main.go:204-211

Resource Usage Issues

Unintended high memory consumption can impact node stability and performance.

High Memory Usage

Symptoms:

Solutions:

  • Ensure that applications unsubscribe from Pub/Sub topics when they no longer need to receive messages. Sources: README.md:183, AI_CONTEXT.md:254
  • Monitor the connection pool size to the database and other services. Sources: README.md:183
  • Review message retention policies if applicable. Sources: README.md:183

Authentication Issues

Authentication failures can prevent access to network services and data.

Authentication Issues

Symptoms:

Solutions:

  • Wallet Signature: Check that the wallet signature format is a 65-byte r||s||v hex string. Sources: README.md:187, AI_CONTEXT.md:255
  • Nonce Mismatch: Ensure the nonce used for wallet verification matches exactly. Sources: README.md:188, AI_CONTEXT.md:255
  • Wallet Address Case: Verify wallet address case-insensitivity. Sources: README.md:189, AI_CONTEXT.md:255
  • Expired Tokens: Use the refresh endpoint or re-authenticate for expired JWT tokens. Sources: README.md:190, AI_CONTEXT.md:255
  • Credential Conflicts: Clear the credential cache if multi-wallet conflicts occur by removing ~/.debros/credentials. Sources: README.md:191, AI_CONTEXT.md:255
  • API Key Invalid: Verify the API key format and its namespace mapping in the gateway configuration. Sources: AI_CONTEXT.md:260
  • Circular Auth Dependencies: Ensure public authentication endpoints use an internal authentication context to avoid circular dependencies. Sources: AI_CONTEXT.md:261

Authentication Flow (Gateway):

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant N as Node
    C->>G: POST /v1/auth/challenge
    activate G
    G-->>C: {nonce}
    deactivate G
    Note over C,G: Client signs nonce with wallet
    C->>G: POST /v1/auth/verify {wallet, nonce, signature}
    activate G
    G->>N: Verify signature
    activate N
    N-->>G: Signature valid
    deactivate N
    G-->>C: {jwt_token}
    deactivate G
    C->>G: POST /v1/storage/put (with JWT)
    activate G
    G->>N: Authorized request
    activate N
    N-->>G: Data stored
    deactivate N
    G-->>C: {status: "ok"}
    deactivate G

Sources: README.md:218-228, AI_CONTEXT.md:104-114

Gateway Issues

Problems with the HTTP Gateway can lead to connection refusals, CORS errors, or WebSocket disconnections.

Gateway Issues

Symptoms:

  • Gateway connection refused, CORS errors, WebSocket disconnections. Sources: README.md:193

Solutions:

  • Verify the gateway is running and accessible on its configured port. Sources: README.md:195
  • Check the CORS configuration for web applications interacting with the gateway. Sources: README.md:196
  • Ensure proper authentication headers are provided for protected endpoints. Sources: README.md:197
  • Verify namespace configuration and enforcement if applicable. Sources: README.md:198

Health Checks:

curl http://localhost:8080/health
curl http://localhost:8080/v1/status

Sources: README.md:200-201

Database Migration Issues

Issues during database migrations can cause schema inconsistencies or data loss.

Database Migration Issues

Symptoms:

Solutions:

Migration File Structure:

migrations/
├── 001_initial_schema.sql
├── 002_add_auth_tables.sql
├── 003_add_indexes.sql
└── ...

Sources: AI_CONTEXT.md:231-236

Debugging and Logging

Effective debugging relies on proper logging and health checks.

Debugging & Health Checks

Enable Debug Logging:

export LOG_LEVEL=debug

Sources: README.md:211, AI_CONTEXT.md:262

Service Logs:

Conclusion

Addressing common issues in the DeBros Network involves systematic checks of configuration, network connectivity, service status, and application logic. Utilizing the provided CLI tools and reviewing detailed logs are essential steps in diagnosing and resolving problems efficiently. Sources: README.md, AI_CONTEXT.md

Authentication Troubleshooting

Related Pages

Related topics: Common Issues, CLI Authentication, Gateway Authentication

Relevant source files

The following files were used as context for generating this wiki page:

Authentication Troubleshooting

Authentication in the DeBros Network is critical for securing access to its decentralized services, including storage, database operations, and PubSub messaging. This page provides guidance on troubleshooting common authentication issues, covering wallet-based authentication, JWT tokens, and API keys. The system is designed for enhanced user experience with features like automatic wallet detection and multi-wallet management, but issues can still arise. Sources: AI_CONTEXT.md, README.md

Common Authentication Issues and Solutions

The DeBros Network's authentication system is robust, but users may encounter issues related to wallet connections, token validity, or API key configuration. Understanding the common symptoms and their solutions is key to resolving these problems efficiently. Sources: AI_CONTEXT.md:79-88, README.md:105-114

Wallet Connection Failed

Symptoms:

  • Wallet connection failure during authentication flow.
  • Error messages indicating invalid signature format or nonce mismatch.

Solutions:

JWT Token Expired

Symptoms:

  • API requests fail with JWT token expired errors.
  • Inability to access authenticated endpoints after a period of inactivity.

Solutions:

API Key Invalid

Symptoms:

  • API requests return API Key Invalid errors.
  • Unauthorized access to resources despite providing an API key.

Solutions:

Authentication Flow and Components

The authentication process involves several components, including the client, the gateway, and the wallet. The flow typically starts with a challenge, followed by wallet signature, and verification, ultimately leading to the issuance of an API key or JWT.

Wallet Authentication Process

The wallet authentication process uses Ethereum EIP-191 for secure verification. It involves generating a challenge, signing it with the user's wallet, and then verifying the signature to grant access.

sequenceDiagram
    participant Client
    participant Gateway
    participant Wallet

    Client->>Gateway: POST /v1/auth/challenge
    activate Gateway
    Gateway-->>Client: Challenge (nonce)
    deactivate Gateway

    Client->>Wallet: Sign Challenge (nonce)
    activate Wallet
    Wallet-->>Client: Signed Signature
    deactivate Wallet

    Client->>Gateway: POST /v1/auth/verify (wallet, nonce, signature)
    activate Gateway
    Gateway-->>Client: API Key / JWT Token
    deactivate Gateway

    Note over Client,Gateway: Subsequent requests use API Key / JWT

Sources: AI_CONTEXT.md:191-197, README.md:280-286

The AuthServer in pkg/auth/wallet.go handles the callback from the gateway after successful wallet authentication. It parses query parameters to extract the api_key, refresh_token, namespace, wallet, plan, and expires_at. Sources: pkg/auth/wallet.go:193-200

Authentication Endpoints

The gateway exposes several public and authenticated endpoints for managing authentication.

Endpoint Method Description Authentication Required Source
/v1/auth/challenge POST Generate wallet challenge No AI_CONTEXT.md:164
/v1/auth/verify POST Verify wallet signature No AI_CONTEXT.md:165
/v1/auth/register POST Register new wallet No AI_CONTEXT.md:166
/v1/auth/refresh POST Refresh JWT token No AI_CONTEXT.md:167
/v1/auth/logout POST Clear authentication state Yes AI_CONTEXT.md:168
/v1/auth/whoami GET Returns current authentication status Yes AI_CONTEXT.md:169
/v1/auth/api-key POST Generate API keys for authenticated users Yes AI_CONTEXT.md:170
/v1/auth/jwks GET JWKS endpoint for JWT verification No AI_CONTEXT.md:347
/.well-known/jwks.json GET Standard JWKS endpoint for JWT verification No AI_CONTEXT.md:347

Debugging Authentication Issues

Effective debugging involves enabling detailed logging and using CLI tools to inspect the network and authentication status.

Enabling Debug Logging

To get more detailed information about authentication processes and potential errors, set the LOG_LEVEL environment variable to debug. Sources: AI_CONTEXT.md:99, README.md:364

export LOG_LEVEL=debug

Sources: AI_CONTEXT.md:99, README.md:364

Checking Service Logs

Review the logs of the debros-node.service and debros-gateway.service (if running as services) for authentication-related errors or warnings. Sources: AI_CONTEXT.md:100-101, README.md:370-373

sudo journalctl -u debros-node.service -f
sudo journalctl -u debros-gateway.service -f

Sources: AI_CONTEXT.md:100-101, README.md:370-373

Using the CLI for Health and Peer Checks

The network-cli tool can be used to check the health of the node and the status of connected peers, which can indirectly help diagnose network-related authentication issues. Sources: AI_CONTEXT.md:102-103, README.md:365-366

./bin/network-cli health
./bin/network-cli peers

Sources: AI_CONTEXT.md:102-103, README.md:365-366

Testing Authentication Flow with CLI

The CLI automatically handles wallet authentication when performing operations that require credentials. This can be used to test the end-to-end authentication flow. Sources: AI_CONTEXT.md:300-305

# First time - will prompt for wallet authentication
./bin/network-cli storage put user:123 "John Doe"

# Subsequent calls - uses saved credentials automatically
./bin/network-cli storage get user:123
./bin/network-cli pubsub publish notifications "Hello World"

Sources: AI_CONTEXT.md:300-305

Conclusion

Troubleshooting authentication issues in the DeBros Network involves understanding the various authentication methods (wallet, JWT, API keys), common error symptoms, and utilizing debugging tools and logs. The enhanced authentication system aims to streamline the user experience, but knowing how to diagnose and resolve problems is essential for maintaining secure and functional access to the network's decentralized services.

Database Migration Troubleshooting

Related Pages

Related topics: Database Layer

Relevant source files

Database Migration Troubleshooting

This document provides a guide to troubleshooting common issues encountered during database migrations in the DeBros Network. The gateway component includes a robust database migration system designed for automatic schema management and version control, ensuring data integrity and proper upgrade paths. Understanding the system's behavior and common failure points is crucial for effective debugging. Sources: AI_CONTEXT.md:23-26, AI_CONTEXT.md:73-76

Overview of the Migration System

The DeBros Network's gateway features an enhanced database migration system that automatically initializes and updates the database schema using SQL migration files. Each migration runs within a transaction to ensure atomicity. The system tracks migration versions in a schema_migrations table to prevent duplicate execution. Sources: AI_CONTEXT.md:23-26, AI_CONTEXT.md:73-76, pkg/gateway/migrate.go:103-106

The migration process involves:

  1. Checking for the existence of the schema_migrations table and creating it if necessary. Sources: pkg/gateway/migrate.go:103-106
  2. Locating SQL migration files in the migrations/ directory. Sources: pkg/gateway/migrate.go:109-112
  3. Parsing migration versions from filenames (e.g., 001_initial_schema.sql). Sources: pkg/gateway/migrate.go:129-130, pkg/gateway/migrate.go:174-187
  4. Sorting migrations by version. Sources: pkg/gateway/migrate.go:136
  5. Applying unapplied migrations sequentially, each within a transaction. Sources: pkg/gateway/migrate.go:140-169
graph TD
    A[Gateway Startup] --> B{Client connected?};
    B -- Yes --> C[Apply Auto Migrations];
    C --> D[Apply File-based Migrations];
    D --> E[Check schema_migrations table];
    E -- Not Exists --> F[Create schema_migrations table];
    F --> G[Locate SQL Migration Files];
    G --> H[Parse & Sort Migrations];
    H --> I{Migration Applied?};
    I -- No --> J[Read SQL Content];
    J --> K[Split SQL Statements];
    K --> L[Execute Statements in Transaction];
    L -- Success --> M[Mark Migration as Applied];
    M --> I;
    I -- Yes / All Applied --> N[Migration Complete];
    B -- No --> N;

Sources: pkg/gateway/migrate.go:30-33, pkg/gateway/migrate.go:99-102, pkg/gateway/migrate.go:103-106, pkg/gateway/migrate.go:109-136, pkg/gateway/migrate.go:140-169

Common Migration Issues and Solutions

Migration Failures

Symptoms: Migration failed, SQL syntax error, Version conflict Sources: README.md:20-21, AI_CONTEXT.md:144

Causes:

  • Incorrect SQL syntax in migration files.
  • Missing statement terminators.
  • Improper migration file naming or sequential order.
  • Issues with transaction rollbacks or database locks. Sources: AI_CONTEXT.md:145

Solutions:

Incomplete Migrations

Symptoms: Database schema does not reflect expected changes after migration, or some migrations appear to have run partially. Sources: AI_CONTEXT.md:145

Causes:

  • Transaction rollbacks due to an error within a migration.
  • Database locks preventing schema changes.
  • Interruption of the migration process.

Solutions:

  • Check for Transaction Rollbacks: The system runs each migration in a transaction. If an error occurs, the entire migration should roll back. If a partial change persists, investigate the specific SQL statement and its atomicity. Sources: AI_CONTEXT.md:25, AI_CONTEXT.md:75, pkg/gateway/migrate.go:160-162
  • Inspect Database Locks: If schema changes are blocked, check for any lingering database connections or locks on the RQLite instance.
  • Review schema_migrations table: Query the schema_migrations table to see which versions have been successfully applied. If a version is missing, it indicates the migration for that version did not complete. Sources: pkg/gateway/migrate.go:150-155

Debugging Migration Issues

For deeper debugging, several tools and techniques are available.

Logging

Enable debug logging to get more verbose output from the node and gateway services.

export LOG_LEVEL=debug

Sources: AI_CONTEXT.md:152, README.md:30

Check service logs:

sudo journalctl -u debros-node.service -f
sudo journalctl -u debros-gateway.service -f

Sources: AI_CONTEXT.md:154-155, README.md:46-50

CLI Health Checks

Use the network-cli tool to check the health and status of the network and database.

./bin/network-cli health
./bin/network-cli peers
./bin/network-cli query "SELECT 1"

Sources: AI_CONTEXT.md:153, README.md:31-33

End-to-End Tests

The project includes end-to-end tests that simulate database operations, including table creation and transactions, which can be useful for understanding expected behavior and replicating issues. The e2e/client_e2e_test.go and e2e/gateway_e2e_test.go files contain tests for database interactions, including schema creation and transactions, which are analogous to migration steps. Sources: e2e/client_e2e_test.go:46-60, e2e/gateway_e2e_test.go:120-131

An example of a database creation and transaction test:

func TestClient_Database_CreateQueryMigrate(t *testing.T) {
	// ... client setup ...
	table := fmt.Sprintf("e2e_items_client_%d", time.Now().UnixNano())
	schema := fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP)", table)
	ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
	defer cancel()
	if err := c.Database().CreateTable(ctx, schema); err != nil {
		t.Fatalf("create table: %v", err)
	}
	// Insert via transaction
	stmts := []string{
		fmt.Sprintf("INSERT INTO %s(name) VALUES ('alpha')", table),
		fmt.Sprintf("INSERT INTO %s(name) VALUES ('beta')", table),
	}
	ctx2, cancel2 := context.WithTimeout(context.Background(), 15*time.Second)
	defer cancel2()
	if err := c.Database().Transaction(ctx2, stmts); err != nil {
		t.Fatalf("transaction: %v", err)
	}
	// ... query ...
}

Sources: e2e/client_e2e_test.go:62-89

Migration File Structure

Migration files are standard SQL files with a .sql extension, located in the migrations/ directory. They are prefixed with a version number to ensure correct execution order.

migrations/
├── 001_initial_schema.sql     # Initial database setup
├── 002_add_auth_tables.sql    # Authentication tables
├── 003_add_indexes.sql        # Performance indexes
└── ...                        # Additional migrations

Sources: AI_CONTEXT.md:14-19, AI_CONTEXT.md:77-82

The parseMigrationVersion function extracts the numerical version from the filename:

func parseMigrationVersion(name string) (int, bool) {
	i := 0
	for i < len(name) && name[i] >= '0' && name[i] <= '9' {
		i++
	}
	if i == 0 {
		return 0, false
	}
	v, err := strconv.Atoi(name[:i])
	if err != nil {
		return 0, false
	}
	return v, true
}

Sources: pkg/gateway/migrate.go:174-187

The splitSQLStatements function handles parsing the SQL content, removing comments, and splitting into individual statements for execution. It specifically ignores BEGIN;, COMMIT; and INSERT statements into SCHEMA_MIGRATIONS if present within the file, as these are handled internally by the migration system.

func splitSQLStatements(sqlText string) []string {
	lines := strings.Split(sqlText, "\n")
	cleaned := make([]string, 0, len(lines))
	for _, ln := range lines {
		s := strings.TrimSpace(ln)
		if s == "" {
			continue
		}
		// Handle inline comments by removing everything after --
		if commentIdx := strings.Index(s, "--"); commentIdx >= 0 {
			s = strings.TrimSpace(s[:commentIdx])
			if s == "" {
				continue // line was only a comment
			}
		}
		upper := strings.ToUpper(s)
		if upper == "BEGIN;" || upper == "COMMIT;" || upper == "BEGIN" || upper == "COMMIT" {
			continue
		}
		if strings.HasPrefix(upper, "INSERT") && strings.Contains(upper, "SCHEMA_MIGRATIONS") {
			// ignore in-file migration markers
			continue
		}
		cleaned = append(cleaned, s)
	}
	// Join and split by ';'
	joined := strings.Join(cleaned, "\n")
	parts := strings.Split(joined, ";")
	out := make([]string, 0, len(parts))
	for _, p := range parts {
		sp := strings.TrimSpace(p)
		if sp == "" {
			continue
		}
		out = append(out, sp+";")
	}
	return out
}

Sources: pkg/gateway/migrate.go:189-228

Conclusion

Effective troubleshooting of database migrations in the DeBros Network relies on understanding the automated migration system, its version control mechanisms, and the proper format of migration files. By leveraging detailed logs and the provided CLI tools, developers can diagnose and resolve issues related to SQL syntax, version conflicts, or incomplete schema updates, ensuring a stable and consistent database state.

Debugging & Health Checks

Related Pages

Related topics: Service Logs

Relevant source files

Debugging & Health Checks

The DeBros Network provides several tools and methods for debugging issues and performing health checks on both network nodes and the HTTP Gateway. This includes command-line interface (CLI) utilities, direct API calls, and log monitoring, designed to help diagnose common problems such as connection failures, database timeouts, and message delivery issues. Sources: README.md:144-145, AI_CONTEXT.md:144-145

Common Issues and Solutions

The project documentation outlines several common issues that users might encounter and provides specific solutions for each. These range from network connectivity problems to database and authentication issues. Sources: README.md:147-148, AI_CONTEXT.md:147-148

Bootstrap Connection Failed

Symptoms include Failed to connect to bootstrap peer. Solutions involve checking if the node is running, firewall settings, and the validity of the peer ID. Sources: README.md:150-152, AI_CONTEXT.md:150-152

Database Operations Timeout

Symptoms are Query timeout or No RQLite connection available. Solutions require ensuring RQLite ports are open, leader election has completed, and the cluster join configuration is correct. Sources: README.md:154-156, AI_CONTEXT.md:154-156

Message Delivery Failures

Symptoms indicate messages are not received by subscribers. Solutions include verifying topic names, active subscriptions, and network connectivity. Sources: README.md:158-160, AI_CONTEXT.md:158-160

High Memory Usage

Symptoms show memory usage growing continuously. Solutions involve unsubscribing when done, monitoring the connection pool, and reviewing message retention policies. Sources: README.md:162-164, AI_CONTEXT.md:162-164

Authentication Issues

Various symptoms such as Authentication failed, Invalid wallet signature, or JWT token expired. Solutions cover checking wallet signature format (65-byte r||s||v hex), ensuring nonce matches exactly during verification, verifying wallet address case-insensitivity, using the refresh endpoint or re-authenticating for expired tokens, and clearing the credential cache (rm -rf ~/.debros/credentials) if multi-wallet conflicts occur. Sources: README.md:166-173, AI_CONTEXT.md:166-173

Gateway Issues

Symptoms include Gateway connection refused, CORS errors, or WebSocket disconnections. Solutions require verifying the gateway is running and accessible, checking CORS configuration for web applications, ensuring proper authentication headers, and verifying namespace configuration. Sources: README.md:175-180, AI_CONTEXT.md:175-180

Database Migration Issues

Symptoms are Migration failed, SQL syntax error, or Version conflict. Solutions involve checking SQL syntax in migration files, ensuring proper statement termination, verifying migration file naming and sequential order, and reviewing migration logs for transaction rollbacks. Sources: README.md:182-187, AI_CONTEXT.md:182-187

Debugging & Health Check Tools

The DeBros Network provides both CLI commands and direct curl commands for debugging and health checks.

CLI Commands

The network-cli tool offers several commands to inspect the status and health of the network. Sources: README.md:190-191, AI_CONTEXT.md:190-191

export LOG_LEVEL=debug # Enable debug logging
./bin/network-cli health
./bin/network-cli peers
./bin/network-cli query "SELECT 1"
./bin/network-cli pubsub publish test "hello"
./bin/network-cli pubsub subscribe test 10s
./bin/network-cli storage put test-key test-value

Sources: README.md:190-197, AI_CONTEXT.md:190-197

The network-cli also supports global flags for configuration:

Flag Description Source
-b, --bootstrap Specifies the bootstrap peer address. cmd/cli/main.go:88-91
-f, --format Sets the output format (e.g., json, table). cmd/cli/main.go:92-95
-t, --timeout Sets the command timeout duration. cmd/cli/main.go:96-100
--production Enables production mode. cmd/cli/main.go:101-102
--disable-anonrc Disables Anyone proxy routing. cmd/cli/main.go:103-104

CLI Command Flow

The CLI commands typically create a client, connect to the network, perform the requested operation, and then disconnect.

sequenceDiagram
    participant CLI as network-cli
    participant Client as pkg/client
    participant Network as DeBros Network

    CLI->>Client: createClient()
    activate Client
    Client->>Network: Connect()
    activate Network
    Network-->>Client: Connection Status
    deactivate Network
    CLI->>Client: Call Operation (e.g., GetPeers)
    activate Client
    Client->>Network: Request
    activate Network
    Network-->>Client: Response Data
    deactivate Network
    deactivate Client
    CLI->>Client: Disconnect()
    activate Client
    Client->>Network: Disconnect Request
    deactivate Client
    CLI: Print Results

Sources: cmd/cli/main.go:107-111, examples/basic_usage.go:21-25

Gateway Health Checks

For the HTTP Gateway, curl commands can be used to check its health and status. Sources: README.md:199-200, AI_CONTEXT.md:199-200

curl http://localhost:8080/health
curl http://localhost:8080/v1/status

Sources: README.md:199-200, AI_CONTEXT.md:199-200

Service Logs

Monitoring service logs is crucial for debugging. Logs for both the node and gateway services can be accessed using journalctl for systemd services or tail for application-specific log files. Sources: README.md:203-204, AI_CONTEXT.md:203-204

# Node service logs
sudo journalctl -u debros-node.service --since "1 hour ago"

# Gateway service logs (if running as service)
sudo journalctl -u debros-gateway.service --since "1 hour ago"

# Application logs
tail -f ./logs/gateway.log
tail -f ./logs/node.log

Sources: README.md:206-211, AI_CONTEXT.md:206-211

The cmd/node/main.go file shows how logging is set up and used within the node application, including the use of zap for structured logging. Sources: cmd/node/main.go:20-27

Configuration for Debugging

The LOG_LEVEL environment variable can be set to debug to enable more verbose logging, which is helpful during troubleshooting. Sources: README.md:190, AI_CONTEXT.md:190

Node configuration can be loaded from YAML files, and command-line flags can override these settings, providing flexibility for debugging specific scenarios. For example, p2p-port, rqlite-http-port, rqlite-raft-port, and rqlite-join-address can be specified via flags. Sources: cmd/node/main.go:34-40, cmd/node/main.go:120-158

graph TD
    A[Start Node] --> B{Parse Flags};
    B --> C{Load Config from YAML?};
    C -- Yes --> D[Load Config File];
    C -- No --> E[Load Default Config];
    D --> F[Apply Flag Overrides];
    E --> F;
    F --> G[Initialize Logger];
    G --> H[Create and Start Node];
    H --> I[Save Peer Info];
    I --> J[Wait for Shutdown Signal];
    J --> K[Stop Node];

Sources: cmd/node/main.go:34-162

End-to-End Testing for Validation

The project includes end-to-end tests (e2e/client_e2e_test.go) that demonstrate client-side interactions with the network, including database operations (create, query, migrate) and storage. These tests can serve as complex health checks or integration tests to validate the system's behavior. Sources: e2e/client_e2e_test.go:10-46

For example, the TestClient_Database_CreateQueryMigrate test performs the following steps:

  1. Loads API key and namespace from environment variables.
  2. Configures the client with bootstrap peers and database endpoints.
  3. Connects to the network.
  4. Creates a unique table.
  5. Inserts data using a transaction.
  6. Queries the inserted data. Sources: e2e/client_e2e_test.go:48-77

This comprehensive approach to debugging and health checks ensures that developers and operators have the necessary tools to maintain and troubleshoot the DeBros Network effectively.

Service Logs

Related Pages

Related topics: Debugging & Health Checks

Relevant source files

The following files were used as context for generating this wiki page:

Service Logs

Service logs in the DeBros Network provide critical insights into the operation and health of its various components, including the core network node and the HTTP gateway. These logs are essential for monitoring system behavior, troubleshooting issues, and debugging application flows. The logging system supports different levels of detail and output formats to cater to various operational needs. Sources: README.md:95-97, AI_CONTEXT.md:12-14

Log Management

The DeBros Network utilizes journalctl for managing service logs when components are run as systemd services, and direct file tailing for application-specific logs. This dual approach ensures comprehensive log coverage and accessibility. Sources: README.md:95-97, AI_CONTEXT.md:12-14

Systemd Service Logs

When the DeBros Network node or gateway is deployed using the automated production install script, they are configured as systemd services. This allows for centralized log management via journalctl. Sources: README.md:43-46, README.md:95-97

# Node service logs
sudo journalctl -u debros-node.service --since "1 hour ago"

# Gateway service logs (if running as service)
sudo journalctl -u debros-gateway.service --since "1 hour ago"

Sources: README.md:95-97, AI_CONTEXT.md:12-14

Application-Specific Logs

Beyond systemd, specific application logs can be directly accessed from their designated directories, providing granular detail. Sources: README.md:95-97

tail -f ./logs/gateway.log
tail -f ./logs/node.log

Sources: README.md:95-97, AI_CONTEXT.md:12-14

Log Levels and Configuration

The logging system supports configurable log levels, allowing operators to adjust the verbosity of the output. The default log level is "info". Sources: README.md:79-81

logging:
  level: "info"
  format: "console"
  output_file: ""

Sources: README.md:79-81

The log level can be dynamically set for debugging purposes using environment variables. Sources: README.md:90-91

export LOG_LEVEL=debug

Sources: README.md:90-91

Node Logging Setup

The cmd/node/main.go file illustrates how the logger is set up for the network node. It uses a ColoredLogger from the pkg/logging package. Sources: cmd/node/main.go:21-31

func setup_logger(component logging.Component) (logger *logging.ColoredLogger) {
	var err error

	logger, err = logging.NewColoredLogger(component, true)
	if err != nil {
		log.Fatalf("Failed to create logger: %v", err)
	}

	return logger
}

Sources: cmd/node/main.go:21-31

The setup_logger function is called at various points within the main.go to initialize loggers for different components, such as ComponentNode and ComponentAnyone. Sources: cmd/node/main.go:34, cmd/node/main.go:121

Log Flow Overview

The logging process involves initializing a logger, setting its component, and then emitting messages at various levels (e.g., Info, Error).

graph TD
    A[Start Node/Gateway] --> B{Initialize Logger};
    B --> C{Set Logging Component};
    C --> D[Load Configuration];
    D --> E{Apply CLI/Env Overrides};
    E --> F[Start Service];
    F --> G{Log Messages (Info, Error, Debug)};
    G --> H{Output to Console/File};
    H --> I{Systemd Journal (if applicable)};
    I --> J[Monitoring/Troubleshooting];

Sources: cmd/node/main.go:21-31, cmd/node/main.go:121, cmd/node/main.go:183-190

Troubleshooting with Logs

Service logs are the primary tool for diagnosing issues within the DeBros Network. Common troubleshooting steps involve examining logs for specific symptoms. Sources: README.md:83-84, AI_CONTEXT.md:12-14

Common Log-Related Symptoms and Solutions

Symptom Description Log Clues Solution Approach Source
Failed to connect to bootstrap peer Node cannot establish connection with bootstrap peers. Error creating node, failed to create node, failed to start node Check node status, firewall settings, peer ID validity. README.md:85-87, AI_CONTEXT.md:12-14
Query timeout or No RQLite connection available Database operations fail or are unresponsive. Error querying data, Error inserting data, failed to create node, failed to start node Ensure RQLite ports (5001, 7001) are open, leader election completed, cluster join config is correct. README.md:88-89, AI_CONTEXT.md:12-14
Message Delivery Failures Pub/Sub messages are not being received by subscribers. Error subscribing, Error publishing Verify topic names, active subscriptions, network connectivity. README.md:90-91, AI_CONTEXT.md:12-14
High Memory Usage Continuous growth in memory consumption. No specific log error, but resource monitoring shows increasing usage. Unsubscribe from topics when no longer needed, monitor connection pool, review message retention. README.md:92-93, AI_CONTEXT.md:12-14
Authentication failed Issues with user authentication via wallet or API keys. Authentication failed, Invalid wallet signature, JWT token expired Check wallet signature format, nonce match, wallet address case-insensitivity, refresh tokens, clear credential cache. README.md:94-98, AI_CONTEXT.md:12-14
Gateway connection refused Problems connecting to the HTTP Gateway. Gateway connection refused, CORS errors, WebSocket disconnections Verify gateway is running, accessible on configured port, check CORS, authentication headers. README.md:99-102, AI_CONTEXT.md:12-14
Migration failed Database schema migration errors. Migration failed, SQL syntax error, Version conflict Check SQL syntax in migration files, proper statement termination, file naming, and sequential order. Review migration logs for transaction rollbacks. README.md:103-106, AI_CONTEXT.md:12-14

Sources: README.md:85-106, AI_CONTEXT.md:12-14

Conclusion

Service logs are an indispensable part of operating the DeBros Network. By leveraging journalctl for systemd services and direct file access for application logs, combined with configurable log levels, operators and developers can effectively monitor, troubleshoot, and maintain the health and performance of the distributed system. Sources: README.md:95-97, AI_CONTEXT.md:12-14