Enhance AI_CONTEXT.md with additional functional requirements and improve formatting for clarity; include detailed descriptions for database operations, key-value storage, pub/sub messaging, and peer discovery.

This commit is contained in:
anonpenguin 2025-08-06 20:47:03 +03:00
parent 9b69aa4432
commit d32a6934e9

View File

@ -1,6 +1,7 @@
# AI Context - DeBros Network Cluster # AI Context - DeBros Network Cluster
## Table of Contents ## Table of Contents
- [Project Overview](#project-overview) - [Project Overview](#project-overview)
- [Product Requirements Document (PRD)](#product-requirements-document-prd) - [Product Requirements Document (PRD)](#product-requirements-document-prd)
- [Architecture Overview](#architecture-overview) - [Architecture Overview](#architecture-overview)
@ -19,30 +20,36 @@
## Product Requirements Document (PRD) ## Product Requirements Document (PRD)
### Vision ### Vision
Create a robust, decentralized network platform that enables applications to seamlessly share data, communicate, and discover peers in a distributed environment. Create a robust, decentralized network platform that enables applications to seamlessly share data, communicate, and discover peers in a distributed environment.
### Core Requirements ### Core Requirements
#### Functional Requirements #### Functional Requirements
1. **Distributed Database Operations** 1. **Distributed Database Operations**
- SQL query execution across network nodes - SQL query execution across network nodes
- ACID transactions with eventual consistency - ACID transactions with eventual consistency
- Schema management and table operations - Schema management and table operations
- Multi-node resilience with automatic failover - Multi-node resilience with automatic failover
2. **Key-Value Storage** 2. **Key-Value Storage**
- Distributed storage with namespace isolation - Distributed storage with namespace isolation
- CRUD operations with consistency guarantees - CRUD operations with consistency guarantees
- Prefix-based querying and key enumeration - Prefix-based querying and key enumeration
- Data replication across network participants - Data replication across network participants
3. **Pub/Sub Messaging** 3. **Pub/Sub Messaging**
- Topic-based publish/subscribe communication - Topic-based publish/subscribe communication
- Real-time message delivery with ordering guarantees - Real-time message delivery with ordering guarantees
- Subscription management with automatic cleanup - Subscription management with automatic cleanup
- Namespace isolation per application - Namespace isolation per application
4. **Peer Discovery & Management** 4. **Peer Discovery & Management**
- Automatic peer discovery using DHT (Distributed Hash Table) - Automatic peer discovery using DHT (Distributed Hash Table)
- Bootstrap node support for network joining - Bootstrap node support for network joining
- Connection health monitoring and recovery - Connection health monitoring and recovery
@ -54,6 +61,7 @@ Create a robust, decentralized network platform that enables applications to sea
- Independent configuration and lifecycle management - Independent configuration and lifecycle management
#### Non-Functional Requirements #### Non-Functional Requirements
1. **Reliability**: 99.9% uptime with automatic failover 1. **Reliability**: 99.9% uptime with automatic failover
2. **Scalability**: Support 100+ nodes with linear performance 2. **Scalability**: Support 100+ nodes with linear performance
3. **Security**: End-to-end encryption for sensitive data 3. **Security**: End-to-end encryption for sensitive data
@ -61,6 +69,7 @@ Create a robust, decentralized network platform that enables applications to sea
5. **Developer Experience**: Simple client API with comprehensive examples 5. **Developer Experience**: Simple client API with comprehensive examples
### Success Metrics ### Success Metrics
- Network uptime > 99.9% - Network uptime > 99.9%
- Peer discovery time < 30 seconds - Peer discovery time < 30 seconds
- Database operation latency < 500ms - Database operation latency < 500ms
@ -99,6 +108,7 @@ Create a robust, decentralized network platform that enables applications to sea
``` ```
### Key Design Principles ### Key Design Principles
1. **Modularity**: Each component can be developed and tested independently 1. **Modularity**: Each component can be developed and tested independently
2. **Fault Tolerance**: Network continues operating even with node failures 2. **Fault Tolerance**: Network continues operating even with node failures
3. **Consistency**: Strong consistency for database operations, eventual consistency for discovery 3. **Consistency**: Strong consistency for database operations, eventual consistency for discovery
@ -157,9 +167,11 @@ debros-testing/
## Key Components ## Key Components
### 1. Network Client (`pkg/client/`) ### 1. Network Client (`pkg/client/`)
The main entry point for applications to interact with the network. The main entry point for applications to interact with the network.
**Core Interfaces:** **Core Interfaces:**
- `NetworkClient`: Main client interface - `NetworkClient`: Main client interface
- `DatabaseClient`: SQL database operations - `DatabaseClient`: SQL database operations
- `StorageClient`: Key-value storage operations - `StorageClient`: Key-value storage operations
@ -167,57 +179,69 @@ The main entry point for applications to interact with the network.
- `NetworkInfo`: Network status and peer information - `NetworkInfo`: Network status and peer information
**Key Features:** **Key Features:**
- Automatic connection management with retry logic - Automatic connection management with retry logic
- Namespace isolation per application - Namespace isolation per application
- Health monitoring and status reporting - Health monitoring and status reporting
- Graceful shutdown and cleanup - Graceful shutdown and cleanup
### 2. Peer Discovery (`pkg/discovery/`) ### 2. Peer Discovery (`pkg/discovery/`)
Handles automatic peer discovery and network topology management. Handles automatic peer discovery and network topology management.
**Discovery Strategies:** **Discovery Strategies:**
- **DHT-based**: Uses Kademlia DHT for efficient peer routing - **DHT-based**: Uses Kademlia DHT for efficient peer routing
- **Peer Exchange**: Learns about new peers from existing connections - **Peer Exchange**: Learns about new peers from existing connections
- **Bootstrap**: Connects to known bootstrap nodes for network entry - **Bootstrap**: Connects to known bootstrap nodes for network entry
**Configuration:** **Configuration:**
- Discovery interval (default: 10 seconds) - Discovery interval (default: 10 seconds)
- Maximum concurrent connections (default: 3) - Maximum concurrent connections (default: 3)
- Connection timeout and retry policies - Connection timeout and retry policies
### 3. Pub/Sub System (`pkg/pubsub/`) ### 3. Pub/Sub System (`pkg/pubsub/`)
Provides reliable, topic-based messaging with ordering guarantees. Provides reliable, topic-based messaging with ordering guarantees.
**Features:** **Features:**
- Topic-based routing with wildcard support - Topic-based routing with wildcard support
- Namespace isolation per application - Namespace isolation per application
- Automatic subscription management - Automatic subscription management
- Message deduplication and ordering - Message deduplication and ordering
**Message Flow:** **Message Flow:**
1. Client subscribes to topic with handler 1. Client subscribes to topic with handler
2. Publisher sends message to topic 2. Publisher sends message to topic
3. Network propagates message to all subscribers 3. Network propagates message to all subscribers
4. Handlers process messages asynchronously 4. Handlers process messages asynchronously
### 4. Database Layer (`pkg/database/`) ### 4. Database Layer (`pkg/database/`)
Distributed SQL database built on RQLite (Raft-based SQLite). Distributed SQL database built on RQLite (Raft-based SQLite).
**Capabilities:** **Capabilities:**
- ACID transactions with strong consistency - ACID transactions with strong consistency
- Automatic leader election and failover - Automatic leader election and failover
- Multi-node replication with conflict resolution - Multi-node replication with conflict resolution
- Schema management and migrations - Schema management and migrations
**Query Types:** **Query Types:**
- Read operations: Served from any node - Read operations: Served from any node
- Write operations: Routed to leader node - Write operations: Routed to leader node
- Transactions: Atomic across multiple statements - Transactions: Atomic across multiple statements
### 5. Storage System (`pkg/storage/`) ### 5. Storage System (`pkg/storage/`)
Distributed key-value store with eventual consistency. Distributed key-value store with eventual consistency.
**Operations:** **Operations:**
- `Put(key, value)`: Store value with key - `Put(key, value)`: Store value with key
- `Get(key)`: Retrieve value by key - `Get(key)`: Retrieve value by key
- `Delete(key)`: Remove key-value pair - `Delete(key)`: Remove key-value pair
@ -227,18 +251,21 @@ Distributed key-value store with eventual consistency.
## Network Protocol ## Network Protocol
### Connection Establishment ### Connection Establishment
1. **Bootstrap Connection**: New nodes connect to bootstrap peers 1. **Bootstrap Connection**: New nodes connect to bootstrap peers
2. **DHT Bootstrap**: Initialize Kademlia DHT for routing 2. **DHT Bootstrap**: Initialize Kademlia DHT for routing
3. **Peer Discovery**: Discover additional peers through DHT 3. **Peer Discovery**: Discover additional peers through DHT
4. **Service Registration**: Register available services (database, storage, pubsub) 4. **Service Registration**: Register available services (database, storage, pubsub)
### Message Types ### Message Types
- **Control Messages**: Node status, heartbeats, topology updates - **Control Messages**: Node status, heartbeats, topology updates
- **Database Messages**: SQL queries, transactions, schema operations - **Database Messages**: SQL queries, transactions, schema operations
- **Storage Messages**: Key-value operations, replication data - **Storage Messages**: Key-value operations, replication data
- **PubSub Messages**: Topic subscriptions, published content - **PubSub Messages**: Topic subscriptions, published content
### Security Model ### Security Model
- **Transport Security**: All connections use TLS/Noise encryption - **Transport Security**: All connections use TLS/Noise encryption
- **Peer Authentication**: Cryptographic peer identity verification - **Peer Authentication**: Cryptographic peer identity verification
- **Message Integrity**: Hash-based message authentication codes - **Message Integrity**: Hash-based message authentication codes
@ -247,6 +274,7 @@ Distributed key-value store with eventual consistency.
## Data Flow ## Data Flow
### Database Operation Flow ### Database Operation Flow
``` ```
Client App → DatabaseClient → RQLite Leader → Raft Consensus → All Nodes Client App → DatabaseClient → RQLite Leader → Raft Consensus → All Nodes
↑ ↓ ↑ ↓
@ -254,6 +282,7 @@ Client App → DatabaseClient → RQLite Leader → Raft Consensus → All Nodes
``` ```
### Storage Operation Flow ### Storage Operation Flow
``` ```
Client App → StorageClient → DHT Routing → Target Nodes → Replication Client App → StorageClient → DHT Routing → Target Nodes → Replication
↑ ↓ ↑ ↓
@ -261,6 +290,7 @@ Client App → StorageClient → DHT Routing → Target Nodes → Replication
``` ```
### PubSub Message Flow ### PubSub Message Flow
``` ```
Publisher → PubSub Manager → Topic Router → All Subscribers → Message Handlers Publisher → PubSub Manager → Topic Router → All Subscribers → Message Handlers
``` ```
@ -268,11 +298,13 @@ Publisher → PubSub Manager → Topic Router → All Subscribers → Message Ha
## Build & Development ## Build & Development
### Prerequisites ### Prerequisites
- Go 1.19+
- Go 1.19+
- Make - Make
- Git - Git
### Build Commands ### Build Commands
```bash ```bash
# Build all executables # Build all executables
make build make build
@ -283,14 +315,15 @@ make test
# Clean build artifacts # Clean build artifacts
make clean make clean
# Start bootstrap node # Start network node (auto-detects bootstrap vs regular)
make start-bootstrap make run-node
# Start regular node # Start additional node
make start-node make run-node
``` ```
### Development Workflow ### Development Workflow
1. **Local Development**: Use `make run-node` (auto-detects bootstrap vs regular) 1. **Local Development**: Use `make run-node` (auto-detects bootstrap vs regular)
2. **Testing**: Run `make test` for unit tests 2. **Testing**: Run `make test` for unit tests
3. **Integration Testing**: Use `scripts/test-multinode.sh` 3. **Integration Testing**: Use `scripts/test-multinode.sh`
@ -299,6 +332,7 @@ make start-node
### Configuration Files ### Configuration Files
#### Bootstrap Node (`configs/bootstrap.yaml`) #### Bootstrap Node (`configs/bootstrap.yaml`)
```yaml ```yaml
node: node:
data_dir: "./data/bootstrap" data_dir: "./data/bootstrap"
@ -311,6 +345,7 @@ database:
``` ```
#### Regular Node (`configs/node.yaml`) #### Regular Node (`configs/node.yaml`)
```yaml ```yaml
node: node:
data_dir: "./data/node" data_dir: "./data/node"
@ -329,6 +364,7 @@ database:
## API Reference ## API Reference
### Client Creation ### Client Creation
```go ```go
import "network/pkg/client" import "network/pkg/client"
@ -348,6 +384,7 @@ defer client.Disconnect()
``` ```
### Database Operations ### Database Operations
```go ```go
// Create table // Create table
err := client.Database().CreateTable(ctx, ` err := client.Database().CreateTable(ctx, `
@ -359,7 +396,7 @@ err := client.Database().CreateTable(ctx, `
`) `)
// Insert data // Insert data
result, err := client.Database().Query(ctx, result, err := client.Database().Query(ctx,
"INSERT INTO users (name, email) VALUES (?, ?)", "INSERT INTO users (name, email) VALUES (?, ?)",
"Alice", "alice@example.com") "Alice", "alice@example.com")
@ -369,6 +406,7 @@ result, err := client.Database().Query(ctx,
``` ```
### Storage Operations ### Storage Operations
```go ```go
// Store data // Store data
err := client.Storage().Put(ctx, "user:123", []byte(`{"name":"Alice"}`)) err := client.Storage().Put(ctx, "user:123", []byte(`{"name":"Alice"}`))
@ -384,6 +422,7 @@ exists, err := client.Storage().Exists(ctx, "user:123")
``` ```
### PubSub Operations ### PubSub Operations
```go ```go
// Subscribe to messages // Subscribe to messages
handler := func(topic string, data []byte) error { handler := func(topic string, data []byte) error {
@ -400,6 +439,7 @@ topics, err := client.PubSub().ListTopics(ctx)
``` ```
### Network Information ### Network Information
```go ```go
// Get network status // Get network status
status, err := client.Network().GetStatus(ctx) status, err := client.Network().GetStatus(ctx)
@ -420,40 +460,51 @@ err := client.Network().ConnectToPeer(ctx, "/ip4/192.168.1.100/tcp/4002/p2p/{PEE
### Common Issues ### Common Issues
#### 1. Bootstrap Connection Failed #### 1. Bootstrap Connection Failed
**Symptoms**: `Failed to connect to bootstrap peer` **Symptoms**: `Failed to connect to bootstrap peer`
**Solutions**: **Solutions**:
- Verify bootstrap node is running and accessible - Verify bootstrap node is running and accessible
- Check firewall settings and port availability - Check firewall settings and port availability
- Validate peer ID in bootstrap address - Validate peer ID in bootstrap address
#### 2. Database Operations Timeout #### 2. Database Operations Timeout
**Symptoms**: `Query timeout` or `No RQLite connection available` **Symptoms**: `Query timeout` or `No RQLite connection available`
**Solutions**: **Solutions**:
- Ensure RQLite ports are not blocked - Ensure RQLite ports are not blocked
- Check if leader election has completed - Check if leader election has completed
- Verify cluster join configuration - Verify cluster join configuration
#### 3. Message Delivery Failures #### 3. Message Delivery Failures
**Symptoms**: Messages not received by subscribers **Symptoms**: Messages not received by subscribers
**Solutions**: **Solutions**:
- Verify topic names match exactly - Verify topic names match exactly
- Check subscription is active before publishing - Check subscription is active before publishing
- Ensure network connectivity between peers - Ensure network connectivity between peers
#### 4. High Memory Usage #### 4. High Memory Usage
**Symptoms**: Memory usage grows continuously **Symptoms**: Memory usage grows continuously
**Solutions**: **Solutions**:
- Check for subscription leaks (unsubscribe when done) - Check for subscription leaks (unsubscribe when done)
- Monitor connection pool size - Monitor connection pool size
- Review message retention policies - Review message retention policies
### Debug Mode ### Debug Mode
Enable debug logging by setting environment variable: Enable debug logging by setting environment variable:
```bash ```bash
export LOG_LEVEL=debug export LOG_LEVEL=debug
``` ```
### Health Checks ### Health Checks
```go ```go
health, err := client.Health() health, err := client.Health()
if health.Status != "healthy" { if health.Status != "healthy" {
@ -462,6 +513,7 @@ if health.Status != "healthy" {
``` ```
### Network Diagnostics ### Network Diagnostics
```bash ```bash
# Check node connectivity # Check node connectivity
./bin/network-cli peers ./bin/network-cli peers
@ -490,4 +542,4 @@ This serves as both a practical example and a reference implementation for build
--- ---
*This document provides comprehensive context for AI systems to understand the DeBros Network Cluster project architecture, implementation details, and usage patterns.* _This document provides comprehensive context for AI systems to understand the DeBros Network Cluster project architecture, implementation details, and usage patterns._