mirror of
https://github.com/DeBrosOfficial/network.git
synced 2025-10-06 12:09:07 +00:00
Enhance AI_CONTEXT.md with additional functional requirements and improve formatting for clarity; include detailed descriptions for database operations, key-value storage, pub/sub messaging, and peer discovery.
This commit is contained in:
parent
9b69aa4432
commit
d32a6934e9
@ -1,6 +1,7 @@
|
|||||||
# AI Context - DeBros Network Cluster
|
# AI Context - DeBros Network Cluster
|
||||||
|
|
||||||
## Table of Contents
|
## Table of Contents
|
||||||
|
|
||||||
- [Project Overview](#project-overview)
|
- [Project Overview](#project-overview)
|
||||||
- [Product Requirements Document (PRD)](#product-requirements-document-prd)
|
- [Product Requirements Document (PRD)](#product-requirements-document-prd)
|
||||||
- [Architecture Overview](#architecture-overview)
|
- [Architecture Overview](#architecture-overview)
|
||||||
@ -19,30 +20,36 @@
|
|||||||
## Product Requirements Document (PRD)
|
## Product Requirements Document (PRD)
|
||||||
|
|
||||||
### Vision
|
### Vision
|
||||||
|
|
||||||
Create a robust, decentralized network platform that enables applications to seamlessly share data, communicate, and discover peers in a distributed environment.
|
Create a robust, decentralized network platform that enables applications to seamlessly share data, communicate, and discover peers in a distributed environment.
|
||||||
|
|
||||||
### Core Requirements
|
### Core Requirements
|
||||||
|
|
||||||
#### Functional Requirements
|
#### Functional Requirements
|
||||||
|
|
||||||
1. **Distributed Database Operations**
|
1. **Distributed Database Operations**
|
||||||
|
|
||||||
- SQL query execution across network nodes
|
- SQL query execution across network nodes
|
||||||
- ACID transactions with eventual consistency
|
- ACID transactions with eventual consistency
|
||||||
- Schema management and table operations
|
- Schema management and table operations
|
||||||
- Multi-node resilience with automatic failover
|
- Multi-node resilience with automatic failover
|
||||||
|
|
||||||
2. **Key-Value Storage**
|
2. **Key-Value Storage**
|
||||||
|
|
||||||
- Distributed storage with namespace isolation
|
- Distributed storage with namespace isolation
|
||||||
- CRUD operations with consistency guarantees
|
- CRUD operations with consistency guarantees
|
||||||
- Prefix-based querying and key enumeration
|
- Prefix-based querying and key enumeration
|
||||||
- Data replication across network participants
|
- Data replication across network participants
|
||||||
|
|
||||||
3. **Pub/Sub Messaging**
|
3. **Pub/Sub Messaging**
|
||||||
|
|
||||||
- Topic-based publish/subscribe communication
|
- Topic-based publish/subscribe communication
|
||||||
- Real-time message delivery with ordering guarantees
|
- Real-time message delivery with ordering guarantees
|
||||||
- Subscription management with automatic cleanup
|
- Subscription management with automatic cleanup
|
||||||
- Namespace isolation per application
|
- Namespace isolation per application
|
||||||
|
|
||||||
4. **Peer Discovery & Management**
|
4. **Peer Discovery & Management**
|
||||||
|
|
||||||
- Automatic peer discovery using DHT (Distributed Hash Table)
|
- Automatic peer discovery using DHT (Distributed Hash Table)
|
||||||
- Bootstrap node support for network joining
|
- Bootstrap node support for network joining
|
||||||
- Connection health monitoring and recovery
|
- Connection health monitoring and recovery
|
||||||
@ -54,6 +61,7 @@ Create a robust, decentralized network platform that enables applications to sea
|
|||||||
- Independent configuration and lifecycle management
|
- Independent configuration and lifecycle management
|
||||||
|
|
||||||
#### Non-Functional Requirements
|
#### Non-Functional Requirements
|
||||||
|
|
||||||
1. **Reliability**: 99.9% uptime with automatic failover
|
1. **Reliability**: 99.9% uptime with automatic failover
|
||||||
2. **Scalability**: Support 100+ nodes with linear performance
|
2. **Scalability**: Support 100+ nodes with linear performance
|
||||||
3. **Security**: End-to-end encryption for sensitive data
|
3. **Security**: End-to-end encryption for sensitive data
|
||||||
@ -61,6 +69,7 @@ Create a robust, decentralized network platform that enables applications to sea
|
|||||||
5. **Developer Experience**: Simple client API with comprehensive examples
|
5. **Developer Experience**: Simple client API with comprehensive examples
|
||||||
|
|
||||||
### Success Metrics
|
### Success Metrics
|
||||||
|
|
||||||
- Network uptime > 99.9%
|
- Network uptime > 99.9%
|
||||||
- Peer discovery time < 30 seconds
|
- Peer discovery time < 30 seconds
|
||||||
- Database operation latency < 500ms
|
- Database operation latency < 500ms
|
||||||
@ -99,6 +108,7 @@ Create a robust, decentralized network platform that enables applications to sea
|
|||||||
```
|
```
|
||||||
|
|
||||||
### Key Design Principles
|
### Key Design Principles
|
||||||
|
|
||||||
1. **Modularity**: Each component can be developed and tested independently
|
1. **Modularity**: Each component can be developed and tested independently
|
||||||
2. **Fault Tolerance**: Network continues operating even with node failures
|
2. **Fault Tolerance**: Network continues operating even with node failures
|
||||||
3. **Consistency**: Strong consistency for database operations, eventual consistency for discovery
|
3. **Consistency**: Strong consistency for database operations, eventual consistency for discovery
|
||||||
@ -157,9 +167,11 @@ debros-testing/
|
|||||||
## Key Components
|
## Key Components
|
||||||
|
|
||||||
### 1. Network Client (`pkg/client/`)
|
### 1. Network Client (`pkg/client/`)
|
||||||
|
|
||||||
The main entry point for applications to interact with the network.
|
The main entry point for applications to interact with the network.
|
||||||
|
|
||||||
**Core Interfaces:**
|
**Core Interfaces:**
|
||||||
|
|
||||||
- `NetworkClient`: Main client interface
|
- `NetworkClient`: Main client interface
|
||||||
- `DatabaseClient`: SQL database operations
|
- `DatabaseClient`: SQL database operations
|
||||||
- `StorageClient`: Key-value storage operations
|
- `StorageClient`: Key-value storage operations
|
||||||
@ -167,57 +179,69 @@ The main entry point for applications to interact with the network.
|
|||||||
- `NetworkInfo`: Network status and peer information
|
- `NetworkInfo`: Network status and peer information
|
||||||
|
|
||||||
**Key Features:**
|
**Key Features:**
|
||||||
|
|
||||||
- Automatic connection management with retry logic
|
- Automatic connection management with retry logic
|
||||||
- Namespace isolation per application
|
- Namespace isolation per application
|
||||||
- Health monitoring and status reporting
|
- Health monitoring and status reporting
|
||||||
- Graceful shutdown and cleanup
|
- Graceful shutdown and cleanup
|
||||||
|
|
||||||
### 2. Peer Discovery (`pkg/discovery/`)
|
### 2. Peer Discovery (`pkg/discovery/`)
|
||||||
|
|
||||||
Handles automatic peer discovery and network topology management.
|
Handles automatic peer discovery and network topology management.
|
||||||
|
|
||||||
**Discovery Strategies:**
|
**Discovery Strategies:**
|
||||||
|
|
||||||
- **DHT-based**: Uses Kademlia DHT for efficient peer routing
|
- **DHT-based**: Uses Kademlia DHT for efficient peer routing
|
||||||
- **Peer Exchange**: Learns about new peers from existing connections
|
- **Peer Exchange**: Learns about new peers from existing connections
|
||||||
- **Bootstrap**: Connects to known bootstrap nodes for network entry
|
- **Bootstrap**: Connects to known bootstrap nodes for network entry
|
||||||
|
|
||||||
**Configuration:**
|
**Configuration:**
|
||||||
|
|
||||||
- Discovery interval (default: 10 seconds)
|
- Discovery interval (default: 10 seconds)
|
||||||
- Maximum concurrent connections (default: 3)
|
- Maximum concurrent connections (default: 3)
|
||||||
- Connection timeout and retry policies
|
- Connection timeout and retry policies
|
||||||
|
|
||||||
### 3. Pub/Sub System (`pkg/pubsub/`)
|
### 3. Pub/Sub System (`pkg/pubsub/`)
|
||||||
|
|
||||||
Provides reliable, topic-based messaging with ordering guarantees.
|
Provides reliable, topic-based messaging with ordering guarantees.
|
||||||
|
|
||||||
**Features:**
|
**Features:**
|
||||||
|
|
||||||
- Topic-based routing with wildcard support
|
- Topic-based routing with wildcard support
|
||||||
- Namespace isolation per application
|
- Namespace isolation per application
|
||||||
- Automatic subscription management
|
- Automatic subscription management
|
||||||
- Message deduplication and ordering
|
- Message deduplication and ordering
|
||||||
|
|
||||||
**Message Flow:**
|
**Message Flow:**
|
||||||
|
|
||||||
1. Client subscribes to topic with handler
|
1. Client subscribes to topic with handler
|
||||||
2. Publisher sends message to topic
|
2. Publisher sends message to topic
|
||||||
3. Network propagates message to all subscribers
|
3. Network propagates message to all subscribers
|
||||||
4. Handlers process messages asynchronously
|
4. Handlers process messages asynchronously
|
||||||
|
|
||||||
### 4. Database Layer (`pkg/database/`)
|
### 4. Database Layer (`pkg/database/`)
|
||||||
|
|
||||||
Distributed SQL database built on RQLite (Raft-based SQLite).
|
Distributed SQL database built on RQLite (Raft-based SQLite).
|
||||||
|
|
||||||
**Capabilities:**
|
**Capabilities:**
|
||||||
|
|
||||||
- ACID transactions with strong consistency
|
- ACID transactions with strong consistency
|
||||||
- Automatic leader election and failover
|
- Automatic leader election and failover
|
||||||
- Multi-node replication with conflict resolution
|
- Multi-node replication with conflict resolution
|
||||||
- Schema management and migrations
|
- Schema management and migrations
|
||||||
|
|
||||||
**Query Types:**
|
**Query Types:**
|
||||||
|
|
||||||
- Read operations: Served from any node
|
- Read operations: Served from any node
|
||||||
- Write operations: Routed to leader node
|
- Write operations: Routed to leader node
|
||||||
- Transactions: Atomic across multiple statements
|
- Transactions: Atomic across multiple statements
|
||||||
|
|
||||||
### 5. Storage System (`pkg/storage/`)
|
### 5. Storage System (`pkg/storage/`)
|
||||||
|
|
||||||
Distributed key-value store with eventual consistency.
|
Distributed key-value store with eventual consistency.
|
||||||
|
|
||||||
**Operations:**
|
**Operations:**
|
||||||
|
|
||||||
- `Put(key, value)`: Store value with key
|
- `Put(key, value)`: Store value with key
|
||||||
- `Get(key)`: Retrieve value by key
|
- `Get(key)`: Retrieve value by key
|
||||||
- `Delete(key)`: Remove key-value pair
|
- `Delete(key)`: Remove key-value pair
|
||||||
@ -227,18 +251,21 @@ Distributed key-value store with eventual consistency.
|
|||||||
## Network Protocol
|
## Network Protocol
|
||||||
|
|
||||||
### Connection Establishment
|
### Connection Establishment
|
||||||
|
|
||||||
1. **Bootstrap Connection**: New nodes connect to bootstrap peers
|
1. **Bootstrap Connection**: New nodes connect to bootstrap peers
|
||||||
2. **DHT Bootstrap**: Initialize Kademlia DHT for routing
|
2. **DHT Bootstrap**: Initialize Kademlia DHT for routing
|
||||||
3. **Peer Discovery**: Discover additional peers through DHT
|
3. **Peer Discovery**: Discover additional peers through DHT
|
||||||
4. **Service Registration**: Register available services (database, storage, pubsub)
|
4. **Service Registration**: Register available services (database, storage, pubsub)
|
||||||
|
|
||||||
### Message Types
|
### Message Types
|
||||||
|
|
||||||
- **Control Messages**: Node status, heartbeats, topology updates
|
- **Control Messages**: Node status, heartbeats, topology updates
|
||||||
- **Database Messages**: SQL queries, transactions, schema operations
|
- **Database Messages**: SQL queries, transactions, schema operations
|
||||||
- **Storage Messages**: Key-value operations, replication data
|
- **Storage Messages**: Key-value operations, replication data
|
||||||
- **PubSub Messages**: Topic subscriptions, published content
|
- **PubSub Messages**: Topic subscriptions, published content
|
||||||
|
|
||||||
### Security Model
|
### Security Model
|
||||||
|
|
||||||
- **Transport Security**: All connections use TLS/Noise encryption
|
- **Transport Security**: All connections use TLS/Noise encryption
|
||||||
- **Peer Authentication**: Cryptographic peer identity verification
|
- **Peer Authentication**: Cryptographic peer identity verification
|
||||||
- **Message Integrity**: Hash-based message authentication codes
|
- **Message Integrity**: Hash-based message authentication codes
|
||||||
@ -247,6 +274,7 @@ Distributed key-value store with eventual consistency.
|
|||||||
## Data Flow
|
## Data Flow
|
||||||
|
|
||||||
### Database Operation Flow
|
### Database Operation Flow
|
||||||
|
|
||||||
```
|
```
|
||||||
Client App → DatabaseClient → RQLite Leader → Raft Consensus → All Nodes
|
Client App → DatabaseClient → RQLite Leader → Raft Consensus → All Nodes
|
||||||
↑ ↓
|
↑ ↓
|
||||||
@ -254,6 +282,7 @@ Client App → DatabaseClient → RQLite Leader → Raft Consensus → All Nodes
|
|||||||
```
|
```
|
||||||
|
|
||||||
### Storage Operation Flow
|
### Storage Operation Flow
|
||||||
|
|
||||||
```
|
```
|
||||||
Client App → StorageClient → DHT Routing → Target Nodes → Replication
|
Client App → StorageClient → DHT Routing → Target Nodes → Replication
|
||||||
↑ ↓
|
↑ ↓
|
||||||
@ -261,6 +290,7 @@ Client App → StorageClient → DHT Routing → Target Nodes → Replication
|
|||||||
```
|
```
|
||||||
|
|
||||||
### PubSub Message Flow
|
### PubSub Message Flow
|
||||||
|
|
||||||
```
|
```
|
||||||
Publisher → PubSub Manager → Topic Router → All Subscribers → Message Handlers
|
Publisher → PubSub Manager → Topic Router → All Subscribers → Message Handlers
|
||||||
```
|
```
|
||||||
@ -268,11 +298,13 @@ Publisher → PubSub Manager → Topic Router → All Subscribers → Message Ha
|
|||||||
## Build & Development
|
## Build & Development
|
||||||
|
|
||||||
### Prerequisites
|
### Prerequisites
|
||||||
|
|
||||||
- Go 1.19+
|
- Go 1.19+
|
||||||
- Make
|
- Make
|
||||||
- Git
|
- Git
|
||||||
|
|
||||||
### Build Commands
|
### Build Commands
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Build all executables
|
# Build all executables
|
||||||
make build
|
make build
|
||||||
@ -283,14 +315,15 @@ make test
|
|||||||
# Clean build artifacts
|
# Clean build artifacts
|
||||||
make clean
|
make clean
|
||||||
|
|
||||||
# Start bootstrap node
|
# Start network node (auto-detects bootstrap vs regular)
|
||||||
make start-bootstrap
|
make run-node
|
||||||
|
|
||||||
# Start regular node
|
# Start additional node
|
||||||
make start-node
|
make run-node
|
||||||
```
|
```
|
||||||
|
|
||||||
### Development Workflow
|
### Development Workflow
|
||||||
|
|
||||||
1. **Local Development**: Use `make run-node` (auto-detects bootstrap vs regular)
|
1. **Local Development**: Use `make run-node` (auto-detects bootstrap vs regular)
|
||||||
2. **Testing**: Run `make test` for unit tests
|
2. **Testing**: Run `make test` for unit tests
|
||||||
3. **Integration Testing**: Use `scripts/test-multinode.sh`
|
3. **Integration Testing**: Use `scripts/test-multinode.sh`
|
||||||
@ -299,6 +332,7 @@ make start-node
|
|||||||
### Configuration Files
|
### Configuration Files
|
||||||
|
|
||||||
#### Bootstrap Node (`configs/bootstrap.yaml`)
|
#### Bootstrap Node (`configs/bootstrap.yaml`)
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
node:
|
node:
|
||||||
data_dir: "./data/bootstrap"
|
data_dir: "./data/bootstrap"
|
||||||
@ -311,6 +345,7 @@ database:
|
|||||||
```
|
```
|
||||||
|
|
||||||
#### Regular Node (`configs/node.yaml`)
|
#### Regular Node (`configs/node.yaml`)
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
node:
|
node:
|
||||||
data_dir: "./data/node"
|
data_dir: "./data/node"
|
||||||
@ -329,6 +364,7 @@ database:
|
|||||||
## API Reference
|
## API Reference
|
||||||
|
|
||||||
### Client Creation
|
### Client Creation
|
||||||
|
|
||||||
```go
|
```go
|
||||||
import "network/pkg/client"
|
import "network/pkg/client"
|
||||||
|
|
||||||
@ -348,6 +384,7 @@ defer client.Disconnect()
|
|||||||
```
|
```
|
||||||
|
|
||||||
### Database Operations
|
### Database Operations
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// Create table
|
// Create table
|
||||||
err := client.Database().CreateTable(ctx, `
|
err := client.Database().CreateTable(ctx, `
|
||||||
@ -369,6 +406,7 @@ result, err := client.Database().Query(ctx,
|
|||||||
```
|
```
|
||||||
|
|
||||||
### Storage Operations
|
### Storage Operations
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// Store data
|
// Store data
|
||||||
err := client.Storage().Put(ctx, "user:123", []byte(`{"name":"Alice"}`))
|
err := client.Storage().Put(ctx, "user:123", []byte(`{"name":"Alice"}`))
|
||||||
@ -384,6 +422,7 @@ exists, err := client.Storage().Exists(ctx, "user:123")
|
|||||||
```
|
```
|
||||||
|
|
||||||
### PubSub Operations
|
### PubSub Operations
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// Subscribe to messages
|
// Subscribe to messages
|
||||||
handler := func(topic string, data []byte) error {
|
handler := func(topic string, data []byte) error {
|
||||||
@ -400,6 +439,7 @@ topics, err := client.PubSub().ListTopics(ctx)
|
|||||||
```
|
```
|
||||||
|
|
||||||
### Network Information
|
### Network Information
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// Get network status
|
// Get network status
|
||||||
status, err := client.Network().GetStatus(ctx)
|
status, err := client.Network().GetStatus(ctx)
|
||||||
@ -420,40 +460,51 @@ err := client.Network().ConnectToPeer(ctx, "/ip4/192.168.1.100/tcp/4002/p2p/{PEE
|
|||||||
### Common Issues
|
### Common Issues
|
||||||
|
|
||||||
#### 1. Bootstrap Connection Failed
|
#### 1. Bootstrap Connection Failed
|
||||||
|
|
||||||
**Symptoms**: `Failed to connect to bootstrap peer`
|
**Symptoms**: `Failed to connect to bootstrap peer`
|
||||||
**Solutions**:
|
**Solutions**:
|
||||||
|
|
||||||
- Verify bootstrap node is running and accessible
|
- Verify bootstrap node is running and accessible
|
||||||
- Check firewall settings and port availability
|
- Check firewall settings and port availability
|
||||||
- Validate peer ID in bootstrap address
|
- Validate peer ID in bootstrap address
|
||||||
|
|
||||||
#### 2. Database Operations Timeout
|
#### 2. Database Operations Timeout
|
||||||
|
|
||||||
**Symptoms**: `Query timeout` or `No RQLite connection available`
|
**Symptoms**: `Query timeout` or `No RQLite connection available`
|
||||||
**Solutions**:
|
**Solutions**:
|
||||||
|
|
||||||
- Ensure RQLite ports are not blocked
|
- Ensure RQLite ports are not blocked
|
||||||
- Check if leader election has completed
|
- Check if leader election has completed
|
||||||
- Verify cluster join configuration
|
- Verify cluster join configuration
|
||||||
|
|
||||||
#### 3. Message Delivery Failures
|
#### 3. Message Delivery Failures
|
||||||
|
|
||||||
**Symptoms**: Messages not received by subscribers
|
**Symptoms**: Messages not received by subscribers
|
||||||
**Solutions**:
|
**Solutions**:
|
||||||
|
|
||||||
- Verify topic names match exactly
|
- Verify topic names match exactly
|
||||||
- Check subscription is active before publishing
|
- Check subscription is active before publishing
|
||||||
- Ensure network connectivity between peers
|
- Ensure network connectivity between peers
|
||||||
|
|
||||||
#### 4. High Memory Usage
|
#### 4. High Memory Usage
|
||||||
|
|
||||||
**Symptoms**: Memory usage grows continuously
|
**Symptoms**: Memory usage grows continuously
|
||||||
**Solutions**:
|
**Solutions**:
|
||||||
|
|
||||||
- Check for subscription leaks (unsubscribe when done)
|
- Check for subscription leaks (unsubscribe when done)
|
||||||
- Monitor connection pool size
|
- Monitor connection pool size
|
||||||
- Review message retention policies
|
- Review message retention policies
|
||||||
|
|
||||||
### Debug Mode
|
### Debug Mode
|
||||||
|
|
||||||
Enable debug logging by setting environment variable:
|
Enable debug logging by setting environment variable:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export LOG_LEVEL=debug
|
export LOG_LEVEL=debug
|
||||||
```
|
```
|
||||||
|
|
||||||
### Health Checks
|
### Health Checks
|
||||||
|
|
||||||
```go
|
```go
|
||||||
health, err := client.Health()
|
health, err := client.Health()
|
||||||
if health.Status != "healthy" {
|
if health.Status != "healthy" {
|
||||||
@ -462,6 +513,7 @@ if health.Status != "healthy" {
|
|||||||
```
|
```
|
||||||
|
|
||||||
### Network Diagnostics
|
### Network Diagnostics
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Check node connectivity
|
# Check node connectivity
|
||||||
./bin/network-cli peers
|
./bin/network-cli peers
|
||||||
@ -490,4 +542,4 @@ This serves as both a practical example and a reference implementation for build
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
*This document provides comprehensive context for AI systems to understand the DeBros Network Cluster project architecture, implementation details, and usage patterns.*
|
_This document provides comprehensive context for AI systems to understand the DeBros Network Cluster project architecture, implementation details, and usage patterns._
|
||||||
|
Loading…
x
Reference in New Issue
Block a user