"}` | `{status:"ok"}` | [AI_CONTEXT.md:168](), [README.md:108]() |
| `/v1/storage/list` | `GET` | Lists keys matching an optional prefix. | `query: ?prefix=` | `{"keys":["", "..."]}` | [AI_CONTEXT.md:169](), [README.md:107]() |
| `/v1/storage/exists` | `GET` | Checks if a key exists. | `query: ?key=` | `{"exists":boolean}` | [AI_CONTEXT.md:169](), [README.md:106]() |
### Client Library Usage (Go)
The `pkg/client` package provides a Go client for programmatic interaction with the Storage System. Sources: [examples/basic_usage.go:19-21]()
```go
func demonstrateStorage(client client.NetworkClient) {
ctx := context.Background()
storage := client.Storage()
log.Printf("=== Storage Operations ===")
// Store some data
key := "user:123"
value := []byte(`{"name": "Alice", "age": 30}`)
if err := storage.Put(ctx, key, value); err != nil {
log.Printf("Error storing data: %v", err)
return
}
log.Printf("Data stored successfully")
// Retrieve data
retrieved, err := storage.Get(ctx, key)
if err != nil {
log.Printf("Error retrieving data: %v", err)
return
}
log.Printf("Retrieved data: %s", string(retrieved))
// Check if key exists
exists, err := storage.Exists(ctx, key)
if err != nil {
log.Printf("Error checking existence: %v", err)
return
}
log.Printf("Key exists: %v", exists)
// List keys
keys, err := storage.List(ctx, "user:", 10) // The 'limit' parameter (10) is not reflected in the docs
if err != nil {
log.Printf("Error listing keys: %v", err)
return
}
log.Printf("Keys: %v", keys)
}
```
Sources: [examples/basic_usage.go:70-109]()
### Client Library Usage (TypeScript SDK)
A TypeScript SDK client is available for web and Node.js applications, mirroring the Go client's functionality. Sources: [examples/sdk-typescript/src/client.ts:25-63]()
```typescript
// Storage
async put(key: string, value: Uint8Array | string): Promise {
const body = typeof value === 'string' ? new TextEncoder().encode(value) : value;
const r = await this.http(`${this.baseUrl}/v1/storage/put?key=${encodeURIComponent(key)}`, {
method: 'POST', headers: { 'X-API-Key': this.apiKey }, body
});
if (!r.ok) throw new Error(`put failed: ${r.status}`);
}
async get(key: string): Promise {
const r = await this.http(`${this.baseUrl}/v1/storage/get?key=${encodeURIComponent(key)}`, {
headers: { 'X-API-Key': this.apiKey }
});
if (!r.ok) throw new Error(`get failed: ${r.status}`);
const buf = new Uint8Array(await r.arrayBuffer());
return buf;
}
async exists(key: string): Promise {
const r = await this.http(`${this.baseUrl}/v1/storage/exists?key=${encodeURIComponent(key)}`, {
headers: this.headers(false)
});
if (!r.ok) throw new Error(`exists failed: ${r.status}`);
const j = await r.json();
return !!j.exists;
}
async list(prefix = ""): Promise {
const r = await this.http(`${this.baseUrl}/v1/storage/list?prefix=${encodeURIComponent(prefix)}`, {
headers: this.headers(false)
});
if (!r.ok) throw new Error(`list failed: ${r.status}`);
const j = await r.json();
return j.keys || [];
}
async delete(key: string): Promise {
const r = await this.http(`${this.baseUrl}/v1/storage/delete`, {
method: 'POST', headers: this.headers(), body: JSON.stringify({ key })
});
if (!r.ok) throw new Error(`delete failed: ${r.status}`);
}
```
Sources: [examples/sdk-typescript/src/client.ts:25-63]()
### CLI Usage
The `network-cli` tool provides command-line access to storage operations. Sources: [cmd/cli/main.go:60-61]()
```bash
# Store data
network-cli storage put test-key "Hello Network"
# Retrieve data
network-cli storage get test-key
```
Sources: [README.md:65-66]()
## Security and Isolation
The Storage System incorporates several security features:
* **Namespace Enforcement:** All operations are automatically prefixed with a namespace, ensuring data isolation between different applications or tenants. This is enforced by middleware for specific routes. Sources: [README.md:139-140](), [AI_CONTEXT.md:124-125]()
* **Authentication Middleware:** Flexible authentication mechanisms, including JWT and API keys, are used to secure access to storage endpoints. Sources: [README.md:142]()
* **Transport Security:** All network communications leverage Noise/TLS encryption for secure data transfer. Sources: [README.md:141]()
## Conclusion
The DeBros Network's Storage System offers a robust, decentralized, and secure key-value store. Its namespaced design, comprehensive API, and integration with the network's security features make it suitable for distributed applications requiring resilient data persistence and isolation.
---
## Pub/Sub System
### Related Pages
Related topics: [CLI Pub/Sub Operations](#cli-pubsub-ops), [Gateway API Endpoints](#gateway-api-endpoints)
Relevant source files
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
- [pkg/gateway/pubsub_handlers.go](https://git.debros.io/DeBros/network/src/main/pkg/gateway/pubsub_handlers.go)
- [e2e/gateway_e2e_test.go](https://git.debros.io/DeBros/network/src/main/e2e/gateway_e2e_test.go)
- [examples/sdk-typescript/src/client.ts](https://git.debros.io/DeBros/network/src/main/examples/sdk-typescript/src/client.ts)
# Pub/Sub System
The DeBros Network Cluster incorporates a robust Pub/Sub (Publish/Subscribe) messaging system designed for real-time, topic-based communication. This system enables different components or applications within the network to exchange messages asynchronously without direct coupling, facilitating resilient and scalable data flow. It supports both WebSocket and REST interfaces for publishing and subscribing to topics. The Pub/Sub system is namespaced, ensuring application isolation and preventing topic name collisions.
Sources: [README.md:15-16](), [AI_CONTEXT.md:88-89]()
## Architecture and Components
The Pub/Sub system is an integral part of the DeBros Network's client API, allowing applications to interact with it through a lightweight Go client or an HTTP Gateway. Messages are delivered over the underlying LibP2P network.
Sources: [README.md:46-55](), [AI_CONTEXT.md:57-66]()
```mermaid
graph TD
A[Application] -->|Client API| B(DeBros Network Client)
B --> C[PubSub Service]
C --> D[LibP2P Host]
D --> E[Noise/TLS Encryption]
E --> F[Network Peers]
F --> G[Other Applications]
G -->|Client API| H(DeBros Network Client)
H --> I[PubSub Service]
I --> J[LibP2P Host]
J --> K[Noise/TLS Encryption]
```
Sources: [README.md:46-55]()
### Key Features
The Pub/Sub system offers several key features:
Sources: [README.md:15-16]()
| Feature | Description |
| :----------------------- | :------------------------------------------------------------------------------------------------------ |
| Topic-based | Messages are categorized and delivered based on specific topics. |
| Real-time | Designed for low-latency message delivery. |
| Namespaced | Topics are prefixed internally with a namespace (`ns::::...`) for multi-tenancy and isolation. |
| Automatic Cleanup | Implies management of subscriptions and topics. |
| WebSocket Support | Provides a persistent connection for real-time message streaming. |
| REST Interface | Allows publishing messages and listing topics via HTTP endpoints. |
Sources: [README.md:15-16](), [AI_CONTEXT.md:176-177](), [AI_CONTEXT.md:195-197](), [AI_CONTEXT.md:214-216](), [AI_CONTEXT.md:220-222]()
## Data Flow and Operations
The Pub/Sub system supports both publishing messages to topics and subscribing to receive messages from topics. These operations can be performed via the Go client library or through the HTTP Gateway.
Sources: [AI_CONTEXT.md:176-177](), [AI_CONTEXT.md:195-197](), [AI_CONTEXT.md:214-216](), [AI_CONTEXT.md:220-222]()
### Publishing Messages
Messages can be published using either the Go client or the HTTP Gateway's REST endpoint.
#### Go Client Publishing
Applications using the Go client can publish messages by calling the `Publish` method on the `PubSub` client.
Sources: [examples/basic_usage.go:134-137]()
```go
err := client.PubSub().Publish(ctx, "topic", []byte("msg"))
```
Sources: [AI_CONTEXT.md:27]()
#### HTTP Gateway REST Publishing
The Gateway provides a `POST /v1/pubsub/publish` endpoint for publishing messages. The message data must be base64 encoded.
Sources: [AI_CONTEXT.md:195-197](), [pkg/gateway/pubsub_handlers.go:61-64]()
```http
POST /v1/pubsub/publish → body {topic, data_base64} → {status:"ok"}
```
Sources: [AI_CONTEXT.md:195-197]()
```mermaid
sequenceDiagram
participant App as Application
participant GW as HTTP Gateway
participant PS as PubSub Service
participant Net as DeBros Network
App->>GW: POST /v1/pubsub/publish {topic, data_base64}
activate GW
GW->>PS: PubSub().Publish(ctx, topic, data)
activate PS
PS->>Net: Send message to network
deactivate PS
GW-->>App: HTTP 200 OK {status:"ok"}
deactivate GW
```
Sources: [pkg/gateway/pubsub_handlers.go:61-91]()
### Subscribing to Messages
Subscriptions are primarily handled via the Go client or through a WebSocket connection to the HTTP Gateway.
#### Go Client Subscription
The Go client allows applications to subscribe to topics by providing a handler function that processes incoming messages.
Sources: [examples/basic_usage.go:125-132]()
```go
handler := func(topic string, data []byte) error {
log.Printf("Received message on topic '%s': %s", topic, string(data))
return nil
}
if err := pubsub.Subscribe(ctx, topic, handler); err != nil {
log.Printf("Error subscribing: %v", err)
return
}
```
Sources: [examples/basic_usage.go:125-132]()
#### HTTP Gateway WebSocket Subscription
The Gateway offers a WebSocket endpoint `GET /v1/pubsub/ws?topic=` for real-time message reception. Clients connect to this endpoint, and the server sends messages as binary frames. Client text/binary frames sent over the WebSocket are published to the same namespaced topic.
Sources: [AI_CONTEXT.md:191-193](), [pkg/gateway/pubsub_handlers.go:15-32](), [examples/sdk-typescript/src/client.ts:83-87]()
```http
GET /v1/pubsub/ws?topic=
```
Sources: [AI_CONTEXT.md:191-193]()
```mermaid
sequenceDiagram
participant Client as WebSocket Client
participant GW as HTTP Gateway
participant PS as PubSub Service
participant Net as DeBros Network
Client->>GW: GET /v1/pubsub/ws?topic=
activate GW
GW->>PS: PubSub().Subscribe(ctx, topic, handler)
activate PS
PS->>Net: Register subscription
deactivate PS
GW-->>Client: WebSocket connection established
loop Message Flow
Net->>PS: Message received for topic
activate PS
PS->>GW: Forward message to handler
deactivate PS
GW-->>Client: WebSocket binary frame (message)
end
Client->>GW: WebSocket text/binary frame (publish)
activate GW
GW->>PS: PubSub().Publish(ctx, topic, data)
activate PS
PS->>Net: Send message to network
deactivate PS
```
Sources: [pkg/gateway/pubsub_handlers.go:15-32]()
### Listing Topics
Both the Go client and the HTTP Gateway provide functionality to list active topics. The Gateway's `GET /v1/pubsub/topics` endpoint returns topics trimmed to the caller's namespace.
Sources: [AI_CONTEXT.md:198-199](), [pkg/gateway/pubsub_handlers.go:94-110]()
```http
GET /v1/pubsub/topics → {topics:["", ...]}
```
Sources: [AI_CONTEXT.md:198-199]()
## Namespace Enforcement
The Pub/Sub system strictly enforces namespaces. All topics are internally prefixed with `ns::::...` to ensure isolation between different applications or tenants. This means that a client operating within a specific namespace will only interact with topics belonging to that namespace. The `resolveNamespaceFromRequest` function extracts the namespace from the request context, typically set by an authentication middleware.
Sources: [AI_CONTEXT.md:176-177](), [pkg/gateway/pubsub_handlers.go:112-124]()
```go
func namespacePrefix(ns string) string {
return "ns::" + ns + "::"
}
func namespacedTopic(ns, topic string) string {
return namespacePrefix(ns) + topic
}
```
Sources: [pkg/gateway/pubsub_handlers.go:126-133]()
## Troubleshooting Pub/Sub
Common issues related to message delivery failures include incorrect topic names, inactive subscriptions, or general network connectivity problems.
Sources: [AI_CONTEXT.md:38-39](), [AI_CONTEXT.md:209-210]()
| Symptom | Solution |
| :-------------------------- | :---------------------------------------------------------------------------- |
| Messages not received | Verify topic names, active subscriptions, and network connectivity. |
| High Memory Usage | Unsubscribe from topics when no longer needed; monitor connection pool size. |
Sources: [AI_CONTEXT.md:38-39](), [AI_CONTEXT.md:209-210]()
## Conclusion
The DeBros Network's Pub/Sub system provides a fundamental capability for real-time, decentralized communication. Its topic-based, namespaced design, coupled with flexible API access via Go clients and HTTP Gateway endpoints (including WebSockets), makes it suitable for building responsive and isolated distributed applications.
Sources: [README.md:15-16](), [AI_CONTEXT.md:88-89]()
---
## Discovery Component
### Related Pages
Related topics: [Node Component](#node-component)
Error with OpenRouter API: argument of type 'NoneType' is not iterable
Please check that you have set the OPENROUTER_API_KEY environment variable with a valid API key.
---
## CLI Overview
### Related Pages
Related topics: [Quick Start Guide](#quick-start)
Timeout while calling OpenRouter API (120s). The service may be slow or unreachable.
---
## CLI Network Operations
### Related Pages
Related topics: [CLI Overview](#cli-overview)
Relevant source files
The following files were used as context for generating this wiki page:
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [CONTRIBUTING.md](https://git.debros.io/DeBros/network/src/main/CONTRIBUTING.md)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
# CLI Network Operations
The Command Line Interface (CLI) for the DeBros Network provides a direct and efficient way to interact with the decentralized P2P system. It allows users to perform various operations, including network health checks, peer management, distributed SQL queries, key-value storage operations, and Pub/Sub messaging. The CLI is designed to be a lightweight client, connecting to bootstrap peers to consume network services without participating in peer discovery or running full node services. It supports automatic wallet authentication for operations requiring credentials, enhancing user experience. Sources: [README.md:23-24](), [cmd/cli/main.go:37-40](), [AI_CONTEXT.md:65-67]()
## Architecture and Components
The CLI is implemented as a standalone executable (`network-cli`) that communicates with the DeBros Network. It leverages the `pkg/client` library to establish connections and interact with the network's core services. Global flags allow for configuration of bootstrap peers, output format, operation timeouts, and the disabling of anonymous routing. Sources: [cmd/cli/main.go:37-40](), [cmd/cli/main.go:46-51](), [cmd/cli/main.go:94-107]()
The CLI's operational flow involves parsing commands and arguments, creating a network client instance, executing the requested operation, and then formatting and printing the results.
```mermaid
graph TD
A[User] --> B[Execute network-cli command]
B --> C{Parse Command & Global Flags}
C --> D[Initialize Client Configuration]
D --> E[Create Network Client]
E --> F{Connect to Network}
F -- On Success --> G{Perform Requested Operation}
G -- Query/Storage/PubSub --> H[Interact with Network Services]
H -- On Success --> I[Process & Print Results]
F -- On Failure --> J[Handle Error & Exit]
G -- On Failure --> J
```
Sources: [cmd/cli/main.go:53-91](), [cmd/cli/main.go:94-107]()
## Key Features and Commands
The `network-cli` supports a range of commands for interacting with the DeBros Network.
### General Commands
| Command | Description |
| :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `version` | Displays the CLI version, commit hash, and build date. |
| `health` | Checks the overall health status of the connected network. |
| `peers` | Lists the peers currently connected to the network. |
| `status` | Retrieves detailed status information about the network. |
| `connect` | Attempts to connect to a specific peer address. |
| `peer-id` | Displays the local peer ID. |
| `help` | Shows the command-line help message, listing available commands and global flags. |
Sources: [cmd/cli/main.go:61-91]()
### Database Operations
The CLI allows direct interaction with the distributed SQL database, backed by RQLite.
The `query` command is used to execute SQL statements. Operations requiring authentication, such as `query`, will trigger an automatic authentication flow if not already authenticated. Sources: [README.md:46-47](), [cmd/cli/main.go:179-199]()
```bash
./bin/network-cli query "CREATE TABLE users (id INTEGER)"
./bin/network-cli query "SELECT name FROM users WHERE id = ?"
```
Sources: [README.md:46-47](), [AI_CONTEXT.md:364]()
### Storage Operations
Key-value storage operations are supported for managing data within the network.
The `storage` command provides sub-commands for `put`, `get`, and `list` operations.
Sources: [cmd/cli/main.go:201-204]()
| Operation | Usage | Description |
| :-------- | :--------------------------------------------- | :--------------------------------------------- |
| `put` | `network-cli storage put ` | Stores a key-value pair. |
| `get` | `network-cli storage get ` | Retrieves the value associated with a key. |
| `list` | `network-cli storage list [prefix] [limit]` | Lists keys, optionally filtered by a prefix. |
Sources: [cmd/cli/main.go:201-204]()
Example:
```bash
./bin/network-cli storage put test-key "Hello Network"
./bin/network-cli storage get test-key
```
Sources: [README.md:48-49]()
### Pub/Sub Messaging
The CLI facilitates real-time messaging through Pub/Sub functionality.
The `pubsub` command supports `publish`, `subscribe`, and `topics` operations.
Sources: [cmd/cli/main.go:206-209]()
| Operation | Usage | Description |
| :---------- | :--------------------------------------------- | :-------------------------------------------------- |
| `publish` | `network-cli pubsub publish ` | Publishes a message to a specified topic. |
| `subscribe` | `network-cli pubsub subscribe [duration]` | Subscribes to a topic to receive messages. |
| `topics` | `network-cli pubsub topics` | Lists all currently active Pub/Sub topics. |
Sources: [cmd/cli/main.go:206-209]()
Example:
```bash
./bin/network-cli pubsub publish notifications "Hello World"
./bin/network-cli pubsub subscribe notifications 10s
```
Sources: [README.md:50-51]()
## CLI Options
Global flags can be used to modify the behavior of CLI commands. These flags are parsed early in the execution flow.
| Flag | Description
A[Client] --> B[Connect to Network]
B --> C[Establish Connection to Bootstrap Peers]
C --> D[Query]
D[Query] -- SQL Query --> E[RQLite Database]
E[RQLite Database] -- Query Result --> D
A[Client] --> F[Storage]
F[Storage] -- Put/Get/List --> G[Key-Value Storage Service]
G[Key-Value Storage Service] -- Data --> F
A[Client] --> H[Pub/Sub]
H[Pub/Sub] -- Publish/Subscribe --> I[Pub/Sub Messaging Service]
I[Pub/Sub Messaging Service] -- Message --> H
```
Sources: [examples/basic_usage.go:19-24](), [examples/basic_usage.go:34-36](), [examples/basic_usage.go:62-63](), [examples/basic_usage.go:91-92](), [examples/basic_usage.go:127-128]()
## Authentication
The CLI incorporates an enhanced authentication system, particularly for operations requiring user credentials (e.g., storage, database, pubsub). This system features automatic wallet detection, multi-wallet management, and persistent sessions. When an authenticated operation is invoked, the CLI will:
1. Check for existing valid credentials.
2. Prompt for wallet authentication if necessary.
3. Handle signature verification.
4. Persist credentials for future use.
Sources: [README.md:299-315](), [AI_CONTEXT.md:278-294]()
This streamlines the authentication process, making it largely automatic for the user after the initial setup.
```mermaid
sequenceDiagram
participant U as User
participant C as CLI
participant GW as Gateway
participant W as Wallet
U->>C: Execute authenticated command (e.g., storage put)
activate C
C->>C: Check for existing credentials
alt No valid credentials
C->>GW: Request challenge (POST /v1/auth/challenge)
activate GW
GW-->>C: Returns {nonce}
deactivate GW
C->>W: Prompt to sign nonce with wallet
activate W
W-->>C: Returns signature (r||s||v hex)
deactivate W
C->>GW: Verify signature (POST /v1/auth/verify)
activate GW
GW-->>C: Returns JWT token
deactivate GW
C->>C: Save credentials
end
C->>GW: Execute requested operation with JWT
activate GW
GW-->>C: Operation result
deactivate GW
C-->>U: Display result
deactivate C
```
Sources: [README.md:299-315](), [AI_CONTEXT.md:278-294]()
## Development and Building
To build the `network-cli` executable, the `make build` command can be used, which compiles all executables in the project. The CLI's version information (version, commit, date) is populated at build time using `-ldflags`. Sources: [CONTRIBUTING.md:20](), [cmd/cli/main.go:27-31](), [AI_CONTEXT.md:347]()
```bash
make build
```
Sources: [CONTRIBUTING.md:20]()
## Conclusion
The DeBros Network CLI serves as a crucial interface for users to interact with the decentralized network's core services. Its design emphasizes ease of use through streamlined commands and automatic authentication, while providing comprehensive access to distributed database, storage, and messaging capabilities.
---
## CLI Storage Operations
### Related Pages
Related topics: [CLI Overview](#cli-overview), [Storage System](#storage-system)
Relevant source files
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [CONTRIBUTING.md](https://git.debros.io/DeBros/network/src/main/CONTRIBUTING.md)
# CLI Storage Operations
The DeBros Network provides a command-line interface (CLI) for interacting with its various services, including distributed key-value storage. This wiki page details the CLI commands and underlying mechanisms for performing storage operations such as putting, getting, listing, and deleting data. These operations are essential for managing application-specific data within the decentralized network. The CLI interacts with the network via a client library, abstracting the complexities of peer-to-peer communication and authentication. Sources: [README.md:20-21](), [cmd/cli/main.go:46-52](), [examples/basic_usage.go:12-16]()
## Overview of Storage Commands
The `network-cli storage` command provides subcommands for common key-value storage operations. All storage operations require authentication, which the CLI handles automatically, including wallet-based authentication. Sources: [cmd/cli/main.go:148-150](), [README.md:167-172]()
### Available Storage Commands
| Command | Description | Usage Example | Sources |
| :------ | :---------- | :------------ | :------ |
| `put` | Stores data associated with a key. | `network-cli storage put ` | [cmd/cli/main.go:275]() |
| `get` | Retrieves data for a given key. | `network-cli storage get ` | [cmd/cli/main.go:284]() |
| `list` | Lists keys, optionally filtered by a prefix. | `network-cli storage list [prefix] [limit]` | [cmd/cli/main.go:293]() |
| `delete` | Deletes data associated with a key. | `network-cli storage delete ` | [cmd/cli/main.go:302]()
### Command Structure
```mermaid
graph TD
A[network-cli] --> B{storage}
B --> C[put]
B --> D[get]
B --> E[list]
B --> F[delete]
```
Sources: [cmd/cli/main.go:148-150]()
## Data Flow for Storage Operations
CLI storage operations involve the `network-cli` executable, the underlying client library, and the DeBros Network's storage service. The CLI first ensures authentication, then constructs a request using the client library, which communicates with the network. Sources: [cmd/cli/main.go:268-270](), [AI_CONTEXT.md:126-128]()
### `storage put` Operation Flow
When a user executes `network-cli storage put `, the following sequence occurs:
```mermaid
sequenceDiagram
participant U as User
participant C as network-cli
participant A as Authentication System
participant CL as Client Library
participant S as DeBros Network Storage Service
U->>C: `storage put `
C->>A: `ensureAuthenticated()`
activate A
A-->>C: Authentication Status (e.g., JWT)
deactivate A
C->>CL: `NewClient(config)`
activate CL
CL->>CL: Connect to Network
C->>CL: `Storage().Put(ctx, key, value)`
activate CL
CL->>S: Put Request (key, value)
activate S
S-->>CL: Success/Error Response
deactivate S
CL-->>C: Result
deactivate CL
C->>U: Display Result
```
Sources: [cmd/cli/main.go:268-281](), [examples/basic_usage.go:68-75]()
### `storage get` Operation Flow
```mermaid
sequenceDiagram
participant U as User
participant C as network-cli
participant A as Authentication System
participant CL as Client Library
participant S as DeBros Network Storage Service
U->>C: `storage get `
C->>A: `ensureAuthenticated()`
activate A
A-->>C: Authentication Status
deactivate A
C->>CL: `NewClient(config)`
activate CL
CL->>CL: Connect to Network
C->>CL: `Storage().Get(ctx, key)`
activate CL
CL->>S: Get Request (key)
activate S
S-->>CL: Data/Error Response
deactivate S
CL-->>C: Result
deactivate CL
C->>U: Display Data
```
Sources: [cmd/cli/main.go:284-291](), [examples/basic_usage.go:77-84]()
## Authentication for CLI Operations
The CLI features an enhanced authentication system that automatically handles wallet detection and multi-wallet support. When an operation requires authentication (such as storage, database, or pub/sub), the CLI will:
1. Check for existing valid credentials.
2. Prompt for wallet authentication if needed.
3. Handle signature verification.
4. Persist credentials for future use. Sources: [README.md:167-172](), [AI_CONTEXT.md:155-160]()
This streamlined process means users do not need to manually execute separate authentication commands. The `ensureAuthenticated()` function is called before executing authenticated commands. Sources: [cmd/cli/main.go:268](), [README.md:175-179]()
## Configuration and Client Initialization
The `network-cli` tool initializes a client connection to the DeBros Network. This involves setting up configuration parameters such as the bootstrap peer address, timeout, and output format. Sources: [cmd/cli/main.go:20-25](), [cmd/cli/main.go:104-109]()
### Global Flags
The CLI supports global flags that can modify its behavior:
| Flag | Description | Default Value | Sources |
| :--- | :---------- | :------------ | :------ |
| `-b`, `--bootstrap` | Specifies the bootstrap peer address. | `/ip4/127.0.0.1/tcp/4001` | [cmd/cli/main.go:20](), [cmd/cli/main.go:116-119]() |
| `-f`, `--format` | Sets the output format (`table` or `json`). | `table` | [cmd/cli/main.go:22](), [cmd/cli/main.go:120-123]() |
| `-t`, `--timeout` | Sets the operation timeout duration. | `30s` | [cmd/cli/main.go:21](), [cmd/cli/main.go:124-128]() |
| `--production` | Enables production mode settings. | `false` | [cmd/cli/main.go:23](), [cmd/cli/main.go:129]() |
| `--disable-anonrc` | Disables Anyone proxy routing. | `false` | [cmd/cli/main.go:24](), [cmd/cli/main.go:130]()
### Client Creation
The `createClient()` function in `cmd/cli/main.go` is responsible for setting up the `client.NetworkClient` instance. This function configures the client with the parsed global flags, including the bootstrap peer and timeout. Sources: [cmd/cli/main.go:204-210]()
```go
// cmd/cli/main.go
func createClient() (client.NetworkClient, error) {
cfg := client.DefaultClientConfig("cli")
cfg.BootstrapPeers = []string{bootstrapPeer}
cfg.Timeout = timeout
cfg.UseProduction = useProduction
// Set the authentication provider for the client
cfg.AuthProvider = auth.NewCLIAuthProvider()
return client.NewClient(cfg)
}
```
Sources: [cmd/cli/main.go:204-210]()
## Conclusion
The DeBros Network CLI provides a robust and user-friendly interface for interacting with the distributed key-value storage. Through commands like `put`, `get`, `list`, and `delete`, users can manage data efficiently within the decentralized environment. The automatic authentication system simplifies the user experience, while configurable options allow for tailored client behavior. The underlying client library handles the complexities of network communication, ensuring reliable and secure data operations.
---
## CLI Database Operations
### Related Pages
Related topics: [CLI Overview](#cli-overview), [Database Layer](#database-layer)
Relevant source files
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [CONTRIBUTING.md](https://git.debros.io/DeBros/network/src/main/CONTRIBUTING.md)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
- [e2e/gateway_e2e_test.go](https://git.debros.io/DeBros/network/src/main/e2e/gateway_e2e_test.go)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
# CLI Database Operations
The DeBros Network provides a Command Line Interface (CLI) tool, `network-cli`, that allows users to interact with the distributed SQL database functionality of the network. This includes executing SQL queries, managing tables, and performing atomic transactions. The CLI acts as a lightweight client, connecting to bootstrap peers to access the network's services. Sources: [README.md](), [cmd/cli/main.go](), [AI_CONTEXT.md]()
## Overview of Database Interaction
The `network-cli` facilitates direct interaction with the RQLite-backed distributed SQL database. Operations such as `CREATE TABLE`, `INSERT`, `SELECT`, `ALTER TABLE`, and `PRAGMA` can be executed. The CLI handles the underlying network communication and authentication processes, providing a simplified interface for database management and data manipulation. Sources: [README.md:38-44](), [cmd/cli/main.go:142-164](), [examples/basic_usage.go:50-78]()
## Core Commands
The primary command for database interaction via the CLI is `query`. This command allows users to execute arbitrary SQL statements against the distributed database.
### `query` Command
The `network-cli query ` command sends an SQL statement to the network's database. It supports various SQL operations, including Data Definition Language (DDL) and Data Manipulation Language (DML). The CLI automatically handles authentication if required for the operation. Sources: [cmd/cli/main.go:138-164](), [README.md:188]()
#### Usage Example
```bash
./bin/network-cli query "CREATE TABLE users (id INTEGER)"
./bin/network-cli query "INSERT INTO users (id) VALUES (1)"
./bin/network-cli query "SELECT * FROM users"
```
Sources: [README.md:188]()
#### Query Execution Flow
When a `query` command is issued, the CLI performs several steps:
1. **Authentication Check:** Ensures the user is authenticated. If not, it may prompt for wallet authentication. Sources: [cmd/cli/main.go:143](), [README.md:209-218]()
2. **Client Creation:** A network client is created to connect to the DeBros Network. Sources: [cmd/cli/main.go:146-151]()
3. **Context with Timeout:** A context with a defined timeout (default 30 seconds) is established for the operation. Sources: [cmd/cli/main.go:153-154]()
4. **Database Query:** The SQL statement is passed to the client's `Database().Query()` method. Sources: [cmd/cli/main.go:156]()
5. **Result Handling:** The result of the query is processed and printed, either in JSON or table format, based on global flags. Sources: [cmd/cli/main.go:158-163]()
```mermaid
sequenceDiagram
participant User
participant CLI as network-cli
participant Client as pkg/client
participant NetworkNode as DeBros Network Node
User->>CLI: Execute `network-cli query "SQL_STATEMENT"`
activate CLI
CLI->>CLI: Parse global flags (timeout, format)
CLI->>CLI: Check authentication (ensureAuthenticated)
activate CLI
CLI-->>CLI: (Prompts for wallet if needed)
deactivate CLI
CLI->>Client: Create new client (NewClient)
activate Client
Client->>NetworkNode: Connect to network
activate NetworkNode
NetworkNode-->>Client: Connection established
deactivate NetworkNode
CLI->>Client: Call `Database().Query(ctx, SQL_STATEMENT)`
activate Client
Client->>NetworkNode: Send SQL query
activate NetworkNode
NetworkNode-->>Client: Return query result
deactivate NetworkNode
Client-->>CLI: Return query result
deactivate Client
CLI->>CLI: Format and print result (JSON/Table)
CLI-->>User: Display query output
deactivate CLI
```
Sources: [cmd/cli/main.go:142-164](), [pkg/client]() (implied)
## Database Operations via HTTP Gateway
While the `network-cli query` command provides a direct way to execute SQL, the DeBros Network also exposes database operations through its HTTP Gateway. This allows applications and scripts to interact with the database using standard HTTP requests. Sources: [README.md:46-56](), [README.md:195-206](), [e2e/gateway_e2e_test.go:1-40]()
### Key HTTP Endpoints for Database
| Endpoint | Method | Description | Request Body (JSON) | Response (JSON) | Source |
| :------------------- | :----- | :--------------------------------------------- | :------------------------------ | :-------------------------------------- | :--------------- |
| `/v1/db/create-table`| `POST` | Creates a new table based on the provided schema. | `{"schema": "CREATE TABLE ..."}` | `{"status":"ok"}` | [README.md:40]() |
| `/v1/db/drop-table` | `POST` | Drops an existing table. | `{"table": "table_name"}` | `{"status":"ok"}` | [README.md:41]() |
| `/v1/db/query` | `POST` | Executes an SQL query. | `{"sql": "SELECT ...", "args": [...]?}` | `{"columns", "rows", "count"}` | [README.md:42]() |
| `/v1/db/transaction` | `POST` | Executes multiple SQL statements atomically. | `{"statements": ["SQL 1", "SQL 2", ...]}` | `{"status":"ok"}` | [README.md:43]() |
| `/v1/db/schema` | `GET` | Retrieves the current database schema. | None | Schema JSON | [README.md:44]() |
Sources: [README.md:40-44](), [README.md:195-206]()
### Example: Create Table via Gateway
```bash
curl -X POST "$GW/v1/db/create-table" \
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{"schema":"CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT)"}'
```
Sources: [README.md:199-201]()
### Example: Atomic Transaction via Gateway
The `/v1/db/transaction` endpoint is crucial for executing multiple SQL statements as a single, atomic operation, ensuring data integrity. Sources: [README.md:43](), [README.md:203-207]()
```bash
curl -X POST "$GW/v1/db/transaction" \
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{"statements":[
"ALTER TABLE users ADD COLUMN email TEXT",
"CREATE INDEX IF NOT EXISTS idx_users_email ON users(email)"
]}'
```
Sources: [README.md:203-207]()
## Database Migrations
The DeBros Network incorporates a robust migration system, primarily managed by the Gateway service, to handle schema evolution. Migrations are executed automatically during gateway startup. Sources: [AI_CONTEXT.md:11-13]()
### Migration System Features
* **Version Control:** Tracks migration versions to prevent duplicate execution and ensure proper upgrade paths. Sources: [AI_CONTEXT.md:14]()
* **Error Recovery:** Comprehensive error handling with detailed logging for debugging. Sources: [AI_CONTEXT.md:15]()
* **Transaction Safety:** Each migration runs in a transaction to ensure atomicity and data integrity. Sources: [AI_CONTEXT.md:16]()
* **SQL File Support:** Supports standard SQL migration files with `.sql` extension in the `migrations/` directory. Sources: [AI_CONTEXT.md:17-18]()
### Migration File Structure
Migration files are sequentially named SQL files located in the `migrations/` directory. Sources: [AI_CONTEXT.md:20-24]()
```
migrations/
├── 001_initial_schema.sql
├── 002_add_auth_tables.sql
├── 003_add_indexes.sql
└── ...
```
Sources: [AI_CONTEXT.md:20-24]()
### Common Migration Workflow
A common pattern for migrations, especially for changing column types or adding foreign keys, involves recreating tables. This is typically done as an atomic transaction. Sources: [README.md:46-48]()
```mermaid
graph TD
A[Start Migration] --> B{Migration System Initiated};
B --> C{Read Migration Files};
C --> D{For Each Migration File};
D --> E{Begin Transaction};
E --> F[Execute SQL Statements];
F --> G{Commit Transaction};
G --> H{Update Migration Version};
H --> I{Migration Complete};
D -- No More Files --> I;
E -- Error --> J{Rollback Transaction};
J --> K[Log Error];
K --> I;
```
Sources: [AI_CONTEXT.md:14-16]() (implied flow)
## Troubleshooting Database Operations
Common issues related to database operations include:
* **Database Operations Timeout:** Symptoms like "Query timeout" or "No RQLite connection available". Solutions involve ensuring RQLite ports are open (5001 HTTP, 7001 Raft), leader election is complete, and cluster join configuration is correct. Sources: [README.md:139-142](), [AI_CONTEXT.md:79-81]()
* **Database Migration Issues:** Symptoms such as "Migration failed", "SQL syntax error", or "Version conflict". Solutions include checking SQL syntax, ensuring proper statement termination, verifying migration file naming and sequential order, and reviewing migration logs for transaction rollbacks. Sources: [README.md:168-172](), [AI_CONTEXT.md:92-96]()
For debugging, setting `LOG_LEVEL=debug` and checking service logs (`sudo journalctl -u debros-node.service -f`) are recommended. Sources: [README.md:175-176](), [AI_CONTEXT.md:98-99]()
## Conclusion
The `network-cli` provides essential command-line access to the DeBros Network's distributed SQL database, enabling direct query execution and management. Complementing this, the HTTP Gateway offers a RESTful interface for programmatic database interactions, including robust migration capabilities. These tools collectively ensure flexible and reliable management of the network's decentralized data store. Sources: [README.md](), [cmd/cli/main.go](), [AI_CONTEXT.md]()
---
## CLI Pub/Sub Operations
### Related Pages
Related topics: [CLI Overview](#cli-overview), [Pub/Sub System](#pubsub-system)
Relevant source files
The following files were used as context for generating this wiki page:
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
- [examples/sdk-typescript/src/client.ts](https://git.debros.io/DeBros/network/src/main/examples/sdk-typescript/src/client.ts)
# CLI Pub/Sub Operations
The DeBros Network provides a decentralized Pub/Sub messaging system that allows applications and services to publish messages to topics and subscribe to receive messages from those topics. This functionality is exposed via the command-line interface (CLI) tool, `network-cli`, enabling direct interaction with the Pub/Sub system for testing, debugging, and administrative tasks. The CLI Pub/Sub operations are designed to be simple and efficient, allowing users to quickly send and receive messages across the distributed network. Sources: [README.md:1-11](), [AI_CONTEXT.md:1-3](), [cmd/cli/main.go:37-40]()
## Pub/Sub Architecture Overview
The Pub/Sub system in DeBros Network is topic-based and supports real-time messaging. Messages are namespaced, and the system handles automatic cleanup of topics. Clients can subscribe to topics, and publishers can send messages to those topics. This enables many-to-many communication patterns. Sources: [README.md:37-39](), [AI_CONTEXT.md:1-3]()
```mermaid
graph TD
A[Publisher] -->|Publish Message| B(PubSub Service)
B -->|Distribute Message| C[Subscriber 1]
B -->|Distribute Message| D[Subscriber 2]
C -->|Receive Message| E[Application]
D -->|Receive Message| F[Application]
```
Sources: [README.md:37-39]()
## CLI Commands for Pub/Sub
The `network-cli` tool provides specific subcommands under `pubsub` to interact with the messaging system. These include `publish` for sending messages and `subscribe` for receiving them. Sources: [cmd/cli/main.go:102-104]()
### Publishing Messages
To publish a message to a topic, the `publish` subcommand is used. This command requires the topic name and the message content. The message content is typically provided as a string. Sources: [cmd/cli/main.go:102-104](), [cmd/cli/main.go:303-314]()
```bash
./bin/network-cli pubsub publish
```
Sources: [README.md:73](), [AI_CONTEXT.md:71]()
The internal process for publishing involves the CLI client connecting to the network, authenticating if required, and then calling the `Publish` method on the `PubSub` client. The message data is converted to a byte array before being sent. Sources: [cmd/cli/main.go:303-314](), [examples/basic_usage.go:124-128]()
```mermaid
sequenceDiagram
participant CLI as network-cli
participant Client as Client Library
participant Network as DeBros Network
participant PubSubService as Pub/Sub Service
CLI->>Client: Call handlePubSub(publish, topic, message)
activate Client
Client->>Client: Ensure Authenticated
Client->>Network: Connect()
activate Network
Client->>PubSubService: Publish(ctx, topic, []byte(message))
activate PubSubService
PubSubService-->>Network: Message Acknowledged
deactivate PubSubService
Network-->>Client: Publish Result
deactivate Network
Client-->>CLI: Success/Error
deactivate Client
```
Sources: [cmd/cli/main.go:303-314](), [cmd/cli/main.go:264-268](), [examples/basic_usage.go:124-128]()
### Subscribing to Topics
Subscribing to a topic allows the CLI to receive messages published to that topic in real-time. The `subscribe` subcommand is used for this purpose, specifying the topic name and an optional duration for how long to listen. Sources: [cmd/cli/main.go:102-104](), [cmd/cli/main.go:316-340]()
```bash
./bin/network-cli pubsub subscribe [duration]
```
Sources: [README.md:74](), [AI_CONTEXT.md:72]()
When subscribing, the CLI establishes a connection and registers a handler function that processes incoming messages. Messages are received as binary frames over a WebSocket connection when using the HTTP Gateway. Sources: [AI_CONTEXT.md:236-237](), [examples/basic_usage.go:115-121](), [examples/sdk-typescript/src/client.ts:63-66]()
```mermaid
sequenceDiagram
participant CLI as network-cli
participant Client as Client Library
participant Network as DeBros Network
participant PubSubService as Pub/Sub Service
CLI->>Client: Call handlePubSub(subscribe, topic, duration)
activate Client
Client->>Client: Ensure Authenticated
Client->>Network: Connect()
activate Network
Client->>PubSubService: Subscribe(ctx, topic, handler)
activate PubSubService
PubSubService-->>Network: Subscription Confirmed
deactivate PubSubService
Network-->>Client: Subscription Result
deactivate Network
loop Message Flow
PubSubService->>PubSubService: Message Published
PubSubService->>Client: Deliver Message (via handler)
Client->>CLI: Display Message
end
Client->>Network: Disconnect (after duration/interrupt)
deactivate Network
Client-->>CLI: Subscription Ended
deactivate Client
```
Sources: [cmd/cli/main.go:316-340](), [cmd/cli/main.go:264-268](), [examples/basic_usage.go:115-121]()
### Listing Topics
The CLI also supports listing active topics. This is useful for discovering available communication channels within the network. Sources: [AI_CONTEXT.md:240](), [examples/basic_usage.go:133-138]()
```bash
./bin/network-cli pubsub topics
```
The command internally queries the Pub/Sub service to retrieve a list of currently active topics.
## Troubleshooting Pub/Sub Operations
Common issues related to Pub/Sub messaging include message delivery failures. These can often be resolved by verifying topic names, ensuring subscription status, and checking overall network connectivity. High memory usage might occur if subscriptions are not properly managed; users should unsubscribe from topics when they are no longer needed. Sources: [AI_CONTEXT.md:65-67](), [AI_CONTEXT.md:144-146]()
| Symptom | Solution | Source |
| :-------------------------- | :---------------------------------------------------------------------- | :----------------------------------- |
| Messages not received | Verify topic names, active subscriptions, network connectivity. | [AI_CONTEXT.md:66-67](), [AI_CONTEXT.md:144-146]() |
| High Memory Usage | Unsubscribe when done, monitor connection pool, review message retention. | [AI_CONTEXT.md:67](), [AI_CONTEXT.md:146]() |
## Conclusion
The `network-cli` provides essential command-line tools for interacting with the DeBros Network's Pub/Sub messaging system. It allows users to easily publish and subscribe to topics, facilitating real-time communication within the decentralized network. These CLI operations are crucial for development, testing, and managing the messaging flow in a DeBros Network deployment. Sources: [README.md:73-74](), [cmd/cli/main.go:102-104]()
---
## CLI Authentication
### Related Pages
Related topics: [Gateway Authentication](#gateway-authentication), [Authentication Troubleshooting](#authentication-troubleshooting)
Error with OpenRouter API: argument of type 'NoneType' is not iterable
Please check that you have set the OPENROUTER_API_KEY environment variable with a valid API key.
---
## HTTP Gateway Overview
### Related Pages
Related topics: [Gateway API Endpoints](#gateway-api-endpoints)
Relevant source files
The following files were used as context for generating this wiki page:
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [cmd/gateway/config.go](https://git.debros.io/DeBros/network/src/main/cmd/gateway/config.go)
- [pkg/gateway/status_handlers.go](https://git.debros.io/DeBros/network/src/main/pkg/gateway/status_handlers.go)
- [openapi/gateway.yaml](https://git.debros.io/DeBros/network/src/main/openapi/gateway.yaml)
- [examples/sdk-typescript/README.md](https://git.debros.io/DeBros/network/src/main/examples/sdk-typescript/README.md)
# HTTP Gateway Overview
The DeBros Network includes a powerful HTTP/WebSocket gateway that provides a modern REST API and WebSocket interface over the P2P network. It acts as a bridge, enabling external applications and clients to interact with the decentralized services like distributed SQL database, key-value storage, and pub/sub messaging without directly engaging with the underlying P2P protocols. This gateway features an enhanced authentication system with multi-wallet support and namespace enforcement for resource isolation. Sources: [README.md:129-132](), [AI_CONTEXT.md:11-12]()
## Architecture and Components
The HTTP Gateway operates as a service that exposes the core functionalities of the DeBros Network via standard HTTP and WebSocket protocols. It integrates with the network's client library to communicate with the P2P nodes.
```mermaid
graph TD
A[External Client/Application] -->|HTTP/WebSocket| B(HTTP Gateway)
B -->|Client API| C[DeBros Network Client]
C -->|P2P Communication| D[DeBros Network Node]
D --> E{Distributed Services}
E --> F[RQLite Database]
E --> G[Key-Value Storage]
E --> H[Pub/Sub Messaging]
```
Sources: [README.md:129-132](), [AI_CONTEXT.md:11-12]()
### Gateway Configuration
The gateway's behavior can be configured through environment variables or command-line flags. Flags take precedence over environment variables, which in turn override default settings.
Sources: [README.md:141-145](), [cmd/gateway/config.go:1-62]()
| Configuration Option | Environment Variable | Command-line Flag | Description | Default Value |
| :------------------- | :------------------- | :---------------- | :---------- | :------------ |
| Listen Address | `GATEWAY_ADDR` | `--addr` | HTTP listen address. | `:8080` |
| Client Namespace | `GATEWAY_NAMESPACE` | `--namespace` | Client namespace for scoping resources. | `default` |
| Bootstrap Peers | `GATEWAY_BOOTSTRAP_PEERS` | `--bootstrap-peers` | Comma-separated bootstrap peers for network client. | `""` |
| Require Authentication | `GATEWAY_REQUIRE_AUTH` | N/A | Enables/disables authentication requirement. | `true` |
| API Keys | `GATEWAY_API_KEYS` | N/A | Comma-separated API keys and their namespaces. | `""` |
Sources: [README.md:141-145](), [cmd/gateway/config.go:1-62]()
### Health and Status Endpoints
The gateway provides several endpoints to monitor its operational status and the underlying network client's health.
Sources: [pkg/gateway/status_handlers.go:1-74]()
```mermaid
graph TD
A[Client] -->|GET /health| B{healthHandler}
B --> C{server healthResponse}
B --> D{client.Health()}
C & D --> E[JSON Response]
A -->|GET /v1/status| F{statusHandler}
F --> G{client.Network().GetStatus()}
G --> H[JSON Response]
A -->|GET /v1/version| I{versionHandler}
I --> J[JSON Response with Build Info]
```
Sources: [pkg/gateway/status_handlers.go:1-74]()
## Authentication System
The gateway features an enhanced authentication system with automatic wallet detection, multi-wallet support, and API key support. It uses JWT for authenticated sessions and supports Ethereum EIP-191 `personal_sign` for wallet verification.
Sources: [README.md:147-156](), [AI_CONTEXT.md:275-288]()
### Authentication Flow (Wallet-Based)
```mermaid
sequenceDiagram
participant C as Client
participant GW as HTTP Gateway
participant NW as DeBros Network
C->>GW: POST /v1/auth/challenge
activate GW
GW->>GW: Generate Nonce (internal auth context)
GW-->>C: {nonce}
deactivate GW
C->>C: Sign nonce with wallet (EIP-191)
C->>GW: POST /v1/auth/verify {wallet, nonce, signature}
activate GW
GW->>GW: Verify signature (65-byte r||s||v hex)
GW->>NW: Authenticate/Verify Wallet
activate NW
NW-->>GW: Verification Result
deactivate NW
GW->>GW: Mark nonce as used (on success)
GW-->>C: JWT Token / {status:"ok"}
deactivate GW
```
Sources: [README.md:109-117](), [AI_CONTEXT.md:280-285]()
### Authentication Methods
| Method | Description | Usage |
| :----- | :---------- | :---- |
| **Wallet-Based** | Uses Ethereum EIP-191 `personal_sign` for secure verification. Supports automatic wallet switching and credential persistence. | `POST /v1/auth/challenge`, `POST /v1/auth/verify`, `POST /v1/auth/register` |
| **JWT Authentication** | Issued by the gateway upon successful wallet verification. JWKS available for public key retrieval. | `Authorization: Bearer ` |
| **API Key Support** | Static API keys can be configured and mapped to namespaces. | `Authorization: Bearer ` or `X-API-Key: ` |
Sources: [README.md:151-156](), [AI_CONTEXT.md:275-288]()
## API Endpoints
The gateway exposes a comprehensive set of REST and WebSocket endpoints for interacting with the DeBros Network's services.
Sources: [README.md:42-87](), [AI_CONTEXT.md:290-332](), [openapi/gateway.yaml:1-68]()
### General Endpoints
| Endpoint | Method | Description |
| :------- | :----- | :---------- |
| `/health`, `/v1/health` | `GET` | Health check, returns gateway and client status. |
| `/v1/status` | `GET` | Aggregates server uptime and network status. |
| `/v1/version` | `GET` | Returns gateway build/runtime information. |
| `/v1/auth/jwks`, `/.well-known/jwks.json` | `GET` | JSON Web Key Set for JWT verification. |
| `/v1/auth/challenge` | `POST` | Generates a wallet challenge (nonce). |
| `/v1/auth/verify` | `POST` | Verifies a wallet signature against a nonce. |
| `/v1/auth/register` | `POST` | Wallet registration. |
| `/v1/auth/refresh` | `POST` | Refreshes JWT token. |
| `/v1/auth/logout` | `POST` | Clears authentication state. |
| `/v1/auth/whoami` | `GET` | Returns current authentication status. |
| `/v1/auth/api-key` | `POST` | Generates API keys for authenticated users. |
Sources: [README.md:42-87](), [AI_CONTEXT.md:290-332]()
### Storage Operations
The storage endpoints allow for key-value data manipulation with namespace enforcement.
Sources: [README.md:52-56](), [AI_CONTEXT.md:310-313](), [openapi/gateway.yaml:104-124]()
| Endpoint | Method | Description | Request Body/Query | Response |
| :------- | :----- | :---------- | :----------------- | :------- |
| `/v1/storage/put` | `POST` | Stores data by key. | `{"key":"","value":""}` or raw bytes | `{"status":"ok"}` |
| `/v1/storage/get` | `POST` | Retrieves data by key. | `{"key":""}` | `{"value":""}` or raw bytes |
| `/v1/storage/delete` | `POST` | Deletes data by key. | `{"key":""}` | `{"status":"ok"}` |
| `/v1/storage/list` | `GET` | Lists keys with an optional prefix. | `?prefix=` | `{"keys":["key1", "key2"]}` |
| `/v1/storage/exists` | `GET` | Checks if a key exists. | `?key=` | `{"exists":true/false}` |
Sources: [README.md:52-56](), [AI_CONTEXT.md:310-313](), [openapi/gateway.yaml:104-124]()
### Database Operations
The database endpoints provide an HTTP interface for interacting with the distributed SQL database.
Sources: [README.md:45-50](), [AI_CONTEXT.md:326-332](), [openapi/gateway.yaml:69-103]()
| Endpoint | Method | Description | Request Body | Response |
| :------- | :----- | :---------- | :----------- | :------- |
| `/v1/db/create-table` | `POST` | Creates a table using SQL DDL. | `{"schema":"CREATE TABLE users (id INTEGER PRIMARY KEY)"}` | `{"status":"ok"}` |
| `/v1/db/drop-table` | `POST` | Drops a table. | `{"table":"users"}` | `{"status":"ok"}` |
| `/v1/db/query` | `POST` | Executes a single SQL query. | `{"sql":"SELECT * FROM users WHERE id = ?", "args":[1]}` | `{"columns":["id","name"],"rows":[[1,"Alice"]],"count":1}` |
| `/v1/db/transaction` | `POST` | Applies multiple SQL statements atomically. | `{"statements":["INSERT ...", "UPDATE ..."]}` | `{"status":"ok"}` |
| `/v1/db/schema` | `GET` | Returns the database schema. | N/A | JSON schema |
Sources: [README.md:45-50](), [AI_CONTEXT.md:326-332](), [openapi/gateway.yaml:69-103]()
### Pub/Sub Messaging
The gateway supports both WebSocket and REST interfaces for real-time messaging.
Sources: [README.md:59-62](), [AI_CONTEXT.md:315-324]()
| Endpoint | Method | Description | Request Body/Query | Response |
| :------- | :----- | :---------- | :----------------- | :------- |
| `/v1/pubsub/ws` | `GET` | Establishes a WebSocket connection for real-time messaging. | `?topic=` | WebSocket frames |
| `/v1/pubsub/publish` | `POST` | Publishes a message to a topic. | `{"topic":"","data_base64":""}` | `{"status":"ok"}` |
| `/v1/pubsub/topics` | `GET` | Lists active topics (trimmed to caller namespace). | N/A | `{"topics":["topic1", "topic2"]}` |
Sources: [README.md:59-62](), [AI_CONTEXT.md:315-324]()
## SDK Authoring Guidelines
The gateway provides an OpenAPI specification (`openapi/gateway.yaml`) for SDK code generation. SDKs should adhere to the following base concepts:
Sources: [README.md:92-95](), [AI_CONTEXT.md:334-337](), [openapi/gateway.yaml:1-68]()
* **Authentication**: Send `X-API-Key: ` or `Authorization: Bearer ` with every request.
* **Versioning**: All endpoints are under `/v1/`.
* **Responses**: Mutations return `{"status":"ok"}`; queries/lists return JSON; errors return `{"error":"message"}` with appropriate HTTP status.
Sources: [README.md:96-101](), [AI_CONTEXT.md:338-342]()
A minimal TypeScript SDK example demonstrates interaction with the gateway:
```typescript
import { GatewayClient } from './src/client';
const c = new GatewayClient(process.env.GATEWAY_BASE_URL!, process.env.GATEWAY_API_KEY!);
await c.createTable('CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)');
await c.transaction([
'INSERT INTO users (id,name) VALUES (1,\'Alice\')'
]);
const res = await c.query('SELECT name FROM users WHERE id = ?', [1]);
console.log(res.rows);
```
Sources: [examples/sdk-typescript/README.md:7-15]()
## Conclusion
The HTTP Gateway is a critical component of the DeBros Network, simplifying access to its distributed services for external applications. Its robust API, flexible authentication, and clear configuration options make it an essential interface for building decentralized applications on the DeBros platform.
---
## Gateway Configuration
### Related Pages
Related topics: [Configuration System](#configuration-system), [HTTP Gateway Overview](#gateway-overview)
Error with OpenRouter API: argument of type 'NoneType' is not iterable
Please check that you have set the OPENROUTER_API_KEY environment variable with a valid API key.
---
## Gateway Authentication
### Related Pages
Related topics: [CLI Authentication](#cli-authentication), [Authentication Troubleshooting](#authentication-troubleshooting)
Error with OpenRouter API: argument of type 'NoneType' is not iterable
Please check that you have set the OPENROUTER_API_KEY environment variable with a valid API key.
---
## Gateway API Endpoints
### Related Pages
Related topics: [SDK Authoring Guide](#sdk-authoring-guide)
Relevant source files
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [openapi/gateway.yaml](https://git.debros.io/DeBros/network/src/main/openapi/gateway.yaml)
- [pkg/gateway/storage_handlers.go](https://git.debros.io/DeBros/network/src/main/pkg/gateway/storage_handlers.go)
- [pkg/gateway/status_handlers.go](https://git.debros.io/DeBros/network/src/main/pkg/gateway/status_handlers.go)
- [cmd/gateway/config.go](https://git.debros.io/DeBros/network/src/main/cmd/gateway/config.go)
- [pkg/gateway/middleware.go](https://git.debros.io/DeBros/network/src/main/pkg/gateway/middleware.go)
- [pkg/gateway/gateway.go](https://git.debros.io/DeBros/network/src/main/pkg/gateway/gateway.go)
- [pkg/gateway/apps_handlers.go](https://git.debros.io/DeBros/network/src/main/pkg/gateway/apps_handlers.go)
# Gateway API Endpoints
The DeBros Network includes an HTTP/WebSocket gateway that provides a modern REST API and WebSocket interface over the P2P network. This gateway acts as a bridge, allowing external applications and clients to interact with the decentralized network's functionalities, including distributed database operations, key-value storage, pub/sub messaging, and peer management. All API endpoints are versioned under `/v1/`. Responses for mutations typically return `{"status":"ok"}`, while queries and lists return JSON data. Errors are returned as `{"error": "message"}` with appropriate HTTP status codes. Sources: [README.md:135-139](), [AI_CONTEXT.md:275-279]()
## Core API Endpoint Categories
The gateway exposes several categories of API endpoints to manage different aspects of the DeBros Network. These categories include Health & Status, Authentication, Storage Operations, Network Operations, Pub/Sub Messaging, and Database Operations. Sources: [README.md:140-159](), [AI_CONTEXT.md:280-299]()
### Health & Status Endpoints
These endpoints provide information about the gateway's operational status, network health, and version details.
| Endpoint | Method | Description |
| :----------------- | :----- | :--------------------------------------------- |
| `/health` | `GET` | Basic health check. |
| `/v1/health` | `GET` | Detailed health status of gateway and client. |
| `/v1/status` | `GET` | Aggregated server uptime and network status. |
| `/v1/version` | `GET` | Gateway build/runtime information. |
Sources: [README.md:141-144](), [AI_CONTEXT.md:281-284](), [pkg/gateway/status_handlers.go:42-83](), [pkg/gateway/status_handlers.go:85-103](), [pkg/gateway/status_handlers.go:105-114]()
The `healthHandler` (for `/health` and `/v1/health`) provides a JSON response including the server's status, started time, uptime, and optionally the client's health status if initialized. The `statusHandler` (for `/v1/status`) provides server uptime and network status from the client. The `versionHandler` (for `/v1/version`) returns build details such as version, commit, build time, started time, and uptime.
```mermaid
graph TD
A[Client Request] --> B{GET /health or /v1/health};
B --> C{Gateway.healthHandler};
C --> D{Gateway.client.Health()};
D -- HealthStatus --> E{Construct JSON Response};
E --> F[Client Receives Health Data];
G[Client Request] --> H{GET /v1/status};
H --> I{Gateway.statusHandler};
I --> J{Gateway.client.Network().GetStatus()};
J -- NetworkStatus --> K{Construct JSON Response};
K --> L[Client Receives Status Data];
M[Client Request] --> N{GET /v1/version};
N --> O{Gateway.versionHandler};
O --> P{Construct Version Info};
P --> Q[Client Receives Version Data];
```
Sources: [pkg/gateway/status_handlers.go:42-114]()
### Authentication Endpoints
The gateway features an enhanced authentication system supporting wallet-based authentication (Ethereum EIP-191), JWT tokens, and API keys. These endpoints facilitate the authentication flow.
| Endpoint | Method | Description |
| :------------------- | :----- | :----------------------------------------------- |
| `/v1/auth/challenge` | `POST` | Generates a wallet challenge (public endpoint). |
| `/v1/auth/verify` | `POST` | Verifies a wallet signature (public endpoint). |
| `/v1/auth/register` | `POST` | Registers a new wallet (public endpoint). |
| `/v1/auth/refresh` | `POST` | Refreshes a JWT token (public endpoint). |
| `/v1/auth/logout` | `POST` | Clears authentication state. |
| `/v1/auth/whoami` | `GET` | Returns the current authentication status. |
| `/v1/auth/api-key` | `POST` | Generates API keys (requires authentication). |
| `/v1/auth/jwks` | `GET` | JWKS endpoint for JWT verification. |
| `/.well-known/jwks.json` | `GET` | Standard JWKS endpoint. |
Sources: [README.md:146-152](), [AI_CONTEXT.md:286-292](), [AI_CONTEXT.md:330-331]()
Authentication is handled by middleware, which checks for `Authorization: Bearer `, `Authorization: Bearer `, or `X-API-Key`. Public paths bypass authentication. Sources: [pkg/gateway/middleware.go:33-35](), [pkg/gateway/middleware.go:73-82]()
```mermaid
sequenceDiagram
participant C as Client
participant GW as Gateway
participant DB as Database
C->>GW: POST /v1/auth/challenge
activate GW
GW-->>C: {nonce}
deactivate GW
C->>C: Sign nonce with wallet
C->>GW: POST /v1/auth/verify {wallet, nonce, signature}
activate GW
GW->>DB: Verify signature and nonce
DB-->>GW: Verification Result
GW-->>C: JWT Token
deactivate GW
C->>GW: Authenticated Request (e.g., POST /v1/storage/put)
activate GW
Note over GW: Auth Middleware validates JWT/API Key
GW->>DB: Perform operation
DB-->>GW: Operation Result
GW-->>C: {status: "ok"} or data
deactivate GW
```
Sources: [README.md:104-114](), [pkg/gateway/middleware.go:84-129]()
### Storage Operations
These endpoints allow clients to interact with the distributed key-value storage.
| Endpoint | Method | Description |
| :------------------ | :----- | :--------------------------------- |
| `/v1/storage/get` | `POST` | Retrieve data by key. |
| `/v1/storage/put` | `POST` | Store data with a key. |
| `/v1/storage/delete`| `POST` | Delete data by key. |
| `/v1/storage/list` | `GET` | List keys with an optional prefix. |
| `/v1/storage/exists`| `GET` | Check if a key exists. |
Sources: [README.md:154-158](), [AI_CONTEXT.md:294-298](), [README.md:118-125]()
The `storagePutHandler` expects a JSON body with `key` and `value` (base64 encoded). The `storageGetHandler` expects a JSON body with `key`. Namespace enforcement is applied to all storage operations. Sources: [pkg/gateway/storage_handlers.go:10-17](), [pkg/gateway/storage_handlers.go:19-26]()
### Network Operations
These endpoints provide information and control over the peer-to-peer network connections.
| Endpoint | Method | Description |
| :-------------------- | :----- | :---------------------------- |
| `/v1/network/status` | `GET` | Network status. |
| `/v1/network/peers` | `GET` | Connected peers. |
| `/v1/network/connect` | `POST` | Connect to a specified peer. |
| `/v1/network/disconnect`| `POST` | Disconnect from a peer. |
Sources: [README.md:160-163](), [AI_CONTEXT.md:300-303]()
### Pub/Sub Messaging
The gateway supports real-time messaging through both WebSocket and REST interfaces.
#### WebSocket Interface
| Endpoint | Method | Description |
| :--------------------------- | :----- | :---------------------------------------------- |
| `/v1/pubsub/ws?topic=`| `GET` | WebSocket connection for real-time messaging. |
Sources: [README.md:166](), [AI_CONTEXT.md:306](), [README.md:128-132]()
Server sends messages as binary frames; 30s ping keepalive. Client text/binary frames are published to the same namespaced topic. Sources: [AI_CONTEXT.md:347-349]()
#### REST Interface
| Endpoint | Method | Description |
| :--------------------- | :----- | :----------------------------------- |
| `/v1/pubsub/publish` | `POST` | Publish message to a topic. |
| `/v1/pubsub/topics` | `GET` | List active topics. |
Sources: [README.md:169-170](), [AI_CONTEXT.md:309-310]()
The `pubsubPublishHandler` expects a JSON body with `topic` and `data_base64`. The `pubsubTopicsHandler` returns a list of topics. Namespace enforcement is applied. Sources: [pkg/gateway/storage_handlers.go:49-55](), [pkg/gateway/storage_handlers.go:57-63]()
### Database Operations
These endpoints provide an interface for distributed SQL database operations, including table creation, querying, and transactions.
| Endpoint | Method | Description |
| :---------------------- | :----- | :--------------------------------------------- |
| `/v1/db/create-table` | `POST` | Create tables with SQL DDL. |
| `/v1/db/drop-table` | `POST` | Drop a table. |
| `/v1/db/query` | `POST` | Execute a single SQL query. |
| `/v1/db/transaction` | `POST` | Apply multiple statements atomically. |
| `/v1/db/schema` | `GET` | Return database schema (tables and columns). |
Sources: [README.md:14-18](), [AI_CONTEXT.md:313-317]()
All `/v1/db/*` routes enforce authentication and namespace ownership. The gateway uses an internal database context for validation and execution, bypassing circular authentication checks. Migrations can be performed by POSTing DDL statements to `/v1/db/transaction`. Sources: [AI_CONTEXT.md:320-323]()
The `dbQueryHandler` processes SQL queries with optional arguments. The `dbTransactionHandler` executes multiple SQL statements within a single transaction. The `dbSchemaHandler` retrieves the database schema. The `dbCreateTableHandler` creates a new table using a provided schema. Sources: [pkg/gateway/storage_handlers.go:30-40](), [pkg/gateway/storage_handlers.go:42-48](), [pkg/gateway/storage_handlers.go:50-56](), [pkg/gateway/storage_handlers.go:58-64]()
```mermaid
graph TD
C[Client] --> A{POST /v1/db/query};
A --> G[Gateway.dbQueryHandler];
G --> H{g.client.Database().Query};
H -- Result --> I[Return JSON];
C --> B{POST /v1/db/transaction};
B --> J[Gateway.dbTransactionHandler];
J --> K{g.client.Database().Transaction};
K -- Status --> L[Return {status:"ok"}];
C --> D{GET /v1/db/schema};
D --> M[Gateway.dbSchemaHandler];
M --> N{g.client.Database().GetSchema};
N -- Schema --> O[Return JSON];
C --> E{POST /v1/db/create-table};
E --> P[Gateway.dbCreateTableHandler];
P --> Q{g.client.Database().CreateTable};
Q -- Status --> R[Return {status:"ok"}];
```
Sources: [pkg/gateway/storage_handlers.go:30-64]()
## Application Management Endpoints
The gateway also provides endpoints for managing applications within a namespace, including listing, creating, fetching, updating, and deleting application records.
| Endpoint | Method | Description |
| :------------------- | :----- | :----------------------------------------------- |
| `/v1/apps` | `GET` | List applications within the current namespace. |
| `/v1/apps` | `POST` | Create a new application. |
| `/v1/apps/{app_id}` | `GET` | Fetch details of a specific application. |
| `/v1/apps/{app_id}` | `PUT` | Update an application's name or public key. |
| `/v1/apps/{app_id}` | `DELETE`| Delete a specific application. |
Sources: [pkg/gateway/apps_handlers.go:10-15]()
The `appsHandler` manages these operations, using the database client to interact with an `apps` table. Application IDs are generated using a random buffer encoded with `base64.RawURLEncoding`. Sources: [pkg/gateway/apps_handlers.go:66-73]()
## Configuration
The gateway can be configured using environment variables or command-line flags. The precedence for configuration is: Flags > Environment Variables > YAML > Defaults. Sources: [AI_CONTEXT.md:465-468]()
Key configuration parameters include:
| Parameter | Environment Variable | Flag | Default | Description |
| :------------------ | :------------------- | :---------------- | :---------- | :---------------------------------------------- |
| Listen Address | `GATEWAY_ADDR` | `--addr` | `:8080` | HTTP listen address for the gateway. |
| Client Namespace | `GATEWAY_NAMESPACE` | `--namespace` | `default` | Namespace for scoping resources. |
| Bootstrap Peers | `GATEWAY_BOOTSTRAP_PEERS` | `--bootstrap-peers` | `""` | Comma-separated list of bootstrap peers. |
Sources: [AI_CONTEXT.md:255-263](), [cmd/gateway/config.go:40-42]()
The `parseGatewayConfig` function in `cmd/gateway/config.go` handles parsing these flags and environment variables into the `gateway.Config` structure. Sources: [cmd/gateway/config.go:38-66]()
## Conclusion
The Gateway API Endpoints provide a comprehensive and structured interface for external clients to interact with the DeBros Network. By categorizing endpoints for health, authentication, storage, network, pub/sub, and database operations, the gateway ensures modularity and ease of use. The built-in authentication system, namespace enforcement, and clear error handling contribute to a robust and secure interaction layer for the decentralized network.
---
## SDK Authoring Guide
### Related Pages
Related topics: [Gateway API Endpoints](#gateway-api-endpoints)
Relevant source files
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [openapi/gateway.yaml](https://git.debros.io/DeBros/network/src/main/openapi/gateway.yaml)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
- [cmd/node/main.go](https://git.debros.io/DeBros/network/src/main/cmd/node/main.go)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
# SDK Authoring Guide
This guide outlines the fundamental concepts and key HTTP endpoints for authoring Software Development Kits (SDKs) that interact with the DeBros Network. The DeBros Network provides distributed SQL database, key-value storage, and pub/sub messaging capabilities. SDKs enable developers to integrate these features into their applications. The API endpoints are versioned under `/v1/` and support various authentication methods. Sources: [README.md:52-53](), [AI_CONTEXT.md:213-214]()
## Base Concepts for SDKs
SDKs should adhere to several core concepts to ensure proper interaction with the DeBros Network. These include understanding the OpenAPI specification, authentication mechanisms, API versioning, and standard response formats. Sources: [README.md:55-60](), [AI_CONTEXT.md:216-221]()
### OpenAPI Specification
A machine-readable OpenAPI specification is available at `openapi/gateway.yaml`. This file can be used for automated SDK code generation, ensuring consistency and accuracy in API interactions. Sources: [README.md:56](), [AI_CONTEXT.md:217](), [openapi/gateway.yaml]()
### Authentication
All requests to the DeBros Network API require authentication. SDKs must send either an `X-API-Key` header with the API key or an `Authorization` header with a `Bearer` token (API key or JWT) with every request. Sources: [README.md:57](), [AI_CONTEXT.md:218]()
### Versioning
All API endpoints are prefixed with `/v1/`, indicating the current version of the API. This ensures backward compatibility and clear API evolution. Sources: [README.md:58](), [AI_CONTEXT.md:219]()
### Response Formats
The API employs pragmatic response envelopes. Mutations typically return `{status:"ok"}` upon success, while queries and list operations return JSON data. Errors are indicated by an HTTP status code and a JSON body in the format `{ "error": "message" }`. Sources: [README.md:59-60](), [AI_CONTEXT.md:220-221]()
## Key HTTP Endpoints for SDKs
The DeBros Network exposes various HTTP endpoints categorized by their functionality: Storage, Database, and Pub/Sub. SDKs will primarily interact with these endpoints. Sources: [README.md:62-95](), [AI_CONTEXT.md:223-256]()
### Storage Operations
Storage operations allow SDKs to interact with the key-value store. This includes putting, getting, checking existence, listing, and deleting data.
```mermaid
graph TD
A[SDK Client] -->|POST /v1/storage/put| B{Store Data}
A -->|GET /v1/storage/get| C{Retrieve Data}
A -->|GET /v1/storage/exists| D{Check Existence}
A -->|GET /v1/storage/list| E{List Keys}
A -->|POST /v1/storage/delete| F{Delete Data}
```
Sources: [README.md:63-70](), [AI_CONTEXT.md:224-231]()
| Operation | Method | Path | Request Body/Query | Response | Description |
|---|---|---|---|---|---|
| Put | `POST` | `/v1/storage/put?key=` | Raw bytes | `{status:"ok"}` | Stores data associated with a key. |
| Get | `GET` | `/v1/storage/get?key=` | None | Raw bytes (may be base64-encoded) | Retrieves data for a given key. |
| Exists | `GET` | `/v1/storage/exists?key=` | None | `{exists:boolean}` | Checks if a key exists. |
| List | `GET` | `/v1/storage/list?prefix=` | None | `{keys:[...]}` | Lists keys with an optional prefix. |
| Delete | `POST` | `/v1/storage/delete` | `{key}` | `{status:"ok"}` | Deletes data associated with a key. |
Sources: [README.md:63-70](), [AI_CONTEXT.md:224-231](), [openapi/gateway.yaml:127-167]()
### Database Operations
Database operations enable SDKs to manage and query the distributed SQL database. This includes creating/dropping tables, executing queries, performing transactions, and retrieving schema information.
```mermaid
graph TD
A[SDK Client] -->|POST /v1/db/create-table| B{Create Table}
A -->|POST /v1/db/drop-table| C{Drop Table}
A -->|POST /v1/db/query| D{Execute Query}
A -->|POST /v1/db/transaction| E{Execute Transaction}
A -->|GET /v1/db/schema| F{Get Schema}
```
Sources: [README.md:71-78](), [AI_CONTEXT.md:232-239]()
| Operation | Method | Path | Request Body/Query | Response | Description |
|---|---|---|---|---|---|
| Create Table | `POST` | `/v1/db/create-table` | `{schema}` | `{status:"ok"}` | Creates a new table with the specified schema. |
| Drop Table | `POST` | `/v1/db/drop-table` | `{table}` | `{status:"ok"}` | Drops an existing table. |
| Query | `POST` | `/v1/db/query` | `{sql, args?}` | `{columns, rows, count}` | Executes a single SQL query. |
| Transaction | `POST` | `/v1/db/transaction` | `{statements:[...]}` | `{status:"ok"}` | Executes multiple SQL statements atomically. |
| Schema | `GET` | `/v1/db/schema` | None | Schema JSON | Retrieves the database schema. |
Sources: [README.md:71-78](), [AI_CONTEXT.md:232-239](), [openapi/gateway.yaml:169-247]()
### Pub/Sub Messaging
Pub/Sub operations allow SDKs to engage in real-time messaging. This involves connecting via WebSockets, publishing messages, and listing active topics.
```mermaid
graph TD
A[SDK Client] -->|GET /v1/pubsub/ws?topic=| B{WebSocket Connection}
A -->|POST /v1/pubsub/publish| C{Publish Message}
A -->|GET /v1/pubsub/topics| D{List Topics}
```
Sources: [README.md:79-82](), [AI_CONTEXT.md:240-243]()
| Operation | Method | Path | Request Body/Query | Response | Description |
|---|---|---|---|---|---|
| WS Subscribe | `GET` | `/v1/pubsub/ws?topic=` | None | WebSocket connection | Establishes a WebSocket for real-time messaging. |
| Publish | `POST` | `/v1/pubsub/publish` | `{topic, data_base64}` | `{status:"ok"}` | Publishes a message to a specified topic. |
| Topics | `GET` | `/v1/pubsub/topics` | None | `{topics:[...]}` | Lists all active topics. |
Sources: [README.md:79-82](), [AI_CONTEXT.md:240-243](), [openapi/gateway.yaml:249-291]()
## Migrations
Database migrations, such as adding columns or changing table schemas, should be handled by sending DDL statements as part of a `POST /v1/db/transaction` call. This ensures atomicity for schema changes. For complex changes like adding foreign keys or changing types, a "recreate pattern" is suggested: create a new table, copy data, drop the old table, and rename the new one. Sources: [README.md:97-100](), [AI_CONTEXT.md:258-261]()
```mermaid
graph TD
A[Start Migration] --> B{ALTER TABLE users ADD COLUMN age INTEGER}
B --> C{POST /v1/db/transaction}
C --> D{Migration Complete}
E[Start Complex Migration] --> F{CREATE TABLE _new}
F --> G{Copy Data to _new}
G --> H{DROP old Table}
H --> I{RENAME _new to old name}
I --> J{POST /v1/db/transaction}
J --> K{Migration Complete}
```
Sources: [README.md:97-100](), [AI_CONTEXT.md:258-261]()
## Suggested SDK Surface
SDKs should provide a clear and intuitive API surface for developers. This typically involves client objects or functions for Database, Storage, and Pub/Sub operations. Sources: [AI_CONTEXT.md:310-332]()
### Database Client Methods
| Method | Description |
|---|---|
| `createTable(schema: string): Promise` | Creates a new table. |
| `dropTable(name: string): Promise` | Drops an existing table. |
| `query(sql: string, args?: any[]): Promise<{ rows: T[] }>` | Executes a SQL query and returns rows. |
| `transaction(statements: string[]): Promise` | Executes multiple SQL statements in a transaction. |
| `schema(): Promise` | Retrieves the database schema. |
Sources: [AI_CONTEXT.md:311-316]()
### Storage Client Methods
| Method | Description |
|---|---|
| `put(key: string, value: Uint8Array | string): Promise` | Stores data for a key. |
| `get(key: string): Promise` | Retrieves data for a key. |
| `exists(key: string): Promise` | Checks if a key exists. |
| `list(prefix?: string): Promise` | Lists keys with an optional prefix. |
| `delete(key: string): Promise` | Deletes data for a key. |
Sources: [AI_CONTEXT.md:317-322]()
### Pub/Sub Client Methods
| Method | Description |
|---|---|
| `subscribe(topic: string, handler: (msg: Uint8Array) => void): Promise<() => void>` | Subscribes to a topic with a message handler. |
| `publish(topic: string, data: Uint8Array | string): Promise` | Publishes data to a topic. |
| `listTopics(): Promise` | Lists active topics. |
Sources: [AI_CONTEXT.md:323-326]()
## Reliability Guidelines
SDKs should incorporate mechanisms to enhance reliability, such as timeouts, retries, and handling WebSocket reconnections. Sources: [AI_CONTEXT.md:328-332]()
### Timeouts
Implement default timeouts (e.g., 10-30 seconds) for API calls, with the option for per-call overrides. Sources: [AI_CONTEXT.md:329]()
### Retries
Employ exponential backoff for retries on `429 Too Many Requests` and `5xx` server error responses. SDKs should also respect the `Retry-After` header if provided by the server. Sources: [AI_CONTEXT.md:330]()
### Idempotency
For transactional operations, consider client-side implementation of an `Idempotency-Key` header, although the gateway may currently ignore it. Sources: [AI_CONTEXT.md:331]()
### WebSocket
For Pub/Sub WebSocket connections, implement automatic reconnection with jitter to handle disconnections gracefully. Re-subscription to topics should occur after a successful reconnect. Sources: [AI_CONTEXT.md:332]()
## Example SDK Usage
The following Go example demonstrates how an SDK client would connect to the network and perform basic database, storage, and pub/sub operations. Sources: [examples/basic_usage.go]()
```go
package main
import (
"context"
"log"
"time"
"git.debros.io/DeBros/network/pkg/client"
)
func main() {
config := client.DefaultClientConfig("example_app")
config.BootstrapPeers = []string{
"/ip4/127.0.0.1/tcp/4001/p2p/QmBootstrap1",
}
networkClient, err := client.NewClient(config)
if err != nil {
log.Fatalf("Failed to create network client: %v", err)
}
if err := networkClient.Connect(); err != nil {
log.Fatalf("Failed to connect to network: %v", err)
}
defer networkClient.Disconnect()
log.Printf("Connected to network successfully!")
demonstrateDatabase(networkClient)
demonstrateStorage(networkClient)
demonstratePubSub(networkClient)
demonstrateNetworkInfo(networkClient)
log.Printf("Example completed successfully!")
}
func demonstrateDatabase(client client.NetworkClient) {
ctx := context.Background()
db := client.Database()
log.Printf("=== Database Operations ===")
schema := `
CREATE TABLE IF NOT EXISTS messages (
id INTEGER PRIMARY KEY,
content TEXT NOT NULL,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
)
`
if err := db.CreateTable(ctx, schema); err != nil {
log.Printf("Error creating table: %v", err)
return
}
log.Printf("Table created successfully")
insertSQL := "INSERT INTO messages (content) VALUES (?)"
result, err := db.Query(ctx, insertSQL, "Hello, distributed world!")
if err != nil {
log.Printf("Error inserting data: %v", err)
return
}
log.Printf("Data inserted, result: %+v", result)
selectSQL := "SELECT * FROM messages"
result, err = db.Query(ctx, selectSQL)
if err != nil {
log.Printf("Error querying data: %v", err)
return
}
log.Printf("Query result: %+v", result)
}
func demonstrateStorage(client client.NetworkClient) {
ctx := context.Background()
storage := client.Storage()
log.Printf("=== Storage Operations ===")
key := "user:123"
value := []byte(`{"name": "Alice", "age": 30}`)
if err := storage.Put(ctx, key, value); err != nil {
log.Printf("Error storing data: %v", err)
return
}
log.Printf("Data stored successfully")
retrieved, err := storage.Get(ctx, key)
if err != nil {
log.Printf("Error retrieving data: %v", err)
return
}
log.Printf("Retrieved data: %s", string(retrieved))
exists, err := storage.Exists(ctx, key)
if err != nil {
log.Printf("Error checking existence: %v", err)
return
}
log.Printf("Key exists: %v", exists)
keys, err := storage.List(ctx, "user:", 10)
if err != nil {
log.Printf("Error listing keys: %v", err)
return
}
log.Printf("Keys: %v", keys)
}
func demonstratePubSub(client client.NetworkClient) {
ctx := context.Background()
pubsub := client.PubSub()
log.Printf("=== Pub/Sub Operations ===")
topic := "notifications"
handler := func(topic string, data []byte) error {
log.Printf("Received message on topic '%s': %s", topic, string(data))
return nil
}
if err := pubsub.Subscribe(ctx, topic, handler); err != nil {
log.Printf("Error subscribing: %v", err)
return
}
log.Printf("Subscribed to topic: %s", topic)
message := []byte("Hello from pub/sub!")
if err := pubsub.Publish(ctx, topic, message); err != nil {
log.Printf("Error publishing: %v", err)
return
}
log.Printf("Message published")
time.Sleep(time.Millisecond * 100)
topics, err := pubsub.ListTopics(ctx)
if err != nil {
log.Printf("Error listing topics: %v", err)
return
}
log.Printf("Subscribed topics: %v", topics)
}
func demonstrateNetworkInfo(client client.NetworkClient) {
ctx := context.Background()
network := client.Network()
log.Printf("=== Network Information ===")
status, err := network.GetStatus(ctx)
if err != nil {
log.Printf("Error getting status: %v", err)
return
}
log.Printf("Network Status: %+v", status)
peers, err := network.GetPeers(ctx)
if err != nil {
log.Printf("Error getting peers: %v", err)
return
}
log.Printf("Connected Peers: %+v", peers)
}
```
Sources: [examples/basic_usage.go:1-177]()
This SDK Authoring Guide provides the foundational knowledge for building robust and reliable client libraries for the DeBros Network. By adhering to these concepts and utilizing the defined HTTP endpoints, developers can effectively integrate distributed database, storage, and messaging capabilities into their applications.
---
## Development Setup
### Related Pages
Related topics: [Build, Test, and Lint](#build-test-lint)
Relevant source files
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [CONTRIBUTING.md](https://git.debros.io/DeBros/network/src/main/CONTRIBUTING.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [cmd/node/main.go](https://git.debros.io/DeBros/network/src/main/cmd/node/main.go)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
- [examples/sdk-typescript/src/client.ts](https://git.debros.io/DeBros/network/src/main/examples/sdk-typescript/src/client.ts)
# Development Setup
The DeBros Network is a robust, decentralized peer-to-peer system designed for distributed data management and communication. This wiki page outlines the necessary steps and tools for setting up a development environment, building the project, running tests, and interacting with the network locally. Understanding the development setup is crucial for contributing to the project and building applications on top of the network. Sources: [README.md:1-4](), [AI_CONTEXT.md:15-19]()
## Requirements
To set up a development environment for the DeBros Network, the following software components are required:
* **Go:** Version 1.22+ (1.23 recommended) is necessary for building and running the Go-based components.
* **RQLite:** Version 8.x, a distributed SQLite database, is used for data persistence and replication.
* **Git:** Essential for cloning the repository and managing source code.
* **Make:** Recommended for build automation and simplifying common development tasks.
Sources: [CONTRIBUTING.md:7-10](), [README.md:95-98](), [AI_CONTEXT.md:129-132]()
## Setup and Installation
The initial setup involves cloning the repository and installing dependencies.
```bash
git clone https://git.debros.io/DeBros/network.git
cd network
make deps
```
Sources: [CONTRIBUTING.md:14-16](), [README.md:108-110]()
## Project Structure
The DeBros Network project follows a modular structure to organize its various components.
```
network/
├── cmd/
│ ├── node/ # Network node executable
│ └── cli/ # Command-line interface
├── pkg/
│ ├── client/ # Client library
│ ├── node/ # Node implementation
│ ├── database/ # RQLite integration
│ ├── storage/ # Storage service
│ ├── pubsub/ # Pub/Sub messaging
│ ├── config/ # Centralized config
│ └── discovery/ # Peer discovery (node only)
├── scripts/ # Install, test scripts
├── configs/ # YAML configs
├── bin/ # Built executables
```
This structure separates executables, core functionalities (packages), and development utilities.
Sources: [README.md:465-478](), [AI_CONTEXT.md:424-437]()
## Build and Test
The project uses `make` for common build and test operations.
### Build Commands
```bash
make build # Build all executables
make test # Run unit tests
make clean # Clean build artifacts
```
Sources: [README.md:480-483](), [AI_CONTEXT.md:439-442]()
The `make build` command compiles the source code into executables, which are placed in the `bin/` directory. Sources: [CONTRIBUTING.md:20](), [README.md:478]()
### Build Process Flow
The build process typically involves compiling Go modules and generating executables.
```mermaid
graph TD
A[Start Build] --> B{make build};
B --> C[Compile Go Modules];
C --> D[Generate Binaries];
D --> E[Place in bin/ Directory];
E --> F[Build Complete];
```
Sources: [CONTRIBUTING.md:20](), [README.md:478]()
### Testing
Unit tests can be executed using `make test`.
```bash
make test
```
Sources: [CONTRIBUTING.md:21](), [README.md:481]()
For local multi-node testing, a dedicated script is provided:
```bash
scripts/test-multinode.sh
```
Sources: [README.md:486](), [AI_CONTEXT.md:445]()
## Running Nodes and CLI
### Starting a Bootstrap Node
A bootstrap node is essential for the network. It can be started manually or via a `Makefile` command.
```bash
make run-node
# Or manually:
go run ./cmd/node --config configs/bootstrap.yaml
```
Sources: [README.md:118-121]()
The `cmd/node/main.go` file handles the node's startup, configuration loading, and flag parsing.
Sources: [cmd/node/main.go:1-20]()
### Node Configuration Loading Flow
The node's configuration can be loaded from a YAML file, with flags and environment variables overriding settings based on precedence.
```mermaid
graph TD
A[Start Node] --> B{Parse CLI Flags};
B --> C{Load YAML Config};
C --> D{Apply Env Vars};
D --> E[Final Config];
E --> F[Initialize Node];
F --> G[Start Node Services];
```
Sources: [README.md:183-186](), [cmd/node/main.go:64-96]()
### Starting Additional Nodes
Once a bootstrap node is running, additional nodes can join the network.
```bash
make run-node2
# Or manually:
go run ./cmd/node --config configs/node.yaml
```
Sources: [README.md:123-126]()
### CLI Usage
The `network-cli` tool provides commands to interact with the running network.
```bash
./bin/network-cli health
./bin/network-cli peers
./bin/network-cli status
./bin/network-cli storage put test-key "Hello Network"
./bin/network-cli storage get test-key
./bin/network-cli pubsub publish notifications "Hello World"
./bin/network-cli pubsub subscribe notifications 10s
```
Sources: [CONTRIBUTING.md:25-31](), [README.md:128-134]()
The `cmd/cli/main.go` file defines the command-line interface logic, including parsing global flags and handling various commands like `health`, `peers`, `status`, `query`, `storage`, and `pubsub`.
Sources: [cmd/cli/main.go:27-106]()
### Client Configuration
The `pkg/client` library is used to programmatically interact with the network. An example client configuration is shown below.
```go
config := client.DefaultClientConfig("example_app")
config.BootstrapPeers = []string{
"/ip4/127.0.0.1/tcp/4001/p2p/QmBootstrap1",
}
```
Sources: [examples/basic_usage.go:13-16]()
## HTTP Gateway
The DeBros Network provides an HTTP Gateway for external applications to interact with its services.
### Gateway Endpoints
The gateway exposes various endpoints for authentication, storage, network information, Pub/Sub, and database operations.
| Category | Method | Path | Description |
| :---------------- | :----- | :------------------------------------- | :-------------------------------------------- |
| **Auth** | POST | `/v1/auth/challenge` | Generates wallet challenge |
| | POST | `/v1/auth/verify` | Verifies wallet signature |
| | POST | `/v1/auth/register` | Registers new wallet |
| | POST | `/v1/auth/refresh` | Refreshes JWT token |
| | POST | `/v1/auth/logout` | Clears authentication |
| | GET | `/v1/auth/whoami` | Current authentication status |
| | POST | `/v1/auth/api-key` | Generates API key (authenticated) |
| **Storage** | POST | `/v1/storage/get` | Retrieves data |
| | POST | `/v1/storage/put` | Stores data |
| | POST | `/v1/storage/delete` | Deletes data |
| | GET | `/v1/storage/list` | Lists keys with optional prefix |
| | GET | `/v1/storage/exists` | Checks key existence |
| **Network** | GET | `/v1/network/status` | Network status |
| | GET | `/v1/network/peers` | Connected peers |
| | POST | `/v1/network/connect` | Connects to peer |
| | POST | `/v1/network/disconnect` | Disconnects from peer |
| **Pub/Sub (WS)** | GET | `/v1/pubsub/ws?topic=` | WebSocket connection for real-time messaging |
| **Pub/Sub (REST)**| POST | `/v1/pubsub/publish` | Publishes message to topic |
| | GET | `/v1/pubsub/topics` | Lists active topics |
| **Database** | POST | `/v1/db/create-table` | Creates tables with SQL DDL |
| | POST | `/v1/db/drop-table` | Drops a table |
| | POST | `/v1/db/query` | Executes single SQL query |
| | POST | `/v1/db/transaction` | Applies multiple statements atomically |
| | GET | `/v1/db/schema` | Returns tables and columns schema |
Sources: [README.md:374-428](), [AI_CONTEXT.md:73-108](), [AI_CONTEXT.md:762-811]()
### TypeScript SDK Example
An example of interacting with the gateway using a TypeScript SDK client:
```typescript
import { GatewayClient } from "../examples/sdk-typescript/src/client";
const client = new GatewayClient(process.env.GATEWAY_BASE_URL!, process.env.GATEWAY_API_KEY!);
await client.createTable("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)");
const res = await client.query("SELECT name FROM users WHERE id = ?", [1]);
```
Sources: [README.md:337-341]()
The TypeScript client code demonstrates methods for database operations (`createTable`, `query`, `schema`), storage operations (`put`, `get`, `exists`, `list`, `delete`), and Pub/Sub subscription.
Sources: [examples/sdk-typescript/src/client.ts:5-71]()
## Troubleshooting
Common issues and their solutions during development:
* **Bootstrap Connection Failed:** Verify the node is running, firewall settings are correct, and the peer ID is valid.
* **Database Operations Timeout:** Ensure RQLite ports are open, leader election is complete, and the cluster join configuration is correct.
* **Message Delivery Failures:** Check topic names, active subscriptions, and network connectivity.
* **High Memory Usage:** Unsubscribe from topics when no longer needed, monitor connection pools, and review message retention policies.
* **Authentication Issues:**
* Wallet signature format (65-byte r||s||v hex)
* Nonce must match exactly during wallet verification
* Wallet address case-insensitivity
* Use refresh endpoint or re-authenticate for expired tokens
* Clear credential cache (`rm -rf ~/.debros/credentials`) for multi-wallet conflicts.
* **Gateway Issues:** Verify the gateway is running and accessible, check CORS configuration, ensure proper authentication headers, and review namespace configuration.
* **Database Migration Issues:** Check SQL syntax, ensure proper statement termination, verify migration file naming and sequential order, and review migration logs for transaction rollbacks.
Sources: [README.md:493-518](), [AI_CONTEXT.md:167-195](), [AI_CONTEXT.md:496-524]()
### Debugging and Health Checks
Useful commands for debugging and health checks:
```bash
export LOG_LEVEL=debug
./bin/network-cli health
./bin/network-cli peers
./bin/network-cli query "SELECT 1"
./bin/network-cli pubsub publish test "hello"
./bin/network-cli pubsub subscribe test 10s
# Test authentication flow
./bin/network-cli storage put test-key test-value
# Gateway health checks
curl http://localhost:8080/health
curl http://localhost:8080/v1/status
```
Sources: [README.md:520-534]()
### Service Logs
To monitor service logs:
```bash
# Node service logs
sudo journalctl -u debros-node.service --since "1 hour ago"
# Gateway service logs (if running as service)
sudo journalctl -u debros-gateway.service --since "1 hour ago"
# Application logs
tail -f ./logs/gateway.log
tail -f ./logs/node.log
```
Sources: [README.md:537-545]()
## Conclusion
This guide provides a comprehensive overview of setting up a development environment for the DeBros Network, covering installation, building, testing, and interacting with the network components. By following these steps and utilizing the provided tools and documentation, developers can effectively contribute to the project and build decentralized applications.
---
## Build, Test, and Lint
### Related Pages
Related topics: [Development Setup](#development-setup)
Relevant source files
- [CONTRIBUTING.md](https://git.debros.io/DeBros/network/src/main/CONTRIBUTING.md)
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [cmd/node/main.go](https://git.debros.io/DeBros/network/src/main/cmd/node/main.go)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
# Build, Test, and Lint
The DeBros Network project emphasizes a structured approach to software development, ensuring code quality, reliability, and maintainability through defined build, test, and lint processes. These processes are crucial for both local development and continuous integration, helping developers produce robust, decentralized peer-to-peer applications. The project provides tools and guidelines to facilitate these activities, ensuring that contributions adhere to established standards before integration.
## Prerequisites
To build, test, and lint the DeBros Network, specific software versions and tools are required. These ensure a consistent development environment and compatibility with project dependencies.
Sources: [CONTRIBUTING.md:6-8](), [README.md:29-32](), [AI_CONTEXT.md:209-212]()
### Software Requirements
The following software components are necessary for development:
| Software | Recommended Version | Purpose |
| :---------- | :------------------ | :------------------------------------- |
| Go | 1.22+ (1.23 recommended) | Primary programming language runtime |
| RQLite | 8.x (optional for local runs) | Distributed SQLite database dependency |
| Git | Any | Version control system |
| Make | Any | Build automation tool |
Sources: [CONTRIBUTING.md:6-8](), [README.md:29-32](), [AI_CONTEXT.md:209-212]()
## Setup
Before performing any build, test, or lint operations, the project repository must be cloned and dependencies installed.
Sources: [CONTRIBUTING.md:11-13]()
```bash
git clone https://git.debros.io/DeBros/network.git
cd network
make deps
```
Sources: [CONTRIBUTING.md:11-13]()
## Build Process
The build process compiles the Go source code into executable binaries. The project structure includes a `cmd/` directory for executables and a `bin/` directory for compiled binaries.
Sources: [README.md:179-183](), [AI_CONTEXT.md:200-204]()
### Build Commands
To build all executables, use the `make build` command. This command compiles the `node` and `cli` executables, placing them in the `bin/` directory.
Sources: [CONTRIBUTING.md:16](), [README.md:60](), [README.md:186](), [AI_CONTEXT.md:215]()
```bash
make build
```
Sources: [CONTRIBUTING.md:16](), [README.md:60](), [README.md:186](), [AI_CONTEXT.md:215]()
### Build Flow
The `main.go` files within `cmd/node` and `cmd/cli` serve as entry points for their respective executables. The build process involves compiling these Go packages.
Sources: [cmd/node/main.go:1](), [cmd/cli/main.go:1]()
```mermaid
graph TD
A[Start Build Process] --> B{Execute 'make build'};
B --> C[Compile cmd/node/main.go];
B --> D[Compile cmd/cli/main.go];
C --> E[Generate bin/network-node];
D --> F[Generate bin/network-cli];
E --> G[Build Complete];
F --> G;
```
Description: This diagram illustrates the high-level flow of the build process, showing how the `make build` command compiles the node and CLI executables.
## Testing
Testing is a critical part of the development workflow to ensure the correctness and reliability of the DeBros Network. The project provides commands for running unit tests and multi-node integration tests.
### Unit Tests
Unit tests are executed using the `make test` command. This command runs all defined unit tests within the project.
Sources: [CONTRIBUTING.md:17](), [README.md:187](), [AI_CONTEXT.md:216]()
```bash
make test
```
Sources: [CONTRIBUTING.md:17](), [README.md:187](), [AI_CONTEXT.md:216]()
Developers are required to include tests for new functionality and ensure that `make build test` passes before submitting pull requests.
Sources: [CONTRIBUTING.md:27]()
### Local Multi-Node Testing
For more comprehensive testing of network interactions and distributed behavior, a script is provided to set up and test a multi-node environment locally.
Sources: [README.md:190-191](), [AI_CONTEXT.md:219-220]()
```bash
scripts/test-multinode.sh
```
Sources: [README.md:190-191](), [AI_CONTEXT.md:219-220]()
This script is crucial for validating features that rely on peer-to-peer communication, such as peer discovery, distributed database operations, and pub/sub messaging across multiple nodes.
Sources: [README.md:190-191](), [AI_CONTEXT.md:219-220]()
### Testing Flow (Example: `examples/basic_usage.go`)
The `examples/basic_usage.go` file demonstrates how a client interacts with the network for database, storage, and pub/sub operations. This example can be used to manually verify core functionalities.
Sources: [examples/basic_usage.go:1]()
```mermaid
sequenceDiagram
participant C as Client
participant N as Network
participant DB as Database Service
participant S as Storage Service
participant P as PubSub Service
C->>N: Connect()
activate N
N-->>C: Connection Success
deactivate N
C->>DB: CreateTable()
activate DB
DB-->>C: Table Created
deactivate DB
C->>S: Put("key", "value")
activate S
S-->>C: Data Stored
deactivate S
C->>P: Subscribe("topic", handler)
activate P
P-->>C: Subscribed
deactivate P
C->>P: Publish("topic", "message")
activate P
P-->>C: Message Published
deactivate P
P->>C: Received message on topic 'topic': message
```
Description: This sequence diagram illustrates the basic interaction flow of a client with the DeBros Network for database, storage, and pub/sub operations, as demonstrated in `examples/basic_usage.go`.
Sources: [examples/basic_usage.go:30-140]()
## Linting and Formatting
Code linting and formatting ensure that the codebase adheres to a consistent style and catches potential issues early in the development cycle.
### Linting Commands
The `make fmt vet` or `make lint` commands are used for formatting and vetting the Go source code.
Sources: [CONTRIBUTING.md:18]()
```bash
make fmt vet
# or
make lint
```
Sources: [CONTRIBUTING.md:18]()
These commands help maintain code quality and readability, which is important for collaborative development.
## Cleaning Build Artifacts
To remove compiled binaries and other build artifacts, the `make clean` command is available.
Sources: [README.md:188](), [AI_CONTEXT.md:217]()
```bash
make clean
```
Sources: [README.md:188](), [AI_CONTEXT.md:217]()
This command is useful for ensuring a clean build environment, especially when switching branches or resolving build issues.
## Pull Request Guidelines
Before submitting a pull request, contributors must ensure that their changes comply with the project's quality standards.
Sources: [CONTRIBUTING.md:25-29]()
1. **Fork and Branch:** Fork the repository and create a topic branch for your changes.
2. **Pass Tests:** Ensure that `make build test` passes successfully. New functionality must include corresponding tests.
3. **Focus and Description:** Keep pull requests focused on a single logical change and provide a clear description including motivation, approach, and testing details.
4. **Documentation:** Update `README.md` or other documentation if behavior changes are introduced.
Sources: [CONTRIBUTING.md:25-29]()
## Conclusion
The build, test, and lint processes are integral to the development lifecycle of the DeBros Network. By adhering to these guidelines and utilizing the provided `make` commands, developers can contribute high-quality, reliable code that integrates seamlessly into the decentralized P2P system. These practices collectively ensure the stability and robustness of the distributed SQL database, key-value storage, and pub/sub messaging functionalities.
---
## Local Multi-Node Testing
### Related Pages
Related topics: [Build, Test, and Lint](#build-test-lint)
Timeout while calling OpenRouter API (120s). The service may be slow or unreachable.
---
## Contributing to DeBros Network
### Related Pages
Related topics: [Code of Conduct](#code-of-conduct)
Relevant source files
- [CONTRIBUTING.md](https://git.debros.io/DeBros/network/src/main/CONTRIBUTING.md)
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [CODE_OF_CONDUCT.md](https://git.debros.io/DeBros/network/src/main/CODE_OF_CONDUCT.md)
- [cmd/node/main.go](https://git.debros.io/DeBros/network/src/main/cmd/node/main.go)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
# Contributing to DeBros Network
Contributing to the DeBros Network project involves understanding its development workflow, testing procedures, and community guidelines. This guide outlines the necessary steps and resources for setting up a development environment, building the project, running tests, and submitting contributions. The project is built in Go and utilizes RQLite for its distributed database capabilities. Sources: [CONTRIBUTING.md](), [README.md]()
## Requirements
To contribute to the DeBros Network, specific software versions are required for a consistent development environment. Sources: [CONTRIBUTING.md:7-9](), [README.md:43-46]()
### Software Requirements
| Requirement | Version | Description |
| :---------- | :------ | :---------- |
| Go | 1.22+ | Recommended: 1.23. The primary language for the project. Sources: [CONTRIBUTING.md:7](), [README.md:43]() |
| RQLite | 8.x | Distributed SQLite. Optional for local runs, as `Makefile` can start embedded nodes. Sources: [CONTRIBUTING.md:8](), [README.md:44]() |
| Git | N/A | For source management. Sources: [README.md:45]() |
| Make | N/A | For build automation. Optional, but recommended. Sources: [CONTRIBUTING.md:9](), [README.md:46]() |
### Hardware Requirements
| Requirement | Minimum | Recommended | Description |
| :---------- | :------------- | :---------------- | :---------- |
| CPU | 2 cores | 4+ cores | Processing power for nodes. Sources: [README.md:50]() |
| RAM | 4GB | 8GB+ | Memory for running services. Sources: [README.md:50]() |
| Disk | 10GB | 50GB+ SSD | Storage for data and binaries. Sources: [README.md:50]() |
| Network | Stable internet | Low-latency network | Connectivity for P2P communication. Sources: [README.md:50]() |
### Network Ports
The DeBros Network uses specific ports for its operations. These ports must be accessible for proper node functionality. Sources: [README.md:53-56]()
| Port | Protocol | Description |
| :--- | :------- | :---------- |
| 4001 | TCP | LibP2P P2P communication. Sources: [README.md:54]() |
| 5001 | TCP | RQLite HTTP API. Sources: [README.md:55]() |
| 7001 | TCP | RQLite Raft consensus. Sources: [README.md:56]() |
## Setup and Development Workflow
Setting up the development environment involves cloning the repository and installing dependencies. The `Makefile` provides convenient commands for building, testing, and linting the project. Sources: [CONTRIBUTING.md:12-20]()
### Initial Setup
To begin, clone the repository and navigate into the project directory. Sources: [CONTRIBUTING.md:12-14](), [README.md:61-62]()
```bash
git clone https://git.debros.io/DeBros/network.git
cd network
make deps
```
Sources: [CONTRIBUTING.md:12-14]()
### Build, Test, and Lint
The project uses `make` for common development tasks. Sources: [CONTRIBUTING.md:16-18]()
| Command | Description |
| :------------ | :-------------------------- |
| `make build` | Builds all executables. Sources: [CONTRIBUTING.md:16](), [README.md:65]() |
| `make test` | Runs unit tests. Sources: [CONTRIBUTING.md:17](), [CONTRIBUTING.md:31](), [README.md:71]() |
| `make fmt vet`| Formats and vets the code. Equivalent to `make lint`. Sources: [CONTRIBUTING.md:18]() |
| `make lint` | Formats and vets the code. Sources: [CONTRIBUTING.md:18]() |
| `make clean` | Cleans build artifacts. Sources: [README.md:72]() |
### Local Multi-Node Testing
A script is provided to facilitate testing with multiple nodes locally. Sources: [README.md:75]()
```bash
scripts/test-multinode.sh
```
Sources: [README.md:75]()
### Starting Nodes
The `Makefile` includes targets for quickly starting bootstrap and additional nodes for local testing. Sources: [README.md:68-69]()
```bash
make run-node # Starts a bootstrap node or a regular node based on configuration. Sources: [README.md:68](), [AI_CONTEXT.md:275]()
make run-node2 # Starts an additional node. Sources: [README.md:70]()
make run-gateway # Starts the HTTP gateway. Sources: [AI_CONTEXT.md:276]()
```
Alternatively, nodes can be started manually with specific configuration files. Sources: [README.md:69](), [README.md:71]()
```bash
go run ./cmd/node --config configs/bootstrap.yaml
go run ./cmd/node --config configs/node.yaml
```
Sources: [README.md:69](), [README.md:71]()
### CLI Usage
The `network-cli` tool provides commands for interacting with the DeBros Network. Sources: [CONTRIBUTING.md:22-24](), [README.md:79-82]()
```bash
./bin/network-cli health
./bin/network-cli peers
./bin/network-cli status
./bin/network-cli storage put test-key "Hello Network"
./bin/network-cli storage get test-key
./bin/network-cli pubsub publish notifications "Hello World"
./bin/network-cli pubsub subscribe notifications 10s
```
Sources: [CONTRIBUTING.md:22-24](), [README.md:79-82]()
## Architecture Overview
The DeBros Network is a robust, decentralized peer-to-peer system built in Go. It offers distributed SQL database capabilities, key-value storage, and pub/sub messaging. The architecture distinguishes between full nodes and lightweight clients. Sources: [README.md:131-134](), [AI_CONTEXT.md:154-157]()
### Components and Data Flow
The system's architecture can be visualized as layers, with applications interacting through a client API that communicates with the underlying transport layer and RQLite database. Sources: [README.md:137-147]()
```mermaid
graph TD
A[Application Layer] --> B(Client API)
B --> C(Database Client)
B --> D(Storage Client)
B --> E(PubSub Client)
C --> F(Transport Layer)
D --> F
E --> F
F --> G(LibP2P Host)
F --> H(Noise/TLS Encryption)
F --> I(RQLite Database)
```
Sources: [README.md:137-147]()
### Node vs. Client Roles
- **Node:** A full P2P participant. It runs services, handles peer discovery, manages the database, storage, and pub/sub. Sources: [README.md:30]()
- **Client:** A lightweight participant. It connects only to bootstrap peers and consumes services without participating in peer discovery. Sources: [README.md:31]()
## Pull Request Guidelines
To ensure smooth integration of contributions, specific guidelines must be followed when submitting pull requests. Sources: [CONTRIBUTING.md:28-33]()
1. **Fork and Branch:** Fork the repository and create a new topic branch for your changes. Sources: [CONTRIBUTING.md:29]()
2. **Pass Tests:** Ensure that `make build test` passes successfully. New functionality must include corresponding tests. Sources: [CONTRIBUTING.md:30-31]()
3. **Focus and Description:** Keep pull requests focused on a single logical change and provide a clear description including motivation, approach, and testing details. Sources: [CONTRIBUTING.md:32]()
4. **Documentation:** Update `README.md` and other relevant documentation for any changes in behavior. Sources: [CONTRIBUTING.md:33]()
## Code of Conduct
The DeBros Network project adopts the Contributor Covenant Code of Conduct. All participants are expected to adhere to its standards to foster an open and welcoming community. Sources: [CODE_OF_CONDUCT.md:3-4]()
### Our Pledge
Project members, contributors, and leaders pledge to make participation a harassment-free experience for everyone. Sources: [CODE_OF_CONDUCT.md:7]()
### Our Standards
| Positive Behaviors | Unacceptable Behaviors |
| :------------------------------------------------ | :----------------------------------------------------- |
| Demonstrating empathy and kindness | Harassment, trolling, or insulting comments |
| Being respectful of differing opinions | Public or private harassment |
| Gracefully accepting constructive criticism | Publishing others’ private information |
| Focusing on what is best for the community | Other conduct reasonably considered inappropriate |
Sources: [CODE_OF_CONDUCT.md:11-20]()
### Enforcement
Project maintainers are responsible for clarifying and enforcing standards. Instances of unacceptable behavior can be reported to `security@debros.io`. All complaints are reviewed promptly and fairly. This Code applies within all project spaces and when an individual officially represents the project. Sources: [CODE_OF_CONDUCT.md:23-32]()
## Versioning
The CLI reports its version using `network-cli version`. Releases are tagged (e.g., `v0.18.0-beta`) and published via GoReleaser. Sources: [CONTRIBUTING.md:26-27]()
## Conclusion
Contributing to the DeBros Network involves a clear set of guidelines for development, testing, and collaboration. By following these steps and adhering to the Code of Conduct, contributors can effectively help improve the network, ensuring its continued growth and stability as a robust, decentralized P2P system.
---
## Code of Conduct
### Related Pages
Related topics: [Contributing to DeBros Network](#contributing)
Relevant source files
The following files were used as context for generating this wiki page:
- [CODE_OF_CONDUCT.md](https://git.debros.io/DeBros/network/src/main/CODE_OF_CONDUCT.md)
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [CONTRIBUTING.md](https://git.debros.io/DeBros/network/src/main/CONTRIBUTING.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
# Code of Conduct
The DeBros Network project adopts a Code of Conduct to establish standards for a welcoming and harassment-free community. This document outlines the principles, expected behaviors, and enforcement mechanisms for all participants. It is designed to ensure a positive environment for contributors and users of the distributed P2P database system. Sources: [CODE_OF_CONDUCT.md:1-2]()
## Purpose and Scope
The Code of Conduct serves to define acceptable and unacceptable behaviors within the DeBros Network project. It applies to all project spaces and to individuals officially representing the project. This includes, but is not limited to, interactions in repositories, issue trackers, communication channels, and events. Sources: [CODE_OF_CONDUCT.md:3-4](), [CODE_OF_CONDUCT.md:27-28]()
## Our Pledge and Standards
Project members, contributors, and leaders pledge to make participation a harassment-free experience. The Code of Conduct specifies both positive and unacceptable behaviors to guide community interactions.
### Positive Behaviors
Examples of behavior that contributes to a positive environment include:
* Demonstrating empathy and kindness.
* Being respectful of differing opinions.
* Gracefully accepting constructive criticism.
* Focusing on what is best for the community.
Sources: [CODE_OF_CONDUCT.md:9-14]()
### Unacceptable Behaviors
Examples of unacceptable behavior include:
* Harassment, trolling, or insulting comments.
* Public or private harassment.
* Publishing others’ private information.
* Other conduct which could reasonably be considered inappropriate.
Sources: [CODE_OF_CONDUCT.md:16-21]()
## Enforcement Responsibilities and Process
Project maintainers are responsible for clarifying and enforcing the standards outlined in the Code of Conduct. They are authorized to take appropriate and fair corrective action.
### Reporting Instances
Instances of abusive, harassing, or otherwise unacceptable behavior should be reported to the maintainers. All complaints will be reviewed and investigated promptly and fairly.
**Contact for Reporting:** security@debros.io
Sources: [CODE_OF_CONDUCT.md:23-25](), [CODE_OF_CONDUCT.md:30-31]()
### Enforcement Flow
The enforcement process involves reporting, review, and investigation, leading to potential corrective action by maintainers.
```mermaid
graph TD
A[Participant Observes Unacceptable Behavior] --> B{Report to Maintainers};
B --> C[Maintainers Receive Report];
C --> D[Review and Investigate Complaint];
D --> E{Determine Corrective Action};
E --> F[Implement Corrective Action];
F --> G[Resolution];
```
Sources: [CODE_OF_CONDUCT.md:23-25](), [CODE_OF_CONDUCT.md:30-31]()
## Attribution
This Code of Conduct is adapted from the Contributor Covenant, version 2.1. This indicates a commitment to widely recognized open-source community standards. Sources: [CODE_OF_CONDUCT.md:33-34]()
## Conclusion
The Code of Conduct is a foundational document for the DeBros Network project, ensuring a respectful and inclusive environment for all contributors and users. By outlining clear expectations and providing an enforcement mechanism, it supports the project's goal of fostering an open and welcoming community for its distributed P2P database system. Sources: [CODE_OF_CONDUCT.md:1-4]()
---
## Common Issues
### Related Pages
Related topics: [Authentication Troubleshooting](#authentication-troubleshooting)
Relevant source files
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [CONTRIBUTING.md](https://git.debros.io/DeBros/network/src/main/CONTRIBUTING.md)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [cmd/node/main.go](https://git.debros.io/DeBros/network/src/main/cmd/node/main.go)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
# Common Issues
This document outlines common issues encountered when operating the DeBros Network, along with their symptoms and recommended solutions. Understanding these issues is crucial for effective troubleshooting and maintaining a healthy network. This guide covers problems related to network connectivity, database operations, messaging, memory usage, authentication, gateway functionality, and database migrations. Sources: [README.md](), [AI_CONTEXT.md]()
## Network Connectivity Issues
Network connectivity problems often manifest as nodes failing to connect to the network or peers.
### Bootstrap Connection Failed
**Symptoms:**
* `Failed to connect to bootstrap peer` error messages. Sources: [README.md:166](), [AI_CONTEXT.md:251]()
**Solutions:**
* Verify the bootstrap node is running and accessible.
* Check firewall settings on both the node and the bootstrap server to ensure the LibP2P port (default 4001) is open. Sources: [README.md:168](), [AI_CONTEXT.md:251]()
* Confirm the validity of the bootstrap peer ID in the configuration. Sources: [README.md:168]()
**CLI Check:**
```bash
./bin/network-cli peers
```
Sources: [README.md:175](), [cmd/cli/main.go:120-136]()
## Database Operations Issues
Problems with database operations typically involve queries timing out or an inability to connect to the RQLite database.
### Database Operations Timeout
**Symptoms:**
* `Query timeout` or `No RQLite connection available` messages. Sources: [README.md:171](), [AI_CONTEXT.md:252]()
**Solutions:**
* Ensure RQLite ports (default HTTP 5001, Raft 7001) are open and not blocked by a firewall. Sources: [README.md:173](), [AI_CONTEXT.md:252]()
* Verify that RQLite leader election has completed successfully within the cluster. Sources: [README.md:173](), [AI_CONTEXT.md:252]()
* Check that the `rqlite_join_address` in the node configuration is correct, especially for non-bootstrap nodes. Sources: [README.md:173](), [AI_CONTEXT.md:252]()
**CLI Check:**
```bash
./bin/network-cli query "SELECT 1"
```
Sources: [README.md:175](), [cmd/cli/main.go:149-179]()
## Messaging Issues
Issues with the Pub/Sub system can lead to messages not being delivered to subscribers.
### Message Delivery Failures
**Symptoms:**
* Messages are not received by subscribers. Sources: [README.md:177](), [AI_CONTEXT.md:253]()
**Solutions:**
* Verify that the topic names used for publishing and subscribing match exactly. Sources: [README.md:179](), [AI_CONTEXT.md:253]()
* Confirm that there are active subscriptions to the topic. Sources: [README.md:179](), [AI_CONTEXT.md:253]()
* Check general network connectivity between the publisher and subscriber nodes. Sources: [README.md:179](), [AI_CONTEXT.md:253]()
**CLI Check:**
```bash
./bin/network-cli pubsub publish test "hello"
./bin/network-cli pubsub subscribe test 10s
```
Sources: [README.md:175](), [cmd/cli/main.go:204-211]()
## Resource Usage Issues
Unintended high memory consumption can impact node stability and performance.
### High Memory Usage
**Symptoms:**
* Memory usage continuously grows. Sources: [README.md:181]()
**Solutions:**
* Ensure that applications unsubscribe from Pub/Sub topics when they no longer need to receive messages. Sources: [README.md:183](), [AI_CONTEXT.md:254]()
* Monitor the connection pool size to the database and other services. Sources: [README.md:183]()
* Review message retention policies if applicable. Sources: [README.md:183]()
## Authentication Issues
Authentication failures can prevent access to network services and data.
### Authentication Issues
**Symptoms:**
* `Authentication failed`, `Invalid wallet signature`, `JWT token expired`. Sources: [README.md:185](), [AI_CONTEXT.md:255]()
**Solutions:**
* **Wallet Signature:** Check that the wallet signature format is a 65-byte `r||s||v` hex string. Sources: [README.md:187](), [AI_CONTEXT.md:255]()
* **Nonce Mismatch:** Ensure the nonce used for wallet verification matches exactly. Sources: [README.md:188](), [AI_CONTEXT.md:255]()
* **Wallet Address Case:** Verify wallet address case-insensitivity. Sources: [README.md:189](), [AI_CONTEXT.md:255]()
* **Expired Tokens:** Use the refresh endpoint or re-authenticate for expired JWT tokens. Sources: [README.md:190](), [AI_CONTEXT.md:255]()
* **Credential Conflicts:** Clear the credential cache if multi-wallet conflicts occur by removing `~/.debros/credentials`. Sources: [README.md:191](), [AI_CONTEXT.md:255]()
* **API Key Invalid:** Verify the API key format and its namespace mapping in the gateway configuration. Sources: [AI_CONTEXT.md:260]()
* **Circular Auth Dependencies:** Ensure public authentication endpoints use an internal authentication context to avoid circular dependencies. Sources: [AI_CONTEXT.md:261]()
**Authentication Flow (Gateway):**
```mermaid
sequenceDiagram
participant C as Client
participant G as Gateway
participant N as Node
C->>G: POST /v1/auth/challenge
activate G
G-->>C: {nonce}
deactivate G
Note over C,G: Client signs nonce with wallet
C->>G: POST /v1/auth/verify {wallet, nonce, signature}
activate G
G->>N: Verify signature
activate N
N-->>G: Signature valid
deactivate N
G-->>C: {jwt_token}
deactivate G
C->>G: POST /v1/storage/put (with JWT)
activate G
G->>N: Authorized request
activate N
N-->>G: Data stored
deactivate N
G-->>C: {status: "ok"}
deactivate G
```
Sources: [README.md:218-228](), [AI_CONTEXT.md:104-114]()
## Gateway Issues
Problems with the HTTP Gateway can lead to connection refusals, CORS errors, or WebSocket disconnections.
### Gateway Issues
**Symptoms:**
* `Gateway connection refused`, `CORS errors`, `WebSocket disconnections`. Sources: [README.md:193]()
**Solutions:**
* Verify the gateway is running and accessible on its configured port. Sources: [README.md:195]()
* Check the CORS configuration for web applications interacting with the gateway. Sources: [README.md:196]()
* Ensure proper authentication headers are provided for protected endpoints. Sources: [README.md:197]()
* Verify namespace configuration and enforcement if applicable. Sources: [README.md:198]()
**Health Checks:**
```bash
curl http://localhost:8080/health
curl http://localhost:8080/v1/status
```
Sources: [README.md:200-201]()
## Database Migration Issues
Issues during database migrations can cause schema inconsistencies or data loss.
### Database Migration Issues
**Symptoms:**
* `Migration failed`, `SQL syntax error`, `Version conflict`. Sources: [README.md:203](), [AI_CONTEXT.md:258]()
**Solutions:**
* **SQL Syntax:** Check the SQL syntax in migration files for errors. Sources: [README.md:205](), [AI_CONTEXT.md:258]()
* **Statement Termination:** Ensure proper statement termination (e.g., semicolons) in SQL files. Sources: [README.md:206](), [AI_CONTEXT.md:258]()
* **Naming and Order:** Verify migration file naming and sequential order (e.g., `001_initial_schema.sql`, `002_add_auth_tables.sql`). Sources: [README.md:207](), [AI_CONTEXT.md:258]()
* **Review Logs:** Review migration logs for transaction rollbacks, which can indicate incomplete migrations. Sources: [README.md:208](), [AI_CONTEXT.md:258]()
**Migration File Structure:**
```
migrations/
├── 001_initial_schema.sql
├── 002_add_auth_tables.sql
├── 003_add_indexes.sql
└── ...
```
Sources: [AI_CONTEXT.md:231-236]()
## Debugging and Logging
Effective debugging relies on proper logging and health checks.
### Debugging & Health Checks
**Enable Debug Logging:**
```bash
export LOG_LEVEL=debug
```
Sources: [README.md:211](), [AI_CONTEXT.md:262]()
**Service Logs:**
* **Node Service Logs:**
```bash
sudo journalctl -u debros-node.service --since "1 hour ago"
```
Sources: [README.md:214](), [AI_CONTEXT.md:263]()
* **Gateway Service Logs (if running as service):**
```bash
sudo journalctl -u debros-gateway.service --since "1 hour ago"
```
Sources: [README.md:217](), [AI_CONTEXT.md:264]()
* **Application Logs:**
```bash
tail -f ./logs/gateway.log
tail -f ./logs/node.log
```
Sources: [README.md:220-221]()
## Conclusion
Addressing common issues in the DeBros Network involves systematic checks of configuration, network connectivity, service status, and application logic. Utilizing the provided CLI tools and reviewing detailed logs are essential steps in diagnosing and resolving problems efficiently. Sources: [README.md](), [AI_CONTEXT.md]()
---
## Authentication Troubleshooting
### Related Pages
Related topics: [Common Issues](#common-issues), [CLI Authentication](#cli-authentication), [Gateway Authentication](#gateway-authentication)
Relevant source files
The following files were used as context for generating this wiki page:
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [pkg/gateway/auth_handlers.go](https://git.debros.io/DeBros/network/src/main/pkg/gateway/auth_handlers.go)
- [pkg/auth/wallet.go](https://git.debros.io/DeBros/network/src/main/pkg/auth/wallet.go)
- [cmd/node/main.go](https://git.debros.io/DeBros/network/src/main/cmd/node/main.go)
# Authentication Troubleshooting
Authentication in the DeBros Network is critical for securing access to its decentralized services, including storage, database operations, and PubSub messaging. This page provides guidance on troubleshooting common authentication issues, covering wallet-based authentication, JWT tokens, and API keys. The system is designed for enhanced user experience with features like automatic wallet detection and multi-wallet management, but issues can still arise. Sources: [AI_CONTEXT.md](), [README.md]()
## Common Authentication Issues and Solutions
The DeBros Network's authentication system is robust, but users may encounter issues related to wallet connections, token validity, or API key configuration. Understanding the common symptoms and their solutions is key to resolving these problems efficiently. Sources: [AI_CONTEXT.md:79-88](), [README.md:105-114]()
### Wallet Connection Failed
**Symptoms:**
- Wallet connection failure during authentication flow.
- Error messages indicating invalid signature format or nonce mismatch.
**Solutions:**
- Check wallet signature format: It must be a 65-byte `r||s||v` hex string. Sources: [AI_CONTEXT.md:80](), [README.md:106]()
- Ensure the nonce matches exactly during wallet verification. Sources: [AI_CONTEXT.md:81](), [README.md:107]()
- Verify wallet address case-insensitivity. Sources: [AI_CONTEXT.md:82](), [README.md:108]()
- If multi-wallet conflicts occur, clear the credential cache: `rm -rf ~/.debros/credentials`. Sources: [AI_CONTEXT.md:85](), [README.md:111]()
### JWT Token Expired
**Symptoms:**
- API requests fail with `JWT token expired` errors.
- Inability to access authenticated endpoints after a period of inactivity.
**Solutions:**
- Use the refresh endpoint (`POST /v1/auth/refresh`) to obtain a new JWT token. Sources: [AI_CONTEXT.md:83](), [README.md:109]()
- Re-authenticate with the wallet if refreshing the token does not resolve the issue. Sources: [AI_CONTEXT.md:83](), [README.md:109]()
### API Key Invalid
**Symptoms:**
- API requests return `API Key Invalid` errors.
- Unauthorized access to resources despite providing an API key.
**Solutions:**
- Verify the key format and namespace mapping in the gateway configuration. Sources: [AI_CONTEXT.md:84](), [README.md:110]()
- Ensure the API key is correctly included in the `X-API-Key` header or `Authorization: Bearer ` header. Sources: [AI_CONTEXT.md:213](), [README.md:200]()
## Authentication Flow and Components
The authentication process involves several components, including the client, the gateway, and the wallet. The flow typically starts with a challenge, followed by wallet signature, and verification, ultimately leading to the issuance of an API key or JWT.
### Wallet Authentication Process
The wallet authentication process uses Ethereum EIP-191 for secure verification. It involves generating a challenge, signing it with the user's wallet, and then verifying the signature to grant access.
```mermaid
sequenceDiagram
participant Client
participant Gateway
participant Wallet
Client->>Gateway: POST /v1/auth/challenge
activate Gateway
Gateway-->>Client: Challenge (nonce)
deactivate Gateway
Client->>Wallet: Sign Challenge (nonce)
activate Wallet
Wallet-->>Client: Signed Signature
deactivate Wallet
Client->>Gateway: POST /v1/auth/verify (wallet, nonce, signature)
activate Gateway
Gateway-->>Client: API Key / JWT Token
deactivate Gateway
Note over Client,Gateway: Subsequent requests use API Key / JWT
```
Sources: [AI_CONTEXT.md:191-197](), [README.md:280-286]()
The `AuthServer` in `pkg/auth/wallet.go` handles the callback from the gateway after successful wallet authentication. It parses query parameters to extract the `api_key`, `refresh_token`, `namespace`, `wallet`, `plan`, and `expires_at`. Sources: [pkg/auth/wallet.go:193-200]()
### Authentication Endpoints
The gateway exposes several public and authenticated endpoints for managing authentication.
| Endpoint | Method | Description | Authentication Required | Source |
| :------------------------ | :----- | :--------------------------------------------- | :---------------------- | :-------------- |
| `/v1/auth/challenge` | `POST` | Generate wallet challenge | No | [AI_CONTEXT.md:164]() |
| `/v1/auth/verify` | `POST` | Verify wallet signature | No | [AI_CONTEXT.md:165]() |
| `/v1/auth/register` | `POST` | Register new wallet | No | [AI_CONTEXT.md:166]() |
| `/v1/auth/refresh` | `POST` | Refresh JWT token | No | [AI_CONTEXT.md:167]() |
| `/v1/auth/logout` | `POST` | Clear authentication state | Yes | [AI_CONTEXT.md:168]() |
| `/v1/auth/whoami` | `GET` | Returns current authentication status | Yes | [AI_CONTEXT.md:169]() |
| `/v1/auth/api-key` | `POST` | Generate API keys for authenticated users | Yes | [AI_CONTEXT.md:170]() |
| `/v1/auth/jwks` | `GET` | JWKS endpoint for JWT verification | No | [AI_CONTEXT.md:347]() |
| `/.well-known/jwks.json` | `GET` | Standard JWKS endpoint for JWT verification | No | [AI_CONTEXT.md:347]() |
## Debugging Authentication Issues
Effective debugging involves enabling detailed logging and using CLI tools to inspect the network and authentication status.
### Enabling Debug Logging
To get more detailed information about authentication processes and potential errors, set the `LOG_LEVEL` environment variable to `debug`. Sources: [AI_CONTEXT.md:99](), [README.md:364]()
```bash
export LOG_LEVEL=debug
```
Sources: [AI_CONTEXT.md:99](), [README.md:364]()
### Checking Service Logs
Review the logs of the `debros-node.service` and `debros-gateway.service` (if running as services) for authentication-related errors or warnings. Sources: [AI_CONTEXT.md:100-101](), [README.md:370-373]()
```bash
sudo journalctl -u debros-node.service -f
sudo journalctl -u debros-gateway.service -f
```
Sources: [AI_CONTEXT.md:100-101](), [README.md:370-373]()
### Using the CLI for Health and Peer Checks
The `network-cli` tool can be used to check the health of the node and the status of connected peers, which can indirectly help diagnose network-related authentication issues. Sources: [AI_CONTEXT.md:102-103](), [README.md:365-366]()
```bash
./bin/network-cli health
./bin/network-cli peers
```
Sources: [AI_CONTEXT.md:102-103](), [README.md:365-366]()
### Testing Authentication Flow with CLI
The CLI automatically handles wallet authentication when performing operations that require credentials. This can be used to test the end-to-end authentication flow. Sources: [AI_CONTEXT.md:300-305]()
```bash
# First time - will prompt for wallet authentication
./bin/network-cli storage put user:123 "John Doe"
# Subsequent calls - uses saved credentials automatically
./bin/network-cli storage get user:123
./bin/network-cli pubsub publish notifications "Hello World"
```
Sources: [AI_CONTEXT.md:300-305]()
## Conclusion
Troubleshooting authentication issues in the DeBros Network involves understanding the various authentication methods (wallet, JWT, API keys), common error symptoms, and utilizing debugging tools and logs. The enhanced authentication system aims to streamline the user experience, but knowing how to diagnose and resolve problems is essential for maintaining secure and functional access to the network's decentralized services.
---
## Database Migration Troubleshooting
### Related Pages
Related topics: [Database Layer](#database-layer)
Relevant source files
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [pkg/gateway/migrate.go](https://git.debros.io/DeBros/network/src/main/pkg/gateway/migrate.go)
- [e2e/client_e2e_test.go](https://git.debros.io/DeBros/network/src/main/e2e/client_e2e_test.go)
- [e2e/gateway_e2e_test.go](https://git.debros.io/DeBros/network/src/main/e2e/gateway_e2e_test.go)
- [cmd/node/main.go](https://git.debros.io/DeBros/network/src/main/cmd/node/main.go)
# Database Migration Troubleshooting
This document provides a guide to troubleshooting common issues encountered during database migrations in the DeBros Network. The gateway component includes a robust database migration system designed for automatic schema management and version control, ensuring data integrity and proper upgrade paths. Understanding the system's behavior and common failure points is crucial for effective debugging. Sources: [AI_CONTEXT.md:23-26](), [AI_CONTEXT.md:73-76]()
## Overview of the Migration System
The DeBros Network's gateway features an enhanced database migration system that automatically initializes and updates the database schema using SQL migration files. Each migration runs within a transaction to ensure atomicity. The system tracks migration versions in a `schema_migrations` table to prevent duplicate execution. Sources: [AI_CONTEXT.md:23-26](), [AI_CONTEXT.md:73-76](), [pkg/gateway/migrate.go:103-106]()
The migration process involves:
1. Checking for the existence of the `schema_migrations` table and creating it if necessary. Sources: [pkg/gateway/migrate.go:103-106]()
2. Locating SQL migration files in the `migrations/` directory. Sources: [pkg/gateway/migrate.go:109-112]()
3. Parsing migration versions from filenames (e.g., `001_initial_schema.sql`). Sources: [pkg/gateway/migrate.go:129-130](), [pkg/gateway/migrate.go:174-187]()
4. Sorting migrations by version. Sources: [pkg/gateway/migrate.go:136]()
5. Applying unapplied migrations sequentially, each within a transaction. Sources: [pkg/gateway/migrate.go:140-169]()
```mermaid
graph TD
A[Gateway Startup] --> B{Client connected?};
B -- Yes --> C[Apply Auto Migrations];
C --> D[Apply File-based Migrations];
D --> E[Check schema_migrations table];
E -- Not Exists --> F[Create schema_migrations table];
F --> G[Locate SQL Migration Files];
G --> H[Parse & Sort Migrations];
H --> I{Migration Applied?};
I -- No --> J[Read SQL Content];
J --> K[Split SQL Statements];
K --> L[Execute Statements in Transaction];
L -- Success --> M[Mark Migration as Applied];
M --> I;
I -- Yes / All Applied --> N[Migration Complete];
B -- No --> N;
```
Sources: [pkg/gateway/migrate.go:30-33](), [pkg/gateway/migrate.go:99-102](), [pkg/gateway/migrate.go:103-106](), [pkg/gateway/migrate.go:109-136](), [pkg/gateway/migrate.go:140-169]()
## Common Migration Issues and Solutions
### Migration Failures
**Symptoms:** `Migration failed`, `SQL syntax error`, `Version conflict` Sources: [README.md:20-21](), [AI_CONTEXT.md:144]()
**Causes:**
* Incorrect SQL syntax in migration files.
* Missing statement terminators.
* Improper migration file naming or sequential order.
* Issues with transaction rollbacks or database locks. Sources: [AI_CONTEXT.md:145]()
**Solutions:**
* **Check SQL Syntax:** Carefully review the SQL statements in the problematic migration file for any syntax errors. Sources: [README.md:22](), [AI_CONTEXT.md:146]()
* **Ensure Proper Statement Termination:** Each SQL statement within a migration file should be correctly terminated. The system splits statements based on semicolons. Sources: [README.md:23](), [AI_CONTEXT.md:147](), [pkg/gateway/migrate.go:207-217]()
* **Verify File Naming and Order:** Migration files must be named with a numerical prefix (e.g., `001_initial_schema.sql`) and sorted numerically. Ensure no gaps or duplicates in the versioning. Sources: [README.md:24](), [AI_CONTEXT.md:148](), [pkg/gateway/migrate.go:174-187]()
* **Review Migration Logs:** Detailed logging provides insights into which statement failed and why. Check logs for error messages. Sources: [README.md:25](), [AI_CONTEXT.md:149]()
### Incomplete Migrations
**Symptoms:** Database schema does not reflect expected changes after migration, or some migrations appear to have run partially. Sources: [AI_CONTEXT.md:145]()
**Causes:**
* Transaction rollbacks due to an error within a migration.
* Database locks preventing schema changes.
* Interruption of the migration process.
**Solutions:**
* **Check for Transaction Rollbacks:** The system runs each migration in a transaction. If an error occurs, the entire migration should roll back. If a partial change persists, investigate the specific SQL statement and its atomicity. Sources: [AI_CONTEXT.md:25](), [AI_CONTEXT.md:75](), [pkg/gateway/migrate.go:160-162]()
* **Inspect Database Locks:** If schema changes are blocked, check for any lingering database connections or locks on the RQLite instance.
* **Review `schema_migrations` table:** Query the `schema_migrations` table to see which versions have been successfully applied. If a version is missing, it indicates the migration for that version did not complete. Sources: [pkg/gateway/migrate.go:150-155]()
### Debugging Migration Issues
For deeper debugging, several tools and techniques are available.
#### Logging
Enable debug logging to get more verbose output from the node and gateway services.
```bash
export LOG_LEVEL=debug
```
Sources: [AI_CONTEXT.md:152](), [README.md:30]()
Check service logs:
```bash
sudo journalctl -u debros-node.service -f
sudo journalctl -u debros-gateway.service -f
```
Sources: [AI_CONTEXT.md:154-155](), [README.md:46-50]()
#### CLI Health Checks
Use the `network-cli` tool to check the health and status of the network and database.
```bash
./bin/network-cli health
./bin/network-cli peers
./bin/network-cli query "SELECT 1"
```
Sources: [AI_CONTEXT.md:153](), [README.md:31-33]()
#### End-to-End Tests
The project includes end-to-end tests that simulate database operations, including table creation and transactions, which can be useful for understanding expected behavior and replicating issues.
The `e2e/client_e2e_test.go` and `e2e/gateway_e2e_test.go` files contain tests for database interactions, including schema creation and transactions, which are analogous to migration steps. Sources: [e2e/client_e2e_test.go:46-60](), [e2e/gateway_e2e_test.go:120-131]()
An example of a database creation and transaction test:
```go
func TestClient_Database_CreateQueryMigrate(t *testing.T) {
// ... client setup ...
table := fmt.Sprintf("e2e_items_client_%d", time.Now().UnixNano())
schema := fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP)", table)
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := c.Database().CreateTable(ctx, schema); err != nil {
t.Fatalf("create table: %v", err)
}
// Insert via transaction
stmts := []string{
fmt.Sprintf("INSERT INTO %s(name) VALUES ('alpha')", table),
fmt.Sprintf("INSERT INTO %s(name) VALUES ('beta')", table),
}
ctx2, cancel2 := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel2()
if err := c.Database().Transaction(ctx2, stmts); err != nil {
t.Fatalf("transaction: %v", err)
}
// ... query ...
}
```
Sources: [e2e/client_e2e_test.go:62-89]()
## Migration File Structure
Migration files are standard SQL files with a `.sql` extension, located in the `migrations/` directory. They are prefixed with a version number to ensure correct execution order.
```
migrations/
├── 001_initial_schema.sql # Initial database setup
├── 002_add_auth_tables.sql # Authentication tables
├── 003_add_indexes.sql # Performance indexes
└── ... # Additional migrations
```
Sources: [AI_CONTEXT.md:14-19](), [AI_CONTEXT.md:77-82]()
The `parseMigrationVersion` function extracts the numerical version from the filename:
```go
func parseMigrationVersion(name string) (int, bool) {
i := 0
for i < len(name) && name[i] >= '0' && name[i] <= '9' {
i++
}
if i == 0 {
return 0, false
}
v, err := strconv.Atoi(name[:i])
if err != nil {
return 0, false
}
return v, true
}
```
Sources: [pkg/gateway/migrate.go:174-187]()
The `splitSQLStatements` function handles parsing the SQL content, removing comments, and splitting into individual statements for execution. It specifically ignores `BEGIN;`, `COMMIT;` and `INSERT` statements into `SCHEMA_MIGRATIONS` if present within the file, as these are handled internally by the migration system.
```go
func splitSQLStatements(sqlText string) []string {
lines := strings.Split(sqlText, "\n")
cleaned := make([]string, 0, len(lines))
for _, ln := range lines {
s := strings.TrimSpace(ln)
if s == "" {
continue
}
// Handle inline comments by removing everything after --
if commentIdx := strings.Index(s, "--"); commentIdx >= 0 {
s = strings.TrimSpace(s[:commentIdx])
if s == "" {
continue // line was only a comment
}
}
upper := strings.ToUpper(s)
if upper == "BEGIN;" || upper == "COMMIT;" || upper == "BEGIN" || upper == "COMMIT" {
continue
}
if strings.HasPrefix(upper, "INSERT") && strings.Contains(upper, "SCHEMA_MIGRATIONS") {
// ignore in-file migration markers
continue
}
cleaned = append(cleaned, s)
}
// Join and split by ';'
joined := strings.Join(cleaned, "\n")
parts := strings.Split(joined, ";")
out := make([]string, 0, len(parts))
for _, p := range parts {
sp := strings.TrimSpace(p)
if sp == "" {
continue
}
out = append(out, sp+";")
}
return out
}
```
Sources: [pkg/gateway/migrate.go:189-228]()
## Conclusion
Effective troubleshooting of database migrations in the DeBros Network relies on understanding the automated migration system, its version control mechanisms, and the proper format of migration files. By leveraging detailed logs and the provided CLI tools, developers can diagnose and resolve issues related to SQL syntax, version conflicts, or incomplete schema updates, ensuring a stable and consistent database state.
---
## Debugging & Health Checks
### Related Pages
Related topics: [Service Logs](#service-logs)
Relevant source files
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [cmd/node/main.go](https://git.debros.io/DeBros/network/src/main/cmd/node/main.go)
- [examples/basic_usage.go](https://git.debros.io/DeBros/network/src/main/examples/basic_usage.go)
- [e2e/client_e2e_test.go](https://git.debros.io/DeBros/network/src/main/e2e/client_e2e_test.go)
# Debugging & Health Checks
The DeBros Network provides several tools and methods for debugging issues and performing health checks on both network nodes and the HTTP Gateway. This includes command-line interface (CLI) utilities, direct API calls, and log monitoring, designed to help diagnose common problems such as connection failures, database timeouts, and message delivery issues. Sources: [README.md:144-145](), [AI_CONTEXT.md:144-145]()
## Common Issues and Solutions
The project documentation outlines several common issues that users might encounter and provides specific solutions for each. These range from network connectivity problems to database and authentication issues. Sources: [README.md:147-148](), [AI_CONTEXT.md:147-148]()
### Bootstrap Connection Failed
Symptoms include `Failed to connect to bootstrap peer`. Solutions involve checking if the node is running, firewall settings, and the validity of the peer ID. Sources: [README.md:150-152](), [AI_CONTEXT.md:150-152]()
### Database Operations Timeout
Symptoms are `Query timeout` or `No RQLite connection available`. Solutions require ensuring RQLite ports are open, leader election has completed, and the cluster join configuration is correct. Sources: [README.md:154-156](), [AI_CONTEXT.md:154-156]()
### Message Delivery Failures
Symptoms indicate messages are not received by subscribers. Solutions include verifying topic names, active subscriptions, and network connectivity. Sources: [README.md:158-160](), [AI_CONTEXT.md:158-160]()
### High Memory Usage
Symptoms show memory usage growing continuously. Solutions involve unsubscribing when done, monitoring the connection pool, and reviewing message retention policies. Sources: [README.md:162-164](), [AI_CONTEXT.md:162-164]()
### Authentication Issues
Various symptoms such as `Authentication failed`, `Invalid wallet signature`, or `JWT token expired`. Solutions cover checking wallet signature format (65-byte r||s||v hex), ensuring nonce matches exactly during verification, verifying wallet address case-insensitivity, using the refresh endpoint or re-authenticating for expired tokens, and clearing the credential cache (`rm -rf ~/.debros/credentials`) if multi-wallet conflicts occur. Sources: [README.md:166-173](), [AI_CONTEXT.md:166-173]()
### Gateway Issues
Symptoms include `Gateway connection refused`, `CORS errors`, or `WebSocket disconnections`. Solutions require verifying the gateway is running and accessible, checking CORS configuration for web applications, ensuring proper authentication headers, and verifying namespace configuration. Sources: [README.md:175-180](), [AI_CONTEXT.md:175-180]()
### Database Migration Issues
Symptoms are `Migration failed`, `SQL syntax error`, or `Version conflict`. Solutions involve checking SQL syntax in migration files, ensuring proper statement termination, verifying migration file naming and sequential order, and reviewing migration logs for transaction rollbacks. Sources: [README.md:182-187](), [AI_CONTEXT.md:182-187]()
## Debugging & Health Check Tools
The DeBros Network provides both CLI commands and direct `curl` commands for debugging and health checks.
### CLI Commands
The `network-cli` tool offers several commands to inspect the status and health of the network. Sources: [README.md:190-191](), [AI_CONTEXT.md:190-191]()
```bash
export LOG_LEVEL=debug # Enable debug logging
./bin/network-cli health
./bin/network-cli peers
./bin/network-cli query "SELECT 1"
./bin/network-cli pubsub publish test "hello"
./bin/network-cli pubsub subscribe test 10s
./bin/network-cli storage put test-key test-value
```
Sources: [README.md:190-197](), [AI_CONTEXT.md:190-197]()
The `network-cli` also supports global flags for configuration:
| Flag | Description | Source |
|---|---|---|
| `-b`, `--bootstrap` | Specifies the bootstrap peer address. | [cmd/cli/main.go:88-91]() |
| `-f`, `--format` | Sets the output format (e.g., `json`, `table`). | [cmd/cli/main.go:92-95]() |
| `-t`, `--timeout` | Sets the command timeout duration. | [cmd/cli/main.go:96-100]() |
| `--production` | Enables production mode. | [cmd/cli/main.go:101-102]() |
| `--disable-anonrc` | Disables Anyone proxy routing. | [cmd/cli/main.go:103-104]()
#### CLI Command Flow
The CLI commands typically create a client, connect to the network, perform the requested operation, and then disconnect.
```mermaid
sequenceDiagram
participant CLI as network-cli
participant Client as pkg/client
participant Network as DeBros Network
CLI->>Client: createClient()
activate Client
Client->>Network: Connect()
activate Network
Network-->>Client: Connection Status
deactivate Network
CLI->>Client: Call Operation (e.g., GetPeers)
activate Client
Client->>Network: Request
activate Network
Network-->>Client: Response Data
deactivate Network
deactivate Client
CLI->>Client: Disconnect()
activate Client
Client->>Network: Disconnect Request
deactivate Client
CLI: Print Results
```
Sources: [cmd/cli/main.go:107-111](), [examples/basic_usage.go:21-25]()
### Gateway Health Checks
For the HTTP Gateway, `curl` commands can be used to check its health and status. Sources: [README.md:199-200](), [AI_CONTEXT.md:199-200]()
```bash
curl http://localhost:8080/health
curl http://localhost:8080/v1/status
```
Sources: [README.md:199-200](), [AI_CONTEXT.md:199-200]()
## Service Logs
Monitoring service logs is crucial for debugging. Logs for both the node and gateway services can be accessed using `journalctl` for systemd services or `tail` for application-specific log files. Sources: [README.md:203-204](), [AI_CONTEXT.md:203-204]()
```bash
# Node service logs
sudo journalctl -u debros-node.service --since "1 hour ago"
# Gateway service logs (if running as service)
sudo journalctl -u debros-gateway.service --since "1 hour ago"
# Application logs
tail -f ./logs/gateway.log
tail -f ./logs/node.log
```
Sources: [README.md:206-211](), [AI_CONTEXT.md:206-211]()
The `cmd/node/main.go` file shows how logging is set up and used within the node application, including the use of `zap` for structured logging. Sources: [cmd/node/main.go:20-27]()
## Configuration for Debugging
The `LOG_LEVEL` environment variable can be set to `debug` to enable more verbose logging, which is helpful during troubleshooting. Sources: [README.md:190](), [AI_CONTEXT.md:190]()
Node configuration can be loaded from YAML files, and command-line flags can override these settings, providing flexibility for debugging specific scenarios. For example, `p2p-port`, `rqlite-http-port`, `rqlite-raft-port`, and `rqlite-join-address` can be specified via flags. Sources: [cmd/node/main.go:34-40](), [cmd/node/main.go:120-158]()
```mermaid
graph TD
A[Start Node] --> B{Parse Flags};
B --> C{Load Config from YAML?};
C -- Yes --> D[Load Config File];
C -- No --> E[Load Default Config];
D --> F[Apply Flag Overrides];
E --> F;
F --> G[Initialize Logger];
G --> H[Create and Start Node];
H --> I[Save Peer Info];
I --> J[Wait for Shutdown Signal];
J --> K[Stop Node];
```
Sources: [cmd/node/main.go:34-162]()
## End-to-End Testing for Validation
The project includes end-to-end tests (`e2e/client_e2e_test.go`) that demonstrate client-side interactions with the network, including database operations (create, query, migrate) and storage. These tests can serve as complex health checks or integration tests to validate the system's behavior. Sources: [e2e/client_e2e_test.go:10-46]()
For example, the `TestClient_Database_CreateQueryMigrate` test performs the following steps:
1. Loads API key and namespace from environment variables.
2. Configures the client with bootstrap peers and database endpoints.
3. Connects to the network.
4. Creates a unique table.
5. Inserts data using a transaction.
6. Queries the inserted data.
Sources: [e2e/client_e2e_test.go:48-77]()
This comprehensive approach to debugging and health checks ensures that developers and operators have the necessary tools to maintain and troubleshoot the DeBros Network effectively.
---
## Service Logs
### Related Pages
Related topics: [Debugging & Health Checks](#debugging-health-checks)
Relevant source files
The following files were used as context for generating this wiki page:
- [README.md](https://git.debros.io/DeBros/network/src/main/README.md)
- [AI_CONTEXT.md](https://git.debros.io/DeBros/network/src/main/AI_CONTEXT.md)
- [cmd/node/main.go](https://git.debros.io/DeBros/network/src/main/cmd/node/main.go)
- [cmd/cli/main.go](https://git.debros.io/DeBros/network/src/main/cmd/cli/main.go)
- [CONTRIBUTING.md](https://git.debros.io/DeBros/network/src/main/CONTRIBUTING.md)
# Service Logs
Service logs in the DeBros Network provide critical insights into the operation and health of its various components, including the core network node and the HTTP gateway. These logs are essential for monitoring system behavior, troubleshooting issues, and debugging application flows. The logging system supports different levels of detail and output formats to cater to various operational needs. Sources: [README.md:95-97](), [AI_CONTEXT.md:12-14]()
## Log Management
The DeBros Network utilizes `journalctl` for managing service logs when components are run as systemd services, and direct file tailing for application-specific logs. This dual approach ensures comprehensive log coverage and accessibility. Sources: [README.md:95-97](), [AI_CONTEXT.md:12-14]()
### Systemd Service Logs
When the DeBros Network node or gateway is deployed using the automated production install script, they are configured as systemd services. This allows for centralized log management via `journalctl`. Sources: [README.md:43-46](), [README.md:95-97]()
```bash
# Node service logs
sudo journalctl -u debros-node.service --since "1 hour ago"
# Gateway service logs (if running as service)
sudo journalctl -u debros-gateway.service --since "1 hour ago"
```
Sources: [README.md:95-97](), [AI_CONTEXT.md:12-14]()
### Application-Specific Logs
Beyond systemd, specific application logs can be directly accessed from their designated directories, providing granular detail. Sources: [README.md:95-97]()
```bash
tail -f ./logs/gateway.log
tail -f ./logs/node.log
```
Sources: [README.md:95-97](), [AI_CONTEXT.md:12-14]()
## Log Levels and Configuration
The logging system supports configurable log levels, allowing operators to adjust the verbosity of the output. The default log level is "info". Sources: [README.md:79-81]()
```yaml
logging:
level: "info"
format: "console"
output_file: ""
```
Sources: [README.md:79-81]()
The log level can be dynamically set for debugging purposes using environment variables. Sources: [README.md:90-91]()
```bash
export LOG_LEVEL=debug
```
Sources: [README.md:90-91]()
### Node Logging Setup
The `cmd/node/main.go` file illustrates how the logger is set up for the network node. It uses a `ColoredLogger` from the `pkg/logging` package. Sources: [cmd/node/main.go:21-31]()
```go
func setup_logger(component logging.Component) (logger *logging.ColoredLogger) {
var err error
logger, err = logging.NewColoredLogger(component, true)
if err != nil {
log.Fatalf("Failed to create logger: %v", err)
}
return logger
}
```
Sources: [cmd/node/main.go:21-31]()
The `setup_logger` function is called at various points within the `main.go` to initialize loggers for different components, such as `ComponentNode` and `ComponentAnyone`. Sources: [cmd/node/main.go:34](), [cmd/node/main.go:121]()
## Log Flow Overview
The logging process involves initializing a logger, setting its component, and then emitting messages at various levels (e.g., `Info`, `Error`).
```mermaid
graph TD
A[Start Node/Gateway] --> B{Initialize Logger};
B --> C{Set Logging Component};
C --> D[Load Configuration];
D --> E{Apply CLI/Env Overrides};
E --> F[Start Service];
F --> G{Log Messages (Info, Error, Debug)};
G --> H{Output to Console/File};
H --> I{Systemd Journal (if applicable)};
I --> J[Monitoring/Troubleshooting];
```
Sources: [cmd/node/main.go:21-31](), [cmd/node/main.go:121](), [cmd/node/main.go:183-190]()
## Troubleshooting with Logs
Service logs are the primary tool for diagnosing issues within the DeBros Network. Common troubleshooting steps involve examining logs for specific symptoms. Sources: [README.md:83-84](), [AI_CONTEXT.md:12-14]()
### Common Log-Related Symptoms and Solutions
| Symptom | Description | Log Clues | Solution Approach | Source |
| :------ | :---------- | :-------- | :---------------- | :----- |
| `Failed to connect to bootstrap peer` | Node cannot establish connection with bootstrap peers. | `Error creating node`, `failed to create node`, `failed to start node` | Check node status, firewall settings, peer ID validity. | [README.md:85-87](), [AI_CONTEXT.md:12-14]() |
| `Query timeout` or `No RQLite connection available` | Database operations fail or are unresponsive. | `Error querying data`, `Error inserting data`, `failed to create node`, `failed to start node` | Ensure RQLite ports (5001, 7001) are open, leader election completed, cluster join config is correct. | [README.md:88-89](), [AI_CONTEXT.md:12-14]() |
| `Message Delivery Failures` | Pub/Sub messages are not being received by subscribers. | `Error subscribing`, `Error publishing` | Verify topic names, active subscriptions, network connectivity. | [README.md:90-91](), [AI_CONTEXT.md:12-14]() |
| `High Memory Usage` | Continuous growth in memory consumption. | No specific log error, but resource monitoring shows increasing usage. | Unsubscribe from topics when no longer needed, monitor connection pool, review message retention. | [README.md:92-93](), [AI_CONTEXT.md:12-14]() |
| `Authentication failed` | Issues with user authentication via wallet or API keys. | `Authentication failed`, `Invalid wallet signature`, `JWT token expired` | Check wallet signature format, nonce match, wallet address case-insensitivity, refresh tokens, clear credential cache. | [README.md:94-98](), [AI_CONTEXT.md:12-14]() |
| `Gateway connection refused` | Problems connecting to the HTTP Gateway. | `Gateway connection refused`, `CORS errors`, `WebSocket disconnections` | Verify gateway is running, accessible on configured port, check CORS, authentication headers. | [README.md:99-102](), [AI_CONTEXT.md:12-14]() |
| `Migration failed` | Database schema migration errors. | `Migration failed`, `SQL syntax error`, `Version conflict` | Check SQL syntax in migration files, proper statement termination, file naming, and sequential order. Review migration logs for transaction rollbacks. | [README.md:103-106](), [AI_CONTEXT.md:12-14]() |
Sources: [README.md:85-106](), [AI_CONTEXT.md:12-14]()
## Conclusion
Service logs are an indispensable part of operating the DeBros Network. By leveraging `journalctl` for systemd services and direct file access for application logs, combined with configurable log levels, operators and developers can effectively monitor, troubleshoot, and maintain the health and performance of the distributed system. Sources: [README.md:95-97](), [AI_CONTEXT.md:12-14]()
---