Compare commits

..

No commits in common. "main" and "v0.50.0" have entirely different histories.

12 changed files with 244 additions and 353 deletions

View File

@ -16,41 +16,6 @@ The format is based on [Keep a Changelog][keepachangelog] and adheres to [Semant
### Fixed
## [0.51.0] - 2025-09-26
### Added
- Added identity/main.go to generate identity and peer id
- Added encryption module identity.go for reusable identity create, save etc funtions
### Changed
- Updated make file to support identity/main.go
- Updated node/node.go on loadOrCreateIdentity to use encryption.identity
- Updated cli/main.go to remove fallbacks for identity
- Updated install-debros-network.sh script to use new ./cmd/identity and fixed port order on print
### Deprecated
### Removed
### Fixed
## [0.50.1] - 2025-09-23
### Added
### Changed
### Deprecated
### Removed
### Fixed
- Fixed wrong URL /v1/db to /v1/rqlite
### Security
## [0.50.0] - 2025-09-23
@ -67,6 +32,7 @@ The format is based on [Keep a Changelog][keepachangelog] and adheres to [Semant
- Updated node.go to support new rqlite architecture
- Updated readme
### Deprecated
### Removed
@ -99,6 +65,7 @@ The format is based on [Keep a Changelog][keepachangelog] and adheres to [Semant
### Security
## [0.43.6] - 2025-09-20
### Added
@ -121,13 +88,11 @@ The format is based on [Keep a Changelog][keepachangelog] and adheres to [Semant
## [0.43.4] - 2025-09-18
### Added
- Added extra comments on main.go
- Remove backoff_test.go and associated backoff tests
- Created node_test, write tests for CalculateNextBackoff, AddJitter, GetPeerId, LoadOrCreateIdentity, hasBootstrapConnections
### Changed
- replaced git.debros.io with github.com
### Deprecated
@ -141,24 +106,20 @@ The format is based on [Keep a Changelog][keepachangelog] and adheres to [Semant
## [0.43.3] - 2025-09-15
### Added
- User authentication module with OAuth2 support.
### Changed
- Make file version to 0.43.2
### Deprecated
### Removed
- Removed cli, network-cli binaries from project
- Removed AI_CONTEXT.md
- Removed Network.md
- Removed unused log from monitoring.go
### Fixed
- Resolved race condition when saving settings.
### Security

View File

@ -21,7 +21,7 @@ test-e2e:
.PHONY: build clean test run-node run-node2 run-node3 run-example deps tidy fmt vet lint clear-ports
VERSION := 0.51.0-beta
VERSION := 0.50.0-beta
COMMIT ?= $(shell git rev-parse --short HEAD 2>/dev/null || echo unknown)
DATE ?= $(shell date -u +%Y-%m-%dT%H:%M:%SZ)
LDFLAGS := -X 'main.version=$(VERSION)' -X 'main.commit=$(COMMIT)' -X 'main.date=$(DATE)'
@ -30,7 +30,6 @@ LDFLAGS := -X 'main.version=$(VERSION)' -X 'main.commit=$(COMMIT)' -X 'main.date
build: deps
@echo "Building network executables (version=$(VERSION))..."
@mkdir -p bin
go build -ldflags "$(LDFLAGS)" -o bin/identity ./cmd/identity
go build -ldflags "$(LDFLAGS)" -o bin/node ./cmd/node
go build -ldflags "$(LDFLAGS)" -o bin/network-cli cmd/cli/main.go
# Inject gateway build metadata via pkg path variables

126
README.md
View File

@ -143,7 +143,6 @@ curl -sSL https://github.com/DeBrosOfficial/network/raw/main/scripts/install-deb
```
**What the Script Does:**
- Detects OS, installs Go, RQLite, dependencies
- Creates `debros` system user, secure directory structure
- Generates LibP2P identity keys
@ -153,7 +152,6 @@ curl -sSL https://github.com/DeBrosOfficial/network/raw/main/scripts/install-deb
- Generates YAML config in `/opt/debros/configs/node.yaml`
**Directory Structure:**
```
/opt/debros/
├── bin/ # Binaries
@ -165,7 +163,6 @@ curl -sSL https://github.com/DeBrosOfficial/network/raw/main/scripts/install-deb
```
**Service Management:**
```bash
sudo systemctl status debros-node
sudo systemctl start debros-node
@ -264,7 +261,6 @@ logging:
The .yaml files are required in order for the nodes and the gateway to run correctly.
node:
- id (string) Optional node ID. Auto-generated if empty.
- type (string) "bootstrap" or "node". Default: "node".
- listen_addresses (string[]) LibP2P listen multiaddrs. Default: ["/ip4/0.0.0.0/tcp/4001"].
@ -272,7 +268,6 @@ node:
- max_connections (int) Max peer connections. Default: 50.
database:
- data_dir (string) Directory for database files. Default: "./data/db".
- replication_factor (int) Number of replicas. Default: 3.
- shard_count (int) Shards for data distribution. Default: 16.
@ -283,7 +278,6 @@ database:
- rqlite_join_address (string) HTTP address of an existing RQLite node to join. Empty for bootstrap.
discovery:
- bootstrap_peers (string[]) List of LibP2P multiaddrs of bootstrap peers.
- discovery_interval (duration) How often to announce/discover peers. Default: 15s.
- bootstrap_port (int) Default port for bootstrap nodes. Default: 4001.
@ -292,13 +286,11 @@ discovery:
- node_namespace (string) Namespace for node identifiers. Default: "default".
security:
- enable_tls (bool) Enable TLS for externally exposed services. Default: false.
- private_key_file (string) Path to TLS private key (if TLS enabled).
- certificate_file (string) Path to TLS certificate (if TLS enabled).
logging:
- level (string) one of "debug", "info", "warn", "error". Default: "info".
- format (string) "json" or "console". Default: "console".
- output_file (string) Empty for stdout; otherwise path to log file.
@ -355,7 +347,6 @@ logging:
Precedence (gateway): Flags > Environment Variables > YAML > Defaults.
Environment variables:
- GATEWAY_ADDR
- GATEWAY_NAMESPACE
- GATEWAY_BOOTSTRAP_PEERS (comma-separated)
@ -394,6 +385,8 @@ bootstrap_peers:
./bin/network-cli peers # List connected peers
```
### Database Operations
```bash
@ -421,27 +414,27 @@ bootstrap_peers:
### Database Operations (Gateway REST)
```http
POST /v1/rqlite/exec # Body: {"sql": "INSERT/UPDATE/DELETE/DDL ...", "args": [...]}
POST /v1/rqlite/find # Body: {"table":"...", "criteria":{"col":val,...}, "options":{...}}
POST /v1/rqlite/find-one # Body: same as /find, returns a single row (404 if not found)
POST /v1/rqlite/select # Body: {"table":"...", "select":[...], "where":[...], "joins":[...], "order_by":[...], "limit":N, "offset":N, "one":false}
POST /v1/rqlite/transaction # Body: {"ops":[{"kind":"exec|query","sql":"...","args":[...]}], "return_results": true}
POST /v1/rqlite/query # Body: {"sql": "SELECT ...", "args": [..]} (legacy-friendly SELECT)
GET /v1/rqlite/schema # Returns tables/views + create SQL
POST /v1/rqlite/create-table # Body: {"schema": "CREATE TABLE ..."}
POST /v1/rqlite/drop-table # Body: {"table": "table_name"}
POST /v1/db/exec # Body: {"sql": "INSERT/UPDATE/DELETE/DDL ...", "args": [...]}
POST /v1/db/find # Body: {"table":"...", "criteria":{"col":val,...}, "options":{...}}
POST /v1/db/find-one # Body: same as /find, returns a single row (404 if not found)
POST /v1/db/select # Body: {"table":"...", "select":[...], "where":[...], "joins":[...], "order_by":[...], "limit":N, "offset":N, "one":false}
POST /v1/db/transaction # Body: {"ops":[{"kind":"exec|query","sql":"...","args":[...]}], "return_results": true}
POST /v1/db/query # Body: {"sql": "SELECT ...", "args": [..]} (legacy-friendly SELECT)
GET /v1/db/schema # Returns tables/views + create SQL
POST /v1/db/create-table # Body: {"schema": "CREATE TABLE ..."}
POST /v1/db/drop-table # Body: {"table": "table_name"}
```
Common workflows:
```bash
# Exec (INSERT/UPDATE/DELETE/DDL)
curl -X POST "$GW/v1/rqlite/exec" \
curl -X POST "$GW/v1/db/exec" \
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{"sql":"INSERT INTO users(name,email) VALUES(?,?)","args":["Alice","alice@example.com"]}'
# Find (criteria + options)
curl -X POST "$GW/v1/rqlite/find" \
curl -X POST "$GW/v1/db/find" \
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{
"table":"users",
@ -450,7 +443,7 @@ curl -X POST "$GW/v1/rqlite/find" \
}'
# Select (fluent builder via JSON)
curl -X POST "$GW/v1/rqlite/select" \
curl -X POST "$GW/v1/db/select" \
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{
"table":"orders o",
@ -462,7 +455,7 @@ curl -X POST "$GW/v1/rqlite/select" \
}'
# Transaction (atomic batch)
curl -X POST "$GW/v1/rqlite/transaction" \
curl -X POST "$GW/v1/db/transaction" \
-H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{
"return_results": true,
@ -473,12 +466,12 @@ curl -X POST "$GW/v1/rqlite/transaction" \
}'
# Schema
curl "$GW/v1/rqlite/schema" -H "Authorization: Bearer $API_KEY"
curl "$GW/v1/db/schema" -H "Authorization: Bearer $API_KEY"
# DDL helpers
curl -X POST "$GW/v1/rqlite/create-table" -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
curl -X POST "$GW/v1/db/create-table" -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{"schema":"CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT, email TEXT)"}'
curl -X POST "$GW/v1/rqlite/drop-table" -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
curl -X POST "$GW/v1/db/drop-table" -H "Authorization: Bearer $API_KEY" -H 'Content-Type: application/json' \
-d '{"table":"users"}'
```
@ -492,14 +485,12 @@ The CLI features an enhanced authentication system with automatic wallet detecti
- **Enhanced User Experience:** Streamlined authentication flow with better error handling and user feedback
When using operations that require authentication (storage, database, pubsub), the CLI will automatically:
1. Check for existing valid credentials
2. Prompt for wallet authentication if needed
3. Handle signature verification
4. Persist credentials for future use
**Example with automatic authentication:**
```bash
# First time - will prompt for wallet authentication when needed
./bin/network-cli pubsub publish notifications "Hello World"
@ -539,7 +530,6 @@ export GATEWAY_API_KEYS="key1:namespace1,key2:namespace2"
The gateway features a significantly improved authentication system with the following capabilities:
#### Key Features
- **Automatic Authentication:** No manual auth commands required - authentication happens automatically when needed
- **Multi-Wallet Support:** Seamlessly manage multiple wallet credentials with automatic switching
- **Persistent Sessions:** Wallet credentials are automatically saved and restored
@ -548,26 +538,22 @@ The gateway features a significantly improved authentication system with the fol
#### Authentication Methods
**Wallet-Based Authentication (Ethereum EIP-191)**
- Uses `personal_sign` for secure wallet verification
- Supports multiple wallets with automatic detection
- Addresses are case-insensitive with normalized signature handling
**JWT Tokens**
- Issued by the gateway with configurable expiration
- JWKS endpoints available at `/v1/auth/jwks` and `/.well-known/jwks.json`
- Automatic refresh capability
**API Keys**
- Support for pre-configured API keys via `Authorization: Bearer <key>` or `X-API-Key` headers
- Optional namespace mapping for multi-tenant applications
### API Endpoints
#### Health & Status
```http
GET /health # Basic health check
GET /v1/health # Detailed health status
@ -576,7 +562,6 @@ GET /v1/version # Version information
```
#### Authentication (Public Endpoints)
```http
POST /v1/auth/challenge # Generate wallet challenge
POST /v1/auth/verify # Verify wallet signature
@ -593,20 +578,19 @@ The gateway now exposes a full HTTP interface over the Go ORM-like client (see `
- Base path: `/v1/db`
- Endpoints:
- `POST /v1/rqlite/exec` — Execute write/DDL SQL; returns `{ rows_affected, last_insert_id }`
- `POST /v1/rqlite/find` — Map-based criteria; returns `{ items: [...], count: N }`
- `POST /v1/rqlite/find-one` — Single row; 404 if not found
- `POST /v1/rqlite/select` — Fluent SELECT via JSON (joins, where, order, group, limit, offset)
- `POST /v1/rqlite/transaction` — Atomic batch of exec/query ops, optional per-op results
- `POST /v1/rqlite/query` — Arbitrary SELECT (legacy-friendly), returns `items`
- `GET /v1/rqlite/schema` — List user tables/views + create SQL
- `POST /v1/rqlite/create-table` — Convenience for DDL
- `POST /v1/rqlite/drop-table` — Safe drop (identifier validated)
- `POST /v1/db/exec` — Execute write/DDL SQL; returns `{ rows_affected, last_insert_id }`
- `POST /v1/db/find` — Map-based criteria; returns `{ items: [...], count: N }`
- `POST /v1/db/find-one` — Single row; 404 if not found
- `POST /v1/db/select` — Fluent SELECT via JSON (joins, where, order, group, limit, offset)
- `POST /v1/db/transaction` — Atomic batch of exec/query ops, optional per-op results
- `POST /v1/db/query` — Arbitrary SELECT (legacy-friendly), returns `items`
- `GET /v1/db/schema` — List user tables/views + create SQL
- `POST /v1/db/create-table` — Convenience for DDL
- `POST /v1/db/drop-table` — Safe drop (identifier validated)
Payload examples are shown in the [Database Operations (Gateway REST)](#database-operations-gateway-rest) section.
#### Network Operations
```http
GET /v1/network/status # Network status
GET /v1/network/peers # Connected peers
@ -617,13 +601,11 @@ POST /v1/network/disconnect # Disconnect from peer
#### Pub/Sub Messaging
**WebSocket Interface**
```http
GET /v1/pubsub/ws?topic=<topic> # WebSocket connection for real-time messaging
```
**REST Interface**
```http
POST /v1/pubsub/publish # Publish message to topic
GET /v1/pubsub/topics # List active topics
@ -634,34 +616,31 @@ GET /v1/pubsub/topics # List active topics
## SDK Authoring Guide
### Base concepts
- OpenAPI: a machine-readable spec is available at `openapi/gateway.yaml` for SDK code generation.
- **Auth**: send `X-API-Key: <key>` or `Authorization: Bearer <key|JWT>` with every request.
- **Versioning**: all endpoints are under `/v1/`.
- **Responses**: mutations return `{status:"ok"}`; queries/lists return JSON; errors return `{ "error": "message" }` with proper HTTP status.
### Key HTTP endpoints for SDKs
- **Database**
- Exec: `POST /v1/rqlite/exec` `{sql, args?}``{rows_affected,last_insert_id}`
- Find: `POST /v1/rqlite/find` `{table, criteria, options?}``{items,count}`
- FindOne: `POST /v1/rqlite/find-one` `{table, criteria, options?}` → single object or 404
- Select: `POST /v1/rqlite/select` `{table, select?, joins?, where?, order_by?, group_by?, limit?, offset?, one?}`
- Transaction: `POST /v1/rqlite/transaction` `{ops:[{kind,sql,args?}], return_results?}`
- Query: `POST /v1/rqlite/query` `{sql, args?}``{items,count}`
- Schema: `GET /v1/rqlite/schema`
- Create Table: `POST /v1/rqlite/create-table` `{schema}`
- Drop Table: `POST /v1/rqlite/drop-table` `{table}`
- Exec: `POST /v1/db/exec` `{sql, args?}``{rows_affected,last_insert_id}`
- Find: `POST /v1/db/find` `{table, criteria, options?}``{items,count}`
- FindOne: `POST /v1/db/find-one` `{table, criteria, options?}` → single object or 404
- Select: `POST /v1/db/select` `{table, select?, joins?, where?, order_by?, group_by?, limit?, offset?, one?}`
- Transaction: `POST /v1/db/transaction` `{ops:[{kind,sql,args?}], return_results?}`
- Query: `POST /v1/db/query` `{sql, args?}``{items,count}`
- Schema: `GET /v1/db/schema`
- Create Table: `POST /v1/db/create-table` `{schema}`
- Drop Table: `POST /v1/db/drop-table` `{table}`
- **PubSub**
- WS Subscribe: `GET /v1/pubsub/ws?topic=<topic>`
- Publish: `POST /v1/pubsub/publish` `{topic, data_base64}``{status:"ok"}`
- Topics: `GET /v1/pubsub/topics``{topics:[...]}`
### Migrations
- Add column: `ALTER TABLE users ADD COLUMN age INTEGER`
- Change type / add FK (recreate pattern): create `_new` table, copy data, drop old, rename.
- Always send as one `POST /v1/rqlite/transaction`.
- Always send as one `POST /v1/db/transaction`.
### Minimal examples
@ -670,13 +649,8 @@ TypeScript (Node)
```ts
import { GatewayClient } from "../examples/sdk-typescript/src/client";
const client = new GatewayClient(
process.env.GATEWAY_BASE_URL!,
process.env.GATEWAY_API_KEY!
);
await client.createTable(
"CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)"
);
const client = new GatewayClient(process.env.GATEWAY_BASE_URL!, process.env.GATEWAY_API_KEY!);
await client.createTable("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)");
const res = await client.query("SELECT name FROM users WHERE id = ?", [1]);
```
@ -690,7 +664,7 @@ KEY = os.environ['GATEWAY_API_KEY']
H = { 'X-API-Key': KEY, 'Content-Type': 'application/json' }
def query(sql, args=None):
r = requests.post(f'{BASE}/v1/rqlite/query', json={ 'sql': sql, 'args': args or [] }, headers=H, timeout=15)
r = requests.post(f'{BASE}/v1/db/query', json={ 'sql': sql, 'args': args or [] }, headers=H, timeout=15)
r.raise_for_status()
return r.json()['rows']
```
@ -698,7 +672,7 @@ def query(sql, args=None):
Go
```go
req, _ := http.NewRequest(http.MethodPost, base+"/v1/rqlite/create-table", bytes.NewBufferString(`{"schema":"CREATE TABLE ..."}`))
req, _ := http.NewRequest(http.MethodPost, base+"/v1/db/create-table", bytes.NewBufferString(`{"schema":"CREATE TABLE ..."}`))
req.Header.Set("X-API-Key", apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
@ -714,7 +688,6 @@ resp, err := http.DefaultClient.Do(req)
### Usage Examples
#### Wallet Authentication Flow
```bash
# 1. Get challenge (automatic)
curl -X POST http://localhost:6001/v1/auth/challenge
@ -726,26 +699,27 @@ curl -X POST http://localhost:6001/v1/auth/verify \
-d '{"wallet":"0x...","nonce":"...","signature":"0x..."}'
```
#### Real-time Messaging
#### Real-time Messaging
```javascript
// WebSocket connection
const ws = new WebSocket("ws://localhost:6001/v1/pubsub/ws?topic=chat");
const ws = new WebSocket('ws://localhost:6001/v1/pubsub/ws?topic=chat');
ws.onmessage = (event) => {
console.log("Received:", event.data);
console.log('Received:', event.data);
};
// Send message
ws.send("Hello, network!");
ws.send('Hello, network!');
```
---
## Development
</text>
### Project Structure
```
@ -784,7 +758,6 @@ scripts/test-multinode.sh
## Database Client (Go ORM-like)
A lightweight ORM-like client over rqlite using Gos `database/sql`. It provides:
- Query/Exec for raw SQL
- A fluent QueryBuilder (`Where`, `InnerJoin`, `LeftJoin`, `OrderBy`, `GroupBy`, `Limit`, `Offset`)
- Simple repositories with `Find`/`FindOne`
@ -799,7 +772,7 @@ A lightweight ORM-like client over rqlite using Gos `database/sql`. It provid
### Quick Start
````go
```go
package main
import (
@ -861,7 +834,7 @@ type Post struct {
CreatedAt time.Time `db:"created_at"`
}
func (Post) TableName() string { return "posts" }
````
```
### Basic queries
@ -1015,6 +988,7 @@ if err := rqlite.ApplyMigrationsDirs(ctx, db, dirs, logger); err != nil {
}
```
---
## Troubleshooting

View File

@ -2,6 +2,7 @@ package main
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"log"
@ -362,6 +363,50 @@ func handlePeerID() {
}
}
// Fallback: try to extract from local identity files
identityPaths := []string{
"/opt/debros/data/node/identity.key",
"/opt/debros/data/bootstrap/identity.key",
"/opt/debros/keys/node/identity.key",
"./data/node/identity.key",
"./data/bootstrap/identity.key",
}
for _, path := range identityPaths {
if peerID := extractPeerIDFromFile(path); peerID != "" {
if format == "json" {
printJSON(map[string]string{"peer_id": peerID, "source": "local_identity"})
} else {
fmt.Printf("🆔 Peer ID: %s\n", peerID)
fmt.Printf("📂 Source: %s\n", path)
}
return
}
}
// Check peer.info files as last resort
peerInfoPaths := []string{
"/opt/debros/data/node/peer.info",
"/opt/debros/data/bootstrap/peer.info",
"./data/node/peer.info",
"./data/bootstrap/peer.info",
}
for _, path := range peerInfoPaths {
if data, err := os.ReadFile(path); err == nil {
multiaddr := strings.TrimSpace(string(data))
if peerID := extractPeerIDFromMultiaddr(multiaddr); peerID != "" {
if format == "json" {
printJSON(map[string]string{"peer_id": peerID, "source": "peer_info"})
} else {
fmt.Printf("🆔 Peer ID: %s\n", peerID)
fmt.Printf("📂 Source: %s\n", path)
}
return
}
}
}
fmt.Fprintf(os.Stderr, "❌ Could not find peer ID. Make sure the node is running or identity files exist.\n")
os.Exit(1)
}
@ -425,6 +470,20 @@ func discoverBootstrapPeer() string {
return "" // Return empty string if no peer info found
}
func tryDecodeBase64(s string) string {
// Only try to decode if it looks like base64 (no spaces, reasonable length)
if len(s) > 0 && len(s)%4 == 0 && !strings.ContainsAny(s, " \n\r\t") {
if decoded, err := base64.StdEncoding.DecodeString(s); err == nil {
// Check if decoded result looks like readable text
decodedStr := string(decoded)
if isPrintableText(decodedStr) {
return decodedStr
}
}
}
return s
}
func isPrintableText(s string) bool {
printableCount := 0
for _, r := range s {

View File

@ -1,45 +0,0 @@
package main
import (
"flag"
"fmt"
"os"
"github.com/DeBrosOfficial/network/pkg/encryption"
)
func main() {
var outputPath string
var displayOnly bool
flag.StringVar(&outputPath, "output", "", "Output path for identity key")
flag.BoolVar(&displayOnly, "display-only", false, "Only display identity info, don't save")
flag.Parse()
// Generate identity using shared package
info, err := encryption.GenerateIdentity()
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to generate identity: %v\n", err)
os.Exit(1)
}
// If display only, just show the info
if displayOnly {
fmt.Printf("Node Identity: %s\n", info.PeerID.String())
return
}
// Save to file using shared package
if outputPath == "" {
fmt.Fprintln(os.Stderr, "Output path is required")
os.Exit(1)
}
if err := encryption.SaveIdentity(info, outputPath); err != nil {
fmt.Fprintf(os.Stderr, "Failed to save identity: %v\n", err)
os.Exit(1)
}
fmt.Printf("Generated Node Identity: %s\n", info.PeerID.String())
fmt.Printf("Identity saved to: %s\n", outputPath)
}

View File

@ -170,7 +170,7 @@ func TestGateway_Database_CreateQueryMigrate(t *testing.T) {
// Create table
schema := `CREATE TABLE IF NOT EXISTS e2e_items (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP)`
body := fmt.Sprintf(`{"schema":%q}`, schema)
req, _ := http.NewRequest(http.MethodPost, base+"/v1/rqlite/create-table", strings.NewReader(body))
req, _ := http.NewRequest(http.MethodPost, base+"/v1/db/create-table", strings.NewReader(body))
req.Header = authHeader(key)
resp, err := httpClient().Do(req)
if err != nil {
@ -183,7 +183,7 @@ func TestGateway_Database_CreateQueryMigrate(t *testing.T) {
// Insert via transaction (simulate migration/data seed)
txBody := `{"statements":["INSERT INTO e2e_items(name) VALUES ('one')","INSERT INTO e2e_items(name) VALUES ('two')"]}`
req, _ = http.NewRequest(http.MethodPost, base+"/v1/rqlite/transaction", strings.NewReader(txBody))
req, _ = http.NewRequest(http.MethodPost, base+"/v1/db/transaction", strings.NewReader(txBody))
req.Header = authHeader(key)
resp, err = httpClient().Do(req)
if err != nil {
@ -196,7 +196,7 @@ func TestGateway_Database_CreateQueryMigrate(t *testing.T) {
// Query rows
qBody := `{"sql":"SELECT name FROM e2e_items ORDER BY id ASC"}`
req, _ = http.NewRequest(http.MethodPost, base+"/v1/rqlite/query", strings.NewReader(qBody))
req, _ = http.NewRequest(http.MethodPost, base+"/v1/db/query", strings.NewReader(qBody))
req.Header = authHeader(key)
resp, err = httpClient().Do(req)
if err != nil {
@ -219,7 +219,7 @@ func TestGateway_Database_CreateQueryMigrate(t *testing.T) {
}
// Schema endpoint returns tables
req, _ = http.NewRequest(http.MethodGet, base+"/v1/rqlite/schema", nil)
req, _ = http.NewRequest(http.MethodGet, base+"/v1/db/schema", nil)
req.Header = authHeader(key)
resp2, err := httpClient().Do(req)
if err != nil {
@ -239,7 +239,7 @@ func TestGateway_Database_DropTable(t *testing.T) {
schema := fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (id INTEGER PRIMARY KEY, note TEXT)", table)
// create
body := fmt.Sprintf(`{"schema":%q}`, schema)
req, _ := http.NewRequest(http.MethodPost, base+"/v1/rqlite/create-table", strings.NewReader(body))
req, _ := http.NewRequest(http.MethodPost, base+"/v1/db/create-table", strings.NewReader(body))
req.Header = authHeader(key)
resp, err := httpClient().Do(req)
if err != nil {
@ -251,7 +251,7 @@ func TestGateway_Database_DropTable(t *testing.T) {
}
// drop
dbody := fmt.Sprintf(`{"table":%q}`, table)
req, _ = http.NewRequest(http.MethodPost, base+"/v1/rqlite/drop-table", strings.NewReader(dbody))
req, _ = http.NewRequest(http.MethodPost, base+"/v1/db/drop-table", strings.NewReader(dbody))
req.Header = authHeader(key)
resp, err = httpClient().Do(req)
if err != nil {
@ -262,7 +262,7 @@ func TestGateway_Database_DropTable(t *testing.T) {
t.Fatalf("drop-table status: %d", resp.StatusCode)
}
// verify not in schema
req, _ = http.NewRequest(http.MethodGet, base+"/v1/rqlite/schema", nil)
req, _ = http.NewRequest(http.MethodGet, base+"/v1/db/schema", nil)
req.Header = authHeader(key)
resp2, err := httpClient().Do(req)
if err != nil {
@ -298,7 +298,7 @@ func TestGateway_Database_RecreateWithFK(t *testing.T) {
createUsers := fmt.Sprintf(`{"schema":%q}`, fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (id INTEGER PRIMARY KEY, name TEXT, org_id INTEGER, age TEXT)", users))
for _, body := range []string{createOrgs, createUsers} {
req, _ := http.NewRequest(http.MethodPost, base+"/v1/rqlite/create-table", strings.NewReader(body))
req, _ := http.NewRequest(http.MethodPost, base+"/v1/db/create-table", strings.NewReader(body))
req.Header = authHeader(key)
resp, err := httpClient().Do(req)
if err != nil {
@ -311,7 +311,7 @@ func TestGateway_Database_RecreateWithFK(t *testing.T) {
}
// seed data
txSeed := fmt.Sprintf(`{"statements":["INSERT INTO %s(id,name) VALUES (1,'org')","INSERT INTO %s(id,name,org_id,age) VALUES (1,'alice',1,'30')"]}`, orgs, users)
req, _ := http.NewRequest(http.MethodPost, base+"/v1/rqlite/transaction", strings.NewReader(txSeed))
req, _ := http.NewRequest(http.MethodPost, base+"/v1/db/transaction", strings.NewReader(txSeed))
req.Header = authHeader(key)
resp, err := httpClient().Do(req)
if err != nil {
@ -331,7 +331,7 @@ func TestGateway_Database_RecreateWithFK(t *testing.T) {
"DROP TABLE %s",
"ALTER TABLE %s_new RENAME TO %s"
]}`, users, orgs, users, users, users, users, users)
req, _ = http.NewRequest(http.MethodPost, base+"/v1/rqlite/transaction", strings.NewReader(txMig))
req, _ = http.NewRequest(http.MethodPost, base+"/v1/db/transaction", strings.NewReader(txMig))
req.Header = authHeader(key)
resp, err = httpClient().Do(req)
if err != nil {
@ -344,7 +344,7 @@ func TestGateway_Database_RecreateWithFK(t *testing.T) {
// verify schema type change
qBody := fmt.Sprintf(`{"sql":"PRAGMA table_info(%s)"}`, users)
req, _ = http.NewRequest(http.MethodPost, base+"/v1/rqlite/query", strings.NewReader(qBody))
req, _ = http.NewRequest(http.MethodPost, base+"/v1/db/query", strings.NewReader(qBody))
req.Header = authHeader(key)
resp, err = httpClient().Do(req)
if err != nil {
@ -375,7 +375,7 @@ func TestGateway_Database_RecreateWithFK(t *testing.T) {
if !ageIsInt {
// Fallback: inspect CREATE TABLE SQL from sqlite_master
qBody2 := fmt.Sprintf(`{"sql":"SELECT sql FROM sqlite_master WHERE type='table' AND name='%s'"}`, users)
req2, _ := http.NewRequest(http.MethodPost, base+"/v1/rqlite/query", strings.NewReader(qBody2))
req2, _ := http.NewRequest(http.MethodPost, base+"/v1/db/query", strings.NewReader(qBody2))
req2.Header = authHeader(key)
resp3, err := httpClient().Do(req2)
if err != nil {

View File

@ -1,110 +1,81 @@
import WebSocket from "isomorphic-ws";
import WebSocket from 'isomorphic-ws';
export class GatewayClient {
constructor(
private baseUrl: string,
private apiKey: string,
private http = fetch
) {}
constructor(private baseUrl: string, private apiKey: string, private http = fetch) {}
private headers(json = true): Record<string, string> {
const h: Record<string, string> = { "X-API-Key": this.apiKey };
if (json) h["Content-Type"] = "application/json";
const h: Record<string, string> = { 'X-API-Key': this.apiKey };
if (json) h['Content-Type'] = 'application/json';
return h;
}
// Database
async createTable(schema: string): Promise<void> {
const r = await this.http(`${this.baseUrl}/v1/rqlite/create-table`, {
method: "POST",
headers: this.headers(),
body: JSON.stringify({ schema }),
const r = await this.http(`${this.baseUrl}/v1/db/create-table`, {
method: 'POST', headers: this.headers(), body: JSON.stringify({ schema })
});
if (!r.ok) throw new Error(`createTable failed: ${r.status}`);
}
async dropTable(table: string): Promise<void> {
const r = await this.http(`${this.baseUrl}/v1/rqlite/drop-table`, {
method: "POST",
headers: this.headers(),
body: JSON.stringify({ table }),
const r = await this.http(`${this.baseUrl}/v1/db/drop-table`, {
method: 'POST', headers: this.headers(), body: JSON.stringify({ table })
});
if (!r.ok) throw new Error(`dropTable failed: ${r.status}`);
}
async query<T = any>(sql: string, args: any[] = []): Promise<{ rows: T[] }> {
const r = await this.http(`${this.baseUrl}/v1/rqlite/query`, {
method: "POST",
headers: this.headers(),
body: JSON.stringify({ sql, args }),
const r = await this.http(`${this.baseUrl}/v1/db/query`, {
method: 'POST', headers: this.headers(), body: JSON.stringify({ sql, args })
});
if (!r.ok) throw new Error(`query failed: ${r.status}`);
return r.json();
}
async transaction(statements: string[]): Promise<void> {
const r = await this.http(`${this.baseUrl}/v1/rqlite/transaction`, {
method: "POST",
headers: this.headers(),
body: JSON.stringify({ statements }),
const r = await this.http(`${this.baseUrl}/v1/db/transaction`, {
method: 'POST', headers: this.headers(), body: JSON.stringify({ statements })
});
if (!r.ok) throw new Error(`transaction failed: ${r.status}`);
}
async schema(): Promise<any> {
const r = await this.http(`${this.baseUrl}/v1/rqlite/schema`, {
headers: this.headers(false),
});
const r = await this.http(`${this.baseUrl}/v1/db/schema`, { headers: this.headers(false) });
if (!r.ok) throw new Error(`schema failed: ${r.status}`);
return r.json();
}
// Storage
async put(key: string, value: Uint8Array | string): Promise<void> {
const body =
typeof value === "string" ? new TextEncoder().encode(value) : value;
const r = await this.http(
`${this.baseUrl}/v1/storage/put?key=${encodeURIComponent(key)}`,
{
method: "POST",
headers: { "X-API-Key": this.apiKey },
body,
}
);
const body = typeof value === 'string' ? new TextEncoder().encode(value) : value;
const r = await this.http(`${this.baseUrl}/v1/storage/put?key=${encodeURIComponent(key)}`, {
method: 'POST', headers: { 'X-API-Key': this.apiKey }, body
});
if (!r.ok) throw new Error(`put failed: ${r.status}`);
}
async get(key: string): Promise<Uint8Array> {
const r = await this.http(
`${this.baseUrl}/v1/storage/get?key=${encodeURIComponent(key)}`,
{
headers: { "X-API-Key": this.apiKey },
}
);
const r = await this.http(`${this.baseUrl}/v1/storage/get?key=${encodeURIComponent(key)}`, {
headers: { 'X-API-Key': this.apiKey }
});
if (!r.ok) throw new Error(`get failed: ${r.status}`);
const buf = new Uint8Array(await r.arrayBuffer());
return buf;
}
async exists(key: string): Promise<boolean> {
const r = await this.http(
`${this.baseUrl}/v1/storage/exists?key=${encodeURIComponent(key)}`,
{
headers: this.headers(false),
}
);
const r = await this.http(`${this.baseUrl}/v1/storage/exists?key=${encodeURIComponent(key)}`, {
headers: this.headers(false)
});
if (!r.ok) throw new Error(`exists failed: ${r.status}`);
const j = await r.json();
return !!j.exists;
}
async list(prefix = ""): Promise<string[]> {
const r = await this.http(
`${this.baseUrl}/v1/storage/list?prefix=${encodeURIComponent(prefix)}`,
{
headers: this.headers(false),
}
);
const r = await this.http(`${this.baseUrl}/v1/storage/list?prefix=${encodeURIComponent(prefix)}`, {
headers: this.headers(false)
});
if (!r.ok) throw new Error(`list failed: ${r.status}`);
const j = await r.json();
return j.keys || [];
@ -112,42 +83,29 @@ export class GatewayClient {
async delete(key: string): Promise<void> {
const r = await this.http(`${this.baseUrl}/v1/storage/delete`, {
method: "POST",
headers: this.headers(),
body: JSON.stringify({ key }),
method: 'POST', headers: this.headers(), body: JSON.stringify({ key })
});
if (!r.ok) throw new Error(`delete failed: ${r.status}`);
}
// PubSub (minimal)
subscribe(
topic: string,
onMessage: (data: Uint8Array) => void
): { close: () => void } {
const url = new URL(`${this.baseUrl.replace(/^http/, "ws")}/v1/pubsub/ws`);
url.searchParams.set("topic", topic);
const ws = new WebSocket(url.toString(), {
headers: { "X-API-Key": this.apiKey },
} as any);
ws.binaryType = "arraybuffer";
subscribe(topic: string, onMessage: (data: Uint8Array) => void): { close: () => void } {
const url = new URL(`${this.baseUrl.replace(/^http/, 'ws')}/v1/pubsub/ws`);
url.searchParams.set('topic', topic);
const ws = new WebSocket(url.toString(), { headers: { 'X-API-Key': this.apiKey } } as any);
ws.binaryType = 'arraybuffer';
ws.onmessage = (ev: any) => {
const data =
ev.data instanceof ArrayBuffer
? new Uint8Array(ev.data)
: new TextEncoder().encode(String(ev.data));
const data = ev.data instanceof ArrayBuffer ? new Uint8Array(ev.data) : new TextEncoder().encode(String(ev.data));
onMessage(data);
};
return { close: () => ws.close() };
}
async publish(topic: string, data: Uint8Array | string): Promise<void> {
const bytes =
typeof data === "string" ? new TextEncoder().encode(data) : data;
const b64 = Buffer.from(bytes).toString("base64");
const bytes = typeof data === 'string' ? new TextEncoder().encode(data) : data;
const b64 = Buffer.from(bytes).toString('base64');
const r = await this.http(`${this.baseUrl}/v1/pubsub/publish`, {
method: "POST",
headers: this.headers(),
body: JSON.stringify({ topic, data_base64: b64 }),
method: 'POST', headers: this.headers(), body: JSON.stringify({ topic, data_base64: b64 })
});
if (!r.ok) throw new Error(`publish failed: ${r.status}`);
}

View File

@ -192,7 +192,7 @@ paths:
key: { type: string }
responses:
"200": { description: OK }
/v1/rqlite/create-table:
/v1/db/create-table:
post:
summary: Create tables via SQL DDL
requestBody:
@ -220,7 +220,7 @@ paths:
{ schema: { $ref: "#/components/schemas/Error" } },
},
}
/v1/rqlite/drop-table:
/v1/db/drop-table:
post:
summary: Drop a table
requestBody:
@ -230,7 +230,7 @@ paths:
schema: { $ref: "#/components/schemas/DropTableRequest" }
responses:
"200": { description: OK }
/v1/rqlite/query:
/v1/db/query:
post:
summary: Execute a single SQL query
requestBody:
@ -262,7 +262,7 @@ paths:
{ schema: { $ref: "#/components/schemas/Error" } },
},
}
/v1/rqlite/transaction:
/v1/db/transaction:
post:
summary: Execute multiple SQL statements atomically
requestBody:
@ -290,7 +290,7 @@ paths:
{ schema: { $ref: "#/components/schemas/Error" } },
},
}
/v1/rqlite/schema:
/v1/db/schema:
get:
summary: Get current database schema
responses:

View File

@ -1,71 +0,0 @@
package encryption
import (
"crypto/rand"
"os"
"path/filepath"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peer"
)
type IdentityInfo struct {
PrivateKey crypto.PrivKey
PublicKey crypto.PubKey
PeerID peer.ID
}
func GenerateIdentity() (*IdentityInfo, error) {
priv, pub, err := crypto.GenerateKeyPairWithReader(crypto.Ed25519, 2048, rand.Reader)
if err != nil {
return nil, err
}
peerID, err := peer.IDFromPublicKey(pub)
if err != nil {
return nil, err
}
return &IdentityInfo{
PrivateKey: priv,
PublicKey: pub,
PeerID: peerID,
}, nil
}
func SaveIdentity(identity *IdentityInfo, path string) error {
data, err := crypto.MarshalPrivateKey(identity.PrivateKey)
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(path), 0700); err != nil {
return err
}
return os.WriteFile(path, data, 0600)
}
func LoadIdentity(path string) (*IdentityInfo, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, err
}
priv, err := crypto.UnmarshalPrivateKey(data)
if err != nil {
return nil, err
}
pub := priv.GetPublic()
peerID, err := peer.IDFromPublicKey(pub)
if err != nil {
return nil, err
}
return &IdentityInfo{
PrivateKey: priv,
PublicKey: pub,
PeerID: peerID,
}, nil
}

View File

@ -27,7 +27,7 @@ func (g *Gateway) Routes() http.Handler {
mux.HandleFunc("/v1/auth/logout", g.logoutHandler)
mux.HandleFunc("/v1/auth/whoami", g.whoamiHandler)
// rqlite ORM HTTP gateway (mounts /v1/rqlite/* endpoints)
// rqlite ORM HTTP gateway (mounts /v1/db/* endpoints)
if g.ormHTTP != nil {
g.ormHTTP.BasePath = "/v1/rqlite"
g.ormHTTP.RegisterRoutes(mux)

View File

@ -2,6 +2,7 @@ package node
import (
"context"
"crypto/rand"
"fmt"
mathrand "math/rand"
"os"
@ -22,7 +23,6 @@ import (
"go.uber.org/zap"
"github.com/DeBrosOfficial/network/pkg/config"
"github.com/DeBrosOfficial/network/pkg/encryption"
"github.com/DeBrosOfficial/network/pkg/logging"
"github.com/DeBrosOfficial/network/pkg/pubsub"
database "github.com/DeBrosOfficial/network/pkg/rqlite"
@ -374,41 +374,65 @@ func (n *Node) startLibP2P() error {
return nil
}
// loadOrCreateIdentity loads an existing identity or creates a new one
// loadOrCreateIdentity loads an existing identity or creates a new one
func (n *Node) loadOrCreateIdentity() (crypto.PrivKey, error) {
identityFile := filepath.Join(n.config.Node.DataDir, "identity.key")
// Try to load existing identity using the shared package
// Try to load existing identity
if _, err := os.Stat(identityFile); err == nil {
info, err := encryption.LoadIdentity(identityFile)
data, err := os.ReadFile(identityFile)
if err != nil {
n.logger.Warn("Failed to load existing identity, creating new one", zap.Error(err))
return nil, fmt.Errorf("failed to read identity file: %w", err)
}
priv, err := crypto.UnmarshalPrivateKey(data)
if err != nil {
n.logger.Warn("Failed to unmarshal existing identity, creating new one", zap.Error(err))
} else {
n.logger.ComponentInfo(logging.ComponentNode, "Loaded existing identity",
zap.String("file", identityFile),
zap.String("peer_id", info.PeerID.String()))
return info.PrivateKey, nil
// Extract peer ID from private key for logging
peerID, err := peer.IDFromPrivateKey(priv)
if err != nil {
n.logger.ComponentInfo(logging.ComponentNode, "Loaded existing identity",
zap.String("file", identityFile),
zap.String("peer_id", "unable_to_extract"))
} else {
n.logger.ComponentInfo(logging.ComponentNode, "Loaded existing identity",
zap.String("file", identityFile),
zap.String("peer_id", peerID.String()))
}
return priv, nil
}
}
// Create new identity using shared package
// Create new identity
n.logger.Info("Creating new identity", zap.String("file", identityFile))
info, err := encryption.GenerateIdentity()
priv, _, err := crypto.GenerateKeyPairWithReader(crypto.Ed25519, 2048, rand.Reader)
if err != nil {
return nil, fmt.Errorf("failed to generate identity: %w", err)
return nil, fmt.Errorf("failed to generate key pair: %w", err)
}
// Save identity using shared package
if err := encryption.SaveIdentity(info, identityFile); err != nil {
// Extract peer ID from private key for logging
peerID, err := peer.IDFromPrivateKey(priv)
if err != nil {
n.logger.Info("Identity created",
zap.String("peer_id", "unable_to_extract"))
} else {
n.logger.Info("Identity created",
zap.String("peer_id", peerID.String()))
}
// Save identity
data, err := crypto.MarshalPrivateKey(priv)
if err != nil {
return nil, fmt.Errorf("failed to marshal private key: %w", err)
}
if err := os.WriteFile(identityFile, data, 0600); err != nil {
return nil, fmt.Errorf("failed to save identity: %w", err)
}
n.logger.Info("Identity saved",
zap.String("file", identityFile),
zap.String("peer_id", info.PeerID.String()))
return info.PrivateKey, nil
n.logger.Info("Identity saved", zap.String("file", identityFile))
return priv, nil
}
// GetPeerID returns the peer ID of this node

View File

@ -335,8 +335,40 @@ generate_identity() {
fi
log "Generating node identity..."
cd "$INSTALL_DIR/src"
cat > /tmp/generate_identity_custom.go << 'EOF'
package main
import (
"crypto/rand"
"flag"
"fmt"
"os"
"path/filepath"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peer"
)
func main() {
var outputPath string
flag.StringVar(&outputPath, "output", "", "Output path for identity key")
flag.Parse()
if outputPath == "" {
fmt.Println("Usage: go run generate_identity_custom.go -output <path>")
os.Exit(1)
}
priv, pub, err := crypto.GenerateKeyPairWithReader(crypto.Ed25519, 2048, rand.Reader)
if err != nil { panic(err) }
peerID, err := peer.IDFromPublicKey(pub)
if err != nil { panic(err) }
data, err := crypto.MarshalPrivateKey(priv)
if err != nil { panic(err) }
if err := os.MkdirAll(filepath.Dir(outputPath), 0700); err != nil { panic(err) }
if err := os.WriteFile(outputPath, data, 0600); err != nil { panic(err) }
fmt.Printf("Generated Peer ID: %s\n", peerID.String())
fmt.Printf("Identity saved to: %s\n", outputPath)
}
EOF
export PATH=$PATH:/usr/local/go/bin
sudo -u debros env "PATH=$PATH:/usr/local/go/bin" go run ./cmd/identity -output "$identity_file"
sudo -u debros env "PATH=$PATH:/usr/local/go/bin" "GOMOD=$(pwd)" go run /tmp/generate_identity_custom.go -output "$identity_file"
rm /tmp/generate_identity_custom.go
success "Node identity generated"
}
@ -528,10 +560,10 @@ main() {
log "${GREEN}Installation Directory:${NOCOLOR} ${CYAN}$INSTALL_DIR${NOCOLOR}"
log "${GREEN}Configuration:${NOCOLOR} ${CYAN}$INSTALL_DIR/configs/node.yaml${NOCOLOR}"
log "${GREEN}Logs:${NOCOLOR} ${CYAN}$INSTALL_DIR/logs/node.log${NOCOLOR}"
log "${GREEN}LibP2P Port:${NOCOLOR} ${CYAN}$NODE_PORT${NOCOLOR}"
log "${GREEN}Node Port:${NOCOLOR} ${CYAN}$NODE_PORT${NOCOLOR}"
log "${GREEN}RQLite Port:${NOCOLOR} ${CYAN}$RQLITE_PORT${NOCOLOR}"
log "${GREEN}Gateway Port:${NOCOLOR} ${CYAN}$GATEWAY_PORT${NOCOLOR}"
log "${GREEN}Raft Port:${NOCOLOR} ${CYAN}$RAFT_PORT${NOCOLOR}"
log "${GREEN}Gateway Port:${NOCOLOR} ${CYAN}$GATEWAY_PORT${NOCOLOR}"
log "${BLUE}==================================================${NOCOLOR}"
log "${GREEN}Management Commands:${NOCOLOR}"
log "${CYAN} - sudo systemctl status debros-node${NOCOLOR} (Check status)"