DeBros/network
4
0
Fork 0
mirror of https://github.com/DeBrosOfficial/network.git synced 2025-10-06 13:49:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add database migration and auth system enhancements

Browse Source 
The commit adds a new database migration system and improves the
authentication flow with multi-wallet support. The migration system
provides robust SQL handling and versioning, while the auth system now
features automatic wallet detection and credential persistence.
This commit is contained in:
anonpenguin 2025-08-20 12:55:52 +03:00
parent 1fca8cb411
commit 2ced1114d9
2 changed files with 293 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

82
AI_CONTEXT.md
View File

@ -110,6 +110,7 @@ network/
### 4. **Database Layer (`pkg/database/`)**
- **RQLite:** Distributed SQLite with Raft consensus, automatic leader election, and failover.
- **Client API:** SQL queries, transactions, schema management.
- **Migration System:** Robust database migration handling with automatic schema versioning, SQL statement processing, and error recovery. Supports complex SQL files with comments and multiple statements.
### 5. **Storage System (`pkg/storage/`)**
- **Distributed KV:** Namespace-isolated, CRUD operations, prefix queries, replication.
@ -192,12 +193,17 @@ make run-gateway
# GATEWAY_REQUIRE_AUTH, GATEWAY_API_KEYS
```
- **Auth:** When `RequireAuth` is enabled, endpoints require either:
- JWT (issued by this gateway; JWKS: `GET /v1/auth/jwks` or `/.well-known/jwks.json`)
- API Key (via `Authorization: Bearer <key>` or `X-API-Key`), optionally mapped to a namespace
- Wallet verification uses Ethereum EIP-191 `personal_sign`:
- `POST /v1/auth/challenge` returns `{nonce}`. Clients must sign the exact nonce string.
- `POST /v1/auth/verify` expects `{wallet, nonce, signature}` with 65-byte r||s||v hex (0x allowed). `v` normalized (27/28 or 0/1). Address match is case-insensitive. Nonce is marked used only after successful verification.
- **Enhanced Authentication System:** The gateway features an improved authentication system with automatic wallet detection and multi-wallet support:
- **Automatic Authentication:** No manual auth command required - authentication happens automatically when needed
- **Multi-Wallet Support:** Users can manage multiple wallet credentials seamlessly
- **JWT Authentication:** Issued by this gateway; JWKS available at `GET /v1/auth/jwks` or `/.well-known/jwks.json`
- **API Key Support:** Via `Authorization: Bearer <key>` or `X-API-Key`, optionally mapped to a namespace
- **Wallet Verification:** Uses Ethereum EIP-191 `personal_sign` with enhanced flow:
- `POST /v1/auth/challenge` returns `{nonce}` (public endpoint with internal auth context)
- `POST /v1/auth/verify` expects `{wallet, nonce, signature}` with 65-byte r||s||v hex (0x allowed)
- `v` normalized (27/28 or 0/1), address match is case-insensitive
- Nonce is marked used only after successful verification
- Supports automatic wallet switching and credential persistence
- **Namespace Enforcement:** Storage and PubSub are internally prefixed `ns::<namespace>::...`. Ownership of namespace is enforced by middleware for routes under `/v1/storage*`, `/v1/apps*`, and `/v1/pubsub*`.
@ -212,13 +218,14 @@ make run-gateway
- `GET /v1/auth/jwks`
- `GET /.well-known/jwks.json`
- Auth
- `POST /v1/auth/challenge`
- `POST /v1/auth/verify`
- `POST /v1/auth/register`
- `POST /v1/auth/refresh`
- `POST /v1/auth/logout`
- `GET /v1/auth/whoami`
- Auth (Enhanced Multi-Wallet System)
- `POST /v1/auth/challenge` (public endpoint, generates wallet challenge)
- `POST /v1/auth/verify` (public endpoint, verifies wallet signature)
- `POST /v1/auth/register` (public endpoint, wallet registration)
- `POST /v1/auth/refresh` (public endpoint, token refresh)
- `POST /v1/auth/logout` (clears authentication state)
- `GET /v1/auth/whoami` (returns current auth status)
- `POST /v1/auth/api-key` (generates API keys for authenticated users)
- Storage
- `POST /v1/storage/get`, `POST /v1/storage/put`, `POST /v1/storage/delete`
@ -239,7 +246,40 @@ make run-gateway
- `POST /v1/pubsub/publish` → body `{topic, data_base64}` → `{status:"ok"}`
- `GET /v1/pubsub/topics` → `{topics:["<topic>", ...]}` (names trimmed to caller namespace)
Security note: CORS and WS origin checks are permissive for development; harden for production.
### Authentication Improvements
The gateway authentication system has been significantly enhanced with the following features:
- **Database Migration System:** Robust SQL migration handling with proper statement processing and error handling
- **Automatic Wallet Detection:** CLI automatically detects and manages wallet credentials without manual auth commands
- **Multi-Wallet Management:** Support for multiple wallet credentials with automatic switching
- **Enhanced User Experience:** Streamlined authentication flow with credential persistence
- **Internal Auth Context:** Public authentication endpoints use internal auth context to prevent circular dependencies
- **Improved Error Handling:** Better error messages and debugging information for authentication issues
Security note: CORS and WS origin checks are permissive for development; harden for production. The enhanced authentication system maintains security while improving accessibility and user experience.
### Database Migration System
The gateway includes an enhanced database migration system with the following features:
- **Automatic Schema Management:** Database schema is automatically initialized and updated using migration files
- **Robust SQL Processing:** Handles complex SQL files with comments, multiple statements, and proper statement termination
- **Version Control:** Tracks migration versions to prevent duplicate execution and ensure proper upgrade paths
- **Error Recovery:** Comprehensive error handling with detailed logging for debugging migration issues
- **Transaction Safety:** Each migration runs in a transaction to ensure atomicity and data integrity
- **SQL File Support:** Supports standard SQL migration files with `.sql` extension in the `migrations/` directory
**Migration File Structure:**
```
migrations/
├── 001_initial_schema.sql # Initial database setup
├── 002_add_auth_tables.sql # Authentication tables
├── 003_add_indexes.sql # Performance indexes
└── ... # Additional migrations
```
The migration system automatically processes SQL statements, handles comments, and ensures proper execution order during gateway startup.
---
@ -313,10 +353,24 @@ peers, err := client.Network().GetPeers(ctx)
- **Message Delivery Failures:** Verify topic names, subscription status, and network connectivity.
- **High Memory Usage:** Unsubscribe from topics when done, monitor connection pool size.
### **Authentication Issues**
- **Wallet Connection Failed:** Check wallet signature format (65-byte r||s||v hex), ensure nonce matches exactly, verify wallet address case-insensitivity.
- **JWT Token Expired:** Use refresh endpoint or re-authenticate with wallet.
- **API Key Invalid:** Verify key format and namespace mapping in gateway configuration.
- **Multi-Wallet Conflicts:** Clear credential cache and re-authenticate: `rm -rf ~/.debros/credentials`
- **Circular Auth Dependencies:** Ensure public auth endpoints use internal auth context.
### **Database Migration Issues**
- **Migration Failures:** Check SQL syntax, ensure proper statement termination, review migration logs.
- **Version Conflicts:** Verify migration file naming and sequential order.
- **Incomplete Migrations:** Check for transaction rollbacks and database locks.
### **Debugging**
- Enable debug logging: `export LOG_LEVEL=debug`
- Check service logs: `sudo journalctl -u debros-node.service -f`
- Use CLI for health and peer checks: `./bin/network-cli health`, `./bin/network-cli peers`
- Check gateway logs: `sudo journalctl -u debros-gateway.service -f`
- Test authentication flow: `./bin/network-cli storage put test-key test-value`
---

225
README.md
View File

@ -317,9 +317,191 @@ logging:
--disable-anonrc # Disable anonymous routing (Tor/SOCKS5)
```
### Authentication
The CLI features an enhanced authentication system with automatic wallet detection and multi-wallet support:
- **Automatic Authentication:** No manual auth commands required - authentication happens automatically when operations need credentials
- **Multi-Wallet Management:** Seamlessly switch between multiple wallet credentials
- **Persistent Sessions:** Wallet credentials are automatically saved and restored between sessions
- **Enhanced User Experience:** Streamlined authentication flow with better error handling and user feedback
When using operations that require authentication (storage, database, pubsub), the CLI will automatically:
1. Check for existing valid credentials
2. Prompt for wallet authentication if needed
3. Handle signature verification
4. Persist credentials for future use
**Example with automatic authentication:**
```bash
# First time - will prompt for wallet authentication
./bin/network-cli storage put user:123 "John Doe"
# Subsequent calls - uses saved credentials automatically
./bin/network-cli storage get user:123
./bin/network-cli pubsub publish notifications "Hello World"
```
---
## HTTP Gateway
The DeBros Network includes a powerful HTTP/WebSocket gateway that provides a modern REST API and WebSocket interface over the P2P network, featuring an enhanced authentication system with multi-wallet support.
### Quick Start
```bash
make run-gateway
# Or manually:
go run ./cmd/gateway
```
### Configuration
The gateway can be configured via environment variables:
```bash
# Basic Configuration
export GATEWAY_ADDR="0.0.0.0:8080"
export GATEWAY_NAMESPACE="my-app"
export GATEWAY_BOOTSTRAP_PEERS="/ip4/127.0.0.1/tcp/4001/p2p/YOUR_PEER_ID"
# Authentication Configuration
export GATEWAY_REQUIRE_AUTH=true
export GATEWAY_API_KEYS="key1:namespace1,key2:namespace2"
```
### Enhanced Authentication System
The gateway features a significantly improved authentication system with the following capabilities:
#### Key Features
- **Automatic Authentication:** No manual auth commands required - authentication happens automatically when needed
- **Multi-Wallet Support:** Seamlessly manage multiple wallet credentials with automatic switching
- **Persistent Sessions:** Wallet credentials are automatically saved and restored
- **Enhanced User Experience:** Streamlined authentication flow with better error handling
#### Authentication Methods
**Wallet-Based Authentication (Ethereum EIP-191)**
- Uses `personal_sign` for secure wallet verification
- Supports multiple wallets with automatic detection
- Addresses are case-insensitive with normalized signature handling
**JWT Tokens**
- Issued by the gateway with configurable expiration
- JWKS endpoints available at `/v1/auth/jwks` and `/.well-known/jwks.json`
- Automatic refresh capability
**API Keys**
- Support for pre-configured API keys via `Authorization: Bearer <key>` or `X-API-Key` headers
- Optional namespace mapping for multi-tenant applications
### API Endpoints
#### Health & Status
```http
GET /health # Basic health check
GET /v1/health # Detailed health status
GET /v1/status # Network status
GET /v1/version # Version information
```
#### Authentication (Public Endpoints)
```http
POST /v1/auth/challenge # Generate wallet challenge
POST /v1/auth/verify # Verify wallet signature
POST /v1/auth/register # Register new wallet
POST /v1/auth/refresh # Refresh JWT token
POST /v1/auth/logout # Clear authentication
GET /v1/auth/whoami # Current auth status
POST /v1/auth/api-key # Generate API key (authenticated)
```
#### Storage Operations
```http
POST /v1/storage/get # Retrieve data
POST /v1/storage/put # Store data
POST /v1/storage/delete # Delete data
GET /v1/storage/list # List keys with optional prefix
GET /v1/storage/exists # Check key existence
```
#### Network Operations
```http
GET /v1/network/status # Network status
GET /v1/network/peers # Connected peers
POST /v1/network/connect # Connect to peer
POST /v1/network/disconnect # Disconnect from peer
```
#### Pub/Sub Messaging
**WebSocket Interface**
```http
GET /v1/pubsub/ws?topic=<topic> # WebSocket connection for real-time messaging
```
**REST Interface**
```http
POST /v1/pubsub/publish # Publish message to topic
GET /v1/pubsub/topics # List active topics
```
### Security Features
- **Namespace Enforcement:** All operations are automatically prefixed with namespace for isolation
- **CORS Support:** Configurable CORS policies (permissive for development, configurable for production)
- **Transport Security:** All network communications use Noise/TLS encryption
- **Authentication Middleware:** Flexible authentication with support for multiple credential types
### Usage Examples
#### Wallet Authentication Flow
```bash
# 1. Get challenge (automatic)
curl -X POST http://localhost:8080/v1/auth/challenge
# 2. Sign challenge with wallet (handled by client)
# 3. Verify signature (automatic)
curl -X POST http://localhost:8080/v1/auth/verify \
-H "Content-Type: application/json" \
-d '{"wallet":"0x...","nonce":"...","signature":"0x..."}'
```
#### Storage Operations
```bash
# Store data
curl -X POST http://localhost:8080/v1/storage/put \
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{"key":"user:123","value":"eyJuYW1lIjoiSm9obiJ9"}'
# Retrieve data
curl -X POST http://localhost:8080/v1/storage/get \
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{"key":"user:123"}'
```
#### Real-time Messaging
```javascript
// WebSocket connection
const ws = new WebSocket('ws://localhost:8080/v1/pubsub/ws?topic=chat');
ws.onmessage = (event) => {
console.log('Received:', event.data);
};
// Send message
ws.send('Hello, network!');
```
---
## Development
</text>
### Project Structure
@ -381,6 +563,34 @@ scripts/test-multinode.sh
- **Symptoms:** Memory usage grows continuously
- **Solutions:** Unsubscribe when done, monitor connection pool, review message retention.
#### Authentication Issues
- **Symptoms:** `Authentication failed`, `Invalid wallet signature`, `JWT token expired`
- **Solutions:**
- Check wallet signature format (65-byte r||s||v hex)
- Ensure nonce matches exactly during wallet verification
- Verify wallet address case-insensitivity
- Use refresh endpoint or re-authenticate for expired tokens
- Clear credential cache if multi-wallet conflicts occur: `rm -rf ~/.debros/credentials`
#### Gateway Issues
- **Symptoms:** `Gateway connection refused`, `CORS errors`, `WebSocket disconnections`
- **Solutions:**
- Verify gateway is running and accessible on configured port
- Check CORS configuration for web applications
- Ensure proper authentication headers for protected endpoints
- Verify namespace configuration and enforcement
#### Database Migration Issues
- **Symptoms:** `Migration failed`, `SQL syntax error`, `Version conflict`
- **Solutions:**
- Check SQL syntax in migration files
- Ensure proper statement termination
- Verify migration file naming and sequential order
- Review migration logs for transaction rollbacks
### Debugging & Health Checks
```bash
@ -390,12 +600,27 @@ export LOG_LEVEL=debug
./bin/network-cli query "SELECT 1"
./bin/network-cli pubsub publish test "hello"
./bin/network-cli pubsub subscribe test 10s
# Test authentication flow
./bin/network-cli storage put test-key test-value
# Gateway health checks
curl http://localhost:8080/health
curl http://localhost:8080/v1/status
```
### Service Logs
```bash
# Node service logs
sudo journalctl -u debros-node.service --since "1 hour ago"
# Gateway service logs (if running as service)
sudo journalctl -u debros-gateway.service --since "1 hour ago"
# Application logs
tail -f ./logs/gateway.log
tail -f ./logs/node.log
```
---