From 2ced1114d9616d4a76afa97a2c7404eb6be791f9 Mon Sep 17 00:00:00 2001 From: anonpenguin Date: Wed, 20 Aug 2025 12:55:52 +0300 Subject: [PATCH] Add database migration and auth system enhancements The commit adds a new database migration system and improves the authentication flow with multi-wallet support. The migration system provides robust SQL handling and versioning, while the auth system now features automatic wallet detection and credential persistence. --- AI_CONTEXT.md | 82 ++++++++++++++---- README.md | 225 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 293 insertions(+), 14 deletions(-) diff --git a/AI_CONTEXT.md b/AI_CONTEXT.md index 3ce8f1c..57b0bf2 100644 --- a/AI_CONTEXT.md +++ b/AI_CONTEXT.md @@ -110,6 +110,7 @@ network/ ### 4. **Database Layer (`pkg/database/`)** - **RQLite:** Distributed SQLite with Raft consensus, automatic leader election, and failover. - **Client API:** SQL queries, transactions, schema management. +- **Migration System:** Robust database migration handling with automatic schema versioning, SQL statement processing, and error recovery. Supports complex SQL files with comments and multiple statements. ### 5. **Storage System (`pkg/storage/`)** - **Distributed KV:** Namespace-isolated, CRUD operations, prefix queries, replication. @@ -192,12 +193,17 @@ make run-gateway # GATEWAY_REQUIRE_AUTH, GATEWAY_API_KEYS ``` -- **Auth:** When `RequireAuth` is enabled, endpoints require either: - - JWT (issued by this gateway; JWKS: `GET /v1/auth/jwks` or `/.well-known/jwks.json`) - - API Key (via `Authorization: Bearer ` or `X-API-Key`), optionally mapped to a namespace - - Wallet verification uses Ethereum EIP-191 `personal_sign`: - - `POST /v1/auth/challenge` returns `{nonce}`. Clients must sign the exact nonce string. - - `POST /v1/auth/verify` expects `{wallet, nonce, signature}` with 65-byte r||s||v hex (0x allowed). `v` normalized (27/28 or 0/1). Address match is case-insensitive. Nonce is marked used only after successful verification. +- **Enhanced Authentication System:** The gateway features an improved authentication system with automatic wallet detection and multi-wallet support: + - **Automatic Authentication:** No manual auth command required - authentication happens automatically when needed + - **Multi-Wallet Support:** Users can manage multiple wallet credentials seamlessly + - **JWT Authentication:** Issued by this gateway; JWKS available at `GET /v1/auth/jwks` or `/.well-known/jwks.json` + - **API Key Support:** Via `Authorization: Bearer ` or `X-API-Key`, optionally mapped to a namespace + - **Wallet Verification:** Uses Ethereum EIP-191 `personal_sign` with enhanced flow: + - `POST /v1/auth/challenge` returns `{nonce}` (public endpoint with internal auth context) + - `POST /v1/auth/verify` expects `{wallet, nonce, signature}` with 65-byte r||s||v hex (0x allowed) + - `v` normalized (27/28 or 0/1), address match is case-insensitive + - Nonce is marked used only after successful verification + - Supports automatic wallet switching and credential persistence - **Namespace Enforcement:** Storage and PubSub are internally prefixed `ns::::...`. Ownership of namespace is enforced by middleware for routes under `/v1/storage*`, `/v1/apps*`, and `/v1/pubsub*`. @@ -212,13 +218,14 @@ make run-gateway - `GET /v1/auth/jwks` - `GET /.well-known/jwks.json` -- Auth - - `POST /v1/auth/challenge` - - `POST /v1/auth/verify` - - `POST /v1/auth/register` - - `POST /v1/auth/refresh` - - `POST /v1/auth/logout` - - `GET /v1/auth/whoami` +- Auth (Enhanced Multi-Wallet System) + - `POST /v1/auth/challenge` (public endpoint, generates wallet challenge) + - `POST /v1/auth/verify` (public endpoint, verifies wallet signature) + - `POST /v1/auth/register` (public endpoint, wallet registration) + - `POST /v1/auth/refresh` (public endpoint, token refresh) + - `POST /v1/auth/logout` (clears authentication state) + - `GET /v1/auth/whoami` (returns current auth status) + - `POST /v1/auth/api-key` (generates API keys for authenticated users) - Storage - `POST /v1/storage/get`, `POST /v1/storage/put`, `POST /v1/storage/delete` @@ -239,7 +246,40 @@ make run-gateway - `POST /v1/pubsub/publish` → body `{topic, data_base64}` → `{status:"ok"}` - `GET /v1/pubsub/topics` → `{topics:["", ...]}` (names trimmed to caller namespace) -Security note: CORS and WS origin checks are permissive for development; harden for production. +### Authentication Improvements + +The gateway authentication system has been significantly enhanced with the following features: + +- **Database Migration System:** Robust SQL migration handling with proper statement processing and error handling +- **Automatic Wallet Detection:** CLI automatically detects and manages wallet credentials without manual auth commands +- **Multi-Wallet Management:** Support for multiple wallet credentials with automatic switching +- **Enhanced User Experience:** Streamlined authentication flow with credential persistence +- **Internal Auth Context:** Public authentication endpoints use internal auth context to prevent circular dependencies +- **Improved Error Handling:** Better error messages and debugging information for authentication issues + +Security note: CORS and WS origin checks are permissive for development; harden for production. The enhanced authentication system maintains security while improving accessibility and user experience. + +### Database Migration System + +The gateway includes an enhanced database migration system with the following features: + +- **Automatic Schema Management:** Database schema is automatically initialized and updated using migration files +- **Robust SQL Processing:** Handles complex SQL files with comments, multiple statements, and proper statement termination +- **Version Control:** Tracks migration versions to prevent duplicate execution and ensure proper upgrade paths +- **Error Recovery:** Comprehensive error handling with detailed logging for debugging migration issues +- **Transaction Safety:** Each migration runs in a transaction to ensure atomicity and data integrity +- **SQL File Support:** Supports standard SQL migration files with `.sql` extension in the `migrations/` directory + +**Migration File Structure:** +``` +migrations/ +├── 001_initial_schema.sql # Initial database setup +├── 002_add_auth_tables.sql # Authentication tables +├── 003_add_indexes.sql # Performance indexes +└── ... # Additional migrations +``` + +The migration system automatically processes SQL statements, handles comments, and ensures proper execution order during gateway startup. --- @@ -313,10 +353,24 @@ peers, err := client.Network().GetPeers(ctx) - **Message Delivery Failures:** Verify topic names, subscription status, and network connectivity. - **High Memory Usage:** Unsubscribe from topics when done, monitor connection pool size. +### **Authentication Issues** +- **Wallet Connection Failed:** Check wallet signature format (65-byte r||s||v hex), ensure nonce matches exactly, verify wallet address case-insensitivity. +- **JWT Token Expired:** Use refresh endpoint or re-authenticate with wallet. +- **API Key Invalid:** Verify key format and namespace mapping in gateway configuration. +- **Multi-Wallet Conflicts:** Clear credential cache and re-authenticate: `rm -rf ~/.debros/credentials` +- **Circular Auth Dependencies:** Ensure public auth endpoints use internal auth context. + +### **Database Migration Issues** +- **Migration Failures:** Check SQL syntax, ensure proper statement termination, review migration logs. +- **Version Conflicts:** Verify migration file naming and sequential order. +- **Incomplete Migrations:** Check for transaction rollbacks and database locks. + ### **Debugging** - Enable debug logging: `export LOG_LEVEL=debug` - Check service logs: `sudo journalctl -u debros-node.service -f` - Use CLI for health and peer checks: `./bin/network-cli health`, `./bin/network-cli peers` +- Check gateway logs: `sudo journalctl -u debros-gateway.service -f` +- Test authentication flow: `./bin/network-cli storage put test-key test-value` --- diff --git a/README.md b/README.md index 08914e1..da74efb 100644 --- a/README.md +++ b/README.md @@ -317,9 +317,191 @@ logging: --disable-anonrc # Disable anonymous routing (Tor/SOCKS5) ``` +### Authentication + +The CLI features an enhanced authentication system with automatic wallet detection and multi-wallet support: + +- **Automatic Authentication:** No manual auth commands required - authentication happens automatically when operations need credentials +- **Multi-Wallet Management:** Seamlessly switch between multiple wallet credentials +- **Persistent Sessions:** Wallet credentials are automatically saved and restored between sessions +- **Enhanced User Experience:** Streamlined authentication flow with better error handling and user feedback + +When using operations that require authentication (storage, database, pubsub), the CLI will automatically: +1. Check for existing valid credentials +2. Prompt for wallet authentication if needed +3. Handle signature verification +4. Persist credentials for future use + +**Example with automatic authentication:** +```bash +# First time - will prompt for wallet authentication +./bin/network-cli storage put user:123 "John Doe" + +# Subsequent calls - uses saved credentials automatically +./bin/network-cli storage get user:123 +./bin/network-cli pubsub publish notifications "Hello World" +``` + +--- + +## HTTP Gateway + +The DeBros Network includes a powerful HTTP/WebSocket gateway that provides a modern REST API and WebSocket interface over the P2P network, featuring an enhanced authentication system with multi-wallet support. + +### Quick Start + +```bash +make run-gateway +# Or manually: +go run ./cmd/gateway +``` + +### Configuration + +The gateway can be configured via environment variables: + +```bash +# Basic Configuration +export GATEWAY_ADDR="0.0.0.0:8080" +export GATEWAY_NAMESPACE="my-app" +export GATEWAY_BOOTSTRAP_PEERS="/ip4/127.0.0.1/tcp/4001/p2p/YOUR_PEER_ID" + +# Authentication Configuration +export GATEWAY_REQUIRE_AUTH=true +export GATEWAY_API_KEYS="key1:namespace1,key2:namespace2" +``` + +### Enhanced Authentication System + +The gateway features a significantly improved authentication system with the following capabilities: + +#### Key Features +- **Automatic Authentication:** No manual auth commands required - authentication happens automatically when needed +- **Multi-Wallet Support:** Seamlessly manage multiple wallet credentials with automatic switching +- **Persistent Sessions:** Wallet credentials are automatically saved and restored +- **Enhanced User Experience:** Streamlined authentication flow with better error handling + +#### Authentication Methods + +**Wallet-Based Authentication (Ethereum EIP-191)** +- Uses `personal_sign` for secure wallet verification +- Supports multiple wallets with automatic detection +- Addresses are case-insensitive with normalized signature handling + +**JWT Tokens** +- Issued by the gateway with configurable expiration +- JWKS endpoints available at `/v1/auth/jwks` and `/.well-known/jwks.json` +- Automatic refresh capability + +**API Keys** +- Support for pre-configured API keys via `Authorization: Bearer ` or `X-API-Key` headers +- Optional namespace mapping for multi-tenant applications + +### API Endpoints + +#### Health & Status +```http +GET /health # Basic health check +GET /v1/health # Detailed health status +GET /v1/status # Network status +GET /v1/version # Version information +``` + +#### Authentication (Public Endpoints) +```http +POST /v1/auth/challenge # Generate wallet challenge +POST /v1/auth/verify # Verify wallet signature +POST /v1/auth/register # Register new wallet +POST /v1/auth/refresh # Refresh JWT token +POST /v1/auth/logout # Clear authentication +GET /v1/auth/whoami # Current auth status +POST /v1/auth/api-key # Generate API key (authenticated) +``` + +#### Storage Operations +```http +POST /v1/storage/get # Retrieve data +POST /v1/storage/put # Store data +POST /v1/storage/delete # Delete data +GET /v1/storage/list # List keys with optional prefix +GET /v1/storage/exists # Check key existence +``` + +#### Network Operations +```http +GET /v1/network/status # Network status +GET /v1/network/peers # Connected peers +POST /v1/network/connect # Connect to peer +POST /v1/network/disconnect # Disconnect from peer +``` + +#### Pub/Sub Messaging + +**WebSocket Interface** +```http +GET /v1/pubsub/ws?topic= # WebSocket connection for real-time messaging +``` + +**REST Interface** +```http +POST /v1/pubsub/publish # Publish message to topic +GET /v1/pubsub/topics # List active topics +``` + +### Security Features + +- **Namespace Enforcement:** All operations are automatically prefixed with namespace for isolation +- **CORS Support:** Configurable CORS policies (permissive for development, configurable for production) +- **Transport Security:** All network communications use Noise/TLS encryption +- **Authentication Middleware:** Flexible authentication with support for multiple credential types + +### Usage Examples + +#### Wallet Authentication Flow +```bash +# 1. Get challenge (automatic) +curl -X POST http://localhost:8080/v1/auth/challenge + +# 2. Sign challenge with wallet (handled by client) +# 3. Verify signature (automatic) +curl -X POST http://localhost:8080/v1/auth/verify \ + -H "Content-Type: application/json" \ + -d '{"wallet":"0x...","nonce":"...","signature":"0x..."}' +``` + +#### Storage Operations +```bash +# Store data +curl -X POST http://localhost:8080/v1/storage/put \ + -H "Authorization: Bearer YOUR_JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"key":"user:123","value":"eyJuYW1lIjoiSm9obiJ9"}' + +# Retrieve data +curl -X POST http://localhost:8080/v1/storage/get \ + -H "Authorization: Bearer YOUR_JWT_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"key":"user:123"}' +``` + +#### Real-time Messaging +```javascript +// WebSocket connection +const ws = new WebSocket('ws://localhost:8080/v1/pubsub/ws?topic=chat'); + +ws.onmessage = (event) => { + console.log('Received:', event.data); +}; + +// Send message +ws.send('Hello, network!'); +``` + --- ## Development + + ### Project Structure @@ -381,6 +563,34 @@ scripts/test-multinode.sh - **Symptoms:** Memory usage grows continuously - **Solutions:** Unsubscribe when done, monitor connection pool, review message retention. +#### Authentication Issues + +- **Symptoms:** `Authentication failed`, `Invalid wallet signature`, `JWT token expired` +- **Solutions:** + - Check wallet signature format (65-byte r||s||v hex) + - Ensure nonce matches exactly during wallet verification + - Verify wallet address case-insensitivity + - Use refresh endpoint or re-authenticate for expired tokens + - Clear credential cache if multi-wallet conflicts occur: `rm -rf ~/.debros/credentials` + +#### Gateway Issues + +- **Symptoms:** `Gateway connection refused`, `CORS errors`, `WebSocket disconnections` +- **Solutions:** + - Verify gateway is running and accessible on configured port + - Check CORS configuration for web applications + - Ensure proper authentication headers for protected endpoints + - Verify namespace configuration and enforcement + +#### Database Migration Issues + +- **Symptoms:** `Migration failed`, `SQL syntax error`, `Version conflict` +- **Solutions:** + - Check SQL syntax in migration files + - Ensure proper statement termination + - Verify migration file naming and sequential order + - Review migration logs for transaction rollbacks + ### Debugging & Health Checks ```bash @@ -390,12 +600,27 @@ export LOG_LEVEL=debug ./bin/network-cli query "SELECT 1" ./bin/network-cli pubsub publish test "hello" ./bin/network-cli pubsub subscribe test 10s + +# Test authentication flow +./bin/network-cli storage put test-key test-value + +# Gateway health checks +curl http://localhost:8080/health +curl http://localhost:8080/v1/status ``` ### Service Logs ```bash +# Node service logs sudo journalctl -u debros-node.service --since "1 hour ago" + +# Gateway service logs (if running as service) +sudo journalctl -u debros-gateway.service --since "1 hour ago" + +# Application logs +tail -f ./logs/gateway.log +tail -f ./logs/node.log ``` ---