mirror of
https://github.com/DeBrosOfficial/network.git
synced 2025-10-06 06:39:07 +00:00
Add database migration and auth system enhancements
The commit adds a new database migration system and improves the authentication flow with multi-wallet support. The migration system provides robust SQL handling and versioning, while the auth system now features automatic wallet detection and credential persistence.
This commit is contained in:
parent
1fca8cb411
commit
2ced1114d9
@ -110,6 +110,7 @@ network/
|
||||
### 4. **Database Layer (`pkg/database/`)**
|
||||
- **RQLite:** Distributed SQLite with Raft consensus, automatic leader election, and failover.
|
||||
- **Client API:** SQL queries, transactions, schema management.
|
||||
- **Migration System:** Robust database migration handling with automatic schema versioning, SQL statement processing, and error recovery. Supports complex SQL files with comments and multiple statements.
|
||||
|
||||
### 5. **Storage System (`pkg/storage/`)**
|
||||
- **Distributed KV:** Namespace-isolated, CRUD operations, prefix queries, replication.
|
||||
@ -192,12 +193,17 @@ make run-gateway
|
||||
# GATEWAY_REQUIRE_AUTH, GATEWAY_API_KEYS
|
||||
```
|
||||
|
||||
- **Auth:** When `RequireAuth` is enabled, endpoints require either:
|
||||
- JWT (issued by this gateway; JWKS: `GET /v1/auth/jwks` or `/.well-known/jwks.json`)
|
||||
- API Key (via `Authorization: Bearer <key>` or `X-API-Key`), optionally mapped to a namespace
|
||||
- Wallet verification uses Ethereum EIP-191 `personal_sign`:
|
||||
- `POST /v1/auth/challenge` returns `{nonce}`. Clients must sign the exact nonce string.
|
||||
- `POST /v1/auth/verify` expects `{wallet, nonce, signature}` with 65-byte r||s||v hex (0x allowed). `v` normalized (27/28 or 0/1). Address match is case-insensitive. Nonce is marked used only after successful verification.
|
||||
- **Enhanced Authentication System:** The gateway features an improved authentication system with automatic wallet detection and multi-wallet support:
|
||||
- **Automatic Authentication:** No manual auth command required - authentication happens automatically when needed
|
||||
- **Multi-Wallet Support:** Users can manage multiple wallet credentials seamlessly
|
||||
- **JWT Authentication:** Issued by this gateway; JWKS available at `GET /v1/auth/jwks` or `/.well-known/jwks.json`
|
||||
- **API Key Support:** Via `Authorization: Bearer <key>` or `X-API-Key`, optionally mapped to a namespace
|
||||
- **Wallet Verification:** Uses Ethereum EIP-191 `personal_sign` with enhanced flow:
|
||||
- `POST /v1/auth/challenge` returns `{nonce}` (public endpoint with internal auth context)
|
||||
- `POST /v1/auth/verify` expects `{wallet, nonce, signature}` with 65-byte r||s||v hex (0x allowed)
|
||||
- `v` normalized (27/28 or 0/1), address match is case-insensitive
|
||||
- Nonce is marked used only after successful verification
|
||||
- Supports automatic wallet switching and credential persistence
|
||||
|
||||
- **Namespace Enforcement:** Storage and PubSub are internally prefixed `ns::<namespace>::...`. Ownership of namespace is enforced by middleware for routes under `/v1/storage*`, `/v1/apps*`, and `/v1/pubsub*`.
|
||||
|
||||
@ -212,13 +218,14 @@ make run-gateway
|
||||
- `GET /v1/auth/jwks`
|
||||
- `GET /.well-known/jwks.json`
|
||||
|
||||
- Auth
|
||||
- `POST /v1/auth/challenge`
|
||||
- `POST /v1/auth/verify`
|
||||
- `POST /v1/auth/register`
|
||||
- `POST /v1/auth/refresh`
|
||||
- `POST /v1/auth/logout`
|
||||
- `GET /v1/auth/whoami`
|
||||
- Auth (Enhanced Multi-Wallet System)
|
||||
- `POST /v1/auth/challenge` (public endpoint, generates wallet challenge)
|
||||
- `POST /v1/auth/verify` (public endpoint, verifies wallet signature)
|
||||
- `POST /v1/auth/register` (public endpoint, wallet registration)
|
||||
- `POST /v1/auth/refresh` (public endpoint, token refresh)
|
||||
- `POST /v1/auth/logout` (clears authentication state)
|
||||
- `GET /v1/auth/whoami` (returns current auth status)
|
||||
- `POST /v1/auth/api-key` (generates API keys for authenticated users)
|
||||
|
||||
- Storage
|
||||
- `POST /v1/storage/get`, `POST /v1/storage/put`, `POST /v1/storage/delete`
|
||||
@ -239,7 +246,40 @@ make run-gateway
|
||||
- `POST /v1/pubsub/publish` → body `{topic, data_base64}` → `{status:"ok"}`
|
||||
- `GET /v1/pubsub/topics` → `{topics:["<topic>", ...]}` (names trimmed to caller namespace)
|
||||
|
||||
Security note: CORS and WS origin checks are permissive for development; harden for production.
|
||||
### Authentication Improvements
|
||||
|
||||
The gateway authentication system has been significantly enhanced with the following features:
|
||||
|
||||
- **Database Migration System:** Robust SQL migration handling with proper statement processing and error handling
|
||||
- **Automatic Wallet Detection:** CLI automatically detects and manages wallet credentials without manual auth commands
|
||||
- **Multi-Wallet Management:** Support for multiple wallet credentials with automatic switching
|
||||
- **Enhanced User Experience:** Streamlined authentication flow with credential persistence
|
||||
- **Internal Auth Context:** Public authentication endpoints use internal auth context to prevent circular dependencies
|
||||
- **Improved Error Handling:** Better error messages and debugging information for authentication issues
|
||||
|
||||
Security note: CORS and WS origin checks are permissive for development; harden for production. The enhanced authentication system maintains security while improving accessibility and user experience.
|
||||
|
||||
### Database Migration System
|
||||
|
||||
The gateway includes an enhanced database migration system with the following features:
|
||||
|
||||
- **Automatic Schema Management:** Database schema is automatically initialized and updated using migration files
|
||||
- **Robust SQL Processing:** Handles complex SQL files with comments, multiple statements, and proper statement termination
|
||||
- **Version Control:** Tracks migration versions to prevent duplicate execution and ensure proper upgrade paths
|
||||
- **Error Recovery:** Comprehensive error handling with detailed logging for debugging migration issues
|
||||
- **Transaction Safety:** Each migration runs in a transaction to ensure atomicity and data integrity
|
||||
- **SQL File Support:** Supports standard SQL migration files with `.sql` extension in the `migrations/` directory
|
||||
|
||||
**Migration File Structure:**
|
||||
```
|
||||
migrations/
|
||||
├── 001_initial_schema.sql # Initial database setup
|
||||
├── 002_add_auth_tables.sql # Authentication tables
|
||||
├── 003_add_indexes.sql # Performance indexes
|
||||
└── ... # Additional migrations
|
||||
```
|
||||
|
||||
The migration system automatically processes SQL statements, handles comments, and ensures proper execution order during gateway startup.
|
||||
|
||||
---
|
||||
|
||||
@ -313,10 +353,24 @@ peers, err := client.Network().GetPeers(ctx)
|
||||
- **Message Delivery Failures:** Verify topic names, subscription status, and network connectivity.
|
||||
- **High Memory Usage:** Unsubscribe from topics when done, monitor connection pool size.
|
||||
|
||||
### **Authentication Issues**
|
||||
- **Wallet Connection Failed:** Check wallet signature format (65-byte r||s||v hex), ensure nonce matches exactly, verify wallet address case-insensitivity.
|
||||
- **JWT Token Expired:** Use refresh endpoint or re-authenticate with wallet.
|
||||
- **API Key Invalid:** Verify key format and namespace mapping in gateway configuration.
|
||||
- **Multi-Wallet Conflicts:** Clear credential cache and re-authenticate: `rm -rf ~/.debros/credentials`
|
||||
- **Circular Auth Dependencies:** Ensure public auth endpoints use internal auth context.
|
||||
|
||||
### **Database Migration Issues**
|
||||
- **Migration Failures:** Check SQL syntax, ensure proper statement termination, review migration logs.
|
||||
- **Version Conflicts:** Verify migration file naming and sequential order.
|
||||
- **Incomplete Migrations:** Check for transaction rollbacks and database locks.
|
||||
|
||||
### **Debugging**
|
||||
- Enable debug logging: `export LOG_LEVEL=debug`
|
||||
- Check service logs: `sudo journalctl -u debros-node.service -f`
|
||||
- Use CLI for health and peer checks: `./bin/network-cli health`, `./bin/network-cli peers`
|
||||
- Check gateway logs: `sudo journalctl -u debros-gateway.service -f`
|
||||
- Test authentication flow: `./bin/network-cli storage put test-key test-value`
|
||||
|
||||
---
|
||||
|
||||
|
||||
225
README.md
225
README.md
@ -317,9 +317,191 @@ logging:
|
||||
--disable-anonrc # Disable anonymous routing (Tor/SOCKS5)
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
The CLI features an enhanced authentication system with automatic wallet detection and multi-wallet support:
|
||||
|
||||
- **Automatic Authentication:** No manual auth commands required - authentication happens automatically when operations need credentials
|
||||
- **Multi-Wallet Management:** Seamlessly switch between multiple wallet credentials
|
||||
- **Persistent Sessions:** Wallet credentials are automatically saved and restored between sessions
|
||||
- **Enhanced User Experience:** Streamlined authentication flow with better error handling and user feedback
|
||||
|
||||
When using operations that require authentication (storage, database, pubsub), the CLI will automatically:
|
||||
1. Check for existing valid credentials
|
||||
2. Prompt for wallet authentication if needed
|
||||
3. Handle signature verification
|
||||
4. Persist credentials for future use
|
||||
|
||||
**Example with automatic authentication:**
|
||||
```bash
|
||||
# First time - will prompt for wallet authentication
|
||||
./bin/network-cli storage put user:123 "John Doe"
|
||||
|
||||
# Subsequent calls - uses saved credentials automatically
|
||||
./bin/network-cli storage get user:123
|
||||
./bin/network-cli pubsub publish notifications "Hello World"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## HTTP Gateway
|
||||
|
||||
The DeBros Network includes a powerful HTTP/WebSocket gateway that provides a modern REST API and WebSocket interface over the P2P network, featuring an enhanced authentication system with multi-wallet support.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
make run-gateway
|
||||
# Or manually:
|
||||
go run ./cmd/gateway
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
The gateway can be configured via environment variables:
|
||||
|
||||
```bash
|
||||
# Basic Configuration
|
||||
export GATEWAY_ADDR="0.0.0.0:8080"
|
||||
export GATEWAY_NAMESPACE="my-app"
|
||||
export GATEWAY_BOOTSTRAP_PEERS="/ip4/127.0.0.1/tcp/4001/p2p/YOUR_PEER_ID"
|
||||
|
||||
# Authentication Configuration
|
||||
export GATEWAY_REQUIRE_AUTH=true
|
||||
export GATEWAY_API_KEYS="key1:namespace1,key2:namespace2"
|
||||
```
|
||||
|
||||
### Enhanced Authentication System
|
||||
|
||||
The gateway features a significantly improved authentication system with the following capabilities:
|
||||
|
||||
#### Key Features
|
||||
- **Automatic Authentication:** No manual auth commands required - authentication happens automatically when needed
|
||||
- **Multi-Wallet Support:** Seamlessly manage multiple wallet credentials with automatic switching
|
||||
- **Persistent Sessions:** Wallet credentials are automatically saved and restored
|
||||
- **Enhanced User Experience:** Streamlined authentication flow with better error handling
|
||||
|
||||
#### Authentication Methods
|
||||
|
||||
**Wallet-Based Authentication (Ethereum EIP-191)**
|
||||
- Uses `personal_sign` for secure wallet verification
|
||||
- Supports multiple wallets with automatic detection
|
||||
- Addresses are case-insensitive with normalized signature handling
|
||||
|
||||
**JWT Tokens**
|
||||
- Issued by the gateway with configurable expiration
|
||||
- JWKS endpoints available at `/v1/auth/jwks` and `/.well-known/jwks.json`
|
||||
- Automatic refresh capability
|
||||
|
||||
**API Keys**
|
||||
- Support for pre-configured API keys via `Authorization: Bearer <key>` or `X-API-Key` headers
|
||||
- Optional namespace mapping for multi-tenant applications
|
||||
|
||||
### API Endpoints
|
||||
|
||||
#### Health & Status
|
||||
```http
|
||||
GET /health # Basic health check
|
||||
GET /v1/health # Detailed health status
|
||||
GET /v1/status # Network status
|
||||
GET /v1/version # Version information
|
||||
```
|
||||
|
||||
#### Authentication (Public Endpoints)
|
||||
```http
|
||||
POST /v1/auth/challenge # Generate wallet challenge
|
||||
POST /v1/auth/verify # Verify wallet signature
|
||||
POST /v1/auth/register # Register new wallet
|
||||
POST /v1/auth/refresh # Refresh JWT token
|
||||
POST /v1/auth/logout # Clear authentication
|
||||
GET /v1/auth/whoami # Current auth status
|
||||
POST /v1/auth/api-key # Generate API key (authenticated)
|
||||
```
|
||||
|
||||
#### Storage Operations
|
||||
```http
|
||||
POST /v1/storage/get # Retrieve data
|
||||
POST /v1/storage/put # Store data
|
||||
POST /v1/storage/delete # Delete data
|
||||
GET /v1/storage/list # List keys with optional prefix
|
||||
GET /v1/storage/exists # Check key existence
|
||||
```
|
||||
|
||||
#### Network Operations
|
||||
```http
|
||||
GET /v1/network/status # Network status
|
||||
GET /v1/network/peers # Connected peers
|
||||
POST /v1/network/connect # Connect to peer
|
||||
POST /v1/network/disconnect # Disconnect from peer
|
||||
```
|
||||
|
||||
#### Pub/Sub Messaging
|
||||
|
||||
**WebSocket Interface**
|
||||
```http
|
||||
GET /v1/pubsub/ws?topic=<topic> # WebSocket connection for real-time messaging
|
||||
```
|
||||
|
||||
**REST Interface**
|
||||
```http
|
||||
POST /v1/pubsub/publish # Publish message to topic
|
||||
GET /v1/pubsub/topics # List active topics
|
||||
```
|
||||
|
||||
### Security Features
|
||||
|
||||
- **Namespace Enforcement:** All operations are automatically prefixed with namespace for isolation
|
||||
- **CORS Support:** Configurable CORS policies (permissive for development, configurable for production)
|
||||
- **Transport Security:** All network communications use Noise/TLS encryption
|
||||
- **Authentication Middleware:** Flexible authentication with support for multiple credential types
|
||||
|
||||
### Usage Examples
|
||||
|
||||
#### Wallet Authentication Flow
|
||||
```bash
|
||||
# 1. Get challenge (automatic)
|
||||
curl -X POST http://localhost:8080/v1/auth/challenge
|
||||
|
||||
# 2. Sign challenge with wallet (handled by client)
|
||||
# 3. Verify signature (automatic)
|
||||
curl -X POST http://localhost:8080/v1/auth/verify \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"wallet":"0x...","nonce":"...","signature":"0x..."}'
|
||||
```
|
||||
|
||||
#### Storage Operations
|
||||
```bash
|
||||
# Store data
|
||||
curl -X POST http://localhost:8080/v1/storage/put \
|
||||
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"key":"user:123","value":"eyJuYW1lIjoiSm9obiJ9"}'
|
||||
|
||||
# Retrieve data
|
||||
curl -X POST http://localhost:8080/v1/storage/get \
|
||||
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"key":"user:123"}'
|
||||
```
|
||||
|
||||
#### Real-time Messaging
|
||||
```javascript
|
||||
// WebSocket connection
|
||||
const ws = new WebSocket('ws://localhost:8080/v1/pubsub/ws?topic=chat');
|
||||
|
||||
ws.onmessage = (event) => {
|
||||
console.log('Received:', event.data);
|
||||
};
|
||||
|
||||
// Send message
|
||||
ws.send('Hello, network!');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
</text>
|
||||
|
||||
|
||||
### Project Structure
|
||||
|
||||
@ -381,6 +563,34 @@ scripts/test-multinode.sh
|
||||
- **Symptoms:** Memory usage grows continuously
|
||||
- **Solutions:** Unsubscribe when done, monitor connection pool, review message retention.
|
||||
|
||||
#### Authentication Issues
|
||||
|
||||
- **Symptoms:** `Authentication failed`, `Invalid wallet signature`, `JWT token expired`
|
||||
- **Solutions:**
|
||||
- Check wallet signature format (65-byte r||s||v hex)
|
||||
- Ensure nonce matches exactly during wallet verification
|
||||
- Verify wallet address case-insensitivity
|
||||
- Use refresh endpoint or re-authenticate for expired tokens
|
||||
- Clear credential cache if multi-wallet conflicts occur: `rm -rf ~/.debros/credentials`
|
||||
|
||||
#### Gateway Issues
|
||||
|
||||
- **Symptoms:** `Gateway connection refused`, `CORS errors`, `WebSocket disconnections`
|
||||
- **Solutions:**
|
||||
- Verify gateway is running and accessible on configured port
|
||||
- Check CORS configuration for web applications
|
||||
- Ensure proper authentication headers for protected endpoints
|
||||
- Verify namespace configuration and enforcement
|
||||
|
||||
#### Database Migration Issues
|
||||
|
||||
- **Symptoms:** `Migration failed`, `SQL syntax error`, `Version conflict`
|
||||
- **Solutions:**
|
||||
- Check SQL syntax in migration files
|
||||
- Ensure proper statement termination
|
||||
- Verify migration file naming and sequential order
|
||||
- Review migration logs for transaction rollbacks
|
||||
|
||||
### Debugging & Health Checks
|
||||
|
||||
```bash
|
||||
@ -390,12 +600,27 @@ export LOG_LEVEL=debug
|
||||
./bin/network-cli query "SELECT 1"
|
||||
./bin/network-cli pubsub publish test "hello"
|
||||
./bin/network-cli pubsub subscribe test 10s
|
||||
|
||||
# Test authentication flow
|
||||
./bin/network-cli storage put test-key test-value
|
||||
|
||||
# Gateway health checks
|
||||
curl http://localhost:8080/health
|
||||
curl http://localhost:8080/v1/status
|
||||
```
|
||||
|
||||
### Service Logs
|
||||
|
||||
```bash
|
||||
# Node service logs
|
||||
sudo journalctl -u debros-node.service --since "1 hour ago"
|
||||
|
||||
# Gateway service logs (if running as service)
|
||||
sudo journalctl -u debros-gateway.service --since "1 hour ago"
|
||||
|
||||
# Application logs
|
||||
tail -f ./logs/gateway.log
|
||||
tail -f ./logs/node.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user