From 10a3f3ed3b3fac2068f2d19c3c47ee3953467b65 Mon Sep 17 00:00:00 2001 From: woikos Date: Fri, 23 Jan 2026 07:29:23 +0100 Subject: [PATCH] Add negentropy sync configuration guide - Create comprehensive docs/NEGENTROPY_SYNC_GUIDE.md covering: - Quick start for standalone and peer sync - Split IPC mode configuration - Neo4j backend setup - Docker/Cloudron deployment with supervisord - Environment variables reference - strfry interoperability - CLI sync command usage - Troubleshooting guide - Add Negentropy Sync section to README.md - Update version badge to v0.55.10 Co-Authored-By: Claude Opus 4.5 --- README.md | 49 ++++- docs/NEGENTROPY_SYNC_GUIDE.md | 325 ++++++++++++++++++++++++++++++++++ 2 files changed, 373 insertions(+), 1 deletion(-) create mode 100644 docs/NEGENTROPY_SYNC_GUIDE.md diff --git a/README.md b/README.md index 3b471d6..c6bf071 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ ![orly.dev](./docs/orly.png) -![Version v0.24.1](https://img.shields.io/badge/version-v0.24.1-blue.svg) +![Version v0.55.10](https://img.shields.io/badge/version-v0.55.10-blue.svg) [![Documentation](https://img.shields.io/badge/godoc-documentation-blue.svg)](https://pkg.go.dev/next.orly.dev) [![Support this project](https://img.shields.io/badge/donate-geyser_crowdfunding_project_page-orange.svg)](https://geyser.fund/project/orly) @@ -64,6 +64,7 @@ See [docs/IPC_SYNC_SERVICES.md](./docs/IPC_SYNC_SERVICES.md) for detailed API do - [Follows ACL](#follows-acl) - [Curation ACL](#curation-acl) - [Cluster Replication](#cluster-replication) +- [Negentropy Sync (NIP-77)](#negentropy-sync-nip-77) - [Developer Notes](#developer-notes) ## ⚠️ Bug Reports & Feature Requests @@ -688,6 +689,52 @@ export ORLY_CLUSTER_PROPAGATE_PRIVILEGED_EVENTS=false **Important:** When disabled, privileged events will not be replicated to peer relays. This provides better privacy but means these events will only be available on the originating relay. Users should be aware that accessing their privileged events may require connecting directly to the relay where they were originally published. +## Negentropy Sync (NIP-77) + +ORLY supports [NIP-77](https://github.com/nostr-protocol/nips/blob/master/77.md) negentropy-based set reconciliation for efficient relay synchronization. + +### Quick Start + +Enable negentropy client support: + +```bash +export ORLY_NEGENTROPY_ENABLED=true +./orly +``` + +Enable peer relay synchronization: + +```bash +export ORLY_NEGENTROPY_ENABLED=true +export ORLY_SYNC_NEGENTROPY_PEERS=wss://relay.orly.dev,wss://other-relay.com +./orly +``` + +### Split IPC Mode + +For production deployments, run negentropy as a separate service: + +```bash +# Build binaries +CGO_ENABLED=0 go build -o orly-sync-negentropy ./cmd/orly-sync-negentropy + +# Configure launcher +export ORLY_LAUNCHER_SYNC_NEGENTROPY_ENABLED=true +export ORLY_LAUNCHER_SYNC_NEGENTROPY_BINARY=/path/to/orly-sync-negentropy +export ORLY_SYNC_NEGENTROPY_PEERS=wss://peer-relay.com +``` + +### strfry Compatibility + +ORLY's negentropy implementation is compatible with strfry: + +```bash +# Pull events from ORLY using strfry +strfry sync wss://your-orly-relay.com --filter '{"kinds": [0, 1, 3]}' --dir down +``` + +For detailed configuration including Docker deployments, filtering options, and troubleshooting, see the [Negentropy Sync Guide](docs/NEGENTROPY_SYNC_GUIDE.md). + ## Developer Notes ### Binary-Optimized Tag Storage diff --git a/docs/NEGENTROPY_SYNC_GUIDE.md b/docs/NEGENTROPY_SYNC_GUIDE.md new file mode 100644 index 0000000..734c2f2 --- /dev/null +++ b/docs/NEGENTROPY_SYNC_GUIDE.md @@ -0,0 +1,325 @@ +# Negentropy Sync Configuration Guide + +ORLY implements [NIP-77](https://github.com/nostr-protocol/nips/blob/master/77.md) negentropy-based set reconciliation for efficient relay synchronization. This guide covers configuration for various deployment scenarios. + +## Overview + +Negentropy sync provides: + +- **Efficient synchronization**: Only exchanges events that differ between relays +- **Bandwidth optimization**: Uses set reconciliation instead of transferring full event lists +- **NIP-77 client support**: Clients can sync directly with your relay using the negentropy protocol +- **Relay-to-relay sync**: Automatically sync with configured peer relays + +## Quick Start + +### Standalone Relay (Simplest) + +For a standalone relay that accepts NIP-77 client connections: + +```bash +export ORLY_NEGENTROPY_ENABLED=true +./orly +``` + +This enables the relay to respond to client `NEG-OPEN`, `NEG-MSG`, and `NEG-CLOSE` messages. + +### Relay with Peer Sync + +To sync events with other relays: + +```bash +export ORLY_NEGENTROPY_ENABLED=true +export ORLY_SYNC_NEGENTROPY_PEERS=wss://relay1.example.com,wss://relay2.example.com +./orly +``` + +The relay will periodically sync with each peer relay. + +## Deployment Scenarios + +### 1. Split IPC Mode (Recommended for Production) + +In split mode, negentropy runs as a separate gRPC service managed by `orly-launcher`: + +```bash +# Build binaries +CGO_ENABLED=0 go build -o orly . +CGO_ENABLED=0 go build -o orly-db-badger ./cmd/orly-db-badger +CGO_ENABLED=0 go build -o orly-sync-negentropy ./cmd/orly-sync-negentropy +CGO_ENABLED=0 go build -o orly-launcher ./cmd/orly-launcher +``` + +**systemd service configuration:** + +```ini +[Service] +ExecStart=/path/to/orly-launcher + +# Database service +Environment=ORLY_LAUNCHER_DB_BACKEND=badger +Environment=ORLY_LAUNCHER_DB_BINARY=/path/to/orly-db-badger +Environment=ORLY_LAUNCHER_DB_LISTEN=127.0.0.1:50051 + +# Negentropy service +Environment=ORLY_LAUNCHER_SYNC_NEGENTROPY_ENABLED=true +Environment=ORLY_LAUNCHER_SYNC_NEGENTROPY_BINARY=/path/to/orly-sync-negentropy +Environment=ORLY_LAUNCHER_SYNC_NEGENTROPY_LISTEN=127.0.0.1:50064 +Environment=ORLY_SYNC_NEGENTROPY_PEERS=wss://relay.orly.dev + +# Relay configuration +Environment=ORLY_LAUNCHER_RELAY_BINARY=/path/to/orly +Environment=ORLY_DB_TYPE=grpc +Environment=ORLY_GRPC_SERVER=127.0.0.1:50051 +Environment=ORLY_NEGENTROPY_ENABLED=true +Environment=ORLY_SYNC_TYPE=grpc +Environment=ORLY_GRPC_SYNC_NEGENTROPY=127.0.0.1:50064 +``` + +### 2. Neo4j Backend with Negentropy + +For Neo4j deployments, use `orly-db-neo4j`: + +```bash +CGO_ENABLED=0 go build -o orly-db-neo4j ./cmd/orly-db-neo4j +``` + +**Configuration:** + +```ini +# Database service (Neo4j) +Environment=ORLY_LAUNCHER_DB_BACKEND=neo4j +Environment=ORLY_LAUNCHER_DB_BINARY=/path/to/orly-db-neo4j +Environment=ORLY_LAUNCHER_DB_LISTEN=127.0.0.1:50051 +Environment=ORLY_NEO4J_URI=bolt://localhost:7687 +Environment=ORLY_NEO4J_USER=neo4j +Environment=ORLY_NEO4J_PASSWORD=your_password +Environment=ORLY_DATA_DIR=/var/lib/orly + +# Negentropy and relay (same as Badger) +Environment=ORLY_LAUNCHER_SYNC_NEGENTROPY_ENABLED=true +... +``` + +### 3. Docker/Cloudron Deployment + +For containerized deployments with supervisord: + +**supervisord.conf:** + +```ini +[program:orly-db] +command=/app/code/orly-db-neo4j +environment=ORLY_DB_LISTEN="127.0.0.1:50051",ORLY_DATA_DIR="/app/data/ORLY",ORLY_DB_TYPE="neo4j",ORLY_NEO4J_URI="bolt://127.0.0.1:7687",ORLY_NEO4J_USER="neo4j",ORLY_NEO4J_PASSWORD="%(ENV_ORLY_NEO4J_PASSWORD)s" +priority=15 + +[program:orly] +command=/app/code/orly +environment=ORLY_DB_TYPE="grpc",ORLY_GRPC_SERVER="127.0.0.1:50051",ORLY_NEGENTROPY_ENABLED="true",ORLY_SYNC_TYPE="grpc",ORLY_GRPC_SYNC_NEGENTROPY="127.0.0.1:50064" +priority=20 + +[program:orly-negentropy] +command=/app/code/orly-sync-negentropy +environment=ORLY_LAUNCHER_SYNC_NEGENTROPY_LISTEN="127.0.0.1:50064",ORLY_DB_TYPE="grpc",ORLY_GRPC_SERVER="127.0.0.1:50051",ORLY_SYNC_NEGENTROPY_PEERS="wss://relay.orly.dev" +priority=25 +``` + +## Environment Variables Reference + +### Relay Settings + +| Variable | Default | Description | +|----------|---------|-------------| +| `ORLY_NEGENTROPY_ENABLED` | false | Enable NIP-77 client support | +| `ORLY_SYNC_TYPE` | local | Sync backend: `local` or `grpc` | +| `ORLY_GRPC_SYNC_NEGENTROPY` | | gRPC address of negentropy service | + +### Negentropy Service Settings + +| Variable | Default | Description | +|----------|---------|-------------| +| `ORLY_LAUNCHER_SYNC_NEGENTROPY_ENABLED` | false | Enable negentropy service | +| `ORLY_LAUNCHER_SYNC_NEGENTROPY_BINARY` | | Path to orly-sync-negentropy binary | +| `ORLY_LAUNCHER_SYNC_NEGENTROPY_LISTEN` | 127.0.0.1:50064 | gRPC listen address | +| `ORLY_SYNC_NEGENTROPY_PEERS` | | Comma-separated peer relay URLs | +| `ORLY_SYNC_NEGENTROPY_INTERVAL` | 60s | Background sync interval | +| `ORLY_SYNC_NEGENTROPY_FRAME_SIZE` | 4096 | Negentropy protocol frame size | +| `ORLY_SYNC_NEGENTROPY_ID_SIZE` | 16 | Event ID truncation size | +| `ORLY_SYNC_NEGENTROPY_SESSION_TIMEOUT` | 5m | Client session timeout | + +### Filter Configuration + +You can limit which events are synced using filter environment variables: + +| Variable | Default | Description | +|----------|---------|-------------| +| `ORLY_SYNC_NEGENTROPY_FILTER_KINDS` | | Comma-separated kind numbers to sync | +| `ORLY_SYNC_NEGENTROPY_FILTER_AUTHORS` | | Comma-separated author pubkeys to sync | +| `ORLY_SYNC_NEGENTROPY_FILTER_SINCE` | | Unix timestamp: only sync events after this time | +| `ORLY_SYNC_NEGENTROPY_FILTER_UNTIL` | | Unix timestamp: only sync events before this time | + +**Example: Sync only social events** + +```bash +export ORLY_SYNC_NEGENTROPY_FILTER_KINDS=0,1,3,6,7,1984,10000,30000 +``` + +## strfry Interoperability + +ORLY's negentropy implementation is compatible with strfry. You can sync events between ORLY and strfry relays. + +### Syncing from strfry to ORLY + +Configure ORLY to sync from a strfry relay: + +```bash +export ORLY_SYNC_NEGENTROPY_PEERS=wss://strfry-relay.example.com +``` + +### Using strfry CLI to sync with ORLY + +strfry can use ORLY as a sync target: + +```bash +# Pull events from ORLY to strfry +strfry sync wss://your-orly-relay.com --filter '{"kinds": [0, 1, 3]}' --dir down + +# Push events from strfry to ORLY +strfry sync wss://your-orly-relay.com --filter '{"kinds": [0, 1, 3]}' --dir up + +# Bidirectional sync +strfry sync wss://your-orly-relay.com --filter '{"kinds": [0, 1, 3]}' --dir both +``` + +### ORLY CLI Sync Command + +ORLY includes a strfry-compatible CLI sync command: + +```bash +# Build the sync binary +CGO_ENABLED=0 go build -o orly-sync ./cmd/orly + +# Pull events from another relay +./orly-sync sync wss://relay.example.com --filter '{"kinds": [0, 1, 3]}' --dir down + +# Push events to another relay +./orly-sync sync wss://relay.example.com --filter '{"kinds": [0, 1, 3]}' --dir up + +# Specify local data directory +./orly-sync sync wss://relay.example.com --data-dir /path/to/db --dir down +``` + +## Testing + +### Docker Compose Test Environment + +A test environment for negentropy interop testing is available in `tests/negentropy/`: + +```bash +cd tests/negentropy +docker compose build +docker compose up -d + +# Run sync test +./test-sync.sh + +# Clean up +docker compose down -v +``` + +### Verifying Negentropy Support + +Check if a relay supports NIP-77: + +```bash +curl -s https://your-relay.com | grep -o '"77"' +``` + +Or check the relay info: + +```bash +curl -s -H "Accept: application/nostr+json" https://your-relay.com | jq '.supported_nips' +``` + +## Troubleshooting + +### Common Issues + +**1. "negentropy sync starting" but no events synced** + +- Verify peer relay URL is correct and accessible +- Check if peer relay supports NIP-77 +- Verify network connectivity between relays + +**2. "failed to connect to gRPC server"** + +- Ensure `orly-db` service is running and ready +- Check `ORLY_GRPC_SERVER` matches the database listen address +- Verify firewall allows connections on gRPC ports + +**3. High memory usage during sync** + +- Reduce `ORLY_SYNC_NEGENTROPY_FRAME_SIZE` to 2048 or lower +- Use filters to limit the event set being reconciled +- Increase sync interval to reduce concurrent operations + +### Logging + +Enable debug logging for negentropy: + +```bash +export ORLY_LOG_LEVEL=debug +``` + +View sync-specific logs: + +```bash +journalctl -u orly | grep negentropy +``` + +## Architecture + +``` + ┌──────────────────┐ + │ Peer Relays │ + │ (wss://...) │ + └────────┬─────────┘ + │ + │ NIP-77 WebSocket + │ +┌─────────────────────────────┼─────────────────────────────┐ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────┐ │ +│ │ orly-sync-negentropy │ │ +│ │ (gRPC :50064) │ │ +│ │ │ │ +│ │ • Background sync with peers │ │ +│ │ • Event reconciliation │ │ +│ │ • Push/pull event exchange │ │ +│ └──────────────────────┬──────────────────────────────┘ │ +│ │ │ +│ │ gRPC │ +│ │ │ +│ ┌──────────────────────▼──────────────────────────────┐ │ +│ │ orly-db │ │ +│ │ (gRPC :50051) │ │ +│ │ │ │ +│ │ • Event storage │ │ +│ │ • Query execution │ │ +│ └─────────────────────────────────────────────────────┘ │ +│ │ +│ ORLY Server │ +└────────────────────────────────────────────────────────────┘ + +NIP-77 Client Flow: + Client ─NEG-OPEN──► Relay ──► Negentropy Service + Client ◄──NEG-MSG── Relay ◄── (reconciliation) + Client ─NEG-MSG───► Relay ──► (continue) + Client ◄──EVENT──── Relay ◄── (missing events) +``` + +## See Also + +- [IPC Sync Services Architecture](./IPC_SYNC_SERVICES.md) - Technical details of gRPC services +- [NIP-77 Specification](https://github.com/nostr-protocol/nips/blob/master/77.md) - Protocol specification +- [strfry sync documentation](https://github.com/hoytech/strfry/blob/master/docs/sync.md) - strfry sync reference