This guide covers migrating your ORLY relay database when changing Badger configuration parameters, specifically for the VLogPercentile and table size optimizations.
## When Migration is Needed
Based on research of Badger v4 source code and documentation:
### Configuration Changes That DON'T Require Migration
The following options can be changed **without migration**:
- `Compression` - New data uses new compression, old data remains as-is
- `BlockSize` - Explicitly stated in Badger source: "Changing BlockSize across DB runs will not break badger"
### Configuration Changes That BENEFIT from Migration
The following options apply to **new writes only** - existing data gradually adopts new settings through compaction:
- `VLogPercentile` - Affects where **new** values are stored (LSM vs vlog)
- `BaseTableSize` - **New** SST files use new size
- `MemTableSize` - Affects new write buffering
- `BaseLevelSize` - Affects new LSM tree structure
- `ValueLogFileSize` - New vlog files use new size
**Migration Impact:** Without migration, existing data remains in its current location (LSM tree or value log). The database will **gradually** adapt through normal compaction, which may take days or weeks depending on write volume.
## Migration Options
### Option 1: No Migration (Let Natural Compaction Handle It)
@ -8,7 +8,7 @@ ORLY is a high-performance Nostr relay written in Go, designed for personal rela
@@ -8,7 +8,7 @@ ORLY is a high-performance Nostr relay written in Go, designed for personal rela
This document tracks the implementation of Dgraph as an alternative database backend for ORLY. The implementation allows switching between Badger (default) and Dgraph via the `ORLY_DB_TYPE` environment variable.
The implementation uses a **client-server architecture** with dual storage:
1. **Dgraph Client** (ORLY)
- Connects to external dgraph via gRPC (default: localhost:9080)
- Applies Nostr schema automatically on startup
- Query/Mutate methods ready for DQL operations
2. **Dgraph Server** (External)
- Run separately via docker or standalone binary
- Stores event graph data (events, authors, tags, relationships)
- Handles all graph queries and mutations
3. **Badger Metadata Store** (Local)
- Stores markers, counters, relay identity
- Provides fast key-value access for non-graph data
- Complements dgraph for hybrid storage benefits
The abstraction layer is complete and the dgraph client integration is functional. Next step is implementing actual DQL query/mutation logic in save-event.go and query-events.go.
Successfully migrated the ORLY relay codebase to use the external `git.mleku.dev/mleku/nostr` library instead of maintaining duplicate protocol code internally.
## Migration Statistics
- **Files Changed**: 449
- **Lines Added**: 624
- **Lines Removed**: 65,132
- **Net Reduction**: **64,508 lines of code** (~30-40% of the codebase)
## Packages Migrated
### Removed from next.orly.dev/pkg/
The following packages were completely removed as they now come from the nostr library:
| khatru-sqlite | 8002 | Khatru with SQLite backend |
| khatru-badger | 8003 | Khatru with Badger backend |
@ -180,7 +177,7 @@ go build -o benchmark main.go
@@ -180,7 +177,7 @@ go build -o benchmark main.go
## Database Backend Comparison
The benchmark suite includes **next.orly.dev** with three different database backends to compare architectural approaches:
The benchmark suite includes **next.orly.dev** with two different database backends to compare architectural approaches:
### Badger Backend (next-orly-badger)
- **Type**: Embedded key-value store
@ -192,16 +189,6 @@ The benchmark suite includes **next.orly.dev** with three different database bac
@@ -192,16 +189,6 @@ The benchmark suite includes **next.orly.dev** with three different database bac
- Simpler deployment
- Limited to single-node scaling
### DGraph Backend (next-orly-dgraph)
- **Type**: Distributed graph database
- **Architecture**: Client-server with dgraph-zero (coordinator) and dgraph-alpha (data node)
- **Architecture**: Client-server with Neo4j Community Edition
@ -218,10 +205,10 @@ The benchmark suite includes **next.orly.dev** with three different database bac
@@ -218,10 +205,10 @@ The benchmark suite includes **next.orly.dev** with three different database bac
### Comparing the Backends
The benchmark results will show:
- **Latency differences**: Embedded vs. distributed overhead, graph traversal efficiency
- **Throughput trade-offs**: Single-process optimization vs. distributed scalability vs. graph query optimization
- **Latency differences**: Embedded vs. client-server overhead, graph traversal efficiency
- **Throughput trade-offs**: Single-process optimization vs. graph query optimization
- **Resource usage**: Memory and CPU patterns for different architectures
- **Query performance**: Graph queries (Neo4j) vs. key-value lookups (Badger) vs. distributed queries (DGraph)
- **Query performance**: Graph queries (Neo4j) vs. key-value lookups (Badger)
This comparison helps determine which backend is appropriate for different deployment scenarios and workload patterns.
@ -4,20 +4,6 @@ This directory contains automation scripts for building, testing, and deploying
@@ -4,20 +4,6 @@ This directory contains automation scripts for building, testing, and deploying
## Quick Reference
### Dgraph Integration Testing
```bash
# Local testing (requires dgraph server)
./dgraph-start.sh # Start dgraph server
./test-dgraph.sh # Run dgraph package tests
./test-dgraph.sh --relay-tester # Run tests + relay-tester
# Docker testing (containers for everything)
./docker-build.sh # Build ORLY docker image
./test-docker.sh # Run integration tests in containers
./test-docker.sh --relay-tester --keep-running # Full test, keep running
```
### Build & Deploy
```bash
@ -26,44 +12,14 @@ This directory contains automation scripts for building, testing, and deploying
@@ -26,44 +12,14 @@ This directory contains automation scripts for building, testing, and deploying
./update-embedded-web.sh # Build and embed web UI
```
## Script Descriptions
### Dgraph Testing Scripts
#### dgraph-start.sh
Starts dgraph server using docker-compose for local testing.
### Testing
**Usage:**
```bash
./dgraph-start.sh
```
**What it does:**
- Checks if dgraph is already running
- Starts dgraph via docker-compose
- Waits for health check
- Shows endpoints and commands
#### dgraph-docker-compose.yml
Docker Compose configuration for standalone dgraph server.
**Ports:**
- 8080: HTTP API
- 9080: gRPC (ORLY connects here)
- 8000: Ratel UI
#### test-dgraph.sh
Runs dgraph package tests against a running dgraph server.