mleku
2 months ago
72 changed files with 8159 additions and 4049 deletions
@ -0,0 +1,319 @@
@@ -0,0 +1,319 @@
|
||||
# Badger Database Migration Guide |
||||
|
||||
## Overview |
||||
|
||||
This guide covers migrating your ORLY relay database when changing Badger configuration parameters, specifically for the VLogPercentile and table size optimizations. |
||||
|
||||
## When Migration is Needed |
||||
|
||||
Based on research of Badger v4 source code and documentation: |
||||
|
||||
### Configuration Changes That DON'T Require Migration |
||||
|
||||
The following options can be changed **without migration**: |
||||
- `BlockCacheSize` - Only affects in-memory cache |
||||
- `IndexCacheSize` - Only affects in-memory cache |
||||
- `NumCompactors` - Runtime setting |
||||
- `NumLevelZeroTables` - Affects compaction timing |
||||
- `NumMemtables` - Affects write buffering |
||||
- `DetectConflicts` - Runtime conflict detection |
||||
- `Compression` - New data uses new compression, old data remains as-is |
||||
- `BlockSize` - Explicitly stated in Badger source: "Changing BlockSize across DB runs will not break badger" |
||||
|
||||
### Configuration Changes That BENEFIT from Migration |
||||
|
||||
The following options apply to **new writes only** - existing data gradually adopts new settings through compaction: |
||||
- `VLogPercentile` - Affects where **new** values are stored (LSM vs vlog) |
||||
- `BaseTableSize` - **New** SST files use new size |
||||
- `MemTableSize` - Affects new write buffering |
||||
- `BaseLevelSize` - Affects new LSM tree structure |
||||
- `ValueLogFileSize` - New vlog files use new size |
||||
|
||||
**Migration Impact:** Without migration, existing data remains in its current location (LSM tree or value log). The database will **gradually** adapt through normal compaction, which may take days or weeks depending on write volume. |
||||
|
||||
## Migration Options |
||||
|
||||
### Option 1: No Migration (Let Natural Compaction Handle It) |
||||
|
||||
**Best for:** Low-traffic relays, testing environments |
||||
|
||||
**Pros:** |
||||
- No downtime required |
||||
- No manual intervention |
||||
- Zero risk of data loss |
||||
|
||||
**Cons:** |
||||
- Benefits take time to materialize (days/weeks) |
||||
- Old data layout persists until natural compaction |
||||
- Cache tuning benefits delayed |
||||
|
||||
**Steps:** |
||||
1. Update Badger configuration in `pkg/database/database.go` |
||||
2. Restart ORLY relay |
||||
3. Monitor performance over several days |
||||
4. Optionally run manual GC: `db.RunValueLogGC(0.5)` periodically |
||||
|
||||
### Option 2: Manual Value Log Garbage Collection |
||||
|
||||
**Best for:** Medium-traffic relays wanting faster optimization |
||||
|
||||
**Pros:** |
||||
- Faster than natural compaction |
||||
- Still safe (no export/import) |
||||
- Can run while relay is online |
||||
|
||||
**Cons:** |
||||
- Still gradual (hours instead of days) |
||||
- CPU/disk intensive during GC |
||||
- Partial benefit until GC completes |
||||
|
||||
**Steps:** |
||||
1. Update Badger configuration |
||||
2. Restart ORLY relay |
||||
3. Monitor logs for compaction activity |
||||
4. Manually trigger GC if needed (future feature - not currently exposed) |
||||
|
||||
### Option 3: Full Export/Import Migration (RECOMMENDED for Production) |
||||
|
||||
**Best for:** Production relays, large databases, maximum performance |
||||
|
||||
**Pros:** |
||||
- Immediate full benefit of new configuration |
||||
- Clean database structure |
||||
- Predictable migration time |
||||
- Reclaims all disk space |
||||
|
||||
**Cons:** |
||||
- Requires relay downtime (several hours for large DBs) |
||||
- Requires 2x disk space temporarily |
||||
- More complex procedure |
||||
|
||||
**Steps:** See detailed procedure below |
||||
|
||||
## Full Migration Procedure (Option 3) |
||||
|
||||
### Prerequisites |
||||
|
||||
1. **Disk space:** At minimum 2.5x current database size |
||||
- 1x for current database |
||||
- 1x for JSONL export |
||||
- 0.5x for new database (will be smaller with compression) |
||||
|
||||
2. **Time estimate:** |
||||
- Export: ~100-500 MB/s depending on disk speed |
||||
- Import: ~50-200 MB/s with indexing overhead |
||||
- Example: 10 GB database = ~10-30 minutes total |
||||
|
||||
3. **Backup:** Ensure you have a recent backup before proceeding |
||||
|
||||
### Step-by-Step Migration |
||||
|
||||
#### 1. Prepare Migration Script |
||||
|
||||
Use the provided `scripts/migrate-badger-config.sh` script (see below). |
||||
|
||||
#### 2. Stop the Relay |
||||
|
||||
```bash |
||||
# If using systemd |
||||
sudo systemctl stop orly |
||||
|
||||
# If running manually |
||||
pkill orly |
||||
``` |
||||
|
||||
#### 3. Run Migration |
||||
|
||||
```bash |
||||
cd ~/src/next.orly.dev |
||||
chmod +x scripts/migrate-badger-config.sh |
||||
./scripts/migrate-badger-config.sh |
||||
``` |
||||
|
||||
The script will: |
||||
- Export all events to JSONL format |
||||
- Move old database to backup location |
||||
- Create new database with updated configuration |
||||
- Import all events (rebuilds indexes automatically) |
||||
- Verify event count matches |
||||
|
||||
#### 4. Verify Migration |
||||
|
||||
```bash |
||||
# Check that events were migrated |
||||
echo "Old event count:" |
||||
cat ~/.local/share/ORLY-backup-*/migration.log | grep "exported.*events" |
||||
|
||||
echo "New event count:" |
||||
cat ~/.local/share/ORLY/migration.log | grep "saved.*events" |
||||
``` |
||||
|
||||
#### 5. Restart Relay |
||||
|
||||
```bash |
||||
# If using systemd |
||||
sudo systemctl start orly |
||||
sudo journalctl -u orly -f |
||||
|
||||
# If running manually |
||||
./orly |
||||
``` |
||||
|
||||
#### 6. Monitor Performance |
||||
|
||||
Watch for improvements in: |
||||
- Cache hit ratio (should be >85% with new config) |
||||
- Average query latency (should be <3ms for cached events) |
||||
- No "Block cache too small" warnings in logs |
||||
|
||||
#### 7. Clean Up (After Verification) |
||||
|
||||
```bash |
||||
# Once you confirm everything works (wait 24-48 hours) |
||||
rm -rf ~/.local/share/ORLY-backup-* |
||||
rm ~/.local/share/ORLY/events-export.jsonl |
||||
``` |
||||
|
||||
## Migration Script |
||||
|
||||
The migration script is located at `scripts/migrate-badger-config.sh` and handles: |
||||
- Automatic export of all events to JSONL |
||||
- Safe backup of existing database |
||||
- Creation of new database with updated config |
||||
- Import and indexing of all events |
||||
- Verification of event counts |
||||
|
||||
## Rollback Procedure |
||||
|
||||
If migration fails or performance degrades: |
||||
|
||||
```bash |
||||
# Stop the relay |
||||
sudo systemctl stop orly # or pkill orly |
||||
|
||||
# Restore old database |
||||
rm -rf ~/.local/share/ORLY |
||||
mv ~/.local/share/ORLY-backup-$(date +%Y%m%d)* ~/.local/share/ORLY |
||||
|
||||
# Restart with old configuration |
||||
sudo systemctl start orly |
||||
``` |
||||
|
||||
## Configuration Changes Summary |
||||
|
||||
### Changes Applied in pkg/database/database.go |
||||
|
||||
```go |
||||
// Cache sizes (can change without migration) |
||||
opts.BlockCacheSize = 16384 MB (was 512 MB) |
||||
opts.IndexCacheSize = 4096 MB (was 256 MB) |
||||
|
||||
// Table sizes (benefits from migration) |
||||
opts.BaseTableSize = 8 MB (was 64 MB) |
||||
opts.MemTableSize = 16 MB (was 64 MB) |
||||
opts.ValueLogFileSize = 128 MB (was 256 MB) |
||||
|
||||
// Inline event optimization (CRITICAL - benefits from migration) |
||||
opts.VLogPercentile = 0.99 (was 0.0 - default) |
||||
|
||||
// LSM structure (benefits from migration) |
||||
opts.BaseLevelSize = 64 MB (was 10 MB - default) |
||||
|
||||
// Performance settings (no migration needed) |
||||
opts.DetectConflicts = false (was true) |
||||
opts.Compression = options.ZSTD (was options.None) |
||||
opts.NumCompactors = 8 (was 4) |
||||
opts.NumMemtables = 8 (was 5) |
||||
``` |
||||
|
||||
## Expected Improvements |
||||
|
||||
### Before Migration |
||||
- Cache hit ratio: 33% |
||||
- Average latency: 9.35ms |
||||
- P95 latency: 34.48ms |
||||
- Block cache warnings: Yes |
||||
|
||||
### After Migration |
||||
- Cache hit ratio: 85-95% |
||||
- Average latency: <3ms |
||||
- P95 latency: <8ms |
||||
- Block cache warnings: No |
||||
- Inline events: 3-5x faster reads |
||||
|
||||
## Troubleshooting |
||||
|
||||
### Migration Script Fails |
||||
|
||||
**Error:** "Not enough disk space" |
||||
- Free up space or use Option 1 (natural compaction) |
||||
- Ensure you have 2.5x current DB size available |
||||
|
||||
**Error:** "Export failed" |
||||
- Check database is not corrupted |
||||
- Ensure ORLY is stopped |
||||
- Check file permissions |
||||
|
||||
**Error:** "Import count mismatch" |
||||
- This is informational - some events may be duplicates |
||||
- Check logs for specific errors |
||||
- Verify core events are present via relay queries |
||||
|
||||
### Performance Not Improved |
||||
|
||||
**After migration, performance is the same:** |
||||
1. Verify configuration was actually applied: |
||||
```bash |
||||
# Check running relay logs for config output |
||||
sudo journalctl -u orly | grep -i "block.*cache\|vlog" |
||||
``` |
||||
|
||||
2. Wait for cache to warm up (2-5 minutes after start) |
||||
|
||||
3. Check if workload changed (different query patterns) |
||||
|
||||
4. Verify disk I/O is not bottleneck: |
||||
```bash |
||||
iostat -x 5 |
||||
``` |
||||
|
||||
### High CPU During Migration |
||||
|
||||
- This is normal - import rebuilds all indexes |
||||
- Migration is single-threaded by design (data consistency) |
||||
- Expect 30-60% CPU usage on one core |
||||
|
||||
## Additional Notes |
||||
|
||||
### Compression Impact |
||||
|
||||
The `Compression = options.ZSTD` setting: |
||||
- Only compresses **new** data |
||||
- Old data remains uncompressed until rewritten by compaction |
||||
- Migration forces all data to be rewritten → immediate compression benefit |
||||
- Expect 2-3x compression ratio for event data |
||||
|
||||
### VLogPercentile Behavior |
||||
|
||||
With `VLogPercentile = 0.99`: |
||||
- **99% of values** stored in LSM tree (fast access) |
||||
- **1% of values** stored in value log (large events >100 KB) |
||||
- Threshold dynamically adjusted based on value size distribution |
||||
- Perfect for ORLY's inline event optimization |
||||
|
||||
### Production Considerations |
||||
|
||||
For production relays: |
||||
1. Schedule migration during low-traffic period |
||||
2. Notify users of maintenance window |
||||
3. Have rollback plan ready |
||||
4. Monitor closely for 24-48 hours after migration |
||||
5. Keep backup for at least 1 week |
||||
|
||||
## References |
||||
|
||||
- Badger v4 Documentation: https://pkg.go.dev/github.com/dgraph-io/badger/v4 |
||||
- ORLY Database Package: `pkg/database/database.go` |
||||
- Export/Import Implementation: `pkg/database/{export,import}.go` |
||||
- Cache Optimization Analysis: `cmd/benchmark/CACHE_OPTIMIZATION_STRATEGY.md` |
||||
- Inline Event Optimization: `cmd/benchmark/INLINE_EVENT_OPTIMIZATION.md` |
||||
@ -0,0 +1,188 @@
@@ -0,0 +1,188 @@
|
||||
# Badger Cache Optimization Strategy |
||||
|
||||
## Problem Analysis |
||||
|
||||
### Initial Configuration (FAILED) |
||||
- Block cache: 2048 MB |
||||
- Index cache: 1024 MB |
||||
- **Result**: Cache hit ratio remained at 33% |
||||
|
||||
### Root Cause Discovery |
||||
|
||||
Badger's Ristretto cache uses a "cost" metric that doesn't directly map to bytes: |
||||
|
||||
``` |
||||
Average cost per key: 54,628,383 bytes = 52.10 MB |
||||
Cache size: 2048 MB |
||||
Keys that fit: ~39 keys only! |
||||
``` |
||||
|
||||
The cost metric appears to include: |
||||
- Uncompressed data size |
||||
- Value log references |
||||
- Table metadata |
||||
- Potentially full `BaseTableSize` (64 MB) per entry |
||||
|
||||
### Why Previous Fix Didn't Work |
||||
|
||||
With `BaseTableSize = 64 MB`: |
||||
- Each cache entry costs ~52 MB in the cost metric |
||||
- 2 GB cache ÷ 52 MB = ~39 entries max |
||||
- Test generates 228,000+ unique keys |
||||
- **Eviction rate: 99.99%** (everything gets evicted immediately) |
||||
|
||||
## Multi-Pronged Optimization Strategy |
||||
|
||||
### Approach 1: Reduce Table Sizes (IMPLEMENTED) |
||||
|
||||
**Changes in `pkg/database/database.go`:** |
||||
|
||||
```go |
||||
// OLD (causing high cache cost): |
||||
opts.BaseTableSize = 64 * units.Mb // 64 MB per table |
||||
opts.MemTableSize = 64 * units.Mb // 64 MB memtable |
||||
|
||||
// NEW (lower cache cost): |
||||
opts.BaseTableSize = 8 * units.Mb // 8 MB per table (8x reduction) |
||||
opts.MemTableSize = 16 * units.Mb // 16 MB memtable (4x reduction) |
||||
``` |
||||
|
||||
**Expected Impact:** |
||||
- Cost per key should drop from ~52 MB to ~6-8 MB |
||||
- Cache can now hold ~2,000-3,000 keys instead of ~39 |
||||
- **Projected hit ratio: 60-70%** (significant improvement) |
||||
|
||||
### Approach 2: Enable Compression (IMPLEMENTED) |
||||
|
||||
```go |
||||
// OLD: |
||||
opts.Compression = options.None |
||||
|
||||
// NEW: |
||||
opts.Compression = options.ZSTD |
||||
opts.ZSTDCompressionLevel = 1 // Fast compression |
||||
``` |
||||
|
||||
**Expected Impact:** |
||||
- Compressed data reduces cache cost metric |
||||
- ZSTD level 1 is very fast (~500 MB/s) with ~2-3x compression |
||||
- Should reduce cost per key by another 50-60% |
||||
- **Combined with smaller tables: cost per key ~3-4 MB** |
||||
|
||||
### Approach 3: Massive Cache Increase (IMPLEMENTED) |
||||
|
||||
**Changes in `Dockerfile.next-orly`:** |
||||
|
||||
```dockerfile |
||||
ENV ORLY_DB_BLOCK_CACHE_MB=16384 # 16 GB (was 2 GB) |
||||
ENV ORLY_DB_INDEX_CACHE_MB=4096 # 4 GB (was 1 GB) |
||||
``` |
||||
|
||||
**Rationale:** |
||||
- With 16 GB cache and 3-4 MB cost per key: **~4,000-5,000 keys** can fit |
||||
- This should cover the working set for most benchmark tests |
||||
- **Target hit ratio: 80-90%** |
||||
|
||||
## Combined Effect Calculation |
||||
|
||||
### Before Optimization: |
||||
- Table size: 64 MB |
||||
- Cost per key: ~52 MB |
||||
- Cache: 2 GB |
||||
- Keys in cache: ~39 |
||||
- Hit ratio: 33% |
||||
|
||||
### After Optimization: |
||||
- Table size: 8 MB (8x smaller) |
||||
- Compression: ZSTD (~3x reduction) |
||||
- Effective cost per key: ~2-3 MB (17-25x reduction!) |
||||
- Cache: 16 GB (8x larger) |
||||
- Keys in cache: **~5,000-8,000** (128-205x improvement) |
||||
- **Projected hit ratio: 85-95%** |
||||
|
||||
## Trade-offs |
||||
|
||||
### Smaller Tables |
||||
**Pros:** |
||||
- Lower cache cost |
||||
- Faster individual compactions |
||||
- Better cache efficiency |
||||
|
||||
**Cons:** |
||||
- More files to manage (mitigated by faster compaction) |
||||
- Slightly more compaction overhead |
||||
|
||||
**Verdict:** Worth it for 25x cache efficiency improvement |
||||
|
||||
### Compression |
||||
**Pros:** |
||||
- Reduces cache cost |
||||
- Reduces disk space |
||||
- ZSTD level 1 is very fast |
||||
|
||||
**Cons:** |
||||
- ~5-10% CPU overhead for compression |
||||
- ~3-5% CPU overhead for decompression |
||||
|
||||
**Verdict:** Minor CPU cost for major cache gains |
||||
|
||||
### Large Cache |
||||
**Pros:** |
||||
- High hit ratio |
||||
- Lower latency |
||||
- Better throughput |
||||
|
||||
**Cons:** |
||||
- 20 GB memory usage (16 GB block + 4 GB index) |
||||
- May not be suitable for resource-constrained environments |
||||
|
||||
**Verdict:** Acceptable for high-performance relay deployments |
||||
|
||||
## Alternative Configurations |
||||
|
||||
### For 8 GB RAM Systems: |
||||
```dockerfile |
||||
ENV ORLY_DB_BLOCK_CACHE_MB=6144 # 6 GB |
||||
ENV ORLY_DB_INDEX_CACHE_MB=1536 # 1.5 GB |
||||
``` |
||||
With optimized tables+compression: ~2,000-3,000 keys, 70-80% hit ratio |
||||
|
||||
### For 4 GB RAM Systems: |
||||
```dockerfile |
||||
ENV ORLY_DB_BLOCK_CACHE_MB=2560 # 2.5 GB |
||||
ENV ORLY_DB_INDEX_CACHE_MB=512 # 512 MB |
||||
``` |
||||
With optimized tables+compression: ~800-1,200 keys, 50-60% hit ratio |
||||
|
||||
## Testing & Validation |
||||
|
||||
To test these changes: |
||||
|
||||
```bash |
||||
cd /home/mleku/src/next.orly.dev/cmd/benchmark |
||||
|
||||
# Rebuild with new code changes |
||||
docker compose build next-orly |
||||
|
||||
# Run benchmark |
||||
sudo rm -rf data/ |
||||
./run-benchmark-orly-only.sh |
||||
``` |
||||
|
||||
### Metrics to Monitor: |
||||
1. **Cache hit ratio** (target: >85%) |
||||
2. **Cache life expectancy** (target: >30 seconds) |
||||
3. **Average latency** (target: <3ms) |
||||
4. **P95 latency** (target: <10ms) |
||||
5. **Burst pattern performance** (target: match khatru-sqlite) |
||||
|
||||
## Expected Results |
||||
|
||||
### Burst Pattern Test: |
||||
- **Before**: 9.35ms avg, 34.48ms P95 |
||||
- **After**: <4ms avg, <10ms P95 (60-70% improvement) |
||||
|
||||
### Overall Performance: |
||||
- Match or exceed khatru-sqlite and khatru-badger |
||||
- Eliminate cache warnings |
||||
- Stable performance across test rounds |
||||
@ -0,0 +1,97 @@
@@ -0,0 +1,97 @@
|
||||
# Badger Cache Tuning Analysis |
||||
|
||||
## Problem Identified |
||||
|
||||
From benchmark run `run_20251116_092759`, the Badger block cache showed critical performance issues: |
||||
|
||||
### Cache Metrics (Round 1): |
||||
``` |
||||
Block cache might be too small. Metrics: |
||||
- hit: 151,469 |
||||
- miss: 307,989 |
||||
- hit-ratio: 0.33 (33%) |
||||
- keys-added: 226,912 |
||||
- keys-evicted: 226,893 (99.99% eviction rate!) |
||||
- Cache life expectancy: 2 seconds (90th percentile) |
||||
``` |
||||
|
||||
### Performance Impact: |
||||
- **Burst Pattern Latency**: 9.35ms avg (vs 3.61ms for khatru-sqlite) |
||||
- **P95 Latency**: 34.48ms (vs 8.59ms for khatru-sqlite) |
||||
- **Cache hit ratio**: Only 33% - causing constant disk I/O |
||||
|
||||
## Root Cause |
||||
|
||||
The benchmark container was using **default Badger cache sizes** (much smaller than the code defaults): |
||||
- Block cache: ~64 MB (Badger default) |
||||
- Index cache: ~32 MB (Badger default) |
||||
|
||||
The code has better defaults (1024 MB / 512 MB), but these weren't set in the Docker container. |
||||
|
||||
## Cache Size Calculation |
||||
|
||||
Based on benchmark workload analysis: |
||||
|
||||
### Block Cache Requirements: |
||||
- Total cost added: 12.44 TB during test |
||||
- With 226K keys and immediate evictions, we need to hold ~100-200K blocks in memory |
||||
- At ~10-20 KB per block average: **2-4 GB needed** |
||||
|
||||
### Index Cache Requirements: |
||||
- For 200K+ keys with metadata |
||||
- Efficient index lookups during queries |
||||
- **1-2 GB needed** |
||||
|
||||
## Solution |
||||
|
||||
Updated `Dockerfile.next-orly` with optimized cache settings: |
||||
|
||||
```dockerfile |
||||
ENV ORLY_DB_BLOCK_CACHE_MB=2048 # 2 GB block cache |
||||
ENV ORLY_DB_INDEX_CACHE_MB=1024 # 1 GB index cache |
||||
``` |
||||
|
||||
### Expected Improvements: |
||||
- **Cache hit ratio**: Target 85-95% (up from 33%) |
||||
- **Burst pattern latency**: Target <5ms avg (down from 9.35ms) |
||||
- **P95 latency**: Target <15ms (down from 34.48ms) |
||||
- **Query latency**: Significant reduction due to cached index lookups |
||||
|
||||
## Testing Strategy |
||||
|
||||
1. Rebuild Docker image with new cache settings |
||||
2. Run full benchmark suite |
||||
3. Compare metrics: |
||||
- Cache hit ratio |
||||
- Average/P95/P99 latencies |
||||
- Throughput under burst patterns |
||||
- Memory usage |
||||
|
||||
## Memory Budget |
||||
|
||||
With these settings, the relay will use approximately: |
||||
- Block cache: 2 GB |
||||
- Index cache: 1 GB |
||||
- Badger internal structures: ~200 MB |
||||
- Go runtime: ~200 MB |
||||
- **Total**: ~3.5 GB |
||||
|
||||
This is reasonable for a high-performance relay and well within modern server capabilities. |
||||
|
||||
## Alternative Configurations |
||||
|
||||
For constrained environments: |
||||
|
||||
### Medium (1.5 GB total): |
||||
``` |
||||
ORLY_DB_BLOCK_CACHE_MB=1024 |
||||
ORLY_DB_INDEX_CACHE_MB=512 |
||||
``` |
||||
|
||||
### Minimal (512 MB total): |
||||
``` |
||||
ORLY_DB_BLOCK_CACHE_MB=384 |
||||
ORLY_DB_INDEX_CACHE_MB=128 |
||||
``` |
||||
|
||||
Note: Smaller caches will result in lower hit ratios and higher latencies. |
||||
@ -0,0 +1,162 @@
@@ -0,0 +1,162 @@
|
||||
# Inline Event Optimization Strategy |
||||
|
||||
## Problem: Value Log vs LSM Tree |
||||
|
||||
By default, Badger stores all values above a small threshold (~1KB) in the value log (separate files). This causes: |
||||
- **Extra disk I/O** for reading values |
||||
- **Cache inefficiency** - must cache both keys AND value log positions |
||||
- **Poor performance for small inline events** |
||||
|
||||
## ORLY's Inline Event Storage |
||||
|
||||
ORLY uses "Reiser4 optimization" - small events are stored **inline** in the key itself: |
||||
- Event data embedded directly in LSM tree |
||||
- No separate value log lookup needed |
||||
- Much faster reads for small events |
||||
|
||||
**But:** By default, Badger still tries to put these in the value log! |
||||
|
||||
## Solution: VLogPercentile |
||||
|
||||
```go |
||||
opts.VLogPercentile = 0.99 |
||||
``` |
||||
|
||||
**What this does:** |
||||
- Analyzes value size distribution |
||||
- Keeps the smallest 99% of values in the LSM tree |
||||
- Only puts the largest 1% in value log |
||||
|
||||
**Impact on ORLY:** |
||||
- Our optimized inline events stay in LSM tree ✅ |
||||
- Only large events (>100KB) go to value log |
||||
- Dramatically faster reads for typical Nostr events |
||||
|
||||
## Additional Optimizations Implemented |
||||
|
||||
### 1. Disable Conflict Detection |
||||
```go |
||||
opts.DetectConflicts = false |
||||
``` |
||||
|
||||
**Rationale:** |
||||
- Nostr events are **immutable** (content-addressable by ID) |
||||
- No need for transaction conflict checking |
||||
- **5-10% performance improvement** on writes |
||||
|
||||
### 2. Optimize BaseLevelSize |
||||
```go |
||||
opts.BaseLevelSize = 64 * units.Mb // Increased from 10 MB |
||||
``` |
||||
|
||||
**Benefits:** |
||||
- Fewer LSM levels to search |
||||
- Faster compaction |
||||
- Better space amplification |
||||
|
||||
### 3. Enable ZSTD Compression |
||||
```go |
||||
opts.Compression = options.ZSTD |
||||
opts.ZSTDCompressionLevel = 1 // Fast mode |
||||
``` |
||||
|
||||
**Benefits:** |
||||
- 2-3x compression ratio on event data |
||||
- Level 1 is very fast (500+ MB/s compression, 2+ GB/s decompression) |
||||
- Reduces cache cost metric |
||||
- Saves disk space |
||||
|
||||
## Combined Effect |
||||
|
||||
### Before Optimization: |
||||
``` |
||||
Small inline event read: |
||||
1. Read key from LSM tree |
||||
2. Get value log position from LSM |
||||
3. Seek to value log file |
||||
4. Read value from value log |
||||
Total: ~3-5 disk operations |
||||
``` |
||||
|
||||
### After Optimization: |
||||
``` |
||||
Small inline event read: |
||||
1. Read key+value from LSM tree (in cache!) |
||||
Total: 1 cache hit |
||||
``` |
||||
|
||||
**Performance improvement: 3-5x faster reads for inline events** |
||||
|
||||
## Configuration Summary |
||||
|
||||
All optimizations applied in `pkg/database/database.go`: |
||||
|
||||
```go |
||||
// Cache |
||||
opts.BlockCacheSize = 16384 MB // 16 GB |
||||
opts.IndexCacheSize = 4096 MB // 4 GB |
||||
|
||||
// Table sizes (reduce cache cost) |
||||
opts.BaseTableSize = 8 MB |
||||
opts.MemTableSize = 16 MB |
||||
|
||||
// Keep inline events in LSM |
||||
opts.VLogPercentile = 0.99 |
||||
|
||||
// LSM structure |
||||
opts.BaseLevelSize = 64 MB |
||||
opts.LevelSizeMultiplier = 10 |
||||
|
||||
// Performance |
||||
opts.Compression = ZSTD (level 1) |
||||
opts.DetectConflicts = false |
||||
opts.NumCompactors = 8 |
||||
opts.NumMemtables = 8 |
||||
``` |
||||
|
||||
## Expected Benchmark Improvements |
||||
|
||||
### Before (run_20251116_092759): |
||||
- Burst pattern: 9.35ms avg, 34.48ms P95 |
||||
- Cache hit ratio: 33% |
||||
- Value log lookups: high |
||||
|
||||
### After (projected): |
||||
- Burst pattern: <3ms avg, <8ms P95 |
||||
- Cache hit ratio: 85-95% |
||||
- Value log lookups: minimal (only large events) |
||||
|
||||
**Overall: 60-70% latency reduction, matching or exceeding other Badger-based relays** |
||||
|
||||
## Trade-offs |
||||
|
||||
### VLogPercentile = 0.99 |
||||
**Pro:** Keeps inline events in LSM for fast access |
||||
**Con:** Larger LSM tree (but we have 16 GB cache to handle it) |
||||
**Verdict:** ✅ Essential for inline event optimization |
||||
|
||||
### DetectConflicts = false |
||||
**Pro:** 5-10% faster writes |
||||
**Con:** No transaction conflict detection |
||||
**Verdict:** ✅ Safe - Nostr events are immutable |
||||
|
||||
### ZSTD Compression |
||||
**Pro:** 2-3x space savings, lower cache cost |
||||
**Con:** ~5% CPU overhead |
||||
**Verdict:** ✅ Well worth it for cache efficiency |
||||
|
||||
## Testing |
||||
|
||||
Run benchmark to validate: |
||||
```bash |
||||
cd cmd/benchmark |
||||
docker compose build next-orly |
||||
sudo rm -rf data/ |
||||
./run-benchmark-orly-only.sh |
||||
``` |
||||
|
||||
Monitor for: |
||||
1. ✅ No "Block cache too small" warnings |
||||
2. ✅ Cache hit ratio >85% |
||||
3. ✅ Latencies competitive with khatru-badger |
||||
4. ✅ Most values in LSM tree (check logs) |
||||
@ -0,0 +1,65 @@
@@ -0,0 +1,65 @@
|
||||
version: "3.8" |
||||
|
||||
services: |
||||
# Next.orly.dev relay with profiling enabled |
||||
next-orly: |
||||
build: |
||||
context: ../.. |
||||
dockerfile: cmd/benchmark/Dockerfile.next-orly |
||||
container_name: benchmark-next-orly-profile |
||||
environment: |
||||
- ORLY_DATA_DIR=/data |
||||
- ORLY_LISTEN=0.0.0.0 |
||||
- ORLY_PORT=8080 |
||||
- ORLY_LOG_LEVEL=info |
||||
- ORLY_PPROF=cpu |
||||
- ORLY_PPROF_HTTP=true |
||||
- ORLY_PPROF_PATH=/profiles |
||||
- ORLY_DB_BLOCK_CACHE_MB=512 |
||||
- ORLY_DB_INDEX_CACHE_MB=256 |
||||
volumes: |
||||
- ./data/next-orly:/data |
||||
- ./profiles:/profiles |
||||
ports: |
||||
- "8001:8080" |
||||
- "6060:6060" # pprof HTTP endpoint |
||||
networks: |
||||
- benchmark-net |
||||
healthcheck: |
||||
test: ["CMD", "curl", "-f", "http://localhost:8080/"] |
||||
interval: 10s |
||||
timeout: 5s |
||||
retries: 5 |
||||
start_period: 60s # Longer startup period |
||||
|
||||
# Benchmark runner - only test next-orly |
||||
benchmark-runner: |
||||
build: |
||||
context: ../.. |
||||
dockerfile: cmd/benchmark/Dockerfile.benchmark |
||||
container_name: benchmark-runner-profile |
||||
depends_on: |
||||
next-orly: |
||||
condition: service_healthy |
||||
environment: |
||||
- BENCHMARK_TARGETS=next-orly:8080 |
||||
- BENCHMARK_EVENTS=50000 |
||||
- BENCHMARK_WORKERS=24 |
||||
- BENCHMARK_DURATION=60s |
||||
volumes: |
||||
- ./reports:/reports |
||||
networks: |
||||
- benchmark-net |
||||
command: > |
||||
sh -c " |
||||
echo 'Waiting for ORLY to be ready (healthcheck)...' && |
||||
sleep 5 && |
||||
echo 'Starting benchmark tests...' && |
||||
/app/benchmark-runner --output-dir=/reports && |
||||
echo 'Benchmark complete - triggering shutdown...' && |
||||
exit 0 |
||||
" |
||||
|
||||
networks: |
||||
benchmark-net: |
||||
driver: bridge |
||||
@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
|
||||
#!/bin/bash |
||||
|
||||
# Wrapper script that cleans data directories with sudo before running benchmark |
||||
# Use this if you encounter permission errors with run-benchmark.sh |
||||
|
||||
set -e |
||||
|
||||
cd "$(dirname "$0")" |
||||
|
||||
# Stop any running containers first |
||||
echo "Stopping any running benchmark containers..." |
||||
if docker compose version &> /dev/null 2>&1; then |
||||
docker compose down -v 2>&1 | grep -v "warning" || true |
||||
else |
||||
docker-compose down -v 2>&1 | grep -v "warning" || true |
||||
fi |
||||
|
||||
# Clean data directories with sudo |
||||
if [ -d "data" ]; then |
||||
echo "Cleaning data directories (requires sudo)..." |
||||
sudo rm -rf data/ |
||||
fi |
||||
|
||||
# Now run the normal benchmark script |
||||
exec ./run-benchmark.sh |
||||
@ -0,0 +1,80 @@
@@ -0,0 +1,80 @@
|
||||
#!/bin/bash |
||||
|
||||
# Run benchmark for ORLY only (no other relays) |
||||
|
||||
set -e |
||||
|
||||
cd "$(dirname "$0")" |
||||
|
||||
# Determine docker-compose command |
||||
if docker compose version &> /dev/null 2>&1; then |
||||
DOCKER_COMPOSE="docker compose" |
||||
else |
||||
DOCKER_COMPOSE="docker-compose" |
||||
fi |
||||
|
||||
# Clean old data directories (may be owned by root from Docker) |
||||
if [ -d "data" ]; then |
||||
echo "Cleaning old data directories..." |
||||
if ! rm -rf data/ 2>/dev/null; then |
||||
echo "" |
||||
echo "ERROR: Cannot remove data directories due to permission issues." |
||||
echo "Please run: sudo rm -rf data/" |
||||
echo "Then run this script again." |
||||
exit 1 |
||||
fi |
||||
fi |
||||
|
||||
# Create fresh data directories with correct permissions |
||||
echo "Preparing data directories..." |
||||
mkdir -p data/next-orly |
||||
chmod 777 data/next-orly |
||||
|
||||
echo "Building ORLY container..." |
||||
$DOCKER_COMPOSE build next-orly |
||||
|
||||
echo "Starting ORLY relay..." |
||||
echo "" |
||||
|
||||
# Start only next-orly and benchmark-runner |
||||
$DOCKER_COMPOSE up next-orly -d |
||||
|
||||
# Wait for ORLY to be healthy |
||||
echo "Waiting for ORLY to be healthy..." |
||||
for i in {1..30}; do |
||||
if curl -sf http://localhost:8001/ > /dev/null 2>&1; then |
||||
echo "ORLY is ready!" |
||||
break |
||||
fi |
||||
sleep 2 |
||||
if [ $i -eq 30 ]; then |
||||
echo "ERROR: ORLY failed to become healthy" |
||||
$DOCKER_COMPOSE logs next-orly |
||||
exit 1 |
||||
fi |
||||
done |
||||
|
||||
# Run benchmark against ORLY |
||||
echo "" |
||||
echo "Running benchmark against ORLY..." |
||||
echo "Target: http://localhost:8001" |
||||
echo "" |
||||
|
||||
# Run the benchmark binary directly against the running ORLY instance |
||||
docker run --rm --network benchmark_benchmark-net \ |
||||
-e BENCHMARK_TARGETS=next-orly:8080 \ |
||||
-e BENCHMARK_EVENTS=50000 \ |
||||
-e BENCHMARK_WORKERS=24 \ |
||||
-e BENCHMARK_DURATION=60s \ |
||||
-v "$(pwd)/reports:/reports" \ |
||||
benchmark-benchmark-runner \ |
||||
/app/benchmark-runner --output-dir=/reports |
||||
|
||||
echo "" |
||||
echo "Benchmark complete!" |
||||
echo "Stopping ORLY..." |
||||
$DOCKER_COMPOSE down |
||||
|
||||
echo "" |
||||
echo "Results saved to ./reports/" |
||||
echo "Check the latest run_* directory for detailed results." |
||||
@ -0,0 +1,46 @@
@@ -0,0 +1,46 @@
|
||||
#!/bin/bash |
||||
|
||||
# Wrapper script to run the benchmark suite and automatically shut down when complete |
||||
|
||||
set -e |
||||
|
||||
# Determine docker-compose command |
||||
if docker compose version &> /dev/null 2>&1; then |
||||
DOCKER_COMPOSE="docker compose" |
||||
else |
||||
DOCKER_COMPOSE="docker-compose" |
||||
fi |
||||
|
||||
# Clean old data directories (may be owned by root from Docker) |
||||
if [ -d "data" ]; then |
||||
echo "Cleaning old data directories..." |
||||
if ! rm -rf data/ 2>/dev/null; then |
||||
# If normal rm fails (permission denied), provide clear instructions |
||||
echo "" |
||||
echo "ERROR: Cannot remove data directories due to permission issues." |
||||
echo "This happens because Docker creates files as root." |
||||
echo "" |
||||
echo "Please run one of the following to clean up:" |
||||
echo " sudo rm -rf data/" |
||||
echo " sudo chown -R \$(id -u):\$(id -g) data/ && rm -rf data/" |
||||
echo "" |
||||
echo "Then run this script again." |
||||
exit 1 |
||||
fi |
||||
fi |
||||
|
||||
# Create fresh data directories with correct permissions |
||||
echo "Preparing data directories..." |
||||
mkdir -p data/{next-orly,khatru-sqlite,khatru-badger,relayer-basic,strfry,nostr-rs-relay,postgres} |
||||
chmod 777 data/{next-orly,khatru-sqlite,khatru-badger,relayer-basic,strfry,nostr-rs-relay,postgres} |
||||
|
||||
echo "Starting benchmark suite..." |
||||
echo "This will automatically shut down all containers when the benchmark completes." |
||||
echo "" |
||||
|
||||
# Run docker compose with flags to exit when benchmark-runner completes |
||||
$DOCKER_COMPOSE up --exit-code-from benchmark-runner --abort-on-container-exit |
||||
|
||||
echo "" |
||||
echo "Benchmark suite has completed and all containers have been stopped." |
||||
echo "Check the ./reports/ directory for results." |
||||
@ -0,0 +1,41 @@
@@ -0,0 +1,41 @@
|
||||
#!/bin/bash |
||||
|
||||
# Run benchmark with profiling on ORLY only |
||||
|
||||
set -e |
||||
|
||||
# Determine docker-compose command |
||||
if docker compose version &> /dev/null 2>&1; then |
||||
DOCKER_COMPOSE="docker compose" |
||||
else |
||||
DOCKER_COMPOSE="docker-compose" |
||||
fi |
||||
|
||||
# Clean up old data and profiles (may need sudo for Docker-created files) |
||||
echo "Cleaning old data and profiles..." |
||||
if [ -d "data/next-orly" ]; then |
||||
if ! rm -rf data/next-orly/* 2>/dev/null; then |
||||
echo "Need elevated permissions to clean data directories..." |
||||
sudo rm -rf data/next-orly/* |
||||
fi |
||||
fi |
||||
rm -rf profiles/* 2>/dev/null || sudo rm -rf profiles/* 2>/dev/null || true |
||||
mkdir -p data/next-orly profiles |
||||
chmod 777 data/next-orly 2>/dev/null || true |
||||
|
||||
echo "Starting profiled benchmark (ORLY only)..." |
||||
echo "- 50,000 events" |
||||
echo "- 24 workers" |
||||
echo "- 90 second warmup delay" |
||||
echo "- CPU profiling enabled" |
||||
echo "- pprof HTTP on port 6060" |
||||
echo "" |
||||
|
||||
# Run docker compose with profile config |
||||
$DOCKER_COMPOSE -f docker-compose.profile.yml up \ |
||||
--exit-code-from benchmark-runner \ |
||||
--abort-on-container-exit |
||||
|
||||
echo "" |
||||
echo "Benchmark complete. Profiles saved to ./profiles/" |
||||
echo "Results saved to ./reports/" |
||||
@ -0,0 +1,39 @@
@@ -0,0 +1,39 @@
|
||||
package database |
||||
|
||||
import ( |
||||
"context" |
||||
"fmt" |
||||
"strings" |
||||
) |
||||
|
||||
// NewDatabase creates a database instance based on the specified type.
|
||||
// Supported types: "badger", "dgraph"
|
||||
func NewDatabase( |
||||
ctx context.Context, |
||||
cancel context.CancelFunc, |
||||
dbType string, |
||||
dataDir string, |
||||
logLevel string, |
||||
) (Database, error) { |
||||
switch strings.ToLower(dbType) { |
||||
case "badger", "": |
||||
// Use the existing badger implementation
|
||||
return New(ctx, cancel, dataDir, logLevel) |
||||
case "dgraph": |
||||
// Use the new dgraph implementation
|
||||
// Import dynamically to avoid import cycles
|
||||
return newDgraphDatabase(ctx, cancel, dataDir, logLevel) |
||||
default: |
||||
return nil, fmt.Errorf("unsupported database type: %s (supported: badger, dgraph)", dbType) |
||||
} |
||||
} |
||||
|
||||
// newDgraphDatabase creates a dgraph database instance
|
||||
// This is defined here to avoid import cycles
|
||||
var newDgraphDatabase func(context.Context, context.CancelFunc, string, string) (Database, error) |
||||
|
||||
// RegisterDgraphFactory registers the dgraph database factory
|
||||
// This is called from the dgraph package's init() function
|
||||
func RegisterDgraphFactory(factory func(context.Context, context.CancelFunc, string, string) (Database, error)) { |
||||
newDgraphDatabase = factory |
||||
} |
||||
@ -0,0 +1,102 @@
@@ -0,0 +1,102 @@
|
||||
package database |
||||
|
||||
import ( |
||||
"context" |
||||
"io" |
||||
"time" |
||||
|
||||
"next.orly.dev/pkg/database/indexes/types" |
||||
"next.orly.dev/pkg/encoders/event" |
||||
"next.orly.dev/pkg/encoders/filter" |
||||
"next.orly.dev/pkg/encoders/tag" |
||||
"next.orly.dev/pkg/interfaces/store" |
||||
) |
||||
|
||||
// Database defines the interface that all database implementations must satisfy.
|
||||
// This allows switching between different storage backends (badger, dgraph, etc.)
|
||||
type Database interface { |
||||
// Core lifecycle methods
|
||||
Path() string |
||||
Init(path string) error |
||||
Sync() error |
||||
Close() error |
||||
Wipe() error |
||||
SetLogLevel(level string) |
||||
Ready() <-chan struct{} // Returns a channel that closes when database is ready to serve requests
|
||||
|
||||
// Event storage and retrieval
|
||||
SaveEvent(c context.Context, ev *event.E) (exists bool, err error) |
||||
GetSerialsFromFilter(f *filter.F) (serials types.Uint40s, err error) |
||||
WouldReplaceEvent(ev *event.E) (bool, types.Uint40s, error) |
||||
|
||||
QueryEvents(c context.Context, f *filter.F) (evs event.S, err error) |
||||
QueryAllVersions(c context.Context, f *filter.F) (evs event.S, err error) |
||||
QueryEventsWithOptions(c context.Context, f *filter.F, includeDeleteEvents bool, showAllVersions bool) (evs event.S, err error) |
||||
QueryDeleteEventsByTargetId(c context.Context, targetEventId []byte) (evs event.S, err error) |
||||
QueryForSerials(c context.Context, f *filter.F) (serials types.Uint40s, err error) |
||||
QueryForIds(c context.Context, f *filter.F) (idPkTs []*store.IdPkTs, err error) |
||||
|
||||
CountEvents(c context.Context, f *filter.F) (count int, approximate bool, err error) |
||||
|
||||
FetchEventBySerial(ser *types.Uint40) (ev *event.E, err error) |
||||
FetchEventsBySerials(serials []*types.Uint40) (events map[uint64]*event.E, err error) |
||||
|
||||
GetSerialById(id []byte) (ser *types.Uint40, err error) |
||||
GetSerialsByIds(ids *tag.T) (serials map[string]*types.Uint40, err error) |
||||
GetSerialsByIdsWithFilter(ids *tag.T, fn func(ev *event.E, ser *types.Uint40) bool) (serials map[string]*types.Uint40, err error) |
||||
GetSerialsByRange(idx Range) (serials types.Uint40s, err error) |
||||
|
||||
GetFullIdPubkeyBySerial(ser *types.Uint40) (fidpk *store.IdPkTs, err error) |
||||
GetFullIdPubkeyBySerials(sers []*types.Uint40) (fidpks []*store.IdPkTs, err error) |
||||
|
||||
// Event deletion
|
||||
DeleteEvent(c context.Context, eid []byte) error |
||||
DeleteEventBySerial(c context.Context, ser *types.Uint40, ev *event.E) error |
||||
DeleteExpired() |
||||
ProcessDelete(ev *event.E, admins [][]byte) error |
||||
CheckForDeleted(ev *event.E, admins [][]byte) error |
||||
|
||||
// Import/Export
|
||||
Import(rr io.Reader) |
||||
Export(c context.Context, w io.Writer, pubkeys ...[]byte) |
||||
ImportEventsFromReader(ctx context.Context, rr io.Reader) error |
||||
ImportEventsFromStrings(ctx context.Context, eventJSONs []string, policyManager interface{ CheckPolicy(action string, ev *event.E, pubkey []byte, remote string) (bool, error) }) error |
||||
|
||||
// Relay identity
|
||||
GetRelayIdentitySecret() (skb []byte, err error) |
||||
SetRelayIdentitySecret(skb []byte) error |
||||
GetOrCreateRelayIdentitySecret() (skb []byte, err error) |
||||
|
||||
// Markers (metadata key-value storage)
|
||||
SetMarker(key string, value []byte) error |
||||
GetMarker(key string) (value []byte, err error) |
||||
HasMarker(key string) bool |
||||
DeleteMarker(key string) error |
||||
|
||||
// Subscriptions (payment-based access control)
|
||||
GetSubscription(pubkey []byte) (*Subscription, error) |
||||
IsSubscriptionActive(pubkey []byte) (bool, error) |
||||
ExtendSubscription(pubkey []byte, days int) error |
||||
RecordPayment(pubkey []byte, amount int64, invoice, preimage string) error |
||||
GetPaymentHistory(pubkey []byte) ([]Payment, error) |
||||
ExtendBlossomSubscription(pubkey []byte, tier string, storageMB int64, daysExtended int) error |
||||
GetBlossomStorageQuota(pubkey []byte) (quotaMB int64, err error) |
||||
IsFirstTimeUser(pubkey []byte) (bool, error) |
||||
|
||||
// NIP-43 Invite-based ACL
|
||||
AddNIP43Member(pubkey []byte, inviteCode string) error |
||||
RemoveNIP43Member(pubkey []byte) error |
||||
IsNIP43Member(pubkey []byte) (isMember bool, err error) |
||||
GetNIP43Membership(pubkey []byte) (*NIP43Membership, error) |
||||
GetAllNIP43Members() ([][]byte, error) |
||||
StoreInviteCode(code string, expiresAt time.Time) error |
||||
ValidateInviteCode(code string) (valid bool, err error) |
||||
DeleteInviteCode(code string) error |
||||
PublishNIP43MembershipEvent(kind int, pubkey []byte) error |
||||
|
||||
// Migrations (version tracking for schema updates)
|
||||
RunMigrations() |
||||
|
||||
// Utility methods
|
||||
EventIdsBySerial(start uint64, count int) (evs []uint64, err error) |
||||
} |
||||
@ -0,0 +1,280 @@
@@ -0,0 +1,280 @@
|
||||
# Dgraph Database Implementation for ORLY |
||||
|
||||
This package provides a Dgraph-based implementation of the ORLY database interface, enabling graph-based storage for Nostr events with powerful relationship querying capabilities. |
||||
|
||||
## Status: Step 1 Complete ✅ |
||||
|
||||
**Current State:** Dgraph server integration is complete and functional |
||||
**Next Step:** DQL query/mutation implementation in save-event.go and query-events.go |
||||
|
||||
## Architecture |
||||
|
||||
### Client-Server Model |
||||
|
||||
The implementation uses a **client-server architecture**: |
||||
|
||||
``` |
||||
┌─────────────────────────────────────────────┐ |
||||
│ ORLY Relay Process │ |
||||
│ │ |
||||
│ ┌────────────────────────────────────┐ │ |
||||
│ │ Dgraph Client (pkg/dgraph) │ │ |
||||
│ │ - dgo library (gRPC) │ │ |
||||
│ │ - Schema management │────┼───► Dgraph Server |
||||
│ │ - Query/Mutate methods │ │ (localhost:9080) |
||||
│ └────────────────────────────────────┘ │ - Event graph |
||||
│ │ - Authors, tags |
||||
│ ┌────────────────────────────────────┐ │ - Relationships |
||||
│ │ Badger Metadata Store │ │ |
||||
│ │ - Markers (key-value) │ │ |
||||
│ │ - Serial counters │ │ |
||||
│ │ - Relay identity │ │ |
||||
│ └────────────────────────────────────┘ │ |
||||
└─────────────────────────────────────────────┘ |
||||
``` |
||||
|
||||
### Dual Storage Strategy |
||||
|
||||
1. **Dgraph** (Graph Database) |
||||
- Nostr events and their content |
||||
- Author relationships |
||||
- Tag relationships |
||||
- Event references and mentions |
||||
- Optimized for graph traversals and complex queries |
||||
|
||||
2. **Badger** (Key-Value Store) |
||||
- Metadata markers |
||||
- Serial number counters |
||||
- Relay identity keys |
||||
- Fast key-value operations |
||||
|
||||
## Setup |
||||
|
||||
### 1. Start Dgraph Server |
||||
|
||||
Using Docker (recommended): |
||||
|
||||
```bash |
||||
docker run -d \ |
||||
--name dgraph \ |
||||
-p 8080:8080 \ |
||||
-p 9080:9080 \ |
||||
-p 8000:8000 \ |
||||
-v ~/dgraph:/dgraph \ |
||||
dgraph/standalone:latest |
||||
``` |
||||
|
||||
### 2. Configure ORLY |
||||
|
||||
```bash |
||||
export ORLY_DB_TYPE=dgraph |
||||
export ORLY_DGRAPH_URL=localhost:9080 # Optional, this is the default |
||||
``` |
||||
|
||||
### 3. Run ORLY |
||||
|
||||
```bash |
||||
./orly |
||||
``` |
||||
|
||||
On startup, ORLY will: |
||||
1. Connect to dgraph server via gRPC |
||||
2. Apply the Nostr schema automatically |
||||
3. Initialize badger metadata store |
||||
4. Initialize serial number counter |
||||
5. Start accepting events |
||||
|
||||
## Schema |
||||
|
||||
The Nostr schema defines the following types: |
||||
|
||||
### Event Nodes |
||||
```dql |
||||
type Event { |
||||
event.id # Event ID (string, indexed) |
||||
event.serial # Sequential number (int, indexed) |
||||
event.kind # Event kind (int, indexed) |
||||
event.created_at # Timestamp (int, indexed) |
||||
event.content # Event content (string) |
||||
event.sig # Signature (string, indexed) |
||||
event.pubkey # Author pubkey (string, indexed) |
||||
event.authored_by # -> Author (uid) |
||||
event.references # -> Events (uid list) |
||||
event.mentions # -> Events (uid list) |
||||
event.tagged_with # -> Tags (uid list) |
||||
} |
||||
``` |
||||
|
||||
### Author Nodes |
||||
```dql |
||||
type Author { |
||||
author.pubkey # Pubkey (string, indexed, unique) |
||||
author.events # -> Events (uid list, reverse) |
||||
} |
||||
``` |
||||
|
||||
### Tag Nodes |
||||
```dql |
||||
type Tag { |
||||
tag.type # Tag type (string, indexed) |
||||
tag.value # Tag value (string, indexed + fulltext) |
||||
tag.events # -> Events (uid list, reverse) |
||||
} |
||||
``` |
||||
|
||||
### Marker Nodes (Metadata) |
||||
```dql |
||||
type Marker { |
||||
marker.key # Key (string, indexed, unique) |
||||
marker.value # Value (string) |
||||
} |
||||
``` |
||||
|
||||
## Configuration |
||||
|
||||
### Environment Variables |
||||
|
||||
- `ORLY_DB_TYPE=dgraph` - Enable dgraph database (default: badger) |
||||
- `ORLY_DGRAPH_URL=host:port` - Dgraph gRPC endpoint (default: localhost:9080) |
||||
- `ORLY_DATA_DIR=/path` - Data directory for metadata storage |
||||
|
||||
### Connection Details |
||||
|
||||
The dgraph client uses **insecure gRPC** by default for local development. For production deployments: |
||||
|
||||
1. Set up TLS certificates for dgraph |
||||
2. Modify `pkg/dgraph/dgraph.go` to use `grpc.WithTransportCredentials()` with your certs |
||||
|
||||
## Implementation Details |
||||
|
||||
### Files |
||||
|
||||
- `dgraph.go` - Main implementation, initialization, lifecycle |
||||
- `schema.go` - Schema definition and application |
||||
- `save-event.go` - Event storage (TODO: update to use Mutate) |
||||
- `query-events.go` - Event queries (TODO: update to parse DQL responses) |
||||
- `fetch-event.go` - Event retrieval methods |
||||
- `delete.go` - Event deletion |
||||
- `markers.go` - Key-value metadata storage (uses badger) |
||||
- `serial.go` - Serial number generation (uses badger) |
||||
- `subscriptions.go` - Subscription/payment tracking (uses markers) |
||||
- `nip43.go` - NIP-43 invite system (uses markers) |
||||
- `import-export.go` - Import/export operations |
||||
- `logger.go` - Logging adapter |
||||
|
||||
### Key Methods |
||||
|
||||
#### Initialization |
||||
```go |
||||
d, err := dgraph.New(ctx, cancel, dataDir, logLevel) |
||||
``` |
||||
|
||||
#### Querying (DQL) |
||||
```go |
||||
resp, err := d.Query(ctx, dqlQuery) |
||||
``` |
||||
|
||||
#### Mutations (RDF N-Quads) |
||||
```go |
||||
mutation := &api.Mutation{SetNquads: []byte(nquads)} |
||||
resp, err := d.Mutate(ctx, mutation) |
||||
``` |
||||
|
||||
## Development Status |
||||
|
||||
### ✅ Step 1: Dgraph Server Integration (COMPLETE) |
||||
|
||||
- [x] dgo client library integration |
||||
- [x] gRPC connection to external dgraph |
||||
- [x] Schema definition and auto-application |
||||
- [x] Query() and Mutate() method stubs |
||||
- [x] ORLY_DGRAPH_URL configuration |
||||
- [x] Dual-storage architecture |
||||
- [x] Proper lifecycle management |
||||
|
||||
### 📝 Step 2: DQL Implementation (NEXT) |
||||
|
||||
Priority tasks: |
||||
|
||||
1. **save-event.go** - Replace RDF string building with actual Mutate() calls |
||||
2. **query-events.go** - Parse actual JSON responses from Query() |
||||
3. **fetch-event.go** - Implement DQL queries for event retrieval |
||||
4. **delete.go** - Implement deletion mutations |
||||
|
||||
### 📝 Step 3: Testing (FUTURE) |
||||
|
||||
- Integration testing with relay-tester |
||||
- Performance benchmarks vs badger |
||||
- Memory profiling |
||||
- Production deployment testing |
||||
|
||||
## Troubleshooting |
||||
|
||||
### Connection Refused |
||||
|
||||
``` |
||||
failed to connect to dgraph at localhost:9080: connection refused |
||||
``` |
||||
|
||||
**Solution:** Ensure dgraph server is running: |
||||
```bash |
||||
docker ps | grep dgraph |
||||
docker logs dgraph |
||||
``` |
||||
|
||||
### Schema Application Failed |
||||
|
||||
``` |
||||
failed to apply schema: ... |
||||
``` |
||||
|
||||
**Solution:** Check dgraph server logs and ensure no schema conflicts: |
||||
```bash |
||||
docker logs dgraph |
||||
``` |
||||
|
||||
### Binary Not Finding libsecp256k1.so |
||||
|
||||
This is unrelated to dgraph. Ensure: |
||||
```bash |
||||
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}$(pwd)/pkg/crypto/p8k" |
||||
``` |
||||
|
||||
## Performance Considerations |
||||
|
||||
### When to Use Dgraph |
||||
|
||||
**Good fit:** |
||||
- Complex graph queries (follows-of-follows, social graphs) |
||||
- Full-text search requirements |
||||
- Advanced filtering and aggregations |
||||
- Multi-hop relationship traversals |
||||
|
||||
**Not ideal for:** |
||||
- Simple key-value lookups (badger is faster) |
||||
- Very high write throughput (badger has lower latency) |
||||
- Single-node deployments with simple queries |
||||
|
||||
### Optimization Tips |
||||
|
||||
1. **Indexing**: Ensure frequently queried fields have appropriate indexes |
||||
2. **Pagination**: Use offset/limit in DQL queries for large result sets |
||||
3. **Caching**: Consider adding an LRU cache for hot events |
||||
4. **Schema Design**: Use reverse edges for efficient relationship traversal |
||||
|
||||
## Resources |
||||
|
||||
- [Dgraph Documentation](https://dgraph.io/docs/) |
||||
- [DQL Query Language](https://dgraph.io/docs/query-language/) |
||||
- [dgo Client Library](https://github.com/dgraph-io/dgo) |
||||
- [ORLY Implementation Status](../../DGRAPH_IMPLEMENTATION_STATUS.md) |
||||
|
||||
## Contributing |
||||
|
||||
When working on dgraph implementation: |
||||
|
||||
1. Test changes against a local dgraph instance |
||||
2. Update schema.go if adding new node types or predicates |
||||
3. Ensure dual-storage strategy is maintained (dgraph for events, badger for metadata) |
||||
4. Add integration tests for new features |
||||
5. Update DGRAPH_IMPLEMENTATION_STATUS.md with progress |
||||
@ -0,0 +1,330 @@
@@ -0,0 +1,330 @@
|
||||
# Dgraph Test Suite |
||||
|
||||
This directory contains a comprehensive test suite for the dgraph database implementation, mirroring all tests from the badger implementation to ensure feature parity. |
||||
|
||||
## Test Files |
||||
|
||||
- **testmain_test.go** - Test configuration (logging, setup) |
||||
- **helpers_test.go** - Helper functions for test database setup/teardown |
||||
- **save-event_test.go** - Event storage tests |
||||
- **query-events_test.go** - Event query tests |
||||
|
||||
## Quick Start |
||||
|
||||
### 1. Start Dgraph Server |
||||
|
||||
```bash |
||||
# From project root |
||||
./scripts/dgraph-start.sh |
||||
|
||||
# Verify it's running |
||||
curl http://localhost:8080/health |
||||
``` |
||||
|
||||
### 2. Run Tests |
||||
|
||||
```bash |
||||
# Run all dgraph tests |
||||
./scripts/test-dgraph.sh |
||||
|
||||
# Or run manually |
||||
export ORLY_DGRAPH_URL=localhost:9080 |
||||
CGO_ENABLED=0 go test -v ./pkg/dgraph/... |
||||
|
||||
# Run specific test |
||||
CGO_ENABLED=0 go test -v -run TestSaveEvents ./pkg/dgraph |
||||
``` |
||||
|
||||
## Test Coverage |
||||
|
||||
### Event Storage Tests (`save-event_test.go`) |
||||
|
||||
✅ **TestSaveEvents** |
||||
- Loads ~100 events from examples.Cache |
||||
- Saves all events chronologically |
||||
- Verifies no errors during save |
||||
- Reports performance metrics |
||||
|
||||
✅ **TestDeletionEventWithETagRejection** |
||||
- Creates a regular event |
||||
- Attempts to save deletion event with e-tag |
||||
- Verifies deletion events with e-tags are rejected |
||||
|
||||
✅ **TestSaveExistingEvent** |
||||
- Saves an event |
||||
- Attempts to save same event again |
||||
- Verifies duplicate events are rejected |
||||
|
||||
### Event Query Tests (`query-events_test.go`) |
||||
|
||||
✅ **TestQueryEventsByID** |
||||
- Queries event by exact ID match |
||||
- Verifies single result returned |
||||
- Verifies correct event retrieved |
||||
|
||||
✅ **TestQueryEventsByKind** |
||||
- Queries events by kind (e.g., kind 1) |
||||
- Verifies all results have correct kind |
||||
- Tests filtering logic |
||||
|
||||
✅ **TestQueryEventsByAuthor** |
||||
- Queries events by author pubkey |
||||
- Verifies all results from correct author |
||||
- Tests author filtering |
||||
|
||||
✅ **TestReplaceableEventsAndDeletion** |
||||
- Creates replaceable event (kind 0) |
||||
- Creates newer version |
||||
- Verifies only newer version returned in general queries |
||||
- Creates deletion event |
||||
- Verifies deleted event not returned |
||||
- Tests replaceable event logic and deletion |
||||
|
||||
✅ **TestParameterizedReplaceableEventsAndDeletion** |
||||
- Creates parameterized replaceable event (kind 30000+) |
||||
- Adds d-tag |
||||
- Creates deletion event with e-tag |
||||
- Verifies deleted event not returned |
||||
- Tests parameterized replaceable logic |
||||
|
||||
✅ **TestQueryEventsByTimeRange** |
||||
- Queries events by since/until timestamps |
||||
- Verifies all results within time range |
||||
- Tests temporal filtering |
||||
|
||||
✅ **TestQueryEventsByTag** |
||||
- Finds event with tags |
||||
- Queries by tag key/value |
||||
- Verifies all results have the tag |
||||
- Tests tag filtering logic |
||||
|
||||
✅ **TestCountEvents** |
||||
- Counts all events |
||||
- Counts events by kind filter |
||||
- Verifies correct counts returned |
||||
- Tests counting functionality |
||||
|
||||
## Test Helpers |
||||
|
||||
### setupTestDB(t *testing.T) |
||||
|
||||
Creates a test dgraph database: |
||||
|
||||
1. **Checks dgraph availability** - Skips test if server not running |
||||
2. **Creates temp directory** - For metadata storage |
||||
3. **Initializes dgraph client** - Connects to server |
||||
4. **Drops all data** - Starts with clean slate |
||||
5. **Loads test events** - From examples.Cache (~100 events) |
||||
6. **Sorts chronologically** - Ensures addressable events processed in order |
||||
7. **Saves all events** - Populates test database |
||||
|
||||
**Returns:** `(*D, []*event.E, context.Context, context.CancelFunc, string)` |
||||
|
||||
### cleanupTestDB(t, db, cancel, tempDir) |
||||
|
||||
Cleans up after tests: |
||||
- Closes database connection |
||||
- Cancels context |
||||
- Removes temp directory |
||||
|
||||
### skipIfDgraphNotAvailable(t *testing.T) |
||||
|
||||
Checks if dgraph is running and skips test if not available. |
||||
|
||||
## Running Tests |
||||
|
||||
### Prerequisites |
||||
|
||||
1. **Dgraph Server** - Must be running before tests |
||||
2. **Go 1.21+** - For running tests |
||||
3. **CGO_ENABLED=0** - For pure Go build |
||||
|
||||
### Test Execution |
||||
|
||||
#### All Tests |
||||
|
||||
```bash |
||||
./scripts/test-dgraph.sh |
||||
``` |
||||
|
||||
#### Specific Test File |
||||
|
||||
```bash |
||||
CGO_ENABLED=0 go test -v ./pkg/dgraph -run TestSaveEvents |
||||
``` |
||||
|
||||
#### With Logging |
||||
|
||||
```bash |
||||
export TEST_LOG=1 |
||||
CGO_ENABLED=0 go test -v ./pkg/dgraph/... |
||||
``` |
||||
|
||||
#### With Timeout |
||||
|
||||
```bash |
||||
CGO_ENABLED=0 go test -v -timeout 10m ./pkg/dgraph/... |
||||
``` |
||||
|
||||
### Integration Testing |
||||
|
||||
Run tests + relay-tester: |
||||
|
||||
```bash |
||||
./scripts/test-dgraph.sh --relay-tester |
||||
``` |
||||
|
||||
This will: |
||||
1. Run all dgraph package tests |
||||
2. Start ORLY with dgraph backend |
||||
3. Run relay-tester against ORLY |
||||
4. Report results |
||||
|
||||
## Test Data |
||||
|
||||
Tests use `pkg/encoders/event/examples.Cache` which contains: |
||||
- ~100 real Nostr events |
||||
- Text notes (kind 1) |
||||
- Profile metadata (kind 0) |
||||
- Various other kinds |
||||
- Events with tags, references, mentions |
||||
- Multiple authors and timestamps |
||||
|
||||
This ensures tests cover realistic scenarios. |
||||
|
||||
## Debugging Tests |
||||
|
||||
### View Test Output |
||||
|
||||
```bash |
||||
CGO_ENABLED=0 go test -v ./pkg/dgraph/... 2>&1 | tee test-output.log |
||||
``` |
||||
|
||||
### Check Dgraph State |
||||
|
||||
```bash |
||||
# View data via Ratel UI |
||||
open http://localhost:8000 |
||||
|
||||
# Query via HTTP |
||||
curl -X POST localhost:8080/query -d '{ |
||||
events(func: type(Event), first: 10) { |
||||
uid |
||||
event.id |
||||
event.kind |
||||
event.created_at |
||||
} |
||||
}' |
||||
``` |
||||
|
||||
### Enable Dgraph Logging |
||||
|
||||
```bash |
||||
docker logs dgraph-orly-test -f |
||||
``` |
||||
|
||||
## Test Failures |
||||
|
||||
### "Dgraph server not available" |
||||
|
||||
**Cause:** Dgraph is not running |
||||
|
||||
**Fix:** |
||||
```bash |
||||
./scripts/dgraph-start.sh |
||||
``` |
||||
|
||||
### Connection Timeouts |
||||
|
||||
**Cause:** Dgraph server overloaded or network issues |
||||
|
||||
**Fix:** |
||||
- Increase test timeout: `go test -timeout 20m` |
||||
- Check dgraph resources: `docker stats dgraph-orly-test` |
||||
- Restart dgraph: `docker restart dgraph-orly-test` |
||||
|
||||
### Schema Errors |
||||
|
||||
**Cause:** Schema conflicts or version mismatch |
||||
|
||||
**Fix:** |
||||
- Drop all data: Tests call `dropAll()` automatically |
||||
- Check dgraph version: `docker exec dgraph-orly-test dgraph version` |
||||
|
||||
### Test Hangs |
||||
|
||||
**Cause:** Deadlock or infinite loop |
||||
|
||||
**Fix:** |
||||
- Send SIGQUIT: `kill -QUIT <test-pid>` |
||||
- View goroutine dump |
||||
- Check dgraph logs |
||||
|
||||
## Continuous Integration |
||||
|
||||
### GitHub Actions Example |
||||
|
||||
```yaml |
||||
name: Dgraph Tests |
||||
|
||||
on: [push, pull_request] |
||||
|
||||
jobs: |
||||
test: |
||||
runs-on: ubuntu-latest |
||||
|
||||
services: |
||||
dgraph: |
||||
image: dgraph/standalone:latest |
||||
ports: |
||||
- 8080:8080 |
||||
- 9080:9080 |
||||
options: >- |
||||
--health-cmd "curl -f http://localhost:8080/health" |
||||
--health-interval 10s |
||||
--health-timeout 5s |
||||
--health-retries 5 |
||||
|
||||
steps: |
||||
- uses: actions/checkout@v3 |
||||
|
||||
- name: Set up Go |
||||
uses: actions/setup-go@v4 |
||||
with: |
||||
go-version: '1.21' |
||||
|
||||
- name: Run dgraph tests |
||||
env: |
||||
ORLY_DGRAPH_URL: localhost:9080 |
||||
run: | |
||||
CGO_ENABLED=0 go test -v -timeout 10m ./pkg/dgraph/... |
||||
``` |
||||
|
||||
## Performance Benchmarks |
||||
|
||||
Compare with badger: |
||||
|
||||
```bash |
||||
# Badger benchmarks |
||||
go test -bench=. -benchmem ./pkg/database/... |
||||
|
||||
# Dgraph benchmarks |
||||
go test -bench=. -benchmem ./pkg/dgraph/... |
||||
``` |
||||
|
||||
## Related Documentation |
||||
|
||||
- [Main Testing Guide](../../scripts/DGRAPH_TESTING.md) |
||||
- [Implementation Status](../../DGRAPH_IMPLEMENTATION_STATUS.md) |
||||
- [Package README](README.md) |
||||
|
||||
## Contributing |
||||
|
||||
When adding new tests: |
||||
|
||||
1. **Mirror badger tests** - Ensure feature parity |
||||
2. **Use test helpers** - setupTestDB() and cleanupTestDB() |
||||
3. **Skip if unavailable** - Call skipIfDgraphNotAvailable(t) |
||||
4. **Clean up resources** - Always defer cleanupTestDB() |
||||
5. **Test chronologically** - Sort events by timestamp for addressable events |
||||
6. **Verify behavior** - Don't just check for no errors, verify correctness |
||||
@ -0,0 +1,190 @@
@@ -0,0 +1,190 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"context" |
||||
"fmt" |
||||
|
||||
"github.com/dgraph-io/dgo/v230/protos/api" |
||||
"next.orly.dev/pkg/database/indexes/types" |
||||
"next.orly.dev/pkg/encoders/event" |
||||
"next.orly.dev/pkg/encoders/hex" |
||||
) |
||||
|
||||
// DeleteEvent deletes an event by its ID
|
||||
func (d *D) DeleteEvent(c context.Context, eid []byte) error { |
||||
idStr := hex.Enc(eid) |
||||
|
||||
// Find the event's UID
|
||||
query := fmt.Sprintf(`{ |
||||
event(func: eq(event.id, %q)) { |
||||
uid |
||||
} |
||||
}`, idStr) |
||||
|
||||
resp, err := d.Query(c, query) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to find event for deletion: %w", err) |
||||
} |
||||
|
||||
// Parse UID
|
||||
var result struct { |
||||
Event []struct { |
||||
UID string `json:"uid"` |
||||
} `json:"event"` |
||||
} |
||||
|
||||
if err = unmarshalJSON(resp.Json, &result); err != nil { |
||||
return err |
||||
} |
||||
|
||||
if len(result.Event) == 0 { |
||||
return nil // Event doesn't exist
|
||||
} |
||||
|
||||
// Delete the event node
|
||||
mutation := &api.Mutation{ |
||||
DelNquads: []byte(fmt.Sprintf("<%s> * * .", result.Event[0].UID)), |
||||
CommitNow: true, |
||||
} |
||||
|
||||
if _, err = d.Mutate(c, mutation); err != nil { |
||||
return fmt.Errorf("failed to delete event: %w", err) |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
|
||||
// DeleteEventBySerial deletes an event by its serial number
|
||||
func (d *D) DeleteEventBySerial(c context.Context, ser *types.Uint40, ev *event.E) error { |
||||
serial := ser.Get() |
||||
|
||||
// Find the event's UID
|
||||
query := fmt.Sprintf(`{ |
||||
event(func: eq(event.serial, %d)) { |
||||
uid |
||||
} |
||||
}`, serial) |
||||
|
||||
resp, err := d.Query(c, query) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to find event for deletion: %w", err) |
||||
} |
||||
|
||||
// Parse UID
|
||||
var result struct { |
||||
Event []struct { |
||||
UID string `json:"uid"` |
||||
} `json:"event"` |
||||
} |
||||
|
||||
if err = unmarshalJSON(resp.Json, &result); err != nil { |
||||
return err |
||||
} |
||||
|
||||
if len(result.Event) == 0 { |
||||
return nil // Event doesn't exist
|
||||
} |
||||
|
||||
// Delete the event node
|
||||
mutation := &api.Mutation{ |
||||
DelNquads: []byte(fmt.Sprintf("<%s> * * .", result.Event[0].UID)), |
||||
CommitNow: true, |
||||
} |
||||
|
||||
if _, err = d.Mutate(c, mutation); err != nil { |
||||
return fmt.Errorf("failed to delete event: %w", err) |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
|
||||
// DeleteExpired removes events that have passed their expiration time
|
||||
func (d *D) DeleteExpired() { |
||||
// Query for events with expiration tags
|
||||
// This is a stub - full implementation would:
|
||||
// 1. Find events with "expiration" tag
|
||||
// 2. Check if current time > expiration time
|
||||
// 3. Delete those events
|
||||
} |
||||
|
||||
// ProcessDelete processes a kind 5 deletion event
|
||||
func (d *D) ProcessDelete(ev *event.E, admins [][]byte) (err error) { |
||||
if ev.Kind != 5 { |
||||
return fmt.Errorf("event is not a deletion event (kind 5)") |
||||
} |
||||
|
||||
// Extract event IDs to delete from tags
|
||||
for _, tag := range *ev.Tags { |
||||
if len(tag.T) >= 2 && string(tag.T[0]) == "e" { |
||||
eventID := tag.T[1] |
||||
|
||||
// Verify the deletion is authorized (author must match or be admin)
|
||||
if err = d.CheckForDeleted(ev, admins); err != nil { |
||||
continue |
||||
} |
||||
|
||||
// Delete the event
|
||||
if err = d.DeleteEvent(context.Background(), eventID); err != nil { |
||||
// Log error but continue with other deletions
|
||||
d.Logger.Errorf("failed to delete event %s: %v", hex.Enc(eventID), err) |
||||
} |
||||
} |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
|
||||
// CheckForDeleted checks if an event has been deleted
|
||||
func (d *D) CheckForDeleted(ev *event.E, admins [][]byte) (err error) { |
||||
// Query for delete events (kind 5) that reference this event
|
||||
evID := hex.Enc(ev.ID[:]) |
||||
|
||||
query := fmt.Sprintf(`{ |
||||
deletes(func: eq(event.kind, 5)) @filter(eq(event.pubkey, %q)) { |
||||
uid |
||||
event.pubkey |
||||
references @filter(eq(event.id, %q)) { |
||||
event.id |
||||
} |
||||
} |
||||
}`, hex.Enc(ev.Pubkey), evID) |
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to check for deletions: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Deletes []struct { |
||||
UID string `json:"uid"` |
||||
Pubkey string `json:"event.pubkey"` |
||||
References []struct { |
||||
ID string `json:"event.id"` |
||||
} `json:"references"` |
||||
} `json:"deletes"` |
||||
} |
||||
|
||||
if err = unmarshalJSON(resp.Json, &result); err != nil { |
||||
return err |
||||
} |
||||
|
||||
// Check if any delete events reference this event
|
||||
for _, del := range result.Deletes { |
||||
if len(del.References) > 0 { |
||||
// Check if deletion is from the author or an admin
|
||||
delPubkey, _ := hex.Dec(del.Pubkey) |
||||
if string(delPubkey) == string(ev.Pubkey) { |
||||
return fmt.Errorf("event has been deleted by author") |
||||
} |
||||
|
||||
// Check admins
|
||||
for _, admin := range admins { |
||||
if string(delPubkey) == string(admin) { |
||||
return fmt.Errorf("event has been deleted by admin") |
||||
} |
||||
} |
||||
} |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
@ -0,0 +1,285 @@
@@ -0,0 +1,285 @@
|
||||
// Package dgraph provides a Dgraph-based implementation of the database interface.
|
||||
// This is a simplified implementation for testing - full dgraph integration to be completed later.
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"context" |
||||
"fmt" |
||||
"os" |
||||
"path/filepath" |
||||
|
||||
"github.com/dgraph-io/badger/v4" |
||||
"github.com/dgraph-io/dgo/v230" |
||||
"github.com/dgraph-io/dgo/v230/protos/api" |
||||
"google.golang.org/grpc" |
||||
"google.golang.org/grpc/credentials/insecure" |
||||
"lol.mleku.dev" |
||||
"lol.mleku.dev/chk" |
||||
"next.orly.dev/pkg/database" |
||||
"next.orly.dev/pkg/utils/apputil" |
||||
) |
||||
|
||||
// D implements the database.Database interface using Dgraph as the storage backend
|
||||
type D struct { |
||||
ctx context.Context |
||||
cancel context.CancelFunc |
||||
dataDir string |
||||
Logger *logger |
||||
|
||||
// Dgraph client connection
|
||||
client *dgo.Dgraph |
||||
conn *grpc.ClientConn |
||||
|
||||
// Fallback badger storage for metadata
|
||||
pstore *badger.DB |
||||
|
||||
// Configuration
|
||||
dgraphURL string |
||||
enableGraphQL bool |
||||
enableIntrospection bool |
||||
|
||||
ready chan struct{} // Closed when database is ready to serve requests
|
||||
} |
||||
|
||||
// Ensure D implements database.Database interface at compile time
|
||||
var _ database.Database = (*D)(nil) |
||||
|
||||
// init registers the dgraph database factory
|
||||
func init() { |
||||
database.RegisterDgraphFactory(func( |
||||
ctx context.Context, |
||||
cancel context.CancelFunc, |
||||
dataDir string, |
||||
logLevel string, |
||||
) (database.Database, error) { |
||||
return New(ctx, cancel, dataDir, logLevel) |
||||
}) |
||||
} |
||||
|
||||
// Config holds configuration options for the Dgraph database
|
||||
type Config struct { |
||||
DataDir string |
||||
LogLevel string |
||||
DgraphURL string // Dgraph gRPC endpoint (e.g., "localhost:9080")
|
||||
EnableGraphQL bool |
||||
EnableIntrospection bool |
||||
} |
||||
|
||||
// New creates a new Dgraph-based database instance
|
||||
func New( |
||||
ctx context.Context, cancel context.CancelFunc, dataDir, logLevel string, |
||||
) ( |
||||
d *D, err error, |
||||
) { |
||||
// Get dgraph URL from environment, default to localhost
|
||||
dgraphURL := os.Getenv("ORLY_DGRAPH_URL") |
||||
if dgraphURL == "" { |
||||
dgraphURL = "localhost:9080" |
||||
} |
||||
|
||||
d = &D{ |
||||
ctx: ctx, |
||||
cancel: cancel, |
||||
dataDir: dataDir, |
||||
Logger: NewLogger(lol.GetLogLevel(logLevel), dataDir), |
||||
dgraphURL: dgraphURL, |
||||
enableGraphQL: false, |
||||
enableIntrospection: false, |
||||
ready: make(chan struct{}), |
||||
} |
||||
|
||||
// Ensure the data directory exists
|
||||
if err = os.MkdirAll(dataDir, 0755); chk.E(err) { |
||||
return |
||||
} |
||||
|
||||
// Ensure directory structure
|
||||
dummyFile := filepath.Join(dataDir, "dummy.sst") |
||||
if err = apputil.EnsureDir(dummyFile); chk.E(err) { |
||||
return |
||||
} |
||||
|
||||
// Initialize dgraph client connection
|
||||
if err = d.initDgraphClient(); chk.E(err) { |
||||
return |
||||
} |
||||
|
||||
// Initialize badger for metadata storage
|
||||
if err = d.initStorage(); chk.E(err) { |
||||
return |
||||
} |
||||
|
||||
// Apply Nostr schema to dgraph
|
||||
if err = d.applySchema(ctx); chk.E(err) { |
||||
return |
||||
} |
||||
|
||||
// Initialize serial counter
|
||||
if err = d.initSerialCounter(); chk.E(err) { |
||||
return |
||||
} |
||||
|
||||
// Start warmup goroutine to signal when database is ready
|
||||
go d.warmup() |
||||
|
||||
// Setup shutdown handler
|
||||
go func() { |
||||
<-d.ctx.Done() |
||||
d.cancel() |
||||
if d.conn != nil { |
||||
d.conn.Close() |
||||
} |
||||
if d.pstore != nil { |
||||
d.pstore.Close() |
||||
} |
||||
}() |
||||
|
||||
return |
||||
} |
||||
|
||||
// initDgraphClient establishes connection to dgraph server
|
||||
func (d *D) initDgraphClient() error { |
||||
d.Logger.Infof("connecting to dgraph at %s", d.dgraphURL) |
||||
|
||||
// Establish gRPC connection
|
||||
conn, err := grpc.Dial(d.dgraphURL, grpc.WithTransportCredentials(insecure.NewCredentials())) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to connect to dgraph at %s: %w", d.dgraphURL, err) |
||||
} |
||||
|
||||
d.conn = conn |
||||
d.client = dgo.NewDgraphClient(api.NewDgraphClient(conn)) |
||||
|
||||
d.Logger.Infof("successfully connected to dgraph") |
||||
return nil |
||||
} |
||||
|
||||
// initStorage opens Badger database for metadata storage
|
||||
func (d *D) initStorage() error { |
||||
metadataDir := filepath.Join(d.dataDir, "metadata") |
||||
|
||||
if err := os.MkdirAll(metadataDir, 0755); err != nil { |
||||
return fmt.Errorf("failed to create metadata directory: %w", err) |
||||
} |
||||
|
||||
opts := badger.DefaultOptions(metadataDir) |
||||
|
||||
var err error |
||||
d.pstore, err = badger.Open(opts) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to open badger metadata store: %w", err) |
||||
} |
||||
|
||||
d.Logger.Infof("metadata storage initialized") |
||||
return nil |
||||
} |
||||
|
||||
// Query executes a DQL query against dgraph
|
||||
func (d *D) Query(ctx context.Context, query string) (*api.Response, error) { |
||||
txn := d.client.NewReadOnlyTxn() |
||||
defer txn.Discard(ctx) |
||||
|
||||
resp, err := txn.Query(ctx, query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("dgraph query failed: %w", err) |
||||
} |
||||
|
||||
return resp, nil |
||||
} |
||||
|
||||
// Mutate executes a mutation against dgraph
|
||||
func (d *D) Mutate(ctx context.Context, mutation *api.Mutation) (*api.Response, error) { |
||||
txn := d.client.NewTxn() |
||||
defer txn.Discard(ctx) |
||||
|
||||
resp, err := txn.Mutate(ctx, mutation) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("dgraph mutation failed: %w", err) |
||||
} |
||||
|
||||
if err := txn.Commit(ctx); err != nil { |
||||
return nil, fmt.Errorf("dgraph commit failed: %w", err) |
||||
} |
||||
|
||||
return resp, nil |
||||
} |
||||
|
||||
// Path returns the data directory path
|
||||
func (d *D) Path() string { return d.dataDir } |
||||
|
||||
// Init initializes the database with a given path (no-op, path set in New)
|
||||
func (d *D) Init(path string) (err error) { |
||||
// Path already set in New()
|
||||
return nil |
||||
} |
||||
|
||||
// Sync flushes pending writes
|
||||
func (d *D) Sync() (err error) { |
||||
if d.pstore != nil { |
||||
return d.pstore.Sync() |
||||
} |
||||
return nil |
||||
} |
||||
|
||||
// Close closes the database
|
||||
func (d *D) Close() (err error) { |
||||
d.cancel() |
||||
if d.conn != nil { |
||||
if e := d.conn.Close(); e != nil { |
||||
err = e |
||||
} |
||||
} |
||||
if d.pstore != nil { |
||||
if e := d.pstore.Close(); e != nil && err == nil { |
||||
err = e |
||||
} |
||||
} |
||||
return |
||||
} |
||||
|
||||
// Wipe removes all data
|
||||
func (d *D) Wipe() (err error) { |
||||
if d.pstore != nil { |
||||
if err = d.pstore.Close(); chk.E(err) { |
||||
return |
||||
} |
||||
} |
||||
if err = os.RemoveAll(d.dataDir); chk.E(err) { |
||||
return |
||||
} |
||||
return d.initStorage() |
||||
} |
||||
|
||||
// SetLogLevel sets the logging level
|
||||
func (d *D) SetLogLevel(level string) { |
||||
// d.Logger.SetLevel(lol.GetLogLevel(level))
|
||||
} |
||||
|
||||
// EventIdsBySerial retrieves event IDs by serial range (stub)
|
||||
func (d *D) EventIdsBySerial(start uint64, count int) ( |
||||
evs []uint64, err error, |
||||
) { |
||||
err = fmt.Errorf("not implemented") |
||||
return |
||||
} |
||||
|
||||
// RunMigrations runs database migrations (no-op for dgraph)
|
||||
func (d *D) RunMigrations() { |
||||
// No-op for dgraph
|
||||
} |
||||
|
||||
// Ready returns a channel that closes when the database is ready to serve requests.
|
||||
// This allows callers to wait for database warmup to complete.
|
||||
func (d *D) Ready() <-chan struct{} { |
||||
return d.ready |
||||
} |
||||
|
||||
// warmup performs database warmup operations and closes the ready channel when complete.
|
||||
// For Dgraph, warmup ensures the connection is healthy and schema is applied.
|
||||
func (d *D) warmup() { |
||||
defer close(d.ready) |
||||
|
||||
// Dgraph connection and schema are already verified during initialization
|
||||
// Just give a brief moment for any background processes to settle
|
||||
d.Logger.Infof("dgraph database warmup complete, ready to serve requests") |
||||
} |
||||
@ -0,0 +1,270 @@
@@ -0,0 +1,270 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"context" |
||||
"encoding/json" |
||||
"fmt" |
||||
|
||||
"next.orly.dev/pkg/database" |
||||
"next.orly.dev/pkg/database/indexes/types" |
||||
"next.orly.dev/pkg/encoders/event" |
||||
"next.orly.dev/pkg/encoders/hex" |
||||
"next.orly.dev/pkg/encoders/tag" |
||||
"next.orly.dev/pkg/interfaces/store" |
||||
) |
||||
|
||||
// FetchEventBySerial retrieves an event by its serial number
|
||||
func (d *D) FetchEventBySerial(ser *types.Uint40) (ev *event.E, err error) { |
||||
serial := ser.Get() |
||||
|
||||
query := fmt.Sprintf(`{ |
||||
event(func: eq(event.serial, %d)) { |
||||
event.id |
||||
event.kind |
||||
event.created_at |
||||
event.content |
||||
event.sig |
||||
event.pubkey |
||||
event.tags |
||||
} |
||||
}`, serial) |
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to fetch event by serial: %w", err) |
||||
} |
||||
|
||||
evs, err := d.parseEventsFromResponse(resp.Json) |
||||
if err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
if len(evs) == 0 { |
||||
return nil, fmt.Errorf("event not found") |
||||
} |
||||
|
||||
return evs[0], nil |
||||
} |
||||
|
||||
// FetchEventsBySerials retrieves multiple events by their serial numbers
|
||||
func (d *D) FetchEventsBySerials(serials []*types.Uint40) ( |
||||
events map[uint64]*event.E, err error, |
||||
) { |
||||
if len(serials) == 0 { |
||||
return make(map[uint64]*event.E), nil |
||||
} |
||||
|
||||
// Build query for multiple serials
|
||||
serialStrs := make([]string, len(serials)) |
||||
for i, ser := range serials { |
||||
serialStrs[i] = fmt.Sprintf("%d", ser.Get()) |
||||
} |
||||
|
||||
// Use uid() function for efficient multi-get
|
||||
query := fmt.Sprintf(`{ |
||||
events(func: uid(%s)) { |
||||
event.id |
||||
event.kind |
||||
event.created_at |
||||
event.content |
||||
event.sig |
||||
event.pubkey |
||||
event.tags |
||||
event.serial |
||||
} |
||||
}`, serialStrs[0]) // Simplified - in production you'd handle multiple UIDs properly
|
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to fetch events by serials: %w", err) |
||||
} |
||||
|
||||
evs, err := d.parseEventsFromResponse(resp.Json) |
||||
if err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
// Map events by serial
|
||||
events = make(map[uint64]*event.E) |
||||
for i, ser := range serials { |
||||
if i < len(evs) { |
||||
events[ser.Get()] = evs[i] |
||||
} |
||||
} |
||||
|
||||
return events, nil |
||||
} |
||||
|
||||
// GetSerialById retrieves the serial number for an event ID
|
||||
func (d *D) GetSerialById(id []byte) (ser *types.Uint40, err error) { |
||||
idStr := hex.Enc(id) |
||||
|
||||
query := fmt.Sprintf(`{ |
||||
event(func: eq(event.id, %q)) { |
||||
event.serial |
||||
} |
||||
}`, idStr) |
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to get serial by ID: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Event []struct { |
||||
Serial int64 `json:"event.serial"` |
||||
} `json:"event"` |
||||
} |
||||
|
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
if len(result.Event) == 0 { |
||||
return nil, fmt.Errorf("event not found") |
||||
} |
||||
|
||||
ser = &types.Uint40{} |
||||
ser.Set(uint64(result.Event[0].Serial)) |
||||
|
||||
return ser, nil |
||||
} |
||||
|
||||
// GetSerialsByIds retrieves serial numbers for multiple event IDs
|
||||
func (d *D) GetSerialsByIds(ids *tag.T) ( |
||||
serials map[string]*types.Uint40, err error, |
||||
) { |
||||
serials = make(map[string]*types.Uint40) |
||||
|
||||
if len(ids.T) == 0 { |
||||
return serials, nil |
||||
} |
||||
|
||||
// Query each ID individually (simplified implementation)
|
||||
for _, id := range ids.T { |
||||
if len(id) >= 2 { |
||||
idStr := string(id[1]) |
||||
serial, err := d.GetSerialById([]byte(idStr)) |
||||
if err == nil { |
||||
serials[idStr] = serial |
||||
} |
||||
} |
||||
} |
||||
|
||||
return serials, nil |
||||
} |
||||
|
||||
// GetSerialsByIdsWithFilter retrieves serials with a filter function
|
||||
func (d *D) GetSerialsByIdsWithFilter( |
||||
ids *tag.T, fn func(ev *event.E, ser *types.Uint40) bool, |
||||
) (serials map[string]*types.Uint40, err error) { |
||||
serials = make(map[string]*types.Uint40) |
||||
|
||||
if fn == nil { |
||||
// No filter, just return all
|
||||
return d.GetSerialsByIds(ids) |
||||
} |
||||
|
||||
// With filter, need to fetch events
|
||||
for _, id := range ids.T { |
||||
if len(id) > 0 { |
||||
serial, err := d.GetSerialById(id) |
||||
if err != nil { |
||||
continue |
||||
} |
||||
|
||||
ev, err := d.FetchEventBySerial(serial) |
||||
if err != nil { |
||||
continue |
||||
} |
||||
|
||||
if fn(ev, serial) { |
||||
serials[string(id)] = serial |
||||
} |
||||
} |
||||
} |
||||
|
||||
return serials, nil |
||||
} |
||||
|
||||
// GetSerialsByRange retrieves serials within a range
|
||||
func (d *D) GetSerialsByRange(idx database.Range) ( |
||||
serials types.Uint40s, err error, |
||||
) { |
||||
// This would need to be implemented based on how ranges are defined
|
||||
// For now, returning not implemented
|
||||
err = fmt.Errorf("not implemented") |
||||
return |
||||
} |
||||
|
||||
// GetFullIdPubkeyBySerial retrieves ID and pubkey for a serial number
|
||||
func (d *D) GetFullIdPubkeyBySerial(ser *types.Uint40) ( |
||||
fidpk *store.IdPkTs, err error, |
||||
) { |
||||
serial := ser.Get() |
||||
|
||||
query := fmt.Sprintf(`{ |
||||
event(func: eq(event.serial, %d)) { |
||||
event.id |
||||
event.pubkey |
||||
event.created_at |
||||
} |
||||
}`, serial) |
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to get ID and pubkey by serial: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Event []struct { |
||||
ID string `json:"event.id"` |
||||
Pubkey string `json:"event.pubkey"` |
||||
CreatedAt int64 `json:"event.created_at"` |
||||
} `json:"event"` |
||||
} |
||||
|
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
if len(result.Event) == 0 { |
||||
return nil, fmt.Errorf("event not found") |
||||
} |
||||
|
||||
id, err := hex.Dec(result.Event[0].ID) |
||||
if err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
pubkey, err := hex.Dec(result.Event[0].Pubkey) |
||||
if err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
fidpk = &store.IdPkTs{ |
||||
Id: id, |
||||
Pub: pubkey, |
||||
Ts: result.Event[0].CreatedAt, |
||||
Ser: serial, |
||||
} |
||||
|
||||
return fidpk, nil |
||||
} |
||||
|
||||
// GetFullIdPubkeyBySerials retrieves IDs and pubkeys for multiple serials
|
||||
func (d *D) GetFullIdPubkeyBySerials(sers []*types.Uint40) ( |
||||
fidpks []*store.IdPkTs, err error, |
||||
) { |
||||
fidpks = make([]*store.IdPkTs, 0, len(sers)) |
||||
|
||||
for _, ser := range sers { |
||||
fidpk, err := d.GetFullIdPubkeyBySerial(ser) |
||||
if err != nil { |
||||
continue // Skip errors, continue with others
|
||||
} |
||||
fidpks = append(fidpks, fidpk) |
||||
} |
||||
|
||||
return fidpks, nil |
||||
} |
||||
@ -0,0 +1,144 @@
@@ -0,0 +1,144 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"bufio" |
||||
"bytes" |
||||
"context" |
||||
"net" |
||||
"os" |
||||
"sort" |
||||
"testing" |
||||
"time" |
||||
|
||||
"lol.mleku.dev/chk" |
||||
"next.orly.dev/pkg/encoders/event" |
||||
"next.orly.dev/pkg/encoders/event/examples" |
||||
) |
||||
|
||||
// isDgraphAvailable checks if a dgraph server is running
|
||||
func isDgraphAvailable() bool { |
||||
dgraphURL := os.Getenv("ORLY_DGRAPH_URL") |
||||
if dgraphURL == "" { |
||||
dgraphURL = "localhost:9080" |
||||
} |
||||
|
||||
conn, err := net.DialTimeout("tcp", dgraphURL, 2*time.Second) |
||||
if err != nil { |
||||
return false |
||||
} |
||||
conn.Close() |
||||
return true |
||||
} |
||||
|
||||
// skipIfDgraphNotAvailable skips the test if dgraph is not available
|
||||
func skipIfDgraphNotAvailable(t *testing.T) { |
||||
if !isDgraphAvailable() { |
||||
dgraphURL := os.Getenv("ORLY_DGRAPH_URL") |
||||
if dgraphURL == "" { |
||||
dgraphURL = "localhost:9080" |
||||
} |
||||
t.Skipf("Dgraph server not available at %s. Start with: docker run -p 9080:9080 dgraph/standalone:latest", dgraphURL) |
||||
} |
||||
} |
||||
|
||||
// setupTestDB creates a new test dgraph database and loads example events
|
||||
func setupTestDB(t *testing.T) ( |
||||
*D, []*event.E, context.Context, context.CancelFunc, string, |
||||
) { |
||||
skipIfDgraphNotAvailable(t) |
||||
|
||||
// Create a temporary directory for metadata storage
|
||||
tempDir, err := os.MkdirTemp("", "test-dgraph-*") |
||||
if err != nil { |
||||
t.Fatalf("Failed to create temporary directory: %v", err) |
||||
} |
||||
|
||||
// Create a context and cancel function for the database
|
||||
ctx, cancel := context.WithCancel(context.Background()) |
||||
|
||||
// Initialize the dgraph database
|
||||
db, err := New(ctx, cancel, tempDir, "info") |
||||
if err != nil { |
||||
cancel() |
||||
os.RemoveAll(tempDir) |
||||
t.Fatalf("Failed to create dgraph database: %v", err) |
||||
} |
||||
|
||||
// Drop all data to start fresh
|
||||
if err := db.dropAll(ctx); err != nil { |
||||
db.Close() |
||||
cancel() |
||||
os.RemoveAll(tempDir) |
||||
t.Fatalf("Failed to drop all data: %v", err) |
||||
} |
||||
|
||||
// Create a scanner to read events from examples.Cache
|
||||
scanner := bufio.NewScanner(bytes.NewBuffer(examples.Cache)) |
||||
scanner.Buffer(make([]byte, 0, 1_000_000_000), 1_000_000_000) |
||||
|
||||
var events []*event.E |
||||
|
||||
// First, collect all events from examples.Cache
|
||||
for scanner.Scan() { |
||||
chk.E(scanner.Err()) |
||||
b := scanner.Bytes() |
||||
ev := event.New() |
||||
|
||||
// Unmarshal the event
|
||||
if _, err = ev.Unmarshal(b); chk.E(err) { |
||||
ev.Free() |
||||
db.Close() |
||||
cancel() |
||||
os.RemoveAll(tempDir) |
||||
t.Fatal(err) |
||||
} |
||||
|
||||
events = append(events, ev) |
||||
} |
||||
|
||||
// Check for scanner errors
|
||||
if err = scanner.Err(); err != nil { |
||||
db.Close() |
||||
cancel() |
||||
os.RemoveAll(tempDir) |
||||
t.Fatalf("Scanner error: %v", err) |
||||
} |
||||
|
||||
// Sort events by CreatedAt to ensure addressable events are processed in chronological order
|
||||
sort.Slice(events, func(i, j int) bool { |
||||
return events[i].CreatedAt < events[j].CreatedAt |
||||
}) |
||||
|
||||
// Count the number of events processed
|
||||
eventCount := 0 |
||||
|
||||
// Now process each event in chronological order
|
||||
for _, ev := range events { |
||||
// Save the event to the database
|
||||
if _, err = db.SaveEvent(ctx, ev); err != nil { |
||||
db.Close() |
||||
cancel() |
||||
os.RemoveAll(tempDir) |
||||
t.Fatalf("Failed to save event #%d: %v", eventCount+1, err) |
||||
} |
||||
|
||||
eventCount++ |
||||
} |
||||
|
||||
t.Logf("Successfully saved %d events to dgraph database", eventCount) |
||||
|
||||
return db, events, ctx, cancel, tempDir |
||||
} |
||||
|
||||
// cleanupTestDB cleans up the test database
|
||||
func cleanupTestDB(t *testing.T, db *D, cancel context.CancelFunc, tempDir string) { |
||||
if db != nil { |
||||
db.Close() |
||||
} |
||||
if cancel != nil { |
||||
cancel() |
||||
} |
||||
if tempDir != "" { |
||||
os.RemoveAll(tempDir) |
||||
} |
||||
} |
||||
@ -0,0 +1,44 @@
@@ -0,0 +1,44 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"fmt" |
||||
|
||||
"next.orly.dev/pkg/crypto/keys" |
||||
) |
||||
|
||||
// Relay identity methods
|
||||
// We use the marker system to store the relay's private key
|
||||
|
||||
const relayIdentityMarkerKey = "relay_identity_secret" |
||||
|
||||
// GetRelayIdentitySecret retrieves the relay's identity secret key
|
||||
func (d *D) GetRelayIdentitySecret() (skb []byte, err error) { |
||||
return d.GetMarker(relayIdentityMarkerKey) |
||||
} |
||||
|
||||
// SetRelayIdentitySecret sets the relay's identity secret key
|
||||
func (d *D) SetRelayIdentitySecret(skb []byte) error { |
||||
return d.SetMarker(relayIdentityMarkerKey, skb) |
||||
} |
||||
|
||||
// GetOrCreateRelayIdentitySecret retrieves or creates the relay identity
|
||||
func (d *D) GetOrCreateRelayIdentitySecret() (skb []byte, err error) { |
||||
skb, err = d.GetRelayIdentitySecret() |
||||
if err == nil { |
||||
return skb, nil |
||||
} |
||||
|
||||
// Generate new identity
|
||||
skb, err = keys.GenerateSecretKey() |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to generate identity: %w", err) |
||||
} |
||||
|
||||
// Store it
|
||||
if err = d.SetRelayIdentitySecret(skb); err != nil { |
||||
return nil, fmt.Errorf("failed to store identity: %w", err) |
||||
} |
||||
|
||||
d.Logger.Infof("generated new relay identity") |
||||
return skb, nil |
||||
} |
||||
@ -0,0 +1,97 @@
@@ -0,0 +1,97 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"bufio" |
||||
"context" |
||||
"encoding/json" |
||||
"fmt" |
||||
"io" |
||||
|
||||
"next.orly.dev/pkg/encoders/event" |
||||
) |
||||
|
||||
// Import imports events from a reader (JSONL format)
|
||||
func (d *D) Import(rr io.Reader) { |
||||
d.ImportEventsFromReader(context.Background(), rr) |
||||
} |
||||
|
||||
// Export exports events to a writer (JSONL format)
|
||||
func (d *D) Export(c context.Context, w io.Writer, pubkeys ...[]byte) { |
||||
// Query all events or events for specific pubkeys
|
||||
// Write as JSONL
|
||||
|
||||
// Stub implementation
|
||||
fmt.Fprintf(w, "# Export not yet implemented for dgraph\n") |
||||
} |
||||
|
||||
// ImportEventsFromReader imports events from a reader
|
||||
func (d *D) ImportEventsFromReader(ctx context.Context, rr io.Reader) error { |
||||
scanner := bufio.NewScanner(rr) |
||||
scanner.Buffer(make([]byte, 1024*1024), 10*1024*1024) // 10MB max line size
|
||||
|
||||
count := 0 |
||||
for scanner.Scan() { |
||||
line := scanner.Bytes() |
||||
if len(line) == 0 { |
||||
continue |
||||
} |
||||
|
||||
// Skip comments
|
||||
if line[0] == '#' { |
||||
continue |
||||
} |
||||
|
||||
// Parse event
|
||||
ev := &event.E{} |
||||
if err := json.Unmarshal(line, ev); err != nil { |
||||
d.Logger.Warningf("failed to parse event: %v", err) |
||||
continue |
||||
} |
||||
|
||||
// Save event
|
||||
if _, err := d.SaveEvent(ctx, ev); err != nil { |
||||
d.Logger.Warningf("failed to import event: %v", err) |
||||
continue |
||||
} |
||||
|
||||
count++ |
||||
if count%1000 == 0 { |
||||
d.Logger.Infof("imported %d events", count) |
||||
} |
||||
} |
||||
|
||||
if err := scanner.Err(); err != nil { |
||||
return fmt.Errorf("scanner error: %w", err) |
||||
} |
||||
|
||||
d.Logger.Infof("import complete: %d events", count) |
||||
return nil |
||||
} |
||||
|
||||
// ImportEventsFromStrings imports events from JSON strings
|
||||
func (d *D) ImportEventsFromStrings( |
||||
ctx context.Context, |
||||
eventJSONs []string, |
||||
policyManager interface{ CheckPolicy(action string, ev *event.E, pubkey []byte, remote string) (bool, error) }, |
||||
) error { |
||||
for _, eventJSON := range eventJSONs { |
||||
ev := &event.E{} |
||||
if err := json.Unmarshal([]byte(eventJSON), ev); err != nil { |
||||
continue |
||||
} |
||||
|
||||
// Check policy if manager is provided
|
||||
if policyManager != nil { |
||||
if allowed, err := policyManager.CheckPolicy("write", ev, ev.Pubkey[:], "import"); err != nil || !allowed { |
||||
continue |
||||
} |
||||
} |
||||
|
||||
// Save event
|
||||
if _, err := d.SaveEvent(ctx, ev); err != nil { |
||||
d.Logger.Warningf("failed to import event: %v", err) |
||||
} |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
@ -0,0 +1,68 @@
@@ -0,0 +1,68 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"fmt" |
||||
"runtime" |
||||
"strings" |
||||
|
||||
"go.uber.org/atomic" |
||||
"lol.mleku.dev" |
||||
"lol.mleku.dev/log" |
||||
) |
||||
|
||||
// NewLogger creates a new dgraph logger.
|
||||
func NewLogger(logLevel int, label string) (l *logger) { |
||||
l = &logger{Label: label} |
||||
l.Level.Store(int32(logLevel)) |
||||
return |
||||
} |
||||
|
||||
type logger struct { |
||||
Level atomic.Int32 |
||||
Label string |
||||
} |
||||
|
||||
// SetLogLevel atomically adjusts the log level to the given log level code.
|
||||
func (l *logger) SetLogLevel(level int) { |
||||
l.Level.Store(int32(level)) |
||||
} |
||||
|
||||
// Errorf is a log printer for this level of message.
|
||||
func (l *logger) Errorf(s string, i ...interface{}) { |
||||
if l.Level.Load() >= lol.Error { |
||||
s = l.Label + ": " + s |
||||
txt := fmt.Sprintf(s, i...) |
||||
_, file, line, _ := runtime.Caller(2) |
||||
log.E.F("%s\n%s:%d", strings.TrimSpace(txt), file, line) |
||||
} |
||||
} |
||||
|
||||
// Warningf is a log printer for this level of message.
|
||||
func (l *logger) Warningf(s string, i ...interface{}) { |
||||
if l.Level.Load() >= lol.Warn { |
||||
s = l.Label + ": " + s |
||||
txt := fmt.Sprintf(s, i...) |
||||
_, file, line, _ := runtime.Caller(2) |
||||
log.W.F("%s\n%s:%d", strings.TrimSpace(txt), file, line) |
||||
} |
||||
} |
||||
|
||||
// Infof is a log printer for this level of message.
|
||||
func (l *logger) Infof(s string, i ...interface{}) { |
||||
if l.Level.Load() >= lol.Info { |
||||
s = l.Label + ": " + s |
||||
txt := fmt.Sprintf(s, i...) |
||||
_, file, line, _ := runtime.Caller(2) |
||||
log.I.F("%s\n%s:%d", strings.TrimSpace(txt), file, line) |
||||
} |
||||
} |
||||
|
||||
// Debugf is a log printer for this level of message.
|
||||
func (l *logger) Debugf(s string, i ...interface{}) { |
||||
if l.Level.Load() >= lol.Debug { |
||||
s = l.Label + ": " + s |
||||
txt := fmt.Sprintf(s, i...) |
||||
_, file, line, _ := runtime.Caller(2) |
||||
log.D.F("%s\n%s:%d", strings.TrimSpace(txt), file, line) |
||||
} |
||||
} |
||||
@ -0,0 +1,120 @@
@@ -0,0 +1,120 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"context" |
||||
"encoding/json" |
||||
"fmt" |
||||
|
||||
"github.com/dgraph-io/dgo/v230/protos/api" |
||||
"next.orly.dev/pkg/encoders/hex" |
||||
) |
||||
|
||||
// Markers provide metadata key-value storage using Dgraph predicates
|
||||
// We store markers as special nodes with type "Marker"
|
||||
|
||||
// SetMarker sets a metadata marker
|
||||
func (d *D) SetMarker(key string, value []byte) error { |
||||
// Create or update a marker node
|
||||
markerID := "marker_" + key |
||||
valueHex := hex.Enc(value) |
||||
|
||||
nquads := fmt.Sprintf(` |
||||
_:%s <dgraph.type> "Marker" . |
||||
_:%s <marker.key> %q . |
||||
_:%s <marker.value> %q . |
||||
`, markerID, markerID, key, markerID, valueHex) |
||||
|
||||
mutation := &api.Mutation{ |
||||
SetNquads: []byte(nquads), |
||||
CommitNow: true, |
||||
} |
||||
|
||||
if _, err := d.Mutate(context.Background(), mutation); err != nil { |
||||
return fmt.Errorf("failed to set marker: %w", err) |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
|
||||
// GetMarker retrieves a metadata marker
|
||||
func (d *D) GetMarker(key string) (value []byte, err error) { |
||||
query := fmt.Sprintf(`{ |
||||
marker(func: eq(marker.key, %q)) { |
||||
marker.value |
||||
} |
||||
}`, key) |
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to get marker: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Marker []struct { |
||||
Value string `json:"marker.value"` |
||||
} `json:"marker"` |
||||
} |
||||
|
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return nil, fmt.Errorf("failed to parse marker response: %w", err) |
||||
} |
||||
|
||||
if len(result.Marker) == 0 { |
||||
return nil, fmt.Errorf("marker not found: %s", key) |
||||
} |
||||
|
||||
// Decode hex value
|
||||
value, err = hex.Dec(result.Marker[0].Value) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to decode marker value: %w", err) |
||||
} |
||||
|
||||
return value, nil |
||||
} |
||||
|
||||
// HasMarker checks if a marker exists
|
||||
func (d *D) HasMarker(key string) bool { |
||||
_, err := d.GetMarker(key) |
||||
return err == nil |
||||
} |
||||
|
||||
// DeleteMarker removes a metadata marker
|
||||
func (d *D) DeleteMarker(key string) error { |
||||
// Find the marker's UID
|
||||
query := fmt.Sprintf(`{ |
||||
marker(func: eq(marker.key, %q)) { |
||||
uid |
||||
} |
||||
}`, key) |
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to find marker: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Marker []struct { |
||||
UID string `json:"uid"` |
||||
} `json:"marker"` |
||||
} |
||||
|
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return fmt.Errorf("failed to parse marker query: %w", err) |
||||
} |
||||
|
||||
if len(result.Marker) == 0 { |
||||
return nil // Marker doesn't exist
|
||||
} |
||||
|
||||
// Delete the marker node
|
||||
mutation := &api.Mutation{ |
||||
DelNquads: []byte(fmt.Sprintf("<%s> * * .", result.Marker[0].UID)), |
||||
CommitNow: true, |
||||
} |
||||
|
||||
if _, err = d.Mutate(context.Background(), mutation); err != nil { |
||||
return fmt.Errorf("failed to delete marker: %w", err) |
||||
} |
||||
|
||||
return nil |
||||
} |
||||
@ -0,0 +1,211 @@
@@ -0,0 +1,211 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"encoding/json" |
||||
"fmt" |
||||
"time" |
||||
|
||||
"next.orly.dev/pkg/database" |
||||
"next.orly.dev/pkg/encoders/hex" |
||||
) |
||||
|
||||
// NIP-43 Invite-based ACL methods
|
||||
// Simplified implementation using marker-based storage
|
||||
|
||||
// AddNIP43Member adds a member using an invite code
|
||||
func (d *D) AddNIP43Member(pubkey []byte, inviteCode string) error { |
||||
key := "nip43_" + hex.Enc(pubkey) |
||||
|
||||
member := database.NIP43Membership{ |
||||
InviteCode: inviteCode, |
||||
AddedAt: time.Now(), |
||||
} |
||||
copy(member.Pubkey[:], pubkey) |
||||
|
||||
data, err := json.Marshal(member) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to marshal membership: %w", err) |
||||
} |
||||
|
||||
// Also add to members list
|
||||
if err := d.addToMembersList(pubkey); err != nil { |
||||
return err |
||||
} |
||||
|
||||
return d.SetMarker(key, data) |
||||
} |
||||
|
||||
// RemoveNIP43Member removes a member
|
||||
func (d *D) RemoveNIP43Member(pubkey []byte) error { |
||||
key := "nip43_" + hex.Enc(pubkey) |
||||
|
||||
// Remove from members list
|
||||
if err := d.removeFromMembersList(pubkey); err != nil { |
||||
return err |
||||
} |
||||
|
||||
return d.DeleteMarker(key) |
||||
} |
||||
|
||||
// IsNIP43Member checks if a pubkey is a member
|
||||
func (d *D) IsNIP43Member(pubkey []byte) (isMember bool, err error) { |
||||
_, err = d.GetNIP43Membership(pubkey) |
||||
return err == nil, nil |
||||
} |
||||
|
||||
// GetNIP43Membership retrieves membership information
|
||||
func (d *D) GetNIP43Membership(pubkey []byte) (*database.NIP43Membership, error) { |
||||
key := "nip43_" + hex.Enc(pubkey) |
||||
|
||||
data, err := d.GetMarker(key) |
||||
if err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
var member database.NIP43Membership |
||||
if err := json.Unmarshal(data, &member); err != nil { |
||||
return nil, fmt.Errorf("failed to unmarshal membership: %w", err) |
||||
} |
||||
|
||||
return &member, nil |
||||
} |
||||
|
||||
// GetAllNIP43Members retrieves all member pubkeys
|
||||
func (d *D) GetAllNIP43Members() ([][]byte, error) { |
||||
data, err := d.GetMarker("nip43_members_list") |
||||
if err != nil { |
||||
return nil, nil // No members = empty list
|
||||
} |
||||
|
||||
var members []string |
||||
if err := json.Unmarshal(data, &members); err != nil { |
||||
return nil, fmt.Errorf("failed to unmarshal members list: %w", err) |
||||
} |
||||
|
||||
result := make([][]byte, 0, len(members)) |
||||
for _, hexPubkey := range members { |
||||
pubkey, err := hex.Dec(hexPubkey) |
||||
if err != nil { |
||||
continue |
||||
} |
||||
result = append(result, pubkey) |
||||
} |
||||
|
||||
return result, nil |
||||
} |
||||
|
||||
// StoreInviteCode stores an invite code with expiration
|
||||
func (d *D) StoreInviteCode(code string, expiresAt time.Time) error { |
||||
key := "invite_" + code |
||||
|
||||
inviteData := map[string]interface{}{ |
||||
"code": code, |
||||
"expiresAt": expiresAt, |
||||
} |
||||
|
||||
data, err := json.Marshal(inviteData) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to marshal invite: %w", err) |
||||
} |
||||
|
||||
return d.SetMarker(key, data) |
||||
} |
||||
|
||||
// ValidateInviteCode checks if an invite code is valid
|
||||
func (d *D) ValidateInviteCode(code string) (valid bool, err error) { |
||||
key := "invite_" + code |
||||
|
||||
data, err := d.GetMarker(key) |
||||
if err != nil { |
||||
return false, nil // Code doesn't exist
|
||||
} |
||||
|
||||
var inviteData map[string]interface{} |
||||
if err := json.Unmarshal(data, &inviteData); err != nil { |
||||
return false, fmt.Errorf("failed to unmarshal invite: %w", err) |
||||
} |
||||
|
||||
// Check expiration
|
||||
if expiresStr, ok := inviteData["expiresAt"].(string); ok { |
||||
expiresAt, err := time.Parse(time.RFC3339, expiresStr) |
||||
if err == nil && time.Now().After(expiresAt) { |
||||
return false, nil // Expired
|
||||
} |
||||
} |
||||
|
||||
return true, nil |
||||
} |
||||
|
||||
// DeleteInviteCode removes an invite code
|
||||
func (d *D) DeleteInviteCode(code string) error { |
||||
key := "invite_" + code |
||||
return d.DeleteMarker(key) |
||||
} |
||||
|
||||
// PublishNIP43MembershipEvent publishes a membership event
|
||||
func (d *D) PublishNIP43MembershipEvent(kind int, pubkey []byte) error { |
||||
// This would require publishing an actual Nostr event
|
||||
// For now, just log it
|
||||
d.Logger.Infof("would publish NIP-43 event kind %d for %s", kind, hex.Enc(pubkey)) |
||||
return nil |
||||
} |
||||
|
||||
// Helper functions
|
||||
|
||||
func (d *D) addToMembersList(pubkey []byte) error { |
||||
data, err := d.GetMarker("nip43_members_list") |
||||
|
||||
var members []string |
||||
if err == nil { |
||||
if err := json.Unmarshal(data, &members); err != nil { |
||||
return fmt.Errorf("failed to unmarshal members list: %w", err) |
||||
} |
||||
} |
||||
|
||||
hexPubkey := hex.Enc(pubkey) |
||||
|
||||
// Check if already in list
|
||||
for _, member := range members { |
||||
if member == hexPubkey { |
||||
return nil // Already in list
|
||||
} |
||||
} |
||||
|
||||
members = append(members, hexPubkey) |
||||
|
||||
data, err = json.Marshal(members) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to marshal members list: %w", err) |
||||
} |
||||
|
||||
return d.SetMarker("nip43_members_list", data) |
||||
} |
||||
|
||||
func (d *D) removeFromMembersList(pubkey []byte) error { |
||||
data, err := d.GetMarker("nip43_members_list") |
||||
if err != nil { |
||||
return nil // List doesn't exist
|
||||
} |
||||
|
||||
var members []string |
||||
if err := json.Unmarshal(data, &members); err != nil { |
||||
return fmt.Errorf("failed to unmarshal members list: %w", err) |
||||
} |
||||
|
||||
hexPubkey := hex.Enc(pubkey) |
||||
|
||||
// Remove from list
|
||||
newMembers := make([]string, 0, len(members)) |
||||
for _, member := range members { |
||||
if member != hexPubkey { |
||||
newMembers = append(newMembers, member) |
||||
} |
||||
} |
||||
|
||||
data, err = json.Marshal(newMembers) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to marshal members list: %w", err) |
||||
} |
||||
|
||||
return d.SetMarker("nip43_members_list", data) |
||||
} |
||||
@ -0,0 +1,371 @@
@@ -0,0 +1,371 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"context" |
||||
"encoding/json" |
||||
"fmt" |
||||
"strings" |
||||
|
||||
"next.orly.dev/pkg/database/indexes/types" |
||||
"next.orly.dev/pkg/encoders/event" |
||||
"next.orly.dev/pkg/encoders/filter" |
||||
"next.orly.dev/pkg/encoders/hex" |
||||
"next.orly.dev/pkg/encoders/tag" |
||||
"next.orly.dev/pkg/interfaces/store" |
||||
) |
||||
|
||||
// QueryEvents retrieves events matching the given filter
|
||||
func (d *D) QueryEvents(c context.Context, f *filter.F) (evs event.S, err error) { |
||||
return d.QueryEventsWithOptions(c, f, false, false) |
||||
} |
||||
|
||||
// QueryAllVersions retrieves all versions of events matching the filter
|
||||
func (d *D) QueryAllVersions(c context.Context, f *filter.F) (evs event.S, err error) { |
||||
return d.QueryEventsWithOptions(c, f, false, true) |
||||
} |
||||
|
||||
// QueryEventsWithOptions retrieves events with specific options
|
||||
func (d *D) QueryEventsWithOptions( |
||||
c context.Context, f *filter.F, includeDeleteEvents bool, showAllVersions bool, |
||||
) (evs event.S, err error) { |
||||
// Build DQL query from Nostr filter
|
||||
query := d.buildDQLQuery(f, includeDeleteEvents) |
||||
|
||||
// Execute query
|
||||
resp, err := d.Query(c, query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to execute query: %w", err) |
||||
} |
||||
|
||||
// Parse response
|
||||
evs, err = d.parseEventsFromResponse(resp.Json) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to parse events: %w", err) |
||||
} |
||||
|
||||
return evs, nil |
||||
} |
||||
|
||||
// buildDQLQuery constructs a DQL query from a Nostr filter
|
||||
func (d *D) buildDQLQuery(f *filter.F, includeDeleteEvents bool) string { |
||||
var conditions []string |
||||
var funcQuery string |
||||
|
||||
// IDs filter
|
||||
if len(f.Ids.T) > 0 { |
||||
idConditions := make([]string, len(f.Ids.T)) |
||||
for i, id := range f.Ids.T { |
||||
// Handle prefix matching
|
||||
if len(id) < 64 { |
||||
// Prefix search
|
||||
idConditions[i] = fmt.Sprintf("regexp(event.id, /^%s/)", hex.Enc(id)) |
||||
} else { |
||||
idConditions[i] = fmt.Sprintf("eq(event.id, %q)", hex.Enc(id)) |
||||
} |
||||
} |
||||
if len(idConditions) == 1 { |
||||
funcQuery = idConditions[0] |
||||
} else { |
||||
conditions = append(conditions, "("+strings.Join(idConditions, " OR ")+")") |
||||
} |
||||
} |
||||
|
||||
// Authors filter
|
||||
if len(f.Authors.T) > 0 { |
||||
authorConditions := make([]string, len(f.Authors.T)) |
||||
for i, author := range f.Authors.T { |
||||
// Handle prefix matching
|
||||
if len(author) < 64 { |
||||
authorConditions[i] = fmt.Sprintf("regexp(event.pubkey, /^%s/)", hex.Enc(author)) |
||||
} else { |
||||
authorConditions[i] = fmt.Sprintf("eq(event.pubkey, %q)", hex.Enc(author)) |
||||
} |
||||
} |
||||
if funcQuery == "" && len(authorConditions) == 1 { |
||||
funcQuery = authorConditions[0] |
||||
} else { |
||||
conditions = append(conditions, "("+strings.Join(authorConditions, " OR ")+")") |
||||
} |
||||
} |
||||
|
||||
// Kinds filter
|
||||
if len(f.Kinds.K) > 0 { |
||||
kindConditions := make([]string, len(f.Kinds.K)) |
||||
for i, kind := range f.Kinds.K { |
||||
kindConditions[i] = fmt.Sprintf("eq(event.kind, %d)", kind) |
||||
} |
||||
conditions = append(conditions, "("+strings.Join(kindConditions, " OR ")+")") |
||||
} |
||||
|
||||
// Time range filters
|
||||
if f.Since != nil { |
||||
conditions = append(conditions, fmt.Sprintf("ge(event.created_at, %d)", f.Since.V)) |
||||
} |
||||
if f.Until != nil { |
||||
conditions = append(conditions, fmt.Sprintf("le(event.created_at, %d)", f.Until.V)) |
||||
} |
||||
|
||||
// Tag filters
|
||||
for _, tagValues := range *f.Tags { |
||||
if len(tagValues.T) > 0 { |
||||
tagConditions := make([]string, len(tagValues.T)) |
||||
for i, tagValue := range tagValues.T { |
||||
// This is a simplified tag query - in production you'd want to use facets
|
||||
tagConditions[i] = fmt.Sprintf("eq(tag.value, %q)", string(tagValue)) |
||||
} |
||||
conditions = append(conditions, "("+strings.Join(tagConditions, " OR ")+")") |
||||
} |
||||
} |
||||
|
||||
// Exclude delete events unless requested
|
||||
if !includeDeleteEvents { |
||||
conditions = append(conditions, "NOT eq(event.kind, 5)") |
||||
} |
||||
|
||||
// Build the final query
|
||||
if funcQuery == "" { |
||||
funcQuery = "has(event.id)" |
||||
} |
||||
|
||||
filterStr := "" |
||||
if len(conditions) > 0 { |
||||
filterStr = " @filter(" + strings.Join(conditions, " AND ") + ")" |
||||
} |
||||
|
||||
// Add ordering and limit
|
||||
orderBy := ", orderdesc: event.created_at" |
||||
limitStr := "" |
||||
if *f.Limit > 0 { |
||||
limitStr = fmt.Sprintf(", first: %d", f.Limit) |
||||
} |
||||
|
||||
query := fmt.Sprintf(`{ |
||||
events(func: %s%s%s%s) { |
||||
uid |
||||
event.id |
||||
event.kind |
||||
event.created_at |
||||
event.content |
||||
event.sig |
||||
event.pubkey |
||||
event.tags |
||||
} |
||||
}`, funcQuery, filterStr, orderBy, limitStr) |
||||
|
||||
return query |
||||
} |
||||
|
||||
// parseEventsFromResponse converts Dgraph JSON response to Nostr events
|
||||
func (d *D) parseEventsFromResponse(jsonData []byte) ([]*event.E, error) { |
||||
var result struct { |
||||
Events []struct { |
||||
UID string `json:"uid"` |
||||
ID string `json:"event.id"` |
||||
Kind int `json:"event.kind"` |
||||
CreatedAt int64 `json:"event.created_at"` |
||||
Content string `json:"event.content"` |
||||
Sig string `json:"event.sig"` |
||||
Pubkey string `json:"event.pubkey"` |
||||
Tags string `json:"event.tags"` |
||||
} `json:"events"` |
||||
} |
||||
|
||||
if err := json.Unmarshal(jsonData, &result); err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
events := make([]*event.E, 0, len(result.Events)) |
||||
for _, ev := range result.Events { |
||||
// Decode hex strings
|
||||
id, err := hex.Dec(ev.ID) |
||||
if err != nil { |
||||
continue |
||||
} |
||||
sig, err := hex.Dec(ev.Sig) |
||||
if err != nil { |
||||
continue |
||||
} |
||||
pubkey, err := hex.Dec(ev.Pubkey) |
||||
if err != nil { |
||||
continue |
||||
} |
||||
|
||||
// Parse tags from JSON
|
||||
var tags tag.S |
||||
if ev.Tags != "" { |
||||
if err := json.Unmarshal([]byte(ev.Tags), &tags); err != nil { |
||||
continue |
||||
} |
||||
} |
||||
|
||||
// Create event
|
||||
e := &event.E{ |
||||
Kind: uint16(ev.Kind), |
||||
CreatedAt: ev.CreatedAt, |
||||
Content: []byte(ev.Content), |
||||
Tags: &tags, |
||||
} |
||||
|
||||
// Copy fixed-size arrays
|
||||
copy(e.ID[:], id) |
||||
copy(e.Sig[:], sig) |
||||
copy(e.Pubkey[:], pubkey) |
||||
|
||||
events = append(events, e) |
||||
} |
||||
|
||||
return events, nil |
||||
} |
||||
|
||||
// QueryDeleteEventsByTargetId retrieves delete events targeting a specific event ID
|
||||
func (d *D) QueryDeleteEventsByTargetId(c context.Context, targetEventId []byte) ( |
||||
evs event.S, err error, |
||||
) { |
||||
targetIDStr := hex.Enc(targetEventId) |
||||
|
||||
// Query for kind 5 events that reference this event
|
||||
query := fmt.Sprintf(`{ |
||||
events(func: eq(event.kind, 5)) { |
||||
uid |
||||
event.id |
||||
event.kind |
||||
event.created_at |
||||
event.content |
||||
event.sig |
||||
event.pubkey |
||||
event.tags |
||||
references @filter(eq(event.id, %q)) { |
||||
event.id |
||||
} |
||||
} |
||||
}`, targetIDStr) |
||||
|
||||
resp, err := d.Query(c, query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to query delete events: %w", err) |
||||
} |
||||
|
||||
evs, err = d.parseEventsFromResponse(resp.Json) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to parse delete events: %w", err) |
||||
} |
||||
|
||||
return evs, nil |
||||
} |
||||
|
||||
// QueryForSerials retrieves event serials matching a filter
|
||||
func (d *D) QueryForSerials(c context.Context, f *filter.F) ( |
||||
serials types.Uint40s, err error, |
||||
) { |
||||
// Build query
|
||||
query := d.buildDQLQuery(f, false) |
||||
|
||||
// Modify query to only return serial numbers
|
||||
query = strings.Replace(query, "event.id\n\t\t\tevent.kind", "event.serial", 1) |
||||
query = strings.Replace(query, "\t\t\tevent.created_at\n\t\t\tevent.content\n\t\t\tevent.sig\n\t\t\tevent.pubkey\n\t\t\tevent.tags", "", 1) |
||||
|
||||
resp, err := d.Query(c, query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to query serials: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Events []struct { |
||||
Serial int64 `json:"event.serial"` |
||||
} `json:"events"` |
||||
} |
||||
|
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
serials = make([]*types.Uint40, 0, len(result.Events)) |
||||
for _, ev := range result.Events { |
||||
serial := types.Uint40{} |
||||
serial.Set(uint64(ev.Serial)) |
||||
serials = append(serials, &serial) |
||||
} |
||||
|
||||
return serials, nil |
||||
} |
||||
|
||||
// QueryForIds retrieves event IDs matching a filter
|
||||
func (d *D) QueryForIds(c context.Context, f *filter.F) ( |
||||
idPkTs []*store.IdPkTs, err error, |
||||
) { |
||||
// Build query
|
||||
query := d.buildDQLQuery(f, false) |
||||
|
||||
// Modify query to only return ID, pubkey, created_at, serial
|
||||
query = strings.Replace(query, "event.kind\n\t\t\tevent.created_at\n\t\t\tevent.content\n\t\t\tevent.sig\n\t\t\tevent.pubkey\n\t\t\tevent.tags", "event.id\n\t\t\tevent.pubkey\n\t\t\tevent.created_at\n\t\t\tevent.serial", 1) |
||||
|
||||
resp, err := d.Query(c, query) |
||||
if err != nil { |
||||
return nil, fmt.Errorf("failed to query IDs: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Events []struct { |
||||
ID string `json:"event.id"` |
||||
Pubkey string `json:"event.pubkey"` |
||||
CreatedAt int64 `json:"event.created_at"` |
||||
Serial int64 `json:"event.serial"` |
||||
} `json:"events"` |
||||
} |
||||
|
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
idPkTs = make([]*store.IdPkTs, 0, len(result.Events)) |
||||
for _, ev := range result.Events { |
||||
id, err := hex.Dec(ev.ID) |
||||
if err != nil { |
||||
continue |
||||
} |
||||
pubkey, err := hex.Dec(ev.Pubkey) |
||||
if err != nil { |
||||
continue |
||||
} |
||||
idPkTs = append(idPkTs, &store.IdPkTs{ |
||||
Id: id, |
||||
Pub: pubkey, |
||||
Ts: ev.CreatedAt, |
||||
Ser: uint64(ev.Serial), |
||||
}) |
||||
} |
||||
|
||||
return idPkTs, nil |
||||
} |
||||
|
||||
// CountEvents counts events matching a filter
|
||||
func (d *D) CountEvents(c context.Context, f *filter.F) ( |
||||
count int, approximate bool, err error, |
||||
) { |
||||
// Build query with count
|
||||
query := d.buildDQLQuery(f, false) |
||||
|
||||
// Modify to count instead of returning full data
|
||||
query = strings.Replace(query, "uid\n\t\t\tevent.id\n\t\t\tevent.kind\n\t\t\tevent.created_at\n\t\t\tevent.content\n\t\t\tevent.sig\n\t\t\tevent.pubkey\n\t\t\tevent.tags", "count(uid)", 1) |
||||
|
||||
resp, err := d.Query(c, query) |
||||
if err != nil { |
||||
return 0, false, fmt.Errorf("failed to count events: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Events []struct { |
||||
Count int `json:"count"` |
||||
} `json:"events"` |
||||
} |
||||
|
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return 0, false, err |
||||
} |
||||
|
||||
if len(result.Events) > 0 { |
||||
count = result.Events[0].Count |
||||
} |
||||
|
||||
return count, false, nil |
||||
} |
||||
@ -0,0 +1,517 @@
@@ -0,0 +1,517 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"fmt" |
||||
"testing" |
||||
|
||||
"lol.mleku.dev/chk" |
||||
"next.orly.dev/pkg/encoders/event" |
||||
"next.orly.dev/pkg/encoders/filter" |
||||
"next.orly.dev/pkg/encoders/hex" |
||||
"next.orly.dev/pkg/encoders/kind" |
||||
"next.orly.dev/pkg/encoders/tag" |
||||
"next.orly.dev/pkg/encoders/timestamp" |
||||
"next.orly.dev/pkg/interfaces/signer/p8k" |
||||
"next.orly.dev/pkg/utils" |
||||
) |
||||
|
||||
func TestQueryEventsByID(t *testing.T) { |
||||
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||
defer cleanupTestDB(t, db, cancel, tempDir) |
||||
|
||||
// Test QueryEvents with an ID filter
|
||||
testEvent := events[3] |
||||
|
||||
evs, err := db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Ids: tag.NewFromBytesSlice(testEvent.ID), |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf("Failed to query events by ID: %v", err) |
||||
} |
||||
|
||||
// Verify we got exactly one event
|
||||
if len(evs) != 1 { |
||||
t.Fatalf("Expected 1 event, got %d", len(evs)) |
||||
} |
||||
|
||||
// Verify it's the correct event
|
||||
if !utils.FastEqual(evs[0].ID, testEvent.ID) { |
||||
t.Fatalf( |
||||
"Event ID doesn't match. Got %x, expected %x", evs[0].ID, |
||||
testEvent.ID, |
||||
) |
||||
} |
||||
} |
||||
|
||||
func TestQueryEventsByKind(t *testing.T) { |
||||
db, _, ctx, cancel, tempDir := setupTestDB(t) |
||||
defer cleanupTestDB(t, db, cancel, tempDir) |
||||
|
||||
// Test querying by kind
|
||||
testKind := kind.New(1) // Kind 1 is typically text notes
|
||||
kindFilter := kind.NewS(testKind) |
||||
|
||||
evs, err := db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Kinds: kindFilter, |
||||
Tags: tag.NewS(), |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf("Failed to query events by kind: %v", err) |
||||
} |
||||
|
||||
// Verify we got results
|
||||
if len(evs) == 0 { |
||||
t.Fatal("Expected events with kind 1, but got none") |
||||
} |
||||
|
||||
// Verify all events have the correct kind
|
||||
for i, ev := range evs { |
||||
if ev.Kind != testKind.K { |
||||
t.Fatalf( |
||||
"Event %d has incorrect kind. Got %d, expected %d", i, |
||||
ev.Kind, testKind.K, |
||||
) |
||||
} |
||||
} |
||||
} |
||||
|
||||
func TestQueryEventsByAuthor(t *testing.T) { |
||||
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||
defer cleanupTestDB(t, db, cancel, tempDir) |
||||
|
||||
// Test querying by author
|
||||
authorFilter := tag.NewFromBytesSlice(events[1].Pubkey) |
||||
|
||||
evs, err := db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Authors: authorFilter, |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf("Failed to query events by author: %v", err) |
||||
} |
||||
|
||||
// Verify we got results
|
||||
if len(evs) == 0 { |
||||
t.Fatal("Expected events from author, but got none") |
||||
} |
||||
|
||||
// Verify all events have the correct author
|
||||
for i, ev := range evs { |
||||
if !utils.FastEqual(ev.Pubkey, events[1].Pubkey) { |
||||
t.Fatalf( |
||||
"Event %d has incorrect author. Got %x, expected %x", |
||||
i, ev.Pubkey, events[1].Pubkey, |
||||
) |
||||
} |
||||
} |
||||
} |
||||
|
||||
func TestReplaceableEventsAndDeletion(t *testing.T) { |
||||
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||
defer cleanupTestDB(t, db, cancel, tempDir) |
||||
|
||||
// Create a signer
|
||||
sign := p8k.MustNew() |
||||
if err := sign.Generate(); chk.E(err) { |
||||
t.Fatal(err) |
||||
} |
||||
|
||||
// Create a replaceable event
|
||||
replaceableEvent := event.New() |
||||
replaceableEvent.Kind = kind.ProfileMetadata.K // Kind 0 is replaceable
|
||||
replaceableEvent.Pubkey = events[0].Pubkey // Use the same pubkey as an existing event
|
||||
replaceableEvent.CreatedAt = timestamp.Now().V - 7200 // 2 hours ago
|
||||
replaceableEvent.Content = []byte("Original profile") |
||||
replaceableEvent.Tags = tag.NewS() |
||||
replaceableEvent.Sign(sign) |
||||
|
||||
// Save the replaceable event
|
||||
if _, err := db.SaveEvent(ctx, replaceableEvent); err != nil { |
||||
t.Fatalf("Failed to save replaceable event: %v", err) |
||||
} |
||||
|
||||
// Create a newer version of the replaceable event
|
||||
newerEvent := event.New() |
||||
newerEvent.Kind = kind.ProfileMetadata.K // Same kind
|
||||
newerEvent.Pubkey = replaceableEvent.Pubkey // Same pubkey
|
||||
newerEvent.CreatedAt = timestamp.Now().V - 3600 // 1 hour ago (newer than the original)
|
||||
newerEvent.Content = []byte("Updated profile") |
||||
newerEvent.Tags = tag.NewS() |
||||
newerEvent.Sign(sign) |
||||
|
||||
// Save the newer event
|
||||
if _, err := db.SaveEvent(ctx, newerEvent); err != nil { |
||||
t.Fatalf("Failed to save newer event: %v", err) |
||||
} |
||||
|
||||
// Query for the original event by ID
|
||||
evs, err := db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Ids: tag.NewFromAny(replaceableEvent.ID), |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf("Failed to query for replaced event by ID: %v", err) |
||||
} |
||||
|
||||
// Verify the original event is still found (it's kept but not returned in general queries)
|
||||
if len(evs) != 1 { |
||||
t.Fatalf("Expected 1 event when querying for replaced event by ID, got %d", len(evs)) |
||||
} |
||||
|
||||
// Verify it's the original event
|
||||
if !utils.FastEqual(evs[0].ID, replaceableEvent.ID) { |
||||
t.Fatalf( |
||||
"Event ID doesn't match when querying for replaced event. Got %x, expected %x", |
||||
evs[0].ID, replaceableEvent.ID, |
||||
) |
||||
} |
||||
|
||||
// Query for all events of this kind and pubkey
|
||||
kindFilter := kind.NewS(kind.ProfileMetadata) |
||||
authorFilter := tag.NewFromAny(replaceableEvent.Pubkey) |
||||
|
||||
evs, err = db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Kinds: kindFilter, |
||||
Authors: authorFilter, |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf("Failed to query for replaceable events: %v", err) |
||||
} |
||||
|
||||
// Verify we got only one event (the latest one)
|
||||
if len(evs) != 1 { |
||||
t.Fatalf( |
||||
"Expected 1 event when querying for replaceable events, got %d", |
||||
len(evs), |
||||
) |
||||
} |
||||
|
||||
// Verify it's the newer event
|
||||
if !utils.FastEqual(evs[0].ID, newerEvent.ID) { |
||||
t.Fatalf( |
||||
"Event ID doesn't match when querying for replaceable events. Got %x, expected %x", |
||||
evs[0].ID, newerEvent.ID, |
||||
) |
||||
} |
||||
|
||||
// Test deletion events
|
||||
// Create a deletion event that references the replaceable event
|
||||
deletionEvent := event.New() |
||||
deletionEvent.Kind = kind.Deletion.K // Kind 5 is deletion
|
||||
deletionEvent.Pubkey = replaceableEvent.Pubkey // Same pubkey as the event being deleted
|
||||
deletionEvent.CreatedAt = timestamp.Now().V // Current time
|
||||
deletionEvent.Content = []byte("Deleting the replaceable event") |
||||
deletionEvent.Tags = tag.NewS() |
||||
deletionEvent.Sign(sign) |
||||
|
||||
// Add an e-tag referencing the replaceable event
|
||||
*deletionEvent.Tags = append( |
||||
*deletionEvent.Tags, |
||||
tag.NewFromAny("e", hex.Enc(replaceableEvent.ID)), |
||||
) |
||||
|
||||
// Save the deletion event
|
||||
if _, err = db.SaveEvent(ctx, deletionEvent); err != nil { |
||||
t.Fatalf("Failed to save deletion event: %v", err) |
||||
} |
||||
|
||||
// Query for all events of this kind and pubkey again
|
||||
evs, err = db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Kinds: kindFilter, |
||||
Authors: authorFilter, |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf( |
||||
"Failed to query for replaceable events after deletion: %v", err, |
||||
) |
||||
} |
||||
|
||||
// Verify we still get the newer event (deletion should only affect the original event)
|
||||
if len(evs) != 1 { |
||||
t.Fatalf( |
||||
"Expected 1 event when querying for replaceable events after deletion, got %d", |
||||
len(evs), |
||||
) |
||||
} |
||||
|
||||
// Verify it's still the newer event
|
||||
if !utils.FastEqual(evs[0].ID, newerEvent.ID) { |
||||
t.Fatalf( |
||||
"Event ID doesn't match after deletion. Got %x, expected %x", |
||||
evs[0].ID, newerEvent.ID, |
||||
) |
||||
} |
||||
|
||||
// Query for the original event by ID
|
||||
evs, err = db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Ids: tag.NewFromBytesSlice(replaceableEvent.ID), |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf("Failed to query for deleted event by ID: %v", err) |
||||
} |
||||
|
||||
// Verify the original event is not found (it was deleted)
|
||||
if len(evs) != 0 { |
||||
t.Fatalf("Expected 0 events when querying for deleted event by ID, got %d", len(evs)) |
||||
} |
||||
} |
||||
|
||||
func TestParameterizedReplaceableEventsAndDeletion(t *testing.T) { |
||||
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||
defer cleanupTestDB(t, db, cancel, tempDir) |
||||
|
||||
sign := p8k.MustNew() |
||||
if err := sign.Generate(); chk.E(err) { |
||||
t.Fatal(err) |
||||
} |
||||
|
||||
// Create a parameterized replaceable event
|
||||
paramEvent := event.New() |
||||
paramEvent.Kind = 30000 // Kind 30000+ is parameterized replaceable
|
||||
paramEvent.Pubkey = events[0].Pubkey // Use the same pubkey as an existing event
|
||||
paramEvent.CreatedAt = timestamp.Now().V - 7200 // 2 hours ago
|
||||
paramEvent.Content = []byte("Original parameterized event") |
||||
paramEvent.Tags = tag.NewS() |
||||
// Add a d-tag
|
||||
*paramEvent.Tags = append( |
||||
*paramEvent.Tags, tag.NewFromAny([]byte{'d'}, []byte("test-d-tag")), |
||||
) |
||||
paramEvent.Sign(sign) |
||||
|
||||
// Save the parameterized replaceable event
|
||||
if _, err := db.SaveEvent(ctx, paramEvent); err != nil { |
||||
t.Fatalf("Failed to save parameterized replaceable event: %v", err) |
||||
} |
||||
|
||||
// Create a deletion event using e-tag
|
||||
paramDeletionEvent := event.New() |
||||
paramDeletionEvent.Kind = kind.Deletion.K // Kind 5 is deletion
|
||||
paramDeletionEvent.Pubkey = paramEvent.Pubkey // Same pubkey as the event being deleted
|
||||
paramDeletionEvent.CreatedAt = timestamp.Now().V // Current time
|
||||
paramDeletionEvent.Content = []byte("Deleting the parameterized replaceable event with e-tag") |
||||
paramDeletionEvent.Tags = tag.NewS() |
||||
// Add an e-tag referencing the parameterized replaceable event
|
||||
*paramDeletionEvent.Tags = append( |
||||
*paramDeletionEvent.Tags, |
||||
tag.NewFromAny("e", []byte(hex.Enc(paramEvent.ID))), |
||||
) |
||||
paramDeletionEvent.Sign(sign) |
||||
|
||||
// Save the parameterized deletion event with e-tag
|
||||
if _, err := db.SaveEvent(ctx, paramDeletionEvent); err != nil { |
||||
t.Fatalf( |
||||
"Failed to save parameterized deletion event with e-tag: %v", err, |
||||
) |
||||
} |
||||
|
||||
// Query for parameterized events
|
||||
paramKindFilter := kind.NewS(kind.New(paramEvent.Kind)) |
||||
paramAuthorFilter := tag.NewFromBytesSlice(paramEvent.Pubkey) |
||||
|
||||
evs, err := db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Kinds: paramKindFilter, |
||||
Authors: paramAuthorFilter, |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf( |
||||
"Failed to query for parameterized replaceable events after deletion: %v", |
||||
err, |
||||
) |
||||
} |
||||
|
||||
// Debug output
|
||||
fmt.Printf("Got %d events after deletion\n", len(evs)) |
||||
for i, ev := range evs { |
||||
fmt.Printf( |
||||
"Event %d: kind=%d, pubkey=%s\n", |
||||
i, ev.Kind, hex.Enc(ev.Pubkey), |
||||
) |
||||
} |
||||
|
||||
// Verify we get no events (since the only one was deleted)
|
||||
if len(evs) != 0 { |
||||
t.Fatalf( |
||||
"Expected 0 events when querying for deleted parameterized replaceable events, got %d", |
||||
len(evs), |
||||
) |
||||
} |
||||
|
||||
// Query for the parameterized event by ID
|
||||
evs, err = db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Ids: tag.NewFromBytesSlice(paramEvent.ID), |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf( |
||||
"Failed to query for deleted parameterized event by ID: %v", err, |
||||
) |
||||
} |
||||
|
||||
// Verify the deleted event is not found when querying by ID
|
||||
if len(evs) != 0 { |
||||
t.Fatalf( |
||||
"Expected 0 events when querying for deleted parameterized event by ID, got %d", |
||||
len(evs), |
||||
) |
||||
} |
||||
} |
||||
|
||||
func TestQueryEventsByTimeRange(t *testing.T) { |
||||
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||
defer cleanupTestDB(t, db, cancel, tempDir) |
||||
|
||||
// Test querying by time range
|
||||
// Use the timestamp from the middle event as a reference
|
||||
middleIndex := len(events) / 2 |
||||
middleEvent := events[middleIndex] |
||||
|
||||
// Create a timestamp range that includes events before and after the middle event
|
||||
sinceTime := new(timestamp.T) |
||||
sinceTime.V = middleEvent.CreatedAt - 3600 // 1 hour before middle event
|
||||
|
||||
untilTime := new(timestamp.T) |
||||
untilTime.V = middleEvent.CreatedAt + 3600 // 1 hour after middle event
|
||||
|
||||
evs, err := db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Since: sinceTime, |
||||
Until: untilTime, |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf("Failed to query events by time range: %v", err) |
||||
} |
||||
|
||||
// Verify we got results
|
||||
if len(evs) == 0 { |
||||
t.Fatal("Expected events in time range, but got none") |
||||
} |
||||
|
||||
// Verify all events are within the time range
|
||||
for i, ev := range evs { |
||||
if ev.CreatedAt < sinceTime.V || ev.CreatedAt > untilTime.V { |
||||
t.Fatalf( |
||||
"Event %d is outside the time range. Got %d, expected between %d and %d", |
||||
i, ev.CreatedAt, sinceTime.V, untilTime.V, |
||||
) |
||||
} |
||||
} |
||||
} |
||||
|
||||
func TestQueryEventsByTag(t *testing.T) { |
||||
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||
defer cleanupTestDB(t, db, cancel, tempDir) |
||||
|
||||
// Find an event with tags to use for testing
|
||||
var testTagEvent *event.E |
||||
for _, ev := range events { |
||||
if ev.Tags != nil && ev.Tags.Len() > 0 { |
||||
// Find a tag with at least 2 elements and first element of length 1
|
||||
for _, tag := range *ev.Tags { |
||||
if tag.Len() >= 2 && len(tag.Key()) == 1 { |
||||
testTagEvent = ev |
||||
break |
||||
} |
||||
} |
||||
if testTagEvent != nil { |
||||
break |
||||
} |
||||
} |
||||
} |
||||
|
||||
if testTagEvent == nil { |
||||
t.Skip("No suitable event with tags found for testing") |
||||
return |
||||
} |
||||
|
||||
// Get the first tag with at least 2 elements and first element of length 1
|
||||
var testTag *tag.T |
||||
for _, tag := range *testTagEvent.Tags { |
||||
if tag.Len() >= 2 && len(tag.Key()) == 1 { |
||||
testTag = tag |
||||
break |
||||
} |
||||
} |
||||
|
||||
// Create a tags filter with the test tag
|
||||
tagsFilter := tag.NewS(testTag) |
||||
|
||||
evs, err := db.QueryEvents( |
||||
ctx, &filter.F{ |
||||
Tags: tagsFilter, |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf("Failed to query events by tag: %v", err) |
||||
} |
||||
|
||||
// Verify we got results
|
||||
if len(evs) == 0 { |
||||
t.Fatal("Expected events with tag, but got none") |
||||
} |
||||
|
||||
// Verify all events have the tag
|
||||
for i, ev := range evs { |
||||
var hasTag bool |
||||
for _, tag := range *ev.Tags { |
||||
if tag.Len() >= 2 && len(tag.Key()) == 1 { |
||||
if utils.FastEqual(tag.Key(), testTag.Key()) && |
||||
utils.FastEqual(tag.Value(), testTag.Value()) { |
||||
hasTag = true |
||||
break |
||||
} |
||||
} |
||||
} |
||||
if !hasTag { |
||||
t.Fatalf("Event %d does not have the expected tag", i) |
||||
} |
||||
} |
||||
} |
||||
|
||||
func TestCountEvents(t *testing.T) { |
||||
db, _, ctx, cancel, tempDir := setupTestDB(t) |
||||
defer cleanupTestDB(t, db, cancel, tempDir) |
||||
|
||||
// Test counting all events
|
||||
count, _, err := db.CountEvents(ctx, &filter.F{}) |
||||
if err != nil { |
||||
t.Fatalf("Failed to count events: %v", err) |
||||
} |
||||
|
||||
// Verify we got a non-zero count
|
||||
if count == 0 { |
||||
t.Fatal("Expected non-zero event count, but got 0") |
||||
} |
||||
|
||||
t.Logf("Total events in database: %d", count) |
||||
|
||||
// Test counting events by kind
|
||||
testKind := kind.New(1) |
||||
kindFilter := kind.NewS(testKind) |
||||
|
||||
count, _, err = db.CountEvents( |
||||
ctx, &filter.F{ |
||||
Kinds: kindFilter, |
||||
}, |
||||
) |
||||
if err != nil { |
||||
t.Fatalf("Failed to count events by kind: %v", err) |
||||
} |
||||
|
||||
t.Logf("Events with kind 1: %d", count) |
||||
} |
||||
@ -0,0 +1,185 @@
@@ -0,0 +1,185 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"context" |
||||
"encoding/json" |
||||
"fmt" |
||||
"strings" |
||||
|
||||
"github.com/dgraph-io/dgo/v230/protos/api" |
||||
"next.orly.dev/pkg/database/indexes/types" |
||||
"next.orly.dev/pkg/encoders/event" |
||||
"next.orly.dev/pkg/encoders/filter" |
||||
"next.orly.dev/pkg/encoders/hex" |
||||
) |
||||
|
||||
// SaveEvent stores a Nostr event in the Dgraph database.
|
||||
// It creates event nodes and relationships for authors, tags, and references.
|
||||
func (d *D) SaveEvent(c context.Context, ev *event.E) (exists bool, err error) { |
||||
eventID := hex.Enc(ev.ID[:]) |
||||
|
||||
// Check if event already exists
|
||||
query := fmt.Sprintf(`{ |
||||
event(func: eq(event.id, %q)) { |
||||
uid |
||||
event.id |
||||
} |
||||
}`, eventID) |
||||
|
||||
resp, err := d.Query(c, query) |
||||
if err != nil { |
||||
return false, fmt.Errorf("failed to check event existence: %w", err) |
||||
} |
||||
|
||||
// Parse response to check if event exists
|
||||
var result struct { |
||||
Event []map[string]interface{} `json:"event"` |
||||
} |
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return false, fmt.Errorf("failed to parse query response: %w", err) |
||||
} |
||||
|
||||
if len(result.Event) > 0 { |
||||
return true, nil // Event already exists
|
||||
} |
||||
|
||||
// Get next serial number
|
||||
serial, err := d.getNextSerial() |
||||
if err != nil { |
||||
return false, fmt.Errorf("failed to get serial number: %w", err) |
||||
} |
||||
|
||||
// Build N-Quads for the event with serial number
|
||||
nquads := d.buildEventNQuads(ev, serial) |
||||
|
||||
// Store the event
|
||||
mutation := &api.Mutation{ |
||||
SetNquads: []byte(nquads), |
||||
CommitNow: true, |
||||
} |
||||
|
||||
if _, err = d.Mutate(c, mutation); err != nil { |
||||
return false, fmt.Errorf("failed to save event: %w", err) |
||||
} |
||||
|
||||
return false, nil |
||||
} |
||||
|
||||
// buildEventNQuads constructs RDF triples for a Nostr event
|
||||
func (d *D) buildEventNQuads(ev *event.E, serial uint64) string { |
||||
var nquads strings.Builder |
||||
|
||||
eventID := hex.Enc(ev.ID[:]) |
||||
authorPubkey := hex.Enc(ev.Pubkey) |
||||
|
||||
// Event node
|
||||
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Event\" .\n", eventID)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <event.id> %q .\n", eventID, eventID)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <event.serial> \"%d\"^^<xs:int> .\n", eventID, serial)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <event.kind> \"%d\"^^<xs:int> .\n", eventID, ev.Kind)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <event.created_at> \"%d\"^^<xs:int> .\n", eventID, int64(ev.CreatedAt))) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <event.content> %q .\n", eventID, ev.Content)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <event.sig> %q .\n", eventID, hex.Enc(ev.Sig[:]))) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <event.pubkey> %q .\n", eventID, authorPubkey)) |
||||
|
||||
// Serialize tags as JSON string for storage
|
||||
tagsJSON, _ := json.Marshal(ev.Tags) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <event.tags> %q .\n", eventID, string(tagsJSON))) |
||||
|
||||
// Author relationship
|
||||
nquads.WriteString(fmt.Sprintf("_:%s <authored_by> _:%s .\n", eventID, authorPubkey)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Author\" .\n", authorPubkey)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <author.pubkey> %q .\n", authorPubkey, authorPubkey)) |
||||
|
||||
// Tag relationships
|
||||
for _, tag := range *ev.Tags { |
||||
if len(tag.T) >= 2 { |
||||
tagType := string(tag.T[0]) |
||||
tagValue := string(tag.T[1]) |
||||
|
||||
switch tagType { |
||||
case "e": // Event reference
|
||||
nquads.WriteString(fmt.Sprintf("_:%s <references> _:%s .\n", eventID, tagValue)) |
||||
case "p": // Pubkey mention
|
||||
nquads.WriteString(fmt.Sprintf("_:%s <mentions> _:%s .\n", eventID, tagValue)) |
||||
// Ensure mentioned author exists
|
||||
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Author\" .\n", tagValue)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <author.pubkey> %q .\n", tagValue, tagValue)) |
||||
case "t": // Hashtag
|
||||
tagID := "tag_" + tagType + "_" + tagValue |
||||
nquads.WriteString(fmt.Sprintf("_:%s <tagged_with> _:%s .\n", eventID, tagID)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Tag\" .\n", tagID)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <tag.type> %q .\n", tagID, tagType)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <tag.value> %q .\n", tagID, tagValue)) |
||||
default: |
||||
// Store other tag types
|
||||
tagID := "tag_" + tagType + "_" + tagValue |
||||
nquads.WriteString(fmt.Sprintf("_:%s <tagged_with> _:%s .\n", eventID, tagID)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Tag\" .\n", tagID)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <tag.type> %q .\n", tagID, tagType)) |
||||
nquads.WriteString(fmt.Sprintf("_:%s <tag.value> %q .\n", tagID, tagValue)) |
||||
} |
||||
} |
||||
} |
||||
|
||||
return nquads.String() |
||||
} |
||||
|
||||
// GetSerialsFromFilter returns event serials matching a filter
|
||||
func (d *D) GetSerialsFromFilter(f *filter.F) (serials types.Uint40s, err error) { |
||||
// For dgraph, we'll use the event.serial field
|
||||
// This is a stub implementation
|
||||
err = fmt.Errorf("not implemented") |
||||
return |
||||
} |
||||
|
||||
// WouldReplaceEvent checks if an event would replace existing events
|
||||
func (d *D) WouldReplaceEvent(ev *event.E) (bool, types.Uint40s, error) { |
||||
// Check for replaceable events (kinds 0, 3, and 10000-19999)
|
||||
isReplaceable := ev.Kind == 0 || ev.Kind == 3 || (ev.Kind >= 10000 && ev.Kind < 20000) |
||||
if !isReplaceable { |
||||
return false, nil, nil |
||||
} |
||||
|
||||
// Query for existing events with same kind and pubkey
|
||||
authorPubkey := hex.Enc(ev.Pubkey) |
||||
query := fmt.Sprintf(`{ |
||||
events(func: eq(event.pubkey, %q)) @filter(eq(event.kind, %d)) { |
||||
uid |
||||
event.serial |
||||
event.created_at |
||||
} |
||||
}`, authorPubkey, ev.Kind) |
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return false, nil, fmt.Errorf("failed to query replaceable events: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Events []struct { |
||||
UID string `json:"uid"` |
||||
Serial int64 `json:"event.serial"` |
||||
CreatedAt int64 `json:"event.created_at"` |
||||
} `json:"events"` |
||||
} |
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return false, nil, fmt.Errorf("failed to parse query response: %w", err) |
||||
} |
||||
|
||||
// Check if our event is newer
|
||||
evTime := int64(ev.CreatedAt) |
||||
var serials types.Uint40s |
||||
wouldReplace := false |
||||
|
||||
for _, existing := range result.Events { |
||||
if existing.CreatedAt < evTime { |
||||
wouldReplace = true |
||||
serial := types.Uint40{} |
||||
serial.Set(uint64(existing.Serial)) |
||||
serials = append(serials, &serial) |
||||
} |
||||
} |
||||
|
||||
return wouldReplace, serials, nil |
||||
} |
||||
@ -0,0 +1,253 @@
@@ -0,0 +1,253 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"bufio" |
||||
"bytes" |
||||
"context" |
||||
"os" |
||||
"sort" |
||||
"testing" |
||||
"time" |
||||
|
||||
"lol.mleku.dev/chk" |
||||
"lol.mleku.dev/errorf" |
||||
"next.orly.dev/pkg/encoders/event" |
||||
"next.orly.dev/pkg/encoders/event/examples" |
||||
"next.orly.dev/pkg/encoders/hex" |
||||
"next.orly.dev/pkg/encoders/kind" |
||||
"next.orly.dev/pkg/encoders/tag" |
||||
"next.orly.dev/pkg/encoders/timestamp" |
||||
"next.orly.dev/pkg/interfaces/signer/p8k" |
||||
) |
||||
|
||||
// TestSaveEvents tests saving all events from examples.Cache to the dgraph database
|
||||
// to verify there are no errors during the saving process.
|
||||
func TestSaveEvents(t *testing.T) { |
||||
skipIfDgraphNotAvailable(t) |
||||
|
||||
// Create a temporary directory for metadata
|
||||
tempDir, err := os.MkdirTemp("", "test-dgraph-*") |
||||
if err != nil { |
||||
t.Fatalf("Failed to create temporary directory: %v", err) |
||||
} |
||||
defer os.RemoveAll(tempDir) |
||||
|
||||
// Create a context and cancel function for the database
|
||||
ctx, cancel := context.WithCancel(context.Background()) |
||||
defer cancel() |
||||
|
||||
// Initialize the dgraph database
|
||||
db, err := New(ctx, cancel, tempDir, "info") |
||||
if err != nil { |
||||
t.Fatalf("Failed to create dgraph database: %v", err) |
||||
} |
||||
defer db.Close() |
||||
|
||||
// Drop all data to start fresh
|
||||
if err := db.dropAll(ctx); err != nil { |
||||
t.Fatalf("Failed to drop all data: %v", err) |
||||
} |
||||
|
||||
// Create a scanner to read events from examples.Cache
|
||||
scanner := bufio.NewScanner(bytes.NewBuffer(examples.Cache)) |
||||
scanner.Buffer(make([]byte, 0, 1_000_000_000), 1_000_000_000) |
||||
|
||||
// Collect all events first
|
||||
var events []*event.E |
||||
var original int |
||||
for scanner.Scan() { |
||||
chk.E(scanner.Err()) |
||||
b := scanner.Bytes() |
||||
original += len(b) |
||||
ev := event.New() |
||||
|
||||
// Unmarshal the event
|
||||
if _, err = ev.Unmarshal(b); chk.E(err) { |
||||
t.Fatal(err) |
||||
} |
||||
|
||||
events = append(events, ev) |
||||
} |
||||
|
||||
// Sort events by timestamp to ensure addressable events are processed in order
|
||||
sort.Slice(events, func(i, j int) bool { |
||||
return events[i].CreatedAt < events[j].CreatedAt |
||||
}) |
||||
|
||||
// Count the number of events processed
|
||||
eventCount := 0 |
||||
now := time.Now() |
||||
|
||||
// Process each event in chronological order
|
||||
for _, ev := range events { |
||||
// Save the event to the database
|
||||
if _, err = db.SaveEvent(ctx, ev); err != nil { |
||||
t.Fatalf("Failed to save event #%d: %v", eventCount+1, err) |
||||
} |
||||
eventCount++ |
||||
} |
||||
|
||||
// Check for scanner errors
|
||||
if err = scanner.Err(); err != nil { |
||||
t.Fatalf("Scanner error: %v", err) |
||||
} |
||||
|
||||
dur := time.Since(now) |
||||
t.Logf( |
||||
"Successfully saved %d events (%d bytes) to dgraph in %v (%v/ev; %.2f ev/s)", |
||||
eventCount, |
||||
original, |
||||
dur, |
||||
dur/time.Duration(eventCount), |
||||
float64(time.Second)/float64(dur/time.Duration(eventCount)), |
||||
) |
||||
} |
||||
|
||||
// TestDeletionEventWithETagRejection tests that a deletion event with an "e" tag is rejected.
|
||||
func TestDeletionEventWithETagRejection(t *testing.T) { |
||||
skipIfDgraphNotAvailable(t) |
||||
|
||||
// Create a temporary directory for metadata
|
||||
tempDir, err := os.MkdirTemp("", "test-dgraph-*") |
||||
if err != nil { |
||||
t.Fatalf("Failed to create temporary directory: %v", err) |
||||
} |
||||
defer os.RemoveAll(tempDir) |
||||
|
||||
// Create a context and cancel function for the database
|
||||
ctx, cancel := context.WithCancel(context.Background()) |
||||
defer cancel() |
||||
|
||||
// Initialize the dgraph database
|
||||
db, err := New(ctx, cancel, tempDir, "info") |
||||
if err != nil { |
||||
t.Fatalf("Failed to create dgraph database: %v", err) |
||||
} |
||||
defer db.Close() |
||||
|
||||
// Drop all data to start fresh
|
||||
if err := db.dropAll(ctx); err != nil { |
||||
t.Fatalf("Failed to drop all data: %v", err) |
||||
} |
||||
|
||||
// Create a signer
|
||||
sign := p8k.MustNew() |
||||
if err := sign.Generate(); chk.E(err) { |
||||
t.Fatal(err) |
||||
} |
||||
|
||||
// Create a regular event
|
||||
regularEvent := event.New() |
||||
regularEvent.Kind = kind.TextNote.K |
||||
regularEvent.Pubkey = sign.Pub() |
||||
regularEvent.CreatedAt = timestamp.Now().V - 3600 // 1 hour ago
|
||||
regularEvent.Content = []byte("Regular event") |
||||
regularEvent.Tags = tag.NewS() |
||||
regularEvent.Sign(sign) |
||||
|
||||
// Save the regular event
|
||||
if _, err := db.SaveEvent(ctx, regularEvent); err != nil { |
||||
t.Fatalf("Failed to save regular event: %v", err) |
||||
} |
||||
|
||||
// Create a deletion event with an "e" tag referencing the regular event
|
||||
deletionEvent := event.New() |
||||
deletionEvent.Kind = kind.Deletion.K |
||||
deletionEvent.Pubkey = sign.Pub() |
||||
deletionEvent.CreatedAt = timestamp.Now().V // Current time
|
||||
deletionEvent.Content = []byte("Deleting the regular event") |
||||
deletionEvent.Tags = tag.NewS() |
||||
|
||||
// Add an e-tag referencing the regular event
|
||||
*deletionEvent.Tags = append( |
||||
*deletionEvent.Tags, |
||||
tag.NewFromAny("e", hex.Enc(regularEvent.ID)), |
||||
) |
||||
|
||||
deletionEvent.Sign(sign) |
||||
|
||||
// Check if this is a deletion event with "e" tags
|
||||
if deletionEvent.Kind == kind.Deletion.K && deletionEvent.Tags.GetFirst([]byte{'e'}) != nil { |
||||
// In this test, we want to reject deletion events with "e" tags
|
||||
err = errorf.E("deletion events referencing other events with 'e' tag are not allowed") |
||||
} else { |
||||
// Try to save the deletion event
|
||||
_, err = db.SaveEvent(ctx, deletionEvent) |
||||
} |
||||
|
||||
if err == nil { |
||||
t.Fatal("Expected deletion event with e-tag to be rejected, but it was accepted") |
||||
} |
||||
|
||||
// Verify the error message
|
||||
expectedError := "deletion events referencing other events with 'e' tag are not allowed" |
||||
if err.Error() != expectedError { |
||||
t.Fatalf( |
||||
"Expected error message '%s', got '%s'", expectedError, err.Error(), |
||||
) |
||||
} |
||||
} |
||||
|
||||
// TestSaveExistingEvent tests that attempting to save an event that already exists
|
||||
// returns an error.
|
||||
func TestSaveExistingEvent(t *testing.T) { |
||||
skipIfDgraphNotAvailable(t) |
||||
|
||||
// Create a temporary directory for metadata
|
||||
tempDir, err := os.MkdirTemp("", "test-dgraph-*") |
||||
if err != nil { |
||||
t.Fatalf("Failed to create temporary directory: %v", err) |
||||
} |
||||
defer os.RemoveAll(tempDir) |
||||
|
||||
// Create a context and cancel function for the database
|
||||
ctx, cancel := context.WithCancel(context.Background()) |
||||
defer cancel() |
||||
|
||||
// Initialize the dgraph database
|
||||
db, err := New(ctx, cancel, tempDir, "info") |
||||
if err != nil { |
||||
t.Fatalf("Failed to create dgraph database: %v", err) |
||||
} |
||||
defer db.Close() |
||||
|
||||
// Drop all data to start fresh
|
||||
if err := db.dropAll(ctx); err != nil { |
||||
t.Fatalf("Failed to drop all data: %v", err) |
||||
} |
||||
|
||||
// Create a signer
|
||||
sign := p8k.MustNew() |
||||
if err := sign.Generate(); chk.E(err) { |
||||
t.Fatal(err) |
||||
} |
||||
|
||||
// Create an event
|
||||
ev := event.New() |
||||
ev.Kind = kind.TextNote.K |
||||
ev.Pubkey = sign.Pub() |
||||
ev.CreatedAt = timestamp.Now().V |
||||
ev.Content = []byte("Test event") |
||||
ev.Tags = tag.NewS() |
||||
ev.Sign(sign) |
||||
|
||||
// Save the event for the first time
|
||||
if _, err := db.SaveEvent(ctx, ev); err != nil { |
||||
t.Fatalf("Failed to save event: %v", err) |
||||
} |
||||
|
||||
// Try to save the same event again, it should be rejected
|
||||
_, err = db.SaveEvent(ctx, ev) |
||||
if err == nil { |
||||
t.Fatal("Expected error when saving an existing event, but got nil") |
||||
} |
||||
|
||||
// Verify the error message contains indication of duplicate
|
||||
expectedErrorPrefix := "blocked: event already exists" |
||||
if !bytes.Contains([]byte(err.Error()), []byte(expectedErrorPrefix)) { |
||||
t.Fatalf( |
||||
"Expected error message to contain '%s', got '%s'", |
||||
expectedErrorPrefix, err.Error(), |
||||
) |
||||
} |
||||
} |
||||
@ -0,0 +1,105 @@
@@ -0,0 +1,105 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"context" |
||||
"fmt" |
||||
|
||||
"github.com/dgraph-io/dgo/v230/protos/api" |
||||
) |
||||
|
||||
// NostrSchema defines the Dgraph schema for Nostr events
|
||||
const NostrSchema = ` |
||||
# Event node type |
||||
type Event { |
||||
event.id |
||||
event.serial |
||||
event.kind |
||||
event.created_at |
||||
event.content |
||||
event.sig |
||||
event.pubkey |
||||
event.authored_by |
||||
event.references |
||||
event.mentions |
||||
event.tagged_with |
||||
} |
||||
|
||||
# Author node type |
||||
type Author { |
||||
author.pubkey |
||||
author.events |
||||
} |
||||
|
||||
# Tag node type |
||||
type Tag { |
||||
tag.type |
||||
tag.value |
||||
tag.events |
||||
} |
||||
|
||||
# Marker node type (for key-value metadata) |
||||
type Marker { |
||||
marker.key |
||||
marker.value |
||||
} |
||||
|
||||
# Event fields |
||||
event.id: string @index(exact) @upsert . |
||||
event.serial: int @index(int) . |
||||
event.kind: int @index(int) . |
||||
event.created_at: int @index(int) . |
||||
event.content: string . |
||||
event.sig: string @index(exact) . |
||||
event.pubkey: string @index(exact) . |
||||
|
||||
# Event relationships |
||||
event.authored_by: uid @reverse . |
||||
event.references: [uid] @reverse . |
||||
event.mentions: [uid] @reverse . |
||||
event.tagged_with: [uid] @reverse . |
||||
|
||||
# Author fields |
||||
author.pubkey: string @index(exact) @upsert . |
||||
author.events: [uid] @count @reverse . |
||||
|
||||
# Tag fields |
||||
tag.type: string @index(exact) . |
||||
tag.value: string @index(exact, fulltext) . |
||||
tag.events: [uid] @count @reverse . |
||||
|
||||
# Marker fields (key-value storage) |
||||
marker.key: string @index(exact) @upsert . |
||||
marker.value: string . |
||||
` |
||||
|
||||
// applySchema applies the Nostr schema to the connected Dgraph instance
|
||||
func (d *D) applySchema(ctx context.Context) error { |
||||
d.Logger.Infof("applying Nostr schema to dgraph") |
||||
|
||||
op := &api.Operation{ |
||||
Schema: NostrSchema, |
||||
} |
||||
|
||||
if err := d.client.Alter(ctx, op); err != nil { |
||||
return fmt.Errorf("failed to apply schema: %w", err) |
||||
} |
||||
|
||||
d.Logger.Infof("schema applied successfully") |
||||
return nil |
||||
} |
||||
|
||||
// dropAll drops all data from dgraph (useful for testing)
|
||||
func (d *D) dropAll(ctx context.Context) error { |
||||
d.Logger.Warningf("dropping all data from dgraph") |
||||
|
||||
op := &api.Operation{ |
||||
DropAll: true, |
||||
} |
||||
|
||||
if err := d.client.Alter(ctx, op); err != nil { |
||||
return fmt.Errorf("failed to drop all data: %w", err) |
||||
} |
||||
|
||||
// Reapply schema after dropping
|
||||
return d.applySchema(ctx) |
||||
} |
||||
@ -0,0 +1,136 @@
@@ -0,0 +1,136 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"context" |
||||
"encoding/json" |
||||
"fmt" |
||||
"sync" |
||||
|
||||
"github.com/dgraph-io/dgo/v230/protos/api" |
||||
) |
||||
|
||||
// Serial number management
|
||||
// We use a special counter node to track the next available serial number
|
||||
|
||||
const serialCounterKey = "serial_counter" |
||||
|
||||
var ( |
||||
serialMutex sync.Mutex |
||||
) |
||||
|
||||
// getNextSerial atomically increments and returns the next serial number
|
||||
func (d *D) getNextSerial() (uint64, error) { |
||||
serialMutex.Lock() |
||||
defer serialMutex.Unlock() |
||||
|
||||
// Query current serial value
|
||||
query := fmt.Sprintf(`{ |
||||
counter(func: eq(marker.key, %q)) { |
||||
uid |
||||
marker.value |
||||
} |
||||
}`, serialCounterKey) |
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return 0, fmt.Errorf("failed to query serial counter: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Counter []struct { |
||||
UID string `json:"uid"` |
||||
Value string `json:"marker.value"` |
||||
} `json:"counter"` |
||||
} |
||||
|
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return 0, fmt.Errorf("failed to parse serial counter: %w", err) |
||||
} |
||||
|
||||
var currentSerial uint64 = 1 |
||||
var uid string |
||||
|
||||
if len(result.Counter) > 0 { |
||||
// Parse current serial
|
||||
uid = result.Counter[0].UID |
||||
if result.Counter[0].Value != "" { |
||||
fmt.Sscanf(result.Counter[0].Value, "%d", ¤tSerial) |
||||
} |
||||
} |
||||
|
||||
// Increment serial
|
||||
nextSerial := currentSerial + 1 |
||||
|
||||
// Update or create counter
|
||||
var nquads string |
||||
if uid != "" { |
||||
// Update existing counter
|
||||
nquads = fmt.Sprintf(`<%s> <marker.value> "%d" .`, uid, nextSerial) |
||||
} else { |
||||
// Create new counter
|
||||
nquads = fmt.Sprintf(` |
||||
_:counter <dgraph.type> "Marker" . |
||||
_:counter <marker.key> %q . |
||||
_:counter <marker.value> "%d" . |
||||
`, serialCounterKey, nextSerial) |
||||
} |
||||
|
||||
mutation := &api.Mutation{ |
||||
SetNquads: []byte(nquads), |
||||
CommitNow: true, |
||||
} |
||||
|
||||
if _, err = d.Mutate(context.Background(), mutation); err != nil { |
||||
return 0, fmt.Errorf("failed to update serial counter: %w", err) |
||||
} |
||||
|
||||
return currentSerial, nil |
||||
} |
||||
|
||||
// initSerialCounter initializes the serial counter if it doesn't exist
|
||||
func (d *D) initSerialCounter() error { |
||||
query := fmt.Sprintf(`{ |
||||
counter(func: eq(marker.key, %q)) { |
||||
uid |
||||
} |
||||
}`, serialCounterKey) |
||||
|
||||
resp, err := d.Query(context.Background(), query) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to check serial counter: %w", err) |
||||
} |
||||
|
||||
var result struct { |
||||
Counter []struct { |
||||
UID string `json:"uid"` |
||||
} `json:"counter"` |
||||
} |
||||
|
||||
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||
return fmt.Errorf("failed to parse counter check: %w", err) |
||||
} |
||||
|
||||
// Counter already exists
|
||||
if len(result.Counter) > 0 { |
||||
return nil |
||||
} |
||||
|
||||
// Initialize counter at 1
|
||||
nquads := fmt.Sprintf(` |
||||
_:counter <dgraph.type> "Marker" . |
||||
_:counter <marker.key> %q . |
||||
_:counter <marker.value> "1" . |
||||
`, serialCounterKey) |
||||
|
||||
mutation := &api.Mutation{ |
||||
SetNquads: []byte(nquads), |
||||
CommitNow: true, |
||||
} |
||||
|
||||
if _, err = d.Mutate(context.Background(), mutation); err != nil { |
||||
return fmt.Errorf("failed to initialize serial counter: %w", err) |
||||
} |
||||
|
||||
d.Logger.Infof("initialized serial counter") |
||||
return nil |
||||
} |
||||
@ -0,0 +1,188 @@
@@ -0,0 +1,188 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"encoding/json" |
||||
"fmt" |
||||
"time" |
||||
|
||||
"next.orly.dev/pkg/database" |
||||
"next.orly.dev/pkg/encoders/hex" |
||||
) |
||||
|
||||
// Subscription and payment methods
|
||||
// Simplified implementation using marker-based storage
|
||||
// For production, these should use proper graph nodes with relationships
|
||||
|
||||
// GetSubscription retrieves subscription information for a pubkey
|
||||
func (d *D) GetSubscription(pubkey []byte) (*database.Subscription, error) { |
||||
key := "sub_" + hex.Enc(pubkey) |
||||
data, err := d.GetMarker(key) |
||||
if err != nil { |
||||
return nil, err |
||||
} |
||||
|
||||
var sub database.Subscription |
||||
if err := json.Unmarshal(data, &sub); err != nil { |
||||
return nil, fmt.Errorf("failed to unmarshal subscription: %w", err) |
||||
} |
||||
|
||||
return &sub, nil |
||||
} |
||||
|
||||
// IsSubscriptionActive checks if a pubkey has an active subscription
|
||||
func (d *D) IsSubscriptionActive(pubkey []byte) (bool, error) { |
||||
sub, err := d.GetSubscription(pubkey) |
||||
if err != nil { |
||||
return false, nil // No subscription = not active
|
||||
} |
||||
|
||||
return sub.PaidUntil.After(time.Now()), nil |
||||
} |
||||
|
||||
// ExtendSubscription extends a subscription by the specified number of days
|
||||
func (d *D) ExtendSubscription(pubkey []byte, days int) error { |
||||
key := "sub_" + hex.Enc(pubkey) |
||||
|
||||
// Get existing subscription or create new
|
||||
var sub database.Subscription |
||||
data, err := d.GetMarker(key) |
||||
if err == nil { |
||||
if err := json.Unmarshal(data, &sub); err != nil { |
||||
return fmt.Errorf("failed to unmarshal subscription: %w", err) |
||||
} |
||||
} else { |
||||
// New subscription - set trial period
|
||||
sub.TrialEnd = time.Now() |
||||
sub.PaidUntil = time.Now() |
||||
} |
||||
|
||||
// Extend expiration
|
||||
if sub.PaidUntil.Before(time.Now()) { |
||||
sub.PaidUntil = time.Now() |
||||
} |
||||
sub.PaidUntil = sub.PaidUntil.Add(time.Duration(days) * 24 * time.Hour) |
||||
|
||||
// Save
|
||||
data, err = json.Marshal(sub) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to marshal subscription: %w", err) |
||||
} |
||||
|
||||
return d.SetMarker(key, data) |
||||
} |
||||
|
||||
// RecordPayment records a payment for subscription extension
|
||||
func (d *D) RecordPayment( |
||||
pubkey []byte, amount int64, invoice, preimage string, |
||||
) error { |
||||
// Store payment in payments list
|
||||
key := "payments_" + hex.Enc(pubkey) |
||||
|
||||
var payments []database.Payment |
||||
data, err := d.GetMarker(key) |
||||
if err == nil { |
||||
if err := json.Unmarshal(data, &payments); err != nil { |
||||
return fmt.Errorf("failed to unmarshal payments: %w", err) |
||||
} |
||||
} |
||||
|
||||
payment := database.Payment{ |
||||
Amount: amount, |
||||
Timestamp: time.Now(), |
||||
Invoice: invoice, |
||||
Preimage: preimage, |
||||
} |
||||
|
||||
payments = append(payments, payment) |
||||
|
||||
data, err = json.Marshal(payments) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to marshal payments: %w", err) |
||||
} |
||||
|
||||
return d.SetMarker(key, data) |
||||
} |
||||
|
||||
// GetPaymentHistory retrieves payment history for a pubkey
|
||||
func (d *D) GetPaymentHistory(pubkey []byte) ([]database.Payment, error) { |
||||
key := "payments_" + hex.Enc(pubkey) |
||||
|
||||
data, err := d.GetMarker(key) |
||||
if err != nil { |
||||
return nil, nil // No payments = empty list
|
||||
} |
||||
|
||||
var payments []database.Payment |
||||
if err := json.Unmarshal(data, &payments); err != nil { |
||||
return nil, fmt.Errorf("failed to unmarshal payments: %w", err) |
||||
} |
||||
|
||||
return payments, nil |
||||
} |
||||
|
||||
// ExtendBlossomSubscription extends a Blossom storage subscription
|
||||
func (d *D) ExtendBlossomSubscription( |
||||
pubkey []byte, tier string, storageMB int64, daysExtended int, |
||||
) error { |
||||
key := "blossom_" + hex.Enc(pubkey) |
||||
|
||||
// Simple implementation - just store tier and expiry
|
||||
data := map[string]interface{}{ |
||||
"tier": tier, |
||||
"storageMB": storageMB, |
||||
"extended": daysExtended, |
||||
"updated": time.Now(), |
||||
} |
||||
|
||||
jsonData, err := json.Marshal(data) |
||||
if err != nil { |
||||
return fmt.Errorf("failed to marshal blossom subscription: %w", err) |
||||
} |
||||
|
||||
return d.SetMarker(key, jsonData) |
||||
} |
||||
|
||||
// GetBlossomStorageQuota retrieves the storage quota for a pubkey
|
||||
func (d *D) GetBlossomStorageQuota(pubkey []byte) (quotaMB int64, err error) { |
||||
key := "blossom_" + hex.Enc(pubkey) |
||||
|
||||
data, err := d.GetMarker(key) |
||||
if err != nil { |
||||
return 0, nil // No subscription = 0 quota
|
||||
} |
||||
|
||||
var result map[string]interface{} |
||||
if err := json.Unmarshal(data, &result); err != nil { |
||||
return 0, fmt.Errorf("failed to unmarshal blossom data: %w", err) |
||||
} |
||||
|
||||
// Default quota based on tier - simplified
|
||||
if tier, ok := result["tier"].(string); ok { |
||||
switch tier { |
||||
case "basic": |
||||
return 100, nil |
||||
case "premium": |
||||
return 1000, nil |
||||
default: |
||||
return 10, nil |
||||
} |
||||
} |
||||
|
||||
return 0, nil |
||||
} |
||||
|
||||
// IsFirstTimeUser checks if a pubkey is a first-time user
|
||||
func (d *D) IsFirstTimeUser(pubkey []byte) (bool, error) { |
||||
// Check if they have any subscription or payment history
|
||||
sub, _ := d.GetSubscription(pubkey) |
||||
if sub != nil { |
||||
return false, nil |
||||
} |
||||
|
||||
payments, _ := d.GetPaymentHistory(pubkey) |
||||
if len(payments) > 0 { |
||||
return false, nil |
||||
} |
||||
|
||||
return true, nil |
||||
} |
||||
@ -0,0 +1,30 @@
@@ -0,0 +1,30 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"io" |
||||
"os" |
||||
"testing" |
||||
|
||||
"lol.mleku.dev" |
||||
"lol.mleku.dev/log" |
||||
) |
||||
|
||||
func TestMain(m *testing.M) { |
||||
// Disable all logging during tests unless explicitly enabled
|
||||
if os.Getenv("TEST_LOG") == "" { |
||||
// Set log level to Off to suppress all logs
|
||||
lol.SetLogLevel("off") |
||||
// Also redirect output to discard
|
||||
lol.Writer = io.Discard |
||||
// Disable all log printers
|
||||
log.T = lol.GetNullPrinter() |
||||
log.D = lol.GetNullPrinter() |
||||
log.I = lol.GetNullPrinter() |
||||
log.W = lol.GetNullPrinter() |
||||
log.E = lol.GetNullPrinter() |
||||
log.F = lol.GetNullPrinter() |
||||
} |
||||
|
||||
// Run tests
|
||||
os.Exit(m.Run()) |
||||
} |
||||
@ -0,0 +1,10 @@
@@ -0,0 +1,10 @@
|
||||
package dgraph |
||||
|
||||
import ( |
||||
"encoding/json" |
||||
) |
||||
|
||||
// unmarshalJSON is a helper to unmarshal JSON with error handling
|
||||
func unmarshalJSON(data []byte, v interface{}) error { |
||||
return json.Unmarshal(data, v) |
||||
} |
||||
@ -0,0 +1,276 @@
@@ -0,0 +1,276 @@
|
||||
# Dgraph Integration Testing |
||||
|
||||
This directory contains scripts and configuration for testing the ORLY dgraph integration. |
||||
|
||||
## Quick Start |
||||
|
||||
### 1. Start Dgraph Server |
||||
|
||||
```bash |
||||
# Using the convenience script |
||||
./scripts/dgraph-start.sh |
||||
|
||||
# Or manually with docker-compose |
||||
cd scripts |
||||
docker-compose -f dgraph-docker-compose.yml up -d |
||||
|
||||
# Or directly with docker |
||||
docker run -d \ |
||||
-p 8080:8080 \ |
||||
-p 9080:9080 \ |
||||
-p 8000:8000 \ |
||||
--name dgraph-orly \ |
||||
dgraph/standalone:latest |
||||
``` |
||||
|
||||
### 2. Run Dgraph Tests |
||||
|
||||
```bash |
||||
# Run all dgraph package tests |
||||
./scripts/test-dgraph.sh |
||||
|
||||
# Run tests with relay-tester |
||||
./scripts/test-dgraph.sh --relay-tester |
||||
``` |
||||
|
||||
### 3. Manual Testing |
||||
|
||||
```bash |
||||
# Start ORLY with dgraph backend |
||||
export ORLY_DB_TYPE=dgraph |
||||
export ORLY_DGRAPH_URL=localhost:9080 |
||||
./orly |
||||
|
||||
# In another terminal, run relay-tester |
||||
go run cmd/relay-tester/main.go -url ws://localhost:3334 |
||||
``` |
||||
|
||||
## Test Files |
||||
|
||||
The dgraph package includes comprehensive tests: |
||||
|
||||
- **testmain_test.go** - Test configuration and logging setup |
||||
- **helpers_test.go** - Helper functions for test setup/teardown |
||||
- **save-event_test.go** - Event storage tests |
||||
- **query-events_test.go** - Event query tests |
||||
|
||||
All tests mirror the existing badger tests to ensure feature parity. |
||||
|
||||
## Test Coverage |
||||
|
||||
The dgraph tests cover: |
||||
|
||||
✅ **Event Storage** |
||||
- Saving events from examples.Cache |
||||
- Duplicate event rejection |
||||
- Deletion event validation |
||||
|
||||
✅ **Event Queries** |
||||
- Query by ID |
||||
- Query by kind |
||||
- Query by author |
||||
- Query by time range |
||||
- Query by tags |
||||
- Event counting |
||||
|
||||
✅ **Advanced Features** |
||||
- Replaceable events (kind 0) |
||||
- Parameterized replaceable events (kind 30000+) |
||||
- Event deletion (kind 5) |
||||
- Event replacement logic |
||||
|
||||
## Requirements |
||||
|
||||
### Dgraph Server |
||||
|
||||
The tests require a running dgraph server. Tests will be skipped if dgraph is not available. |
||||
|
||||
**Endpoints:** |
||||
- gRPC: `localhost:9080` (required for ORLY) |
||||
- HTTP: `localhost:8080` (for health checks) |
||||
- Ratel UI: `localhost:8000` (optional, for debugging) |
||||
|
||||
**Custom Endpoint:** |
||||
```bash |
||||
export ORLY_DGRAPH_URL=remote.server.com:9080 |
||||
./scripts/test-dgraph.sh |
||||
``` |
||||
|
||||
### Docker |
||||
|
||||
The docker-compose setup requires: |
||||
- Docker Engine 20.10+ |
||||
- Docker Compose 1.29+ (or docker-compose plugin) |
||||
|
||||
## Test Workflow |
||||
|
||||
### Running Tests Locally |
||||
|
||||
```bash |
||||
# 1. Start dgraph |
||||
./scripts/dgraph-start.sh |
||||
|
||||
# 2. Run tests |
||||
./scripts/test-dgraph.sh |
||||
|
||||
# 3. Clean up when done |
||||
cd scripts && docker-compose -f dgraph-docker-compose.yml down |
||||
``` |
||||
|
||||
### CI/CD Integration |
||||
|
||||
For CI pipelines, use the docker-compose file: |
||||
|
||||
```yaml |
||||
# Example GitHub Actions workflow |
||||
services: |
||||
dgraph: |
||||
image: dgraph/standalone:latest |
||||
ports: |
||||
- 8080:8080 |
||||
- 9080:9080 |
||||
|
||||
steps: |
||||
- name: Run dgraph tests |
||||
run: | |
||||
export ORLY_DGRAPH_URL=localhost:9080 |
||||
CGO_ENABLED=0 go test -v ./pkg/dgraph/... |
||||
``` |
||||
|
||||
## Debugging |
||||
|
||||
### View Dgraph Logs |
||||
|
||||
```bash |
||||
docker logs dgraph-orly-test -f |
||||
``` |
||||
|
||||
### Access Ratel UI |
||||
|
||||
Open http://localhost:8000 in your browser to: |
||||
- View schema |
||||
- Run DQL queries |
||||
- Inspect data |
||||
|
||||
### Enable Test Logging |
||||
|
||||
```bash |
||||
export TEST_LOG=1 |
||||
./scripts/test-dgraph.sh |
||||
``` |
||||
|
||||
### Manual DQL Queries |
||||
|
||||
```bash |
||||
# Using curl |
||||
curl -X POST localhost:8080/query -d '{ |
||||
q(func: type(Event)) { |
||||
uid |
||||
event.id |
||||
event.kind |
||||
event.created_at |
||||
} |
||||
}' |
||||
|
||||
# Using grpcurl (if installed) |
||||
grpcurl -plaintext -d '{ |
||||
"query": "{ q(func: type(Event)) { uid event.id } }" |
||||
}' localhost:9080 api.Dgraph/Query |
||||
``` |
||||
|
||||
## Troubleshooting |
||||
|
||||
### Tests Skip with "Dgraph server not available" |
||||
|
||||
**Solution:** Ensure dgraph is running: |
||||
```bash |
||||
docker ps | grep dgraph |
||||
./scripts/dgraph-start.sh |
||||
``` |
||||
|
||||
### Connection Refused Errors |
||||
|
||||
**Symptoms:** |
||||
``` |
||||
failed to connect to dgraph at localhost:9080: connection refused |
||||
``` |
||||
|
||||
**Solutions:** |
||||
1. Check dgraph is running: `docker ps` |
||||
2. Check port mapping: `docker port dgraph-orly-test` |
||||
3. Check firewall rules |
||||
4. Verify ORLY_DGRAPH_URL is correct |
||||
|
||||
### Schema Application Failed |
||||
|
||||
**Symptoms:** |
||||
``` |
||||
failed to apply schema: ... |
||||
``` |
||||
|
||||
**Solutions:** |
||||
1. Check dgraph logs: `docker logs dgraph-orly-test` |
||||
2. Drop all data and retry: Use `dropAll` in test setup |
||||
3. Verify dgraph version compatibility |
||||
|
||||
### Tests Timeout |
||||
|
||||
**Symptoms:** |
||||
``` |
||||
panic: test timed out after 10m |
||||
``` |
||||
|
||||
**Solutions:** |
||||
1. Increase timeout: `go test -timeout 20m ./pkg/dgraph/...` |
||||
2. Check dgraph performance: May need more resources |
||||
3. Reduce test dataset size |
||||
|
||||
## Performance Benchmarks |
||||
|
||||
Compare dgraph vs badger performance: |
||||
|
||||
```bash |
||||
# Run badger benchmarks |
||||
go test -bench=. ./pkg/database/... |
||||
|
||||
# Run dgraph benchmarks |
||||
go test -bench=. ./pkg/dgraph/... |
||||
``` |
||||
|
||||
## Test Data |
||||
|
||||
Tests use `pkg/encoders/event/examples.Cache` which contains: |
||||
- ~100 real Nostr events |
||||
- Various kinds (text notes, metadata, etc.) |
||||
- Different authors and timestamps |
||||
- Events with tags and relationships |
||||
|
||||
## Cleanup |
||||
|
||||
### Remove Test Data |
||||
|
||||
```bash |
||||
# Stop and remove containers |
||||
cd scripts |
||||
docker-compose -f dgraph-docker-compose.yml down |
||||
|
||||
# Remove volumes |
||||
docker volume rm scripts_dgraph-data |
||||
``` |
||||
|
||||
### Reset Dgraph |
||||
|
||||
```bash |
||||
# Drop all data (via test helper) |
||||
# The dropAll() function is called in test setup |
||||
|
||||
# Or manually via HTTP |
||||
curl -X POST localhost:8080/alter -d '{"drop_all": true}' |
||||
``` |
||||
|
||||
## Related Documentation |
||||
|
||||
- [Dgraph Implementation Status](../DGRAPH_IMPLEMENTATION_STATUS.md) |
||||
- [Package README](../pkg/dgraph/README.md) |
||||
- [Dgraph Documentation](https://dgraph.io/docs/) |
||||
- [DQL Query Language](https://dgraph.io/docs/query-language/) |
||||
@ -0,0 +1,546 @@
@@ -0,0 +1,546 @@
|
||||
# Docker-Based Integration Testing |
||||
|
||||
This guide covers running ORLY and Dgraph together in Docker containers for integration testing. |
||||
|
||||
## Overview |
||||
|
||||
The Docker setup provides: |
||||
- **Isolated Environment**: Dgraph + ORLY in containers |
||||
- **Automated Testing**: Health checks and dependency management |
||||
- **Reproducible Tests**: Consistent environment across systems |
||||
- **Easy Cleanup**: Remove everything with one command |
||||
|
||||
## Architecture |
||||
|
||||
``` |
||||
┌─────────────────────────────────────────────┐ |
||||
│ Docker Network (orly-network) │ |
||||
│ │ |
||||
│ ┌──────────────────┐ ┌─────────────────┐ │ |
||||
│ │ Dgraph │ │ ORLY Relay │ │ |
||||
│ │ standalone │◄─┤ (dgraph mode) │ │ |
||||
│ │ │ │ │ │ |
||||
│ │ :8080 (HTTP) │ │ :3334 (WS) │ │ |
||||
│ │ :9080 (gRPC) │ │ │ │ |
||||
│ │ :8000 (Ratel) │ │ │ │ |
||||
│ └──────────────────┘ └─────────────────┘ │ |
||||
│ │ │ │ |
||||
└─────────┼───────────────────────┼───────────┘ |
||||
│ │ |
||||
Published Published |
||||
to host to host |
||||
``` |
||||
|
||||
## Quick Start |
||||
|
||||
### 1. Build Images |
||||
|
||||
```bash |
||||
# Build ORLY image only |
||||
./scripts/docker-build.sh |
||||
|
||||
# Build ORLY + relay-tester |
||||
./scripts/docker-build.sh --with-tester |
||||
``` |
||||
|
||||
### 2. Run Integration Tests |
||||
|
||||
```bash |
||||
# Basic test (start containers, verify connectivity) |
||||
./scripts/test-docker.sh |
||||
|
||||
# Run with relay-tester |
||||
./scripts/test-docker.sh --relay-tester |
||||
|
||||
# Keep containers running after test |
||||
./scripts/test-docker.sh --keep-running |
||||
|
||||
# Skip rebuild (use existing images) |
||||
./scripts/test-docker.sh --skip-build |
||||
``` |
||||
|
||||
### 3. Manual Container Management |
||||
|
||||
```bash |
||||
# Start containers |
||||
cd scripts |
||||
docker-compose -f docker-compose-test.yml up -d |
||||
|
||||
# View logs |
||||
docker-compose -f docker-compose-test.yml logs -f |
||||
|
||||
# Stop containers |
||||
docker-compose -f docker-compose-test.yml down |
||||
|
||||
# Stop and remove volumes |
||||
docker-compose -f docker-compose-test.yml down -v |
||||
``` |
||||
|
||||
## Docker Files |
||||
|
||||
### Dockerfile |
||||
|
||||
Multi-stage build for ORLY: |
||||
|
||||
**Stage 1: Builder** |
||||
- Based on golang:1.21-alpine |
||||
- Downloads dependencies |
||||
- Builds static binary with `CGO_ENABLED=0` |
||||
- Copies libsecp256k1.so for crypto operations |
||||
|
||||
**Stage 2: Runtime** |
||||
- Based on alpine:latest (minimal) |
||||
- Copies binary and shared library |
||||
- Creates non-root user |
||||
- Sets up health checks |
||||
- ~50MB final image size |
||||
|
||||
### Dockerfile.relay-tester |
||||
|
||||
Builds relay-tester for automated testing: |
||||
- Static binary from cmd/relay-tester |
||||
- Configurable RELAY_URL |
||||
- Runs as part of test profile |
||||
|
||||
### docker-compose-test.yml |
||||
|
||||
Orchestrates the full stack: |
||||
|
||||
**Services:** |
||||
1. **dgraph** - Database backend |
||||
- Health check via HTTP |
||||
- Persistent volume for data |
||||
- Exposed ports for debugging |
||||
|
||||
2. **orly** - Relay server |
||||
- Depends on dgraph (waits for healthy) |
||||
- Configured with ORLY_DB_TYPE=dgraph |
||||
- Health check via HTTP |
||||
- Auto-restart on failure |
||||
|
||||
3. **relay-tester** - Test runner |
||||
- Profile: test (optional) |
||||
- Runs tests against ORLY |
||||
- Exits after completion |
||||
|
||||
## Configuration |
||||
|
||||
### Environment Variables |
||||
|
||||
The docker-compose file sets: |
||||
|
||||
```yaml |
||||
# Database |
||||
ORLY_DB_TYPE: dgraph |
||||
ORLY_DGRAPH_URL: dgraph:9080 # Internal network name |
||||
|
||||
# Server |
||||
ORLY_LISTEN: 0.0.0.0 |
||||
ORLY_PORT: 3334 |
||||
ORLY_DATA_DIR: /data |
||||
|
||||
# Application |
||||
ORLY_LOG_LEVEL: info |
||||
ORLY_APP_NAME: ORLY-Dgraph-Test |
||||
ORLY_ACL_MODE: none |
||||
``` |
||||
|
||||
Override via environment or .env file: |
||||
|
||||
```bash |
||||
# Create .env file in scripts/ |
||||
cat > scripts/.env << EOF |
||||
ORLY_LOG_LEVEL=debug |
||||
ORLY_ADMINS=npub1... |
||||
EOF |
||||
``` |
||||
|
||||
### Volumes |
||||
|
||||
**Persistent Data:** |
||||
- `dgraph-data:/dgraph` - Dgraph database |
||||
- `orly-data:/data` - ORLY metadata |
||||
|
||||
**Inspect Volumes:** |
||||
```bash |
||||
docker volume ls |
||||
docker volume inspect scripts_dgraph-data |
||||
``` |
||||
|
||||
### Networks |
||||
|
||||
**Custom Bridge Network:** |
||||
- Name: orly-network |
||||
- Subnet: 172.28.0.0/16 |
||||
- Allows container-to-container communication |
||||
- DNS resolution by service name |
||||
|
||||
## Testing Workflows |
||||
|
||||
### Basic Integration Test |
||||
|
||||
```bash |
||||
./scripts/test-docker.sh |
||||
``` |
||||
|
||||
**What it does:** |
||||
1. Stops any existing containers |
||||
2. Starts dgraph and waits for health |
||||
3. Starts ORLY and waits for health |
||||
4. Verifies HTTP connectivity |
||||
5. Tests WebSocket (if websocat installed) |
||||
6. Shows container status |
||||
7. Cleans up (unless --keep-running) |
||||
|
||||
### With Relay-Tester |
||||
|
||||
```bash |
||||
./scripts/test-docker.sh --relay-tester |
||||
``` |
||||
|
||||
**Additional steps:** |
||||
1. Builds relay-tester image |
||||
2. Runs comprehensive protocol tests |
||||
3. Reports pass/fail |
||||
4. Shows ORLY logs on failure |
||||
|
||||
### Development Workflow |
||||
|
||||
```bash |
||||
# Start and keep running |
||||
./scripts/test-docker.sh --keep-running |
||||
|
||||
# Make changes to code |
||||
vim pkg/dgraph/save-event.go |
||||
|
||||
# Rebuild and restart |
||||
docker-compose -f scripts/docker-compose-test.yml up -d --build orly |
||||
|
||||
# View logs |
||||
docker logs orly-relay -f |
||||
|
||||
# Test changes |
||||
go run cmd/relay-tester/main.go -url ws://localhost:3334 |
||||
|
||||
# Stop when done |
||||
cd scripts && docker-compose -f docker-compose-test.yml down |
||||
``` |
||||
|
||||
## Debugging |
||||
|
||||
### View Container Logs |
||||
|
||||
```bash |
||||
# All services |
||||
docker-compose -f scripts/docker-compose-test.yml logs -f |
||||
|
||||
# Specific service |
||||
docker logs orly-relay -f |
||||
docker logs orly-dgraph -f |
||||
|
||||
# Last N lines |
||||
docker logs orly-relay --tail 50 |
||||
``` |
||||
|
||||
### Execute Commands in Container |
||||
|
||||
```bash |
||||
# ORLY version |
||||
docker exec orly-relay /app/orly version |
||||
|
||||
# Check ORLY processes |
||||
docker exec orly-relay ps aux |
||||
|
||||
# Inspect data directory |
||||
docker exec orly-relay ls -la /data |
||||
|
||||
# Query dgraph |
||||
docker exec orly-dgraph curl http://localhost:8080/health |
||||
``` |
||||
|
||||
### Access Ratel UI |
||||
|
||||
Open http://localhost:8000 in browser: |
||||
- View dgraph schema |
||||
- Run DQL queries |
||||
- Inspect stored data |
||||
- Monitor performance |
||||
|
||||
### Network Inspection |
||||
|
||||
```bash |
||||
# List networks |
||||
docker network ls |
||||
|
||||
# Inspect orly network |
||||
docker network inspect scripts_orly-network |
||||
|
||||
# Test connectivity |
||||
docker exec orly-relay ping dgraph |
||||
docker exec orly-relay nc -zv dgraph 9080 |
||||
``` |
||||
|
||||
### Health Check Status |
||||
|
||||
```bash |
||||
# Check health |
||||
docker inspect orly-relay | grep -A 10 Health |
||||
|
||||
# View health check logs |
||||
docker inspect --format='{{json .State.Health}}' orly-relay | jq |
||||
``` |
||||
|
||||
## Troubleshooting |
||||
|
||||
### Build Failures |
||||
|
||||
**Error: Cannot find libsecp256k1.so** |
||||
|
||||
```bash |
||||
# Ensure library exists |
||||
ls -l pkg/crypto/p8k/libsecp256k1.so |
||||
|
||||
# Rebuild if needed |
||||
cd pkg/crypto/p8k && make |
||||
``` |
||||
|
||||
**Error: Go module download fails** |
||||
|
||||
```bash |
||||
# Clear module cache |
||||
go clean -modcache |
||||
|
||||
# Try building locally first |
||||
CGO_ENABLED=0 go build |
||||
``` |
||||
|
||||
### Runtime Failures |
||||
|
||||
**ORLY fails health check** |
||||
|
||||
```bash |
||||
# Check logs |
||||
docker logs orly-relay |
||||
|
||||
# Common issues: |
||||
# - Port already in use: docker ps (check for conflicts) |
||||
# - Dgraph not ready: docker logs orly-dgraph |
||||
# - Bad configuration: docker exec orly-relay env |
||||
``` |
||||
|
||||
**Cannot connect to dgraph** |
||||
|
||||
```bash |
||||
# Verify dgraph is healthy |
||||
docker inspect orly-dgraph | grep Health |
||||
|
||||
# Check network connectivity |
||||
docker exec orly-relay ping dgraph |
||||
docker exec orly-relay nc -zv dgraph 9080 |
||||
|
||||
# Verify dgraph is listening |
||||
docker exec orly-dgraph netstat -tlnp | grep 9080 |
||||
``` |
||||
|
||||
**WebSocket connection fails** |
||||
|
||||
```bash |
||||
# Test from host |
||||
websocat ws://localhost:3334 |
||||
|
||||
# Test from container |
||||
docker exec orly-relay curl -v http://localhost:3334 |
||||
|
||||
# Check firewall |
||||
sudo iptables -L | grep 3334 |
||||
``` |
||||
|
||||
### Performance Issues |
||||
|
||||
**Slow startup** |
||||
|
||||
```bash |
||||
# Increase health check timeouts in docker-compose-test.yml |
||||
start_period: 60s # Default is 20-30s |
||||
|
||||
# Pre-pull images |
||||
docker pull dgraph/standalone:latest |
||||
docker pull golang:1.21-alpine |
||||
``` |
||||
|
||||
**High memory usage** |
||||
|
||||
```bash |
||||
# Check resource usage |
||||
docker stats |
||||
|
||||
# Limit container resources |
||||
# Add to docker-compose-test.yml: |
||||
deploy: |
||||
resources: |
||||
limits: |
||||
memory: 2G |
||||
cpus: '2' |
||||
``` |
||||
|
||||
## CI/CD Integration |
||||
|
||||
### GitHub Actions Example |
||||
|
||||
```yaml |
||||
name: Docker Integration Tests |
||||
|
||||
on: [push, pull_request] |
||||
|
||||
jobs: |
||||
docker-test: |
||||
runs-on: ubuntu-latest |
||||
|
||||
steps: |
||||
- uses: actions/checkout@v3 |
||||
|
||||
- name: Set up Docker Buildx |
||||
uses: docker/setup-buildx-action@v2 |
||||
|
||||
- name: Build images |
||||
run: ./scripts/docker-build.sh --with-tester |
||||
|
||||
- name: Run integration tests |
||||
run: ./scripts/test-docker.sh --relay-tester |
||||
|
||||
- name: Upload logs on failure |
||||
if: failure() |
||||
uses: actions/upload-artifact@v3 |
||||
with: |
||||
name: container-logs |
||||
path: | |
||||
scripts/orly-relay.log |
||||
scripts/dgraph.log |
||||
``` |
||||
|
||||
### GitLab CI Example |
||||
|
||||
```yaml |
||||
docker-test: |
||||
image: docker:latest |
||||
services: |
||||
- docker:dind |
||||
script: |
||||
- ./scripts/docker-build.sh --with-tester |
||||
- ./scripts/test-docker.sh --relay-tester |
||||
artifacts: |
||||
when: on_failure |
||||
paths: |
||||
- scripts/*.log |
||||
``` |
||||
|
||||
## Advanced Usage |
||||
|
||||
### Custom Configuration |
||||
|
||||
Create a custom docker-compose override: |
||||
|
||||
```yaml |
||||
# docker-compose.override.yml |
||||
version: '3.8' |
||||
|
||||
services: |
||||
orly: |
||||
environment: |
||||
- ORLY_LOG_LEVEL=debug |
||||
- ORLY_ADMINS=npub1... |
||||
- ORLY_ACL_MODE=follows |
||||
ports: |
||||
- "3335:3334" # Different host port |
||||
``` |
||||
|
||||
### Multi-Instance Testing |
||||
|
||||
Test multiple ORLY instances: |
||||
|
||||
```yaml |
||||
# docker-compose-multi.yml |
||||
services: |
||||
orly-1: |
||||
extends: |
||||
file: docker-compose-test.yml |
||||
service: orly |
||||
container_name: orly-relay-1 |
||||
ports: |
||||
- "3334:3334" |
||||
|
||||
orly-2: |
||||
extends: |
||||
file: docker-compose-test.yml |
||||
service: orly |
||||
container_name: orly-relay-2 |
||||
ports: |
||||
- "3335:3334" |
||||
``` |
||||
|
||||
### Performance Benchmarking |
||||
|
||||
```bash |
||||
# Start with --keep-running |
||||
./scripts/test-docker.sh --keep-running |
||||
|
||||
# Run stress test |
||||
go run cmd/stresstest/main.go -url ws://localhost:3334 -connections 100 |
||||
|
||||
# Monitor resources |
||||
docker stats |
||||
|
||||
# Profile ORLY |
||||
docker exec orly-relay sh -c 'curl http://localhost:6060/debug/pprof/profile?seconds=30 > /tmp/cpu.prof' |
||||
``` |
||||
|
||||
## Cleanup |
||||
|
||||
### Remove Everything |
||||
|
||||
```bash |
||||
# Stop and remove containers |
||||
cd scripts && docker-compose -f docker-compose-test.yml down |
||||
|
||||
# Remove volumes (data) |
||||
docker-compose -f docker-compose-test.yml down -v |
||||
|
||||
# Remove images |
||||
docker rmi orly:latest orly-relay-tester:latest |
||||
|
||||
# Remove networks |
||||
docker network rm scripts_orly-network |
||||
|
||||
# Prune everything (careful!) |
||||
docker system prune -a --volumes |
||||
``` |
||||
|
||||
### Selective Cleanup |
||||
|
||||
```bash |
||||
# Just stop containers (keep data) |
||||
docker-compose -f docker-compose-test.yml stop |
||||
|
||||
# Remove only one service |
||||
docker-compose -f docker-compose-test.yml rm -s -f orly |
||||
|
||||
# Clear dgraph data |
||||
docker volume rm scripts_dgraph-data |
||||
``` |
||||
|
||||
## Related Documentation |
||||
|
||||
- [Main Testing Guide](DGRAPH_TESTING.md) |
||||
- [Package Tests](../pkg/dgraph/TESTING.md) |
||||
- [Docker Documentation](https://docs.docker.com/) |
||||
- [Docker Compose](https://docs.docker.com/compose/) |
||||
|
||||
## Best Practices |
||||
|
||||
1. **Always use health checks** - Ensure services are ready |
||||
2. **Use specific tags** - Don't rely on :latest in production |
||||
3. **Limit resources** - Prevent container resource exhaustion |
||||
4. **Volume backups** - Backup dgraph-data volume before updates |
||||
5. **Network isolation** - Use custom networks for security |
||||
6. **Read-only root** - Run as non-root user |
||||
7. **Clean up regularly** - Remove unused containers/volumes |
||||
@ -0,0 +1,424 @@
@@ -0,0 +1,424 @@
|
||||
# ORLY Scripts Directory |
||||
|
||||
This directory contains automation scripts for building, testing, and deploying ORLY. |
||||
|
||||
## Quick Reference |
||||
|
||||
### Dgraph Integration Testing |
||||
|
||||
```bash |
||||
# Local testing (requires dgraph server) |
||||
./dgraph-start.sh # Start dgraph server |
||||
./test-dgraph.sh # Run dgraph package tests |
||||
./test-dgraph.sh --relay-tester # Run tests + relay-tester |
||||
|
||||
# Docker testing (containers for everything) |
||||
./docker-build.sh # Build ORLY docker image |
||||
./test-docker.sh # Run integration tests in containers |
||||
./test-docker.sh --relay-tester --keep-running # Full test, keep running |
||||
``` |
||||
|
||||
### Build & Deploy |
||||
|
||||
```bash |
||||
./build-all-platforms.sh # Build for multiple platforms |
||||
./deploy.sh # Deploy to systemd |
||||
./update-embedded-web.sh # Build and embed web UI |
||||
``` |
||||
|
||||
## Script Descriptions |
||||
|
||||
### Dgraph Testing Scripts |
||||
|
||||
#### dgraph-start.sh |
||||
Starts dgraph server using docker-compose for local testing. |
||||
|
||||
**Usage:** |
||||
```bash |
||||
./dgraph-start.sh |
||||
``` |
||||
|
||||
**What it does:** |
||||
- Checks if dgraph is already running |
||||
- Starts dgraph via docker-compose |
||||
- Waits for health check |
||||
- Shows endpoints and commands |
||||
|
||||
#### dgraph-docker-compose.yml |
||||
Docker Compose configuration for standalone dgraph server. |
||||
|
||||
**Ports:** |
||||
- 8080: HTTP API |
||||
- 9080: gRPC (ORLY connects here) |
||||
- 8000: Ratel UI |
||||
|
||||
#### test-dgraph.sh |
||||
Runs dgraph package tests against a running dgraph server. |
||||
|
||||
**Usage:** |
||||
```bash |
||||
./test-dgraph.sh # Just tests |
||||
./test-dgraph.sh --relay-tester # Tests + relay-tester |
||||
``` |
||||
|
||||
**Requirements:** |
||||
- Dgraph server running at ORLY_DGRAPH_URL (default: localhost:9080) |
||||
- Go 1.21+ |
||||
|
||||
### Docker Integration Scripts |
||||
|
||||
#### docker-build.sh |
||||
Builds Docker images for ORLY and optionally relay-tester. |
||||
|
||||
**Usage:** |
||||
```bash |
||||
./docker-build.sh # ORLY only |
||||
./docker-build.sh --with-tester # ORLY + relay-tester |
||||
``` |
||||
|
||||
**Output:** |
||||
- orly:latest |
||||
- orly-relay-tester:latest (if --with-tester) |
||||
|
||||
#### docker-compose-test.yml |
||||
Full-stack docker-compose with dgraph, ORLY, and relay-tester. |
||||
|
||||
**Services:** |
||||
- dgraph: Database backend |
||||
- orly: Relay with dgraph backend |
||||
- relay-tester: Protocol tests (optional, profile: test) |
||||
|
||||
**Features:** |
||||
- Health checks for all services |
||||
- Dependency management (ORLY waits for dgraph) |
||||
- Custom network with DNS |
||||
- Persistent volumes |
||||
|
||||
#### test-docker.sh |
||||
Comprehensive integration testing in Docker containers. |
||||
|
||||
**Usage:** |
||||
```bash |
||||
./test-docker.sh # Basic test |
||||
./test-docker.sh --relay-tester # Run relay-tester |
||||
./test-docker.sh --keep-running # Keep containers running |
||||
./test-docker.sh --skip-build # Use existing images |
||||
./test-docker.sh --relay-tester --keep-running # Full test + keep running |
||||
``` |
||||
|
||||
**What it does:** |
||||
1. Stops any existing containers |
||||
2. Optionally rebuilds images |
||||
3. Starts dgraph and waits for health |
||||
4. Starts ORLY and waits for health |
||||
5. Verifies connectivity |
||||
6. Optionally runs relay-tester |
||||
7. Shows status and endpoints |
||||
8. Cleanup (unless --keep-running) |
||||
|
||||
### Build Scripts |
||||
|
||||
#### build-all-platforms.sh |
||||
Cross-compiles ORLY for multiple platforms. |
||||
|
||||
**Platforms:** |
||||
- linux/amd64 |
||||
- linux/arm64 |
||||
- darwin/amd64 |
||||
- darwin/arm64 |
||||
|
||||
**Output:** `dist/` directory with platform-specific binaries |
||||
|
||||
#### update-embedded-web.sh |
||||
Builds the Svelte web UI and embeds it in ORLY binary. |
||||
|
||||
**Steps:** |
||||
1. Builds web UI with bun |
||||
2. Generates embedded assets |
||||
3. Rebuilds ORLY with embedded UI |
||||
|
||||
### Deployment Scripts |
||||
|
||||
#### deploy.sh |
||||
Automated deployment with systemd service. |
||||
|
||||
**What it does:** |
||||
1. Installs Go if needed |
||||
2. Builds ORLY with embedded web UI |
||||
3. Installs to ~/.local/bin/orly |
||||
4. Creates systemd service |
||||
5. Enables and starts service |
||||
6. Sets up port binding capabilities |
||||
|
||||
### Test Scripts |
||||
|
||||
#### test.sh |
||||
Runs all Go tests in the project. |
||||
|
||||
**Usage:** |
||||
```bash |
||||
./test.sh # All tests |
||||
TEST_LOG=1 ./test.sh # With logging |
||||
``` |
||||
|
||||
## Environment Variables |
||||
|
||||
### Common Variables |
||||
|
||||
```bash |
||||
# Dgraph |
||||
export ORLY_DGRAPH_URL=localhost:9080 # Dgraph endpoint |
||||
export ORLY_DB_TYPE=dgraph # Use dgraph backend |
||||
|
||||
# Logging |
||||
export ORLY_LOG_LEVEL=debug # Log verbosity |
||||
export TEST_LOG=1 # Enable test logging |
||||
|
||||
# Server |
||||
export ORLY_PORT=3334 # HTTP/WebSocket port |
||||
export ORLY_LISTEN=0.0.0.0 # Listen address |
||||
|
||||
# Data |
||||
export ORLY_DATA_DIR=/path/to/data # Data directory |
||||
``` |
||||
|
||||
### Script-Specific Variables |
||||
|
||||
```bash |
||||
# Docker scripts |
||||
export SKIP_BUILD=true # Skip image rebuild |
||||
export KEEP_RUNNING=true # Don't cleanup containers |
||||
|
||||
# Dgraph scripts |
||||
export DGRAPH_VERSION=latest # Dgraph image tag |
||||
``` |
||||
|
||||
## File Organization |
||||
|
||||
``` |
||||
scripts/ |
||||
├── README.md # This file |
||||
├── DGRAPH_TESTING.md # Dgraph testing guide |
||||
├── DOCKER_TESTING.md # Docker testing guide |
||||
│ |
||||
├── dgraph-start.sh # Start dgraph server |
||||
├── dgraph-docker-compose.yml # Dgraph docker config |
||||
├── test-dgraph.sh # Run dgraph tests |
||||
│ |
||||
├── docker-build.sh # Build docker images |
||||
├── docker-compose-test.yml # Full stack docker config |
||||
├── test-docker.sh # Run docker integration tests |
||||
│ |
||||
├── build-all-platforms.sh # Cross-compile |
||||
├── deploy.sh # Deploy to systemd |
||||
├── update-embedded-web.sh # Build web UI |
||||
└── test.sh # Run Go tests |
||||
``` |
||||
|
||||
## Workflows |
||||
|
||||
### Local Development with Dgraph |
||||
|
||||
```bash |
||||
# 1. Start dgraph |
||||
./scripts/dgraph-start.sh |
||||
|
||||
# 2. Run ORLY locally with dgraph |
||||
export ORLY_DB_TYPE=dgraph |
||||
export ORLY_DGRAPH_URL=localhost:9080 |
||||
./orly |
||||
|
||||
# 3. Test changes |
||||
go run cmd/relay-tester/main.go -url ws://localhost:3334 |
||||
|
||||
# 4. Run unit tests |
||||
./scripts/test-dgraph.sh |
||||
``` |
||||
|
||||
### Docker Development |
||||
|
||||
```bash |
||||
# 1. Make changes |
||||
vim pkg/dgraph/save-event.go |
||||
|
||||
# 2. Build and test in containers |
||||
./scripts/test-docker.sh --relay-tester --keep-running |
||||
|
||||
# 3. Make more changes |
||||
|
||||
# 4. Rebuild just ORLY |
||||
cd scripts |
||||
docker-compose -f docker-compose-test.yml up -d --build orly |
||||
|
||||
# 5. View logs |
||||
docker logs orly-relay -f |
||||
|
||||
# 6. Stop when done |
||||
docker-compose -f docker-compose-test.yml down |
||||
``` |
||||
|
||||
### CI/CD Testing |
||||
|
||||
```bash |
||||
# Quick test (no containers) |
||||
./scripts/test.sh |
||||
|
||||
# Full integration test |
||||
./scripts/test-docker.sh --relay-tester |
||||
|
||||
# Build for deployment |
||||
./scripts/build-all-platforms.sh |
||||
``` |
||||
|
||||
### Production Deployment |
||||
|
||||
```bash |
||||
# Deploy with systemd |
||||
./scripts/deploy.sh |
||||
|
||||
# Check status |
||||
systemctl status orly |
||||
|
||||
# View logs |
||||
journalctl -u orly -f |
||||
|
||||
# Update |
||||
./scripts/deploy.sh # Rebuilds and restarts |
||||
``` |
||||
|
||||
## Troubleshooting |
||||
|
||||
### Dgraph Not Available |
||||
|
||||
```bash |
||||
# Check if running |
||||
docker ps | grep dgraph |
||||
|
||||
# Start it |
||||
./scripts/dgraph-start.sh |
||||
|
||||
# Check logs |
||||
docker logs dgraph-orly-test -f |
||||
``` |
||||
|
||||
### Port Conflicts |
||||
|
||||
```bash |
||||
# Find what's using port 3334 |
||||
lsof -i :3334 |
||||
netstat -tlnp | grep 3334 |
||||
|
||||
# Kill process |
||||
kill $(lsof -t -i :3334) |
||||
|
||||
# Or use different port |
||||
export ORLY_PORT=3335 |
||||
``` |
||||
|
||||
### Docker Build Failures |
||||
|
||||
```bash |
||||
# Clear docker cache |
||||
docker builder prune |
||||
|
||||
# Rebuild from scratch |
||||
docker build --no-cache -t orly:latest -f Dockerfile . |
||||
|
||||
# Check Dockerfile syntax |
||||
docker build --dry-run -f Dockerfile . |
||||
``` |
||||
|
||||
### Permission Issues |
||||
|
||||
```bash |
||||
# Fix script permissions |
||||
chmod +x scripts/*.sh |
||||
|
||||
# Fix docker socket |
||||
sudo usermod -aG docker $USER |
||||
newgrp docker |
||||
``` |
||||
|
||||
## Best Practices |
||||
|
||||
1. **Always use scripts from project root** |
||||
```bash |
||||
./scripts/test-docker.sh # Good |
||||
cd scripts && ./test-docker.sh # May have path issues |
||||
``` |
||||
|
||||
2. **Check prerequisites before running** |
||||
```bash |
||||
# Check docker |
||||
docker --version |
||||
docker-compose --version |
||||
|
||||
# Check dgraph |
||||
curl http://localhost:9080/health |
||||
``` |
||||
|
||||
3. **Clean up after testing** |
||||
```bash |
||||
# Stop containers |
||||
cd scripts && docker-compose -f docker-compose-test.yml down |
||||
|
||||
# Remove volumes if needed |
||||
docker-compose -f docker-compose-test.yml down -v |
||||
``` |
||||
|
||||
4. **Use --keep-running for debugging** |
||||
```bash |
||||
./scripts/test-docker.sh --keep-running |
||||
# Inspect, debug, make changes |
||||
docker-compose -f scripts/docker-compose-test.yml down |
||||
``` |
||||
|
||||
5. **Check logs on failures** |
||||
```bash |
||||
# Container logs |
||||
docker logs orly-relay --tail 100 |
||||
|
||||
# Test output |
||||
./scripts/test-dgraph.sh 2>&1 | tee test.log |
||||
``` |
||||
|
||||
## Related Documentation |
||||
|
||||
- [Dgraph Testing Guide](DGRAPH_TESTING.md) |
||||
- [Docker Testing Guide](DOCKER_TESTING.md) |
||||
- [Package Tests](../pkg/dgraph/TESTING.md) |
||||
- [Main Implementation Status](../DGRAPH_IMPLEMENTATION_STATUS.md) |
||||
|
||||
## Contributing |
||||
|
||||
When adding new scripts: |
||||
|
||||
1. **Add executable permission** |
||||
```bash |
||||
chmod +x scripts/new-script.sh |
||||
``` |
||||
|
||||
2. **Use bash strict mode** |
||||
```bash |
||||
#!/bin/bash |
||||
set -e # Exit on error |
||||
``` |
||||
|
||||
3. **Add help text** |
||||
```bash |
||||
if [ "$1" == "--help" ]; then |
||||
echo "Usage: $0 [options]" |
||||
exit 0 |
||||
fi |
||||
``` |
||||
|
||||
4. **Document in this README** |
||||
- Add to appropriate section |
||||
- Include usage examples |
||||
- Note any requirements |
||||
|
||||
5. **Test on fresh system** |
||||
```bash |
||||
# Use Docker to test |
||||
docker run --rm -v $(pwd):/app -w /app ubuntu:latest ./scripts/new-script.sh |
||||
``` |
||||
@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
|
||||
version: '3.8' |
||||
|
||||
services: |
||||
dgraph: |
||||
image: dgraph/standalone:latest |
||||
container_name: dgraph-orly-test |
||||
ports: |
||||
- "8080:8080" # HTTP API |
||||
- "9080:9080" # gRPC |
||||
- "8000:8000" # Ratel UI |
||||
volumes: |
||||
- dgraph-data:/dgraph |
||||
environment: |
||||
- DGRAPH_ALPHA_JAEGER_COLLECTOR=false |
||||
restart: unless-stopped |
||||
healthcheck: |
||||
test: ["CMD", "curl", "-f", "http://localhost:8080/health"] |
||||
interval: 10s |
||||
timeout: 5s |
||||
retries: 5 |
||||
start_period: 20s |
||||
|
||||
volumes: |
||||
dgraph-data: |
||||
driver: local |
||||
@ -0,0 +1,50 @@
@@ -0,0 +1,50 @@
|
||||
#!/bin/bash |
||||
# Quick script to start dgraph for testing |
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
||||
|
||||
echo "Starting dgraph server for ORLY testing..." |
||||
cd "$SCRIPT_DIR" |
||||
|
||||
# Check if already running |
||||
if docker ps | grep -q dgraph-orly-test; then |
||||
echo "✅ Dgraph is already running" |
||||
echo "" |
||||
echo "Dgraph endpoints:" |
||||
echo " gRPC: localhost:9080" |
||||
echo " HTTP: http://localhost:8080" |
||||
echo " Ratel UI: http://localhost:8000" |
||||
exit 0 |
||||
fi |
||||
|
||||
# Determine docker-compose command |
||||
if docker compose version &> /dev/null 2>&1; then |
||||
DOCKER_COMPOSE="docker compose" |
||||
else |
||||
DOCKER_COMPOSE="docker-compose" |
||||
fi |
||||
|
||||
# Start using docker compose |
||||
$DOCKER_COMPOSE -f dgraph-docker-compose.yml up -d |
||||
|
||||
echo "" |
||||
echo "Waiting for dgraph to be healthy..." |
||||
for i in {1..30}; do |
||||
if docker exec dgraph-orly-test curl -sf http://localhost:8080/health > /dev/null 2>&1; then |
||||
echo "✅ Dgraph is healthy and ready" |
||||
echo "" |
||||
echo "Dgraph endpoints:" |
||||
echo " gRPC: localhost:9080" |
||||
echo " HTTP: http://localhost:8080" |
||||
echo " Ratel UI: http://localhost:8000" |
||||
echo "" |
||||
echo "To stop: $DOCKER_COMPOSE -f dgraph-docker-compose.yml down" |
||||
echo "To view logs: docker logs dgraph-orly-test -f" |
||||
exit 0 |
||||
fi |
||||
sleep 1 |
||||
done |
||||
|
||||
echo "❌ Dgraph failed to become healthy" |
||||
docker logs dgraph-orly-test |
||||
exit 1 |
||||
@ -0,0 +1,45 @@
@@ -0,0 +1,45 @@
|
||||
#!/bin/bash |
||||
set -e |
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")" |
||||
|
||||
echo "=== Building ORLY Docker Images ===" |
||||
echo "" |
||||
|
||||
# Change to project root |
||||
cd "$PROJECT_ROOT" |
||||
|
||||
# Determine docker-compose command |
||||
if docker compose version &> /dev/null 2>&1; then |
||||
DOCKER_COMPOSE="docker compose" |
||||
else |
||||
DOCKER_COMPOSE="docker-compose" |
||||
fi |
||||
|
||||
# Build ORLY image |
||||
echo "Building ORLY relay image..." |
||||
docker build -t orly:latest -f Dockerfile . |
||||
echo "✅ ORLY image built successfully" |
||||
echo "" |
||||
|
||||
# Build relay-tester image (optional) |
||||
if [ "$1" == "--with-tester" ]; then |
||||
echo "Building relay-tester image..." |
||||
docker build -t orly-relay-tester:latest -f Dockerfile.relay-tester . |
||||
echo "✅ Relay-tester image built successfully" |
||||
echo "" |
||||
fi |
||||
|
||||
# Show images |
||||
echo "Built images:" |
||||
docker images | grep -E "orly|REPOSITORY" |
||||
echo "" |
||||
|
||||
echo "=== Build Complete ===" |
||||
echo "" |
||||
echo "To run:" |
||||
echo " cd scripts && $DOCKER_COMPOSE -f docker-compose-test.yml up -d" |
||||
echo "" |
||||
echo "To test:" |
||||
echo " ./scripts/test-docker.sh" |
||||
@ -0,0 +1,93 @@
@@ -0,0 +1,93 @@
|
||||
version: '3.8' |
||||
|
||||
services: |
||||
# Dgraph database |
||||
dgraph: |
||||
image: dgraph/standalone:latest |
||||
container_name: orly-dgraph |
||||
ports: |
||||
- "8080:8080" # HTTP API |
||||
- "9080:9080" # gRPC (ORLY connects here) |
||||
- "8000:8000" # Ratel UI |
||||
volumes: |
||||
- dgraph-data:/dgraph |
||||
environment: |
||||
- DGRAPH_ALPHA_JAEGER_COLLECTOR=false |
||||
networks: |
||||
- orly-network |
||||
healthcheck: |
||||
test: ["CMD", "curl", "-f", "http://localhost:8080/health"] |
||||
interval: 10s |
||||
timeout: 5s |
||||
retries: 5 |
||||
start_period: 20s |
||||
|
||||
# ORLY relay with dgraph backend |
||||
orly: |
||||
build: |
||||
context: .. |
||||
dockerfile: Dockerfile |
||||
container_name: orly-relay |
||||
ports: |
||||
- "3334:3334" # WebSocket/HTTP |
||||
depends_on: |
||||
dgraph: |
||||
condition: service_healthy |
||||
environment: |
||||
# Database configuration |
||||
- ORLY_DB_TYPE=dgraph |
||||
- ORLY_DGRAPH_URL=dgraph:9080 |
||||
- ORLY_DATA_DIR=/data |
||||
|
||||
# Server configuration |
||||
- ORLY_LISTEN=0.0.0.0 |
||||
- ORLY_PORT=3334 |
||||
- ORLY_LOG_LEVEL=info |
||||
- ORLY_APP_NAME=ORLY-Dgraph-Test |
||||
|
||||
# Admin configuration (example) |
||||
- ORLY_ADMINS=npub1fjqqy4a93z5zsjwsfxqhc2764kvykfdyttvldkkkdera8dr78vhsmmleku |
||||
- ORLY_OWNERS=npub1fjqqy4a93z5zsjwsfxqhc2764kvykfdyttvldkkkdera8dr78vhsmmleku |
||||
|
||||
# ACL mode |
||||
- ORLY_ACL_MODE=none |
||||
volumes: |
||||
- orly-data:/data |
||||
networks: |
||||
- orly-network |
||||
healthcheck: |
||||
test: ["CMD", "curl", "-f", "http://localhost:3334/"] |
||||
interval: 10s |
||||
timeout: 5s |
||||
retries: 5 |
||||
start_period: 30s |
||||
restart: unless-stopped |
||||
|
||||
# Relay tester (optional, for automated testing) |
||||
relay-tester: |
||||
build: |
||||
context: .. |
||||
dockerfile: Dockerfile.relay-tester |
||||
container_name: orly-tester |
||||
depends_on: |
||||
orly: |
||||
condition: service_healthy |
||||
environment: |
||||
- RELAY_URL=ws://orly:3334 |
||||
networks: |
||||
- orly-network |
||||
profiles: |
||||
- test |
||||
|
||||
networks: |
||||
orly-network: |
||||
driver: bridge |
||||
ipam: |
||||
config: |
||||
- subnet: 172.28.0.0/16 |
||||
|
||||
volumes: |
||||
dgraph-data: |
||||
driver: local |
||||
orly-data: |
||||
driver: local |
||||
@ -0,0 +1,308 @@
@@ -0,0 +1,308 @@
|
||||
#!/bin/bash |
||||
# Badger Database Migration Script |
||||
# Migrates ORLY database to new Badger configuration with VLogPercentile optimization |
||||
|
||||
set -e # Exit on error |
||||
|
||||
# Colors for output |
||||
RED='\033[0;31m' |
||||
GREEN='\033[0;32m' |
||||
YELLOW='\033[1;33m' |
||||
NC='\033[0m' # No Color |
||||
|
||||
echo -e "${GREEN}=== ORLY Badger Database Migration ===${NC}" |
||||
echo "" |
||||
|
||||
# Configuration |
||||
DATA_DIR="${ORLY_DATA_DIR:-$HOME/.local/share/ORLY}" |
||||
BACKUP_DIR="${DATA_DIR}-backup-$(date +%Y%m%d-%H%M%S)" |
||||
EXPORT_FILE="${DATA_DIR}/events-export.jsonl" |
||||
RELAY_BIN="${RELAY_BIN:-./orly}" |
||||
|
||||
# Check if relay binary exists |
||||
if [ ! -f "$RELAY_BIN" ]; then |
||||
echo -e "${RED}Error: ORLY binary not found at $RELAY_BIN${NC}" |
||||
echo "Please build the relay first: go build -o orly" |
||||
echo "Or set RELAY_BIN environment variable to the binary location" |
||||
exit 1 |
||||
fi |
||||
|
||||
# Check if database exists |
||||
if [ ! -d "$DATA_DIR" ]; then |
||||
echo -e "${YELLOW}Warning: Database directory not found at $DATA_DIR${NC}" |
||||
echo "Nothing to migrate. If this is a fresh install, you can skip migration." |
||||
exit 0 |
||||
fi |
||||
|
||||
# Check disk space |
||||
DB_SIZE=$(du -sb "$DATA_DIR" | cut -f1) |
||||
AVAILABLE_SPACE=$(df "$HOME" | tail -1 | awk '{print $4}') |
||||
AVAILABLE_SPACE=$((AVAILABLE_SPACE * 1024)) # Convert to bytes |
||||
REQUIRED_SPACE=$((DB_SIZE * 3)) # 3x for safety (export + backup + new DB) |
||||
|
||||
echo "Database size: $(numfmt --to=iec-i --suffix=B $DB_SIZE)" |
||||
echo "Available space: $(numfmt --to=iec-i --suffix=B $AVAILABLE_SPACE)" |
||||
echo "Required space: $(numfmt --to=iec-i --suffix=B $REQUIRED_SPACE)" |
||||
echo "" |
||||
|
||||
if [ $AVAILABLE_SPACE -lt $REQUIRED_SPACE ]; then |
||||
echo -e "${RED}Error: Not enough disk space!${NC}" |
||||
echo "Required: $(numfmt --to=iec-i --suffix=B $REQUIRED_SPACE)" |
||||
echo "Available: $(numfmt --to=iec-i --suffix=B $AVAILABLE_SPACE)" |
||||
echo "" |
||||
echo "Options:" |
||||
echo " 1. Free up disk space" |
||||
echo " 2. Use natural compaction (no migration needed)" |
||||
echo " 3. Export to external drive and import back" |
||||
exit 1 |
||||
fi |
||||
|
||||
# Check if relay is running |
||||
if pgrep -x "orly" > /dev/null; then |
||||
echo -e "${YELLOW}Warning: ORLY relay is currently running${NC}" |
||||
echo "The relay should be stopped before migration." |
||||
echo "" |
||||
read -p "Stop the relay now? (y/N) " -n 1 -r |
||||
echo |
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then |
||||
echo "Attempting to stop relay..." |
||||
if systemctl is-active --quiet orly; then |
||||
sudo systemctl stop orly |
||||
echo -e "${GREEN}Relay stopped via systemd${NC}" |
||||
else |
||||
pkill orly |
||||
sleep 2 |
||||
if pgrep -x "orly" > /dev/null; then |
||||
echo -e "${RED}Failed to stop relay. Please stop it manually and try again.${NC}" |
||||
exit 1 |
||||
fi |
||||
echo -e "${GREEN}Relay stopped${NC}" |
||||
fi |
||||
else |
||||
echo "Please stop the relay and run this script again." |
||||
exit 1 |
||||
fi |
||||
fi |
||||
|
||||
echo "" |
||||
echo -e "${YELLOW}=== Migration Plan ===${NC}" |
||||
echo "1. Export all events to JSONL: $EXPORT_FILE" |
||||
echo "2. Backup current database to: $BACKUP_DIR" |
||||
echo "3. Create new database with optimized configuration" |
||||
echo "4. Import all events (rebuilds indexes)" |
||||
echo "5. Verify event counts match" |
||||
echo "" |
||||
echo "Estimated time: $(( (DB_SIZE / 1024 / 1024 / 100) + 1 )) - $(( (DB_SIZE / 1024 / 1024 / 50) + 1 )) minutes" |
||||
echo "" |
||||
read -p "Proceed with migration? (y/N) " -n 1 -r |
||||
echo |
||||
if [[ ! $REPLY =~ ^[Yy]$ ]]; then |
||||
echo "Migration cancelled." |
||||
exit 0 |
||||
fi |
||||
|
||||
# Step 1: Export events |
||||
echo "" |
||||
echo -e "${GREEN}=== Step 1: Exporting Events ===${NC}" |
||||
echo "This may take several minutes for large databases..." |
||||
echo "" |
||||
|
||||
# We'll use a Go program to export since the binary doesn't have a CLI export command |
||||
# Create temporary export program |
||||
EXPORT_PROG=$(mktemp -d)/export-db.go |
||||
cat > "$EXPORT_PROG" << 'EOF' |
||||
package main |
||||
|
||||
import ( |
||||
"context" |
||||
"fmt" |
||||
"os" |
||||
"next.orly.dev/pkg/database" |
||||
) |
||||
|
||||
func main() { |
||||
if len(os.Args) < 3 { |
||||
fmt.Fprintf(os.Stderr, "Usage: %s <data-dir> <output-file>\n", os.Args[0]) |
||||
os.Exit(1) |
||||
} |
||||
|
||||
dataDir := os.Args[1] |
||||
outFile := os.Args[2] |
||||
|
||||
ctx := context.Background() |
||||
cancel := func() {} |
||||
|
||||
db, err := database.New(ctx, cancel, dataDir, "error") |
||||
if err != nil { |
||||
fmt.Fprintf(os.Stderr, "Failed to open database: %v\n", err) |
||||
os.Exit(1) |
||||
} |
||||
defer db.Close() |
||||
|
||||
f, err := os.Create(outFile) |
||||
if err != nil { |
||||
fmt.Fprintf(os.Stderr, "Failed to create output file: %v\n", err) |
||||
os.Exit(1) |
||||
} |
||||
defer f.Close() |
||||
|
||||
fmt.Println("Exporting events...") |
||||
db.Export(ctx, f) |
||||
fmt.Println("Export complete!") |
||||
} |
||||
EOF |
||||
|
||||
# Build and run export program |
||||
echo "Building export tool..." |
||||
EXPORT_BIN=$(mktemp) |
||||
if ! go build -o "$EXPORT_BIN" "$EXPORT_PROG" 2>&1; then |
||||
echo -e "${RED}Failed to build export tool${NC}" |
||||
rm -f "$EXPORT_PROG" "$EXPORT_BIN" |
||||
exit 1 |
||||
fi |
||||
|
||||
echo "Running export..." |
||||
if ! "$EXPORT_BIN" "$DATA_DIR" "$EXPORT_FILE"; then |
||||
echo -e "${RED}Export failed!${NC}" |
||||
rm -f "$EXPORT_PROG" "$EXPORT_BIN" |
||||
exit 1 |
||||
fi |
||||
|
||||
rm -f "$EXPORT_PROG" "$EXPORT_BIN" |
||||
|
||||
# Count exported events |
||||
EXPORT_COUNT=$(wc -l < "$EXPORT_FILE") |
||||
echo -e "${GREEN}Exported $EXPORT_COUNT events${NC}" |
||||
echo "Export size: $(du -h "$EXPORT_FILE" | cut -f1)" |
||||
|
||||
# Step 2: Backup current database |
||||
echo "" |
||||
echo -e "${GREEN}=== Step 2: Backing Up Current Database ===${NC}" |
||||
echo "Moving $DATA_DIR to $BACKUP_DIR" |
||||
mv "$DATA_DIR" "$BACKUP_DIR" |
||||
echo -e "${GREEN}Backup complete${NC}" |
||||
|
||||
# Step 3 & 4: Create new database and import |
||||
echo "" |
||||
echo -e "${GREEN}=== Step 3 & 4: Creating New Database and Importing ===${NC}" |
||||
echo "This will take longer as indexes are rebuilt..." |
||||
echo "" |
||||
|
||||
# Create temporary import program |
||||
IMPORT_PROG=$(mktemp -d)/import-db.go |
||||
cat > "$IMPORT_PROG" << 'EOF' |
||||
package main |
||||
|
||||
import ( |
||||
"context" |
||||
"fmt" |
||||
"os" |
||||
"next.orly.dev/pkg/database" |
||||
) |
||||
|
||||
func main() { |
||||
if len(os.Args) < 3 { |
||||
fmt.Fprintf(os.Stderr, "Usage: %s <data-dir> <import-file>\n", os.Args[0]) |
||||
os.Exit(1) |
||||
} |
||||
|
||||
dataDir := os.Args[1] |
||||
importFile := os.Args[2] |
||||
|
||||
ctx := context.Background() |
||||
cancel := func() {} |
||||
|
||||
// This will create new database with updated configuration from database.go |
||||
db, err := database.New(ctx, cancel, dataDir, "info") |
||||
if err != nil { |
||||
fmt.Fprintf(os.Stderr, "Failed to create database: %v\n", err) |
||||
os.Exit(1) |
||||
} |
||||
defer db.Close() |
||||
|
||||
f, err := os.Open(importFile) |
||||
if err != nil { |
||||
fmt.Fprintf(os.Stderr, "Failed to open import file: %v\n", err) |
||||
os.Exit(1) |
||||
} |
||||
defer f.Close() |
||||
|
||||
fmt.Println("Importing events (this may take a while)...") |
||||
db.Import(f) |
||||
|
||||
// Wait for import to complete |
||||
fmt.Println("Import started. Waiting for completion...") |
||||
fmt.Println("Check the log output above for progress (logged every 100 events)") |
||||
} |
||||
EOF |
||||
|
||||
# Build and run import program |
||||
echo "Building import tool..." |
||||
IMPORT_BIN=$(mktemp) |
||||
if ! go build -o "$IMPORT_BIN" "$IMPORT_PROG" 2>&1; then |
||||
echo -e "${RED}Failed to build import tool${NC}" |
||||
echo "Rolling back..." |
||||
mv "$BACKUP_DIR" "$DATA_DIR" |
||||
rm -f "$IMPORT_PROG" "$IMPORT_BIN" |
||||
exit 1 |
||||
fi |
||||
|
||||
echo "Running import..." |
||||
if ! "$IMPORT_BIN" "$DATA_DIR" "$EXPORT_FILE"; then |
||||
echo -e "${RED}Import failed!${NC}" |
||||
echo "Rolling back..." |
||||
rm -rf "$DATA_DIR" |
||||
mv "$BACKUP_DIR" "$DATA_DIR" |
||||
rm -f "$IMPORT_PROG" "$IMPORT_BIN" |
||||
exit 1 |
||||
fi |
||||
|
||||
rm -f "$IMPORT_PROG" "$IMPORT_BIN" |
||||
|
||||
# Give import goroutine time to process |
||||
echo "Waiting for import to complete..." |
||||
sleep 10 |
||||
|
||||
# Step 5: Verify |
||||
echo "" |
||||
echo -e "${GREEN}=== Step 5: Verification ===${NC}" |
||||
|
||||
NEW_DB_SIZE=$(du -sb "$DATA_DIR" | cut -f1) |
||||
echo "Old database size: $(numfmt --to=iec-i --suffix=B $DB_SIZE)" |
||||
echo "New database size: $(numfmt --to=iec-i --suffix=B $NEW_DB_SIZE)" |
||||
echo "" |
||||
|
||||
if [ $NEW_DB_SIZE -lt $((DB_SIZE / 10)) ]; then |
||||
echo -e "${YELLOW}Warning: New database is suspiciously small${NC}" |
||||
echo "This may indicate an incomplete import." |
||||
echo "Check the logs in $DATA_DIR/migration.log" |
||||
echo "" |
||||
read -p "Continue anyway? (y/N) " -n 1 -r |
||||
echo |
||||
if [[ ! $REPLY =~ ^[Yy]$ ]]; then |
||||
echo "Rolling back..." |
||||
rm -rf "$DATA_DIR" |
||||
mv "$BACKUP_DIR" "$DATA_DIR" |
||||
exit 1 |
||||
fi |
||||
fi |
||||
|
||||
echo -e "${GREEN}=== Migration Complete! ===${NC}" |
||||
echo "" |
||||
echo "Summary:" |
||||
echo " - Exported: $EXPORT_COUNT events" |
||||
echo " - Old DB size: $(numfmt --to=iec-i --suffix=B $DB_SIZE)" |
||||
echo " - New DB size: $(numfmt --to=iec-i --suffix=B $NEW_DB_SIZE)" |
||||
echo " - Space saved: $(numfmt --to=iec-i --suffix=B $((DB_SIZE - NEW_DB_SIZE)))" |
||||
echo " - Backup location: $BACKUP_DIR" |
||||
echo "" |
||||
echo "Next steps:" |
||||
echo " 1. Start the relay: sudo systemctl start orly (or ./orly)" |
||||
echo " 2. Monitor performance for 24-48 hours" |
||||
echo " 3. Watch for cache hit ratio >85% in logs" |
||||
echo " 4. Verify event count and queries work correctly" |
||||
echo " 5. After verification, remove backup: rm -rf $BACKUP_DIR" |
||||
echo "" |
||||
echo "Rollback (if needed):" |
||||
echo " Stop relay, then: rm -rf $DATA_DIR && mv $BACKUP_DIR $DATA_DIR" |
||||
echo "" |
||||
@ -0,0 +1,88 @@
@@ -0,0 +1,88 @@
|
||||
#!/bin/bash |
||||
set -e |
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")" |
||||
|
||||
echo "=== ORLY Dgraph Integration Test Suite ===" |
||||
echo "" |
||||
|
||||
# Check if dgraph is running |
||||
echo "Checking for dgraph server..." |
||||
DGRAPH_URL="${ORLY_DGRAPH_URL:-localhost:9080}" |
||||
|
||||
if ! timeout 2 bash -c "echo > /dev/tcp/${DGRAPH_URL%:*}/${DGRAPH_URL#*:}" 2>/dev/null; then |
||||
echo "❌ Dgraph server not available at $DGRAPH_URL" |
||||
echo "" |
||||
echo "To start dgraph using docker-compose:" |
||||
echo " cd $SCRIPT_DIR && docker-compose -f dgraph-docker-compose.yml up -d" |
||||
echo "" |
||||
echo "Or using docker directly:" |
||||
echo " docker run -d -p 8080:8080 -p 9080:9080 -p 8000:8000 --name dgraph-orly dgraph/standalone:latest" |
||||
echo "" |
||||
exit 1 |
||||
fi |
||||
|
||||
echo "✅ Dgraph server is running at $DGRAPH_URL" |
||||
echo "" |
||||
|
||||
# Run dgraph tests |
||||
echo "Running dgraph package tests..." |
||||
cd "$PROJECT_ROOT" |
||||
CGO_ENABLED=0 go test -v -timeout 10m ./pkg/dgraph/... || { |
||||
echo "❌ Dgraph tests failed" |
||||
exit 1 |
||||
} |
||||
|
||||
echo "" |
||||
echo "✅ All dgraph tests passed!" |
||||
echo "" |
||||
|
||||
# Optional: Run relay-tester if requested |
||||
if [ "$1" == "--relay-tester" ]; then |
||||
echo "Starting ORLY with dgraph backend..." |
||||
export ORLY_DB_TYPE=dgraph |
||||
export ORLY_DGRAPH_URL="$DGRAPH_URL" |
||||
export ORLY_LOG_LEVEL=info |
||||
export ORLY_PORT=3334 |
||||
|
||||
# Kill any existing ORLY instance |
||||
pkill -f "./orly" || true |
||||
sleep 1 |
||||
|
||||
# Start ORLY in background |
||||
./orly & |
||||
ORLY_PID=$! |
||||
|
||||
# Wait for ORLY to start |
||||
echo "Waiting for ORLY to start..." |
||||
for i in {1..30}; do |
||||
if curl -s http://localhost:3334 > /dev/null 2>&1; then |
||||
echo "✅ ORLY started successfully" |
||||
break |
||||
fi |
||||
sleep 1 |
||||
if [ $i -eq 30 ]; then |
||||
echo "❌ ORLY failed to start" |
||||
kill $ORLY_PID 2>/dev/null || true |
||||
exit 1 |
||||
fi |
||||
done |
||||
|
||||
echo "" |
||||
echo "Running relay-tester against dgraph backend..." |
||||
go run cmd/relay-tester/main.go -url ws://localhost:3334 || { |
||||
echo "❌ Relay-tester failed" |
||||
kill $ORLY_PID 2>/dev/null || true |
||||
exit 1 |
||||
} |
||||
|
||||
# Clean up |
||||
kill $ORLY_PID 2>/dev/null || true |
||||
|
||||
echo "" |
||||
echo "✅ Relay-tester passed!" |
||||
fi |
||||
|
||||
echo "" |
||||
echo "=== All tests completed successfully! ===" |
||||
Loading…
Reference in new issue