mleku
2 months ago
72 changed files with 8159 additions and 4049 deletions
@ -0,0 +1,319 @@ |
|||||||
|
# Badger Database Migration Guide |
||||||
|
|
||||||
|
## Overview |
||||||
|
|
||||||
|
This guide covers migrating your ORLY relay database when changing Badger configuration parameters, specifically for the VLogPercentile and table size optimizations. |
||||||
|
|
||||||
|
## When Migration is Needed |
||||||
|
|
||||||
|
Based on research of Badger v4 source code and documentation: |
||||||
|
|
||||||
|
### Configuration Changes That DON'T Require Migration |
||||||
|
|
||||||
|
The following options can be changed **without migration**: |
||||||
|
- `BlockCacheSize` - Only affects in-memory cache |
||||||
|
- `IndexCacheSize` - Only affects in-memory cache |
||||||
|
- `NumCompactors` - Runtime setting |
||||||
|
- `NumLevelZeroTables` - Affects compaction timing |
||||||
|
- `NumMemtables` - Affects write buffering |
||||||
|
- `DetectConflicts` - Runtime conflict detection |
||||||
|
- `Compression` - New data uses new compression, old data remains as-is |
||||||
|
- `BlockSize` - Explicitly stated in Badger source: "Changing BlockSize across DB runs will not break badger" |
||||||
|
|
||||||
|
### Configuration Changes That BENEFIT from Migration |
||||||
|
|
||||||
|
The following options apply to **new writes only** - existing data gradually adopts new settings through compaction: |
||||||
|
- `VLogPercentile` - Affects where **new** values are stored (LSM vs vlog) |
||||||
|
- `BaseTableSize` - **New** SST files use new size |
||||||
|
- `MemTableSize` - Affects new write buffering |
||||||
|
- `BaseLevelSize` - Affects new LSM tree structure |
||||||
|
- `ValueLogFileSize` - New vlog files use new size |
||||||
|
|
||||||
|
**Migration Impact:** Without migration, existing data remains in its current location (LSM tree or value log). The database will **gradually** adapt through normal compaction, which may take days or weeks depending on write volume. |
||||||
|
|
||||||
|
## Migration Options |
||||||
|
|
||||||
|
### Option 1: No Migration (Let Natural Compaction Handle It) |
||||||
|
|
||||||
|
**Best for:** Low-traffic relays, testing environments |
||||||
|
|
||||||
|
**Pros:** |
||||||
|
- No downtime required |
||||||
|
- No manual intervention |
||||||
|
- Zero risk of data loss |
||||||
|
|
||||||
|
**Cons:** |
||||||
|
- Benefits take time to materialize (days/weeks) |
||||||
|
- Old data layout persists until natural compaction |
||||||
|
- Cache tuning benefits delayed |
||||||
|
|
||||||
|
**Steps:** |
||||||
|
1. Update Badger configuration in `pkg/database/database.go` |
||||||
|
2. Restart ORLY relay |
||||||
|
3. Monitor performance over several days |
||||||
|
4. Optionally run manual GC: `db.RunValueLogGC(0.5)` periodically |
||||||
|
|
||||||
|
### Option 2: Manual Value Log Garbage Collection |
||||||
|
|
||||||
|
**Best for:** Medium-traffic relays wanting faster optimization |
||||||
|
|
||||||
|
**Pros:** |
||||||
|
- Faster than natural compaction |
||||||
|
- Still safe (no export/import) |
||||||
|
- Can run while relay is online |
||||||
|
|
||||||
|
**Cons:** |
||||||
|
- Still gradual (hours instead of days) |
||||||
|
- CPU/disk intensive during GC |
||||||
|
- Partial benefit until GC completes |
||||||
|
|
||||||
|
**Steps:** |
||||||
|
1. Update Badger configuration |
||||||
|
2. Restart ORLY relay |
||||||
|
3. Monitor logs for compaction activity |
||||||
|
4. Manually trigger GC if needed (future feature - not currently exposed) |
||||||
|
|
||||||
|
### Option 3: Full Export/Import Migration (RECOMMENDED for Production) |
||||||
|
|
||||||
|
**Best for:** Production relays, large databases, maximum performance |
||||||
|
|
||||||
|
**Pros:** |
||||||
|
- Immediate full benefit of new configuration |
||||||
|
- Clean database structure |
||||||
|
- Predictable migration time |
||||||
|
- Reclaims all disk space |
||||||
|
|
||||||
|
**Cons:** |
||||||
|
- Requires relay downtime (several hours for large DBs) |
||||||
|
- Requires 2x disk space temporarily |
||||||
|
- More complex procedure |
||||||
|
|
||||||
|
**Steps:** See detailed procedure below |
||||||
|
|
||||||
|
## Full Migration Procedure (Option 3) |
||||||
|
|
||||||
|
### Prerequisites |
||||||
|
|
||||||
|
1. **Disk space:** At minimum 2.5x current database size |
||||||
|
- 1x for current database |
||||||
|
- 1x for JSONL export |
||||||
|
- 0.5x for new database (will be smaller with compression) |
||||||
|
|
||||||
|
2. **Time estimate:** |
||||||
|
- Export: ~100-500 MB/s depending on disk speed |
||||||
|
- Import: ~50-200 MB/s with indexing overhead |
||||||
|
- Example: 10 GB database = ~10-30 minutes total |
||||||
|
|
||||||
|
3. **Backup:** Ensure you have a recent backup before proceeding |
||||||
|
|
||||||
|
### Step-by-Step Migration |
||||||
|
|
||||||
|
#### 1. Prepare Migration Script |
||||||
|
|
||||||
|
Use the provided `scripts/migrate-badger-config.sh` script (see below). |
||||||
|
|
||||||
|
#### 2. Stop the Relay |
||||||
|
|
||||||
|
```bash |
||||||
|
# If using systemd |
||||||
|
sudo systemctl stop orly |
||||||
|
|
||||||
|
# If running manually |
||||||
|
pkill orly |
||||||
|
``` |
||||||
|
|
||||||
|
#### 3. Run Migration |
||||||
|
|
||||||
|
```bash |
||||||
|
cd ~/src/next.orly.dev |
||||||
|
chmod +x scripts/migrate-badger-config.sh |
||||||
|
./scripts/migrate-badger-config.sh |
||||||
|
``` |
||||||
|
|
||||||
|
The script will: |
||||||
|
- Export all events to JSONL format |
||||||
|
- Move old database to backup location |
||||||
|
- Create new database with updated configuration |
||||||
|
- Import all events (rebuilds indexes automatically) |
||||||
|
- Verify event count matches |
||||||
|
|
||||||
|
#### 4. Verify Migration |
||||||
|
|
||||||
|
```bash |
||||||
|
# Check that events were migrated |
||||||
|
echo "Old event count:" |
||||||
|
cat ~/.local/share/ORLY-backup-*/migration.log | grep "exported.*events" |
||||||
|
|
||||||
|
echo "New event count:" |
||||||
|
cat ~/.local/share/ORLY/migration.log | grep "saved.*events" |
||||||
|
``` |
||||||
|
|
||||||
|
#### 5. Restart Relay |
||||||
|
|
||||||
|
```bash |
||||||
|
# If using systemd |
||||||
|
sudo systemctl start orly |
||||||
|
sudo journalctl -u orly -f |
||||||
|
|
||||||
|
# If running manually |
||||||
|
./orly |
||||||
|
``` |
||||||
|
|
||||||
|
#### 6. Monitor Performance |
||||||
|
|
||||||
|
Watch for improvements in: |
||||||
|
- Cache hit ratio (should be >85% with new config) |
||||||
|
- Average query latency (should be <3ms for cached events) |
||||||
|
- No "Block cache too small" warnings in logs |
||||||
|
|
||||||
|
#### 7. Clean Up (After Verification) |
||||||
|
|
||||||
|
```bash |
||||||
|
# Once you confirm everything works (wait 24-48 hours) |
||||||
|
rm -rf ~/.local/share/ORLY-backup-* |
||||||
|
rm ~/.local/share/ORLY/events-export.jsonl |
||||||
|
``` |
||||||
|
|
||||||
|
## Migration Script |
||||||
|
|
||||||
|
The migration script is located at `scripts/migrate-badger-config.sh` and handles: |
||||||
|
- Automatic export of all events to JSONL |
||||||
|
- Safe backup of existing database |
||||||
|
- Creation of new database with updated config |
||||||
|
- Import and indexing of all events |
||||||
|
- Verification of event counts |
||||||
|
|
||||||
|
## Rollback Procedure |
||||||
|
|
||||||
|
If migration fails or performance degrades: |
||||||
|
|
||||||
|
```bash |
||||||
|
# Stop the relay |
||||||
|
sudo systemctl stop orly # or pkill orly |
||||||
|
|
||||||
|
# Restore old database |
||||||
|
rm -rf ~/.local/share/ORLY |
||||||
|
mv ~/.local/share/ORLY-backup-$(date +%Y%m%d)* ~/.local/share/ORLY |
||||||
|
|
||||||
|
# Restart with old configuration |
||||||
|
sudo systemctl start orly |
||||||
|
``` |
||||||
|
|
||||||
|
## Configuration Changes Summary |
||||||
|
|
||||||
|
### Changes Applied in pkg/database/database.go |
||||||
|
|
||||||
|
```go |
||||||
|
// Cache sizes (can change without migration) |
||||||
|
opts.BlockCacheSize = 16384 MB (was 512 MB) |
||||||
|
opts.IndexCacheSize = 4096 MB (was 256 MB) |
||||||
|
|
||||||
|
// Table sizes (benefits from migration) |
||||||
|
opts.BaseTableSize = 8 MB (was 64 MB) |
||||||
|
opts.MemTableSize = 16 MB (was 64 MB) |
||||||
|
opts.ValueLogFileSize = 128 MB (was 256 MB) |
||||||
|
|
||||||
|
// Inline event optimization (CRITICAL - benefits from migration) |
||||||
|
opts.VLogPercentile = 0.99 (was 0.0 - default) |
||||||
|
|
||||||
|
// LSM structure (benefits from migration) |
||||||
|
opts.BaseLevelSize = 64 MB (was 10 MB - default) |
||||||
|
|
||||||
|
// Performance settings (no migration needed) |
||||||
|
opts.DetectConflicts = false (was true) |
||||||
|
opts.Compression = options.ZSTD (was options.None) |
||||||
|
opts.NumCompactors = 8 (was 4) |
||||||
|
opts.NumMemtables = 8 (was 5) |
||||||
|
``` |
||||||
|
|
||||||
|
## Expected Improvements |
||||||
|
|
||||||
|
### Before Migration |
||||||
|
- Cache hit ratio: 33% |
||||||
|
- Average latency: 9.35ms |
||||||
|
- P95 latency: 34.48ms |
||||||
|
- Block cache warnings: Yes |
||||||
|
|
||||||
|
### After Migration |
||||||
|
- Cache hit ratio: 85-95% |
||||||
|
- Average latency: <3ms |
||||||
|
- P95 latency: <8ms |
||||||
|
- Block cache warnings: No |
||||||
|
- Inline events: 3-5x faster reads |
||||||
|
|
||||||
|
## Troubleshooting |
||||||
|
|
||||||
|
### Migration Script Fails |
||||||
|
|
||||||
|
**Error:** "Not enough disk space" |
||||||
|
- Free up space or use Option 1 (natural compaction) |
||||||
|
- Ensure you have 2.5x current DB size available |
||||||
|
|
||||||
|
**Error:** "Export failed" |
||||||
|
- Check database is not corrupted |
||||||
|
- Ensure ORLY is stopped |
||||||
|
- Check file permissions |
||||||
|
|
||||||
|
**Error:** "Import count mismatch" |
||||||
|
- This is informational - some events may be duplicates |
||||||
|
- Check logs for specific errors |
||||||
|
- Verify core events are present via relay queries |
||||||
|
|
||||||
|
### Performance Not Improved |
||||||
|
|
||||||
|
**After migration, performance is the same:** |
||||||
|
1. Verify configuration was actually applied: |
||||||
|
```bash |
||||||
|
# Check running relay logs for config output |
||||||
|
sudo journalctl -u orly | grep -i "block.*cache\|vlog" |
||||||
|
``` |
||||||
|
|
||||||
|
2. Wait for cache to warm up (2-5 minutes after start) |
||||||
|
|
||||||
|
3. Check if workload changed (different query patterns) |
||||||
|
|
||||||
|
4. Verify disk I/O is not bottleneck: |
||||||
|
```bash |
||||||
|
iostat -x 5 |
||||||
|
``` |
||||||
|
|
||||||
|
### High CPU During Migration |
||||||
|
|
||||||
|
- This is normal - import rebuilds all indexes |
||||||
|
- Migration is single-threaded by design (data consistency) |
||||||
|
- Expect 30-60% CPU usage on one core |
||||||
|
|
||||||
|
## Additional Notes |
||||||
|
|
||||||
|
### Compression Impact |
||||||
|
|
||||||
|
The `Compression = options.ZSTD` setting: |
||||||
|
- Only compresses **new** data |
||||||
|
- Old data remains uncompressed until rewritten by compaction |
||||||
|
- Migration forces all data to be rewritten → immediate compression benefit |
||||||
|
- Expect 2-3x compression ratio for event data |
||||||
|
|
||||||
|
### VLogPercentile Behavior |
||||||
|
|
||||||
|
With `VLogPercentile = 0.99`: |
||||||
|
- **99% of values** stored in LSM tree (fast access) |
||||||
|
- **1% of values** stored in value log (large events >100 KB) |
||||||
|
- Threshold dynamically adjusted based on value size distribution |
||||||
|
- Perfect for ORLY's inline event optimization |
||||||
|
|
||||||
|
### Production Considerations |
||||||
|
|
||||||
|
For production relays: |
||||||
|
1. Schedule migration during low-traffic period |
||||||
|
2. Notify users of maintenance window |
||||||
|
3. Have rollback plan ready |
||||||
|
4. Monitor closely for 24-48 hours after migration |
||||||
|
5. Keep backup for at least 1 week |
||||||
|
|
||||||
|
## References |
||||||
|
|
||||||
|
- Badger v4 Documentation: https://pkg.go.dev/github.com/dgraph-io/badger/v4 |
||||||
|
- ORLY Database Package: `pkg/database/database.go` |
||||||
|
- Export/Import Implementation: `pkg/database/{export,import}.go` |
||||||
|
- Cache Optimization Analysis: `cmd/benchmark/CACHE_OPTIMIZATION_STRATEGY.md` |
||||||
|
- Inline Event Optimization: `cmd/benchmark/INLINE_EVENT_OPTIMIZATION.md` |
||||||
@ -0,0 +1,188 @@ |
|||||||
|
# Badger Cache Optimization Strategy |
||||||
|
|
||||||
|
## Problem Analysis |
||||||
|
|
||||||
|
### Initial Configuration (FAILED) |
||||||
|
- Block cache: 2048 MB |
||||||
|
- Index cache: 1024 MB |
||||||
|
- **Result**: Cache hit ratio remained at 33% |
||||||
|
|
||||||
|
### Root Cause Discovery |
||||||
|
|
||||||
|
Badger's Ristretto cache uses a "cost" metric that doesn't directly map to bytes: |
||||||
|
|
||||||
|
``` |
||||||
|
Average cost per key: 54,628,383 bytes = 52.10 MB |
||||||
|
Cache size: 2048 MB |
||||||
|
Keys that fit: ~39 keys only! |
||||||
|
``` |
||||||
|
|
||||||
|
The cost metric appears to include: |
||||||
|
- Uncompressed data size |
||||||
|
- Value log references |
||||||
|
- Table metadata |
||||||
|
- Potentially full `BaseTableSize` (64 MB) per entry |
||||||
|
|
||||||
|
### Why Previous Fix Didn't Work |
||||||
|
|
||||||
|
With `BaseTableSize = 64 MB`: |
||||||
|
- Each cache entry costs ~52 MB in the cost metric |
||||||
|
- 2 GB cache ÷ 52 MB = ~39 entries max |
||||||
|
- Test generates 228,000+ unique keys |
||||||
|
- **Eviction rate: 99.99%** (everything gets evicted immediately) |
||||||
|
|
||||||
|
## Multi-Pronged Optimization Strategy |
||||||
|
|
||||||
|
### Approach 1: Reduce Table Sizes (IMPLEMENTED) |
||||||
|
|
||||||
|
**Changes in `pkg/database/database.go`:** |
||||||
|
|
||||||
|
```go |
||||||
|
// OLD (causing high cache cost): |
||||||
|
opts.BaseTableSize = 64 * units.Mb // 64 MB per table |
||||||
|
opts.MemTableSize = 64 * units.Mb // 64 MB memtable |
||||||
|
|
||||||
|
// NEW (lower cache cost): |
||||||
|
opts.BaseTableSize = 8 * units.Mb // 8 MB per table (8x reduction) |
||||||
|
opts.MemTableSize = 16 * units.Mb // 16 MB memtable (4x reduction) |
||||||
|
``` |
||||||
|
|
||||||
|
**Expected Impact:** |
||||||
|
- Cost per key should drop from ~52 MB to ~6-8 MB |
||||||
|
- Cache can now hold ~2,000-3,000 keys instead of ~39 |
||||||
|
- **Projected hit ratio: 60-70%** (significant improvement) |
||||||
|
|
||||||
|
### Approach 2: Enable Compression (IMPLEMENTED) |
||||||
|
|
||||||
|
```go |
||||||
|
// OLD: |
||||||
|
opts.Compression = options.None |
||||||
|
|
||||||
|
// NEW: |
||||||
|
opts.Compression = options.ZSTD |
||||||
|
opts.ZSTDCompressionLevel = 1 // Fast compression |
||||||
|
``` |
||||||
|
|
||||||
|
**Expected Impact:** |
||||||
|
- Compressed data reduces cache cost metric |
||||||
|
- ZSTD level 1 is very fast (~500 MB/s) with ~2-3x compression |
||||||
|
- Should reduce cost per key by another 50-60% |
||||||
|
- **Combined with smaller tables: cost per key ~3-4 MB** |
||||||
|
|
||||||
|
### Approach 3: Massive Cache Increase (IMPLEMENTED) |
||||||
|
|
||||||
|
**Changes in `Dockerfile.next-orly`:** |
||||||
|
|
||||||
|
```dockerfile |
||||||
|
ENV ORLY_DB_BLOCK_CACHE_MB=16384 # 16 GB (was 2 GB) |
||||||
|
ENV ORLY_DB_INDEX_CACHE_MB=4096 # 4 GB (was 1 GB) |
||||||
|
``` |
||||||
|
|
||||||
|
**Rationale:** |
||||||
|
- With 16 GB cache and 3-4 MB cost per key: **~4,000-5,000 keys** can fit |
||||||
|
- This should cover the working set for most benchmark tests |
||||||
|
- **Target hit ratio: 80-90%** |
||||||
|
|
||||||
|
## Combined Effect Calculation |
||||||
|
|
||||||
|
### Before Optimization: |
||||||
|
- Table size: 64 MB |
||||||
|
- Cost per key: ~52 MB |
||||||
|
- Cache: 2 GB |
||||||
|
- Keys in cache: ~39 |
||||||
|
- Hit ratio: 33% |
||||||
|
|
||||||
|
### After Optimization: |
||||||
|
- Table size: 8 MB (8x smaller) |
||||||
|
- Compression: ZSTD (~3x reduction) |
||||||
|
- Effective cost per key: ~2-3 MB (17-25x reduction!) |
||||||
|
- Cache: 16 GB (8x larger) |
||||||
|
- Keys in cache: **~5,000-8,000** (128-205x improvement) |
||||||
|
- **Projected hit ratio: 85-95%** |
||||||
|
|
||||||
|
## Trade-offs |
||||||
|
|
||||||
|
### Smaller Tables |
||||||
|
**Pros:** |
||||||
|
- Lower cache cost |
||||||
|
- Faster individual compactions |
||||||
|
- Better cache efficiency |
||||||
|
|
||||||
|
**Cons:** |
||||||
|
- More files to manage (mitigated by faster compaction) |
||||||
|
- Slightly more compaction overhead |
||||||
|
|
||||||
|
**Verdict:** Worth it for 25x cache efficiency improvement |
||||||
|
|
||||||
|
### Compression |
||||||
|
**Pros:** |
||||||
|
- Reduces cache cost |
||||||
|
- Reduces disk space |
||||||
|
- ZSTD level 1 is very fast |
||||||
|
|
||||||
|
**Cons:** |
||||||
|
- ~5-10% CPU overhead for compression |
||||||
|
- ~3-5% CPU overhead for decompression |
||||||
|
|
||||||
|
**Verdict:** Minor CPU cost for major cache gains |
||||||
|
|
||||||
|
### Large Cache |
||||||
|
**Pros:** |
||||||
|
- High hit ratio |
||||||
|
- Lower latency |
||||||
|
- Better throughput |
||||||
|
|
||||||
|
**Cons:** |
||||||
|
- 20 GB memory usage (16 GB block + 4 GB index) |
||||||
|
- May not be suitable for resource-constrained environments |
||||||
|
|
||||||
|
**Verdict:** Acceptable for high-performance relay deployments |
||||||
|
|
||||||
|
## Alternative Configurations |
||||||
|
|
||||||
|
### For 8 GB RAM Systems: |
||||||
|
```dockerfile |
||||||
|
ENV ORLY_DB_BLOCK_CACHE_MB=6144 # 6 GB |
||||||
|
ENV ORLY_DB_INDEX_CACHE_MB=1536 # 1.5 GB |
||||||
|
``` |
||||||
|
With optimized tables+compression: ~2,000-3,000 keys, 70-80% hit ratio |
||||||
|
|
||||||
|
### For 4 GB RAM Systems: |
||||||
|
```dockerfile |
||||||
|
ENV ORLY_DB_BLOCK_CACHE_MB=2560 # 2.5 GB |
||||||
|
ENV ORLY_DB_INDEX_CACHE_MB=512 # 512 MB |
||||||
|
``` |
||||||
|
With optimized tables+compression: ~800-1,200 keys, 50-60% hit ratio |
||||||
|
|
||||||
|
## Testing & Validation |
||||||
|
|
||||||
|
To test these changes: |
||||||
|
|
||||||
|
```bash |
||||||
|
cd /home/mleku/src/next.orly.dev/cmd/benchmark |
||||||
|
|
||||||
|
# Rebuild with new code changes |
||||||
|
docker compose build next-orly |
||||||
|
|
||||||
|
# Run benchmark |
||||||
|
sudo rm -rf data/ |
||||||
|
./run-benchmark-orly-only.sh |
||||||
|
``` |
||||||
|
|
||||||
|
### Metrics to Monitor: |
||||||
|
1. **Cache hit ratio** (target: >85%) |
||||||
|
2. **Cache life expectancy** (target: >30 seconds) |
||||||
|
3. **Average latency** (target: <3ms) |
||||||
|
4. **P95 latency** (target: <10ms) |
||||||
|
5. **Burst pattern performance** (target: match khatru-sqlite) |
||||||
|
|
||||||
|
## Expected Results |
||||||
|
|
||||||
|
### Burst Pattern Test: |
||||||
|
- **Before**: 9.35ms avg, 34.48ms P95 |
||||||
|
- **After**: <4ms avg, <10ms P95 (60-70% improvement) |
||||||
|
|
||||||
|
### Overall Performance: |
||||||
|
- Match or exceed khatru-sqlite and khatru-badger |
||||||
|
- Eliminate cache warnings |
||||||
|
- Stable performance across test rounds |
||||||
@ -0,0 +1,97 @@ |
|||||||
|
# Badger Cache Tuning Analysis |
||||||
|
|
||||||
|
## Problem Identified |
||||||
|
|
||||||
|
From benchmark run `run_20251116_092759`, the Badger block cache showed critical performance issues: |
||||||
|
|
||||||
|
### Cache Metrics (Round 1): |
||||||
|
``` |
||||||
|
Block cache might be too small. Metrics: |
||||||
|
- hit: 151,469 |
||||||
|
- miss: 307,989 |
||||||
|
- hit-ratio: 0.33 (33%) |
||||||
|
- keys-added: 226,912 |
||||||
|
- keys-evicted: 226,893 (99.99% eviction rate!) |
||||||
|
- Cache life expectancy: 2 seconds (90th percentile) |
||||||
|
``` |
||||||
|
|
||||||
|
### Performance Impact: |
||||||
|
- **Burst Pattern Latency**: 9.35ms avg (vs 3.61ms for khatru-sqlite) |
||||||
|
- **P95 Latency**: 34.48ms (vs 8.59ms for khatru-sqlite) |
||||||
|
- **Cache hit ratio**: Only 33% - causing constant disk I/O |
||||||
|
|
||||||
|
## Root Cause |
||||||
|
|
||||||
|
The benchmark container was using **default Badger cache sizes** (much smaller than the code defaults): |
||||||
|
- Block cache: ~64 MB (Badger default) |
||||||
|
- Index cache: ~32 MB (Badger default) |
||||||
|
|
||||||
|
The code has better defaults (1024 MB / 512 MB), but these weren't set in the Docker container. |
||||||
|
|
||||||
|
## Cache Size Calculation |
||||||
|
|
||||||
|
Based on benchmark workload analysis: |
||||||
|
|
||||||
|
### Block Cache Requirements: |
||||||
|
- Total cost added: 12.44 TB during test |
||||||
|
- With 226K keys and immediate evictions, we need to hold ~100-200K blocks in memory |
||||||
|
- At ~10-20 KB per block average: **2-4 GB needed** |
||||||
|
|
||||||
|
### Index Cache Requirements: |
||||||
|
- For 200K+ keys with metadata |
||||||
|
- Efficient index lookups during queries |
||||||
|
- **1-2 GB needed** |
||||||
|
|
||||||
|
## Solution |
||||||
|
|
||||||
|
Updated `Dockerfile.next-orly` with optimized cache settings: |
||||||
|
|
||||||
|
```dockerfile |
||||||
|
ENV ORLY_DB_BLOCK_CACHE_MB=2048 # 2 GB block cache |
||||||
|
ENV ORLY_DB_INDEX_CACHE_MB=1024 # 1 GB index cache |
||||||
|
``` |
||||||
|
|
||||||
|
### Expected Improvements: |
||||||
|
- **Cache hit ratio**: Target 85-95% (up from 33%) |
||||||
|
- **Burst pattern latency**: Target <5ms avg (down from 9.35ms) |
||||||
|
- **P95 latency**: Target <15ms (down from 34.48ms) |
||||||
|
- **Query latency**: Significant reduction due to cached index lookups |
||||||
|
|
||||||
|
## Testing Strategy |
||||||
|
|
||||||
|
1. Rebuild Docker image with new cache settings |
||||||
|
2. Run full benchmark suite |
||||||
|
3. Compare metrics: |
||||||
|
- Cache hit ratio |
||||||
|
- Average/P95/P99 latencies |
||||||
|
- Throughput under burst patterns |
||||||
|
- Memory usage |
||||||
|
|
||||||
|
## Memory Budget |
||||||
|
|
||||||
|
With these settings, the relay will use approximately: |
||||||
|
- Block cache: 2 GB |
||||||
|
- Index cache: 1 GB |
||||||
|
- Badger internal structures: ~200 MB |
||||||
|
- Go runtime: ~200 MB |
||||||
|
- **Total**: ~3.5 GB |
||||||
|
|
||||||
|
This is reasonable for a high-performance relay and well within modern server capabilities. |
||||||
|
|
||||||
|
## Alternative Configurations |
||||||
|
|
||||||
|
For constrained environments: |
||||||
|
|
||||||
|
### Medium (1.5 GB total): |
||||||
|
``` |
||||||
|
ORLY_DB_BLOCK_CACHE_MB=1024 |
||||||
|
ORLY_DB_INDEX_CACHE_MB=512 |
||||||
|
``` |
||||||
|
|
||||||
|
### Minimal (512 MB total): |
||||||
|
``` |
||||||
|
ORLY_DB_BLOCK_CACHE_MB=384 |
||||||
|
ORLY_DB_INDEX_CACHE_MB=128 |
||||||
|
``` |
||||||
|
|
||||||
|
Note: Smaller caches will result in lower hit ratios and higher latencies. |
||||||
@ -0,0 +1,162 @@ |
|||||||
|
# Inline Event Optimization Strategy |
||||||
|
|
||||||
|
## Problem: Value Log vs LSM Tree |
||||||
|
|
||||||
|
By default, Badger stores all values above a small threshold (~1KB) in the value log (separate files). This causes: |
||||||
|
- **Extra disk I/O** for reading values |
||||||
|
- **Cache inefficiency** - must cache both keys AND value log positions |
||||||
|
- **Poor performance for small inline events** |
||||||
|
|
||||||
|
## ORLY's Inline Event Storage |
||||||
|
|
||||||
|
ORLY uses "Reiser4 optimization" - small events are stored **inline** in the key itself: |
||||||
|
- Event data embedded directly in LSM tree |
||||||
|
- No separate value log lookup needed |
||||||
|
- Much faster reads for small events |
||||||
|
|
||||||
|
**But:** By default, Badger still tries to put these in the value log! |
||||||
|
|
||||||
|
## Solution: VLogPercentile |
||||||
|
|
||||||
|
```go |
||||||
|
opts.VLogPercentile = 0.99 |
||||||
|
``` |
||||||
|
|
||||||
|
**What this does:** |
||||||
|
- Analyzes value size distribution |
||||||
|
- Keeps the smallest 99% of values in the LSM tree |
||||||
|
- Only puts the largest 1% in value log |
||||||
|
|
||||||
|
**Impact on ORLY:** |
||||||
|
- Our optimized inline events stay in LSM tree ✅ |
||||||
|
- Only large events (>100KB) go to value log |
||||||
|
- Dramatically faster reads for typical Nostr events |
||||||
|
|
||||||
|
## Additional Optimizations Implemented |
||||||
|
|
||||||
|
### 1. Disable Conflict Detection |
||||||
|
```go |
||||||
|
opts.DetectConflicts = false |
||||||
|
``` |
||||||
|
|
||||||
|
**Rationale:** |
||||||
|
- Nostr events are **immutable** (content-addressable by ID) |
||||||
|
- No need for transaction conflict checking |
||||||
|
- **5-10% performance improvement** on writes |
||||||
|
|
||||||
|
### 2. Optimize BaseLevelSize |
||||||
|
```go |
||||||
|
opts.BaseLevelSize = 64 * units.Mb // Increased from 10 MB |
||||||
|
``` |
||||||
|
|
||||||
|
**Benefits:** |
||||||
|
- Fewer LSM levels to search |
||||||
|
- Faster compaction |
||||||
|
- Better space amplification |
||||||
|
|
||||||
|
### 3. Enable ZSTD Compression |
||||||
|
```go |
||||||
|
opts.Compression = options.ZSTD |
||||||
|
opts.ZSTDCompressionLevel = 1 // Fast mode |
||||||
|
``` |
||||||
|
|
||||||
|
**Benefits:** |
||||||
|
- 2-3x compression ratio on event data |
||||||
|
- Level 1 is very fast (500+ MB/s compression, 2+ GB/s decompression) |
||||||
|
- Reduces cache cost metric |
||||||
|
- Saves disk space |
||||||
|
|
||||||
|
## Combined Effect |
||||||
|
|
||||||
|
### Before Optimization: |
||||||
|
``` |
||||||
|
Small inline event read: |
||||||
|
1. Read key from LSM tree |
||||||
|
2. Get value log position from LSM |
||||||
|
3. Seek to value log file |
||||||
|
4. Read value from value log |
||||||
|
Total: ~3-5 disk operations |
||||||
|
``` |
||||||
|
|
||||||
|
### After Optimization: |
||||||
|
``` |
||||||
|
Small inline event read: |
||||||
|
1. Read key+value from LSM tree (in cache!) |
||||||
|
Total: 1 cache hit |
||||||
|
``` |
||||||
|
|
||||||
|
**Performance improvement: 3-5x faster reads for inline events** |
||||||
|
|
||||||
|
## Configuration Summary |
||||||
|
|
||||||
|
All optimizations applied in `pkg/database/database.go`: |
||||||
|
|
||||||
|
```go |
||||||
|
// Cache |
||||||
|
opts.BlockCacheSize = 16384 MB // 16 GB |
||||||
|
opts.IndexCacheSize = 4096 MB // 4 GB |
||||||
|
|
||||||
|
// Table sizes (reduce cache cost) |
||||||
|
opts.BaseTableSize = 8 MB |
||||||
|
opts.MemTableSize = 16 MB |
||||||
|
|
||||||
|
// Keep inline events in LSM |
||||||
|
opts.VLogPercentile = 0.99 |
||||||
|
|
||||||
|
// LSM structure |
||||||
|
opts.BaseLevelSize = 64 MB |
||||||
|
opts.LevelSizeMultiplier = 10 |
||||||
|
|
||||||
|
// Performance |
||||||
|
opts.Compression = ZSTD (level 1) |
||||||
|
opts.DetectConflicts = false |
||||||
|
opts.NumCompactors = 8 |
||||||
|
opts.NumMemtables = 8 |
||||||
|
``` |
||||||
|
|
||||||
|
## Expected Benchmark Improvements |
||||||
|
|
||||||
|
### Before (run_20251116_092759): |
||||||
|
- Burst pattern: 9.35ms avg, 34.48ms P95 |
||||||
|
- Cache hit ratio: 33% |
||||||
|
- Value log lookups: high |
||||||
|
|
||||||
|
### After (projected): |
||||||
|
- Burst pattern: <3ms avg, <8ms P95 |
||||||
|
- Cache hit ratio: 85-95% |
||||||
|
- Value log lookups: minimal (only large events) |
||||||
|
|
||||||
|
**Overall: 60-70% latency reduction, matching or exceeding other Badger-based relays** |
||||||
|
|
||||||
|
## Trade-offs |
||||||
|
|
||||||
|
### VLogPercentile = 0.99 |
||||||
|
**Pro:** Keeps inline events in LSM for fast access |
||||||
|
**Con:** Larger LSM tree (but we have 16 GB cache to handle it) |
||||||
|
**Verdict:** ✅ Essential for inline event optimization |
||||||
|
|
||||||
|
### DetectConflicts = false |
||||||
|
**Pro:** 5-10% faster writes |
||||||
|
**Con:** No transaction conflict detection |
||||||
|
**Verdict:** ✅ Safe - Nostr events are immutable |
||||||
|
|
||||||
|
### ZSTD Compression |
||||||
|
**Pro:** 2-3x space savings, lower cache cost |
||||||
|
**Con:** ~5% CPU overhead |
||||||
|
**Verdict:** ✅ Well worth it for cache efficiency |
||||||
|
|
||||||
|
## Testing |
||||||
|
|
||||||
|
Run benchmark to validate: |
||||||
|
```bash |
||||||
|
cd cmd/benchmark |
||||||
|
docker compose build next-orly |
||||||
|
sudo rm -rf data/ |
||||||
|
./run-benchmark-orly-only.sh |
||||||
|
``` |
||||||
|
|
||||||
|
Monitor for: |
||||||
|
1. ✅ No "Block cache too small" warnings |
||||||
|
2. ✅ Cache hit ratio >85% |
||||||
|
3. ✅ Latencies competitive with khatru-badger |
||||||
|
4. ✅ Most values in LSM tree (check logs) |
||||||
@ -0,0 +1,65 @@ |
|||||||
|
version: "3.8" |
||||||
|
|
||||||
|
services: |
||||||
|
# Next.orly.dev relay with profiling enabled |
||||||
|
next-orly: |
||||||
|
build: |
||||||
|
context: ../.. |
||||||
|
dockerfile: cmd/benchmark/Dockerfile.next-orly |
||||||
|
container_name: benchmark-next-orly-profile |
||||||
|
environment: |
||||||
|
- ORLY_DATA_DIR=/data |
||||||
|
- ORLY_LISTEN=0.0.0.0 |
||||||
|
- ORLY_PORT=8080 |
||||||
|
- ORLY_LOG_LEVEL=info |
||||||
|
- ORLY_PPROF=cpu |
||||||
|
- ORLY_PPROF_HTTP=true |
||||||
|
- ORLY_PPROF_PATH=/profiles |
||||||
|
- ORLY_DB_BLOCK_CACHE_MB=512 |
||||||
|
- ORLY_DB_INDEX_CACHE_MB=256 |
||||||
|
volumes: |
||||||
|
- ./data/next-orly:/data |
||||||
|
- ./profiles:/profiles |
||||||
|
ports: |
||||||
|
- "8001:8080" |
||||||
|
- "6060:6060" # pprof HTTP endpoint |
||||||
|
networks: |
||||||
|
- benchmark-net |
||||||
|
healthcheck: |
||||||
|
test: ["CMD", "curl", "-f", "http://localhost:8080/"] |
||||||
|
interval: 10s |
||||||
|
timeout: 5s |
||||||
|
retries: 5 |
||||||
|
start_period: 60s # Longer startup period |
||||||
|
|
||||||
|
# Benchmark runner - only test next-orly |
||||||
|
benchmark-runner: |
||||||
|
build: |
||||||
|
context: ../.. |
||||||
|
dockerfile: cmd/benchmark/Dockerfile.benchmark |
||||||
|
container_name: benchmark-runner-profile |
||||||
|
depends_on: |
||||||
|
next-orly: |
||||||
|
condition: service_healthy |
||||||
|
environment: |
||||||
|
- BENCHMARK_TARGETS=next-orly:8080 |
||||||
|
- BENCHMARK_EVENTS=50000 |
||||||
|
- BENCHMARK_WORKERS=24 |
||||||
|
- BENCHMARK_DURATION=60s |
||||||
|
volumes: |
||||||
|
- ./reports:/reports |
||||||
|
networks: |
||||||
|
- benchmark-net |
||||||
|
command: > |
||||||
|
sh -c " |
||||||
|
echo 'Waiting for ORLY to be ready (healthcheck)...' && |
||||||
|
sleep 5 && |
||||||
|
echo 'Starting benchmark tests...' && |
||||||
|
/app/benchmark-runner --output-dir=/reports && |
||||||
|
echo 'Benchmark complete - triggering shutdown...' && |
||||||
|
exit 0 |
||||||
|
" |
||||||
|
|
||||||
|
networks: |
||||||
|
benchmark-net: |
||||||
|
driver: bridge |
||||||
@ -0,0 +1,25 @@ |
|||||||
|
#!/bin/bash |
||||||
|
|
||||||
|
# Wrapper script that cleans data directories with sudo before running benchmark |
||||||
|
# Use this if you encounter permission errors with run-benchmark.sh |
||||||
|
|
||||||
|
set -e |
||||||
|
|
||||||
|
cd "$(dirname "$0")" |
||||||
|
|
||||||
|
# Stop any running containers first |
||||||
|
echo "Stopping any running benchmark containers..." |
||||||
|
if docker compose version &> /dev/null 2>&1; then |
||||||
|
docker compose down -v 2>&1 | grep -v "warning" || true |
||||||
|
else |
||||||
|
docker-compose down -v 2>&1 | grep -v "warning" || true |
||||||
|
fi |
||||||
|
|
||||||
|
# Clean data directories with sudo |
||||||
|
if [ -d "data" ]; then |
||||||
|
echo "Cleaning data directories (requires sudo)..." |
||||||
|
sudo rm -rf data/ |
||||||
|
fi |
||||||
|
|
||||||
|
# Now run the normal benchmark script |
||||||
|
exec ./run-benchmark.sh |
||||||
@ -0,0 +1,80 @@ |
|||||||
|
#!/bin/bash |
||||||
|
|
||||||
|
# Run benchmark for ORLY only (no other relays) |
||||||
|
|
||||||
|
set -e |
||||||
|
|
||||||
|
cd "$(dirname "$0")" |
||||||
|
|
||||||
|
# Determine docker-compose command |
||||||
|
if docker compose version &> /dev/null 2>&1; then |
||||||
|
DOCKER_COMPOSE="docker compose" |
||||||
|
else |
||||||
|
DOCKER_COMPOSE="docker-compose" |
||||||
|
fi |
||||||
|
|
||||||
|
# Clean old data directories (may be owned by root from Docker) |
||||||
|
if [ -d "data" ]; then |
||||||
|
echo "Cleaning old data directories..." |
||||||
|
if ! rm -rf data/ 2>/dev/null; then |
||||||
|
echo "" |
||||||
|
echo "ERROR: Cannot remove data directories due to permission issues." |
||||||
|
echo "Please run: sudo rm -rf data/" |
||||||
|
echo "Then run this script again." |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
fi |
||||||
|
|
||||||
|
# Create fresh data directories with correct permissions |
||||||
|
echo "Preparing data directories..." |
||||||
|
mkdir -p data/next-orly |
||||||
|
chmod 777 data/next-orly |
||||||
|
|
||||||
|
echo "Building ORLY container..." |
||||||
|
$DOCKER_COMPOSE build next-orly |
||||||
|
|
||||||
|
echo "Starting ORLY relay..." |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Start only next-orly and benchmark-runner |
||||||
|
$DOCKER_COMPOSE up next-orly -d |
||||||
|
|
||||||
|
# Wait for ORLY to be healthy |
||||||
|
echo "Waiting for ORLY to be healthy..." |
||||||
|
for i in {1..30}; do |
||||||
|
if curl -sf http://localhost:8001/ > /dev/null 2>&1; then |
||||||
|
echo "ORLY is ready!" |
||||||
|
break |
||||||
|
fi |
||||||
|
sleep 2 |
||||||
|
if [ $i -eq 30 ]; then |
||||||
|
echo "ERROR: ORLY failed to become healthy" |
||||||
|
$DOCKER_COMPOSE logs next-orly |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
done |
||||||
|
|
||||||
|
# Run benchmark against ORLY |
||||||
|
echo "" |
||||||
|
echo "Running benchmark against ORLY..." |
||||||
|
echo "Target: http://localhost:8001" |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Run the benchmark binary directly against the running ORLY instance |
||||||
|
docker run --rm --network benchmark_benchmark-net \ |
||||||
|
-e BENCHMARK_TARGETS=next-orly:8080 \ |
||||||
|
-e BENCHMARK_EVENTS=50000 \ |
||||||
|
-e BENCHMARK_WORKERS=24 \ |
||||||
|
-e BENCHMARK_DURATION=60s \ |
||||||
|
-v "$(pwd)/reports:/reports" \ |
||||||
|
benchmark-benchmark-runner \ |
||||||
|
/app/benchmark-runner --output-dir=/reports |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo "Benchmark complete!" |
||||||
|
echo "Stopping ORLY..." |
||||||
|
$DOCKER_COMPOSE down |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo "Results saved to ./reports/" |
||||||
|
echo "Check the latest run_* directory for detailed results." |
||||||
@ -0,0 +1,46 @@ |
|||||||
|
#!/bin/bash |
||||||
|
|
||||||
|
# Wrapper script to run the benchmark suite and automatically shut down when complete |
||||||
|
|
||||||
|
set -e |
||||||
|
|
||||||
|
# Determine docker-compose command |
||||||
|
if docker compose version &> /dev/null 2>&1; then |
||||||
|
DOCKER_COMPOSE="docker compose" |
||||||
|
else |
||||||
|
DOCKER_COMPOSE="docker-compose" |
||||||
|
fi |
||||||
|
|
||||||
|
# Clean old data directories (may be owned by root from Docker) |
||||||
|
if [ -d "data" ]; then |
||||||
|
echo "Cleaning old data directories..." |
||||||
|
if ! rm -rf data/ 2>/dev/null; then |
||||||
|
# If normal rm fails (permission denied), provide clear instructions |
||||||
|
echo "" |
||||||
|
echo "ERROR: Cannot remove data directories due to permission issues." |
||||||
|
echo "This happens because Docker creates files as root." |
||||||
|
echo "" |
||||||
|
echo "Please run one of the following to clean up:" |
||||||
|
echo " sudo rm -rf data/" |
||||||
|
echo " sudo chown -R \$(id -u):\$(id -g) data/ && rm -rf data/" |
||||||
|
echo "" |
||||||
|
echo "Then run this script again." |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
fi |
||||||
|
|
||||||
|
# Create fresh data directories with correct permissions |
||||||
|
echo "Preparing data directories..." |
||||||
|
mkdir -p data/{next-orly,khatru-sqlite,khatru-badger,relayer-basic,strfry,nostr-rs-relay,postgres} |
||||||
|
chmod 777 data/{next-orly,khatru-sqlite,khatru-badger,relayer-basic,strfry,nostr-rs-relay,postgres} |
||||||
|
|
||||||
|
echo "Starting benchmark suite..." |
||||||
|
echo "This will automatically shut down all containers when the benchmark completes." |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Run docker compose with flags to exit when benchmark-runner completes |
||||||
|
$DOCKER_COMPOSE up --exit-code-from benchmark-runner --abort-on-container-exit |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo "Benchmark suite has completed and all containers have been stopped." |
||||||
|
echo "Check the ./reports/ directory for results." |
||||||
@ -0,0 +1,41 @@ |
|||||||
|
#!/bin/bash |
||||||
|
|
||||||
|
# Run benchmark with profiling on ORLY only |
||||||
|
|
||||||
|
set -e |
||||||
|
|
||||||
|
# Determine docker-compose command |
||||||
|
if docker compose version &> /dev/null 2>&1; then |
||||||
|
DOCKER_COMPOSE="docker compose" |
||||||
|
else |
||||||
|
DOCKER_COMPOSE="docker-compose" |
||||||
|
fi |
||||||
|
|
||||||
|
# Clean up old data and profiles (may need sudo for Docker-created files) |
||||||
|
echo "Cleaning old data and profiles..." |
||||||
|
if [ -d "data/next-orly" ]; then |
||||||
|
if ! rm -rf data/next-orly/* 2>/dev/null; then |
||||||
|
echo "Need elevated permissions to clean data directories..." |
||||||
|
sudo rm -rf data/next-orly/* |
||||||
|
fi |
||||||
|
fi |
||||||
|
rm -rf profiles/* 2>/dev/null || sudo rm -rf profiles/* 2>/dev/null || true |
||||||
|
mkdir -p data/next-orly profiles |
||||||
|
chmod 777 data/next-orly 2>/dev/null || true |
||||||
|
|
||||||
|
echo "Starting profiled benchmark (ORLY only)..." |
||||||
|
echo "- 50,000 events" |
||||||
|
echo "- 24 workers" |
||||||
|
echo "- 90 second warmup delay" |
||||||
|
echo "- CPU profiling enabled" |
||||||
|
echo "- pprof HTTP on port 6060" |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Run docker compose with profile config |
||||||
|
$DOCKER_COMPOSE -f docker-compose.profile.yml up \ |
||||||
|
--exit-code-from benchmark-runner \ |
||||||
|
--abort-on-container-exit |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo "Benchmark complete. Profiles saved to ./profiles/" |
||||||
|
echo "Results saved to ./reports/" |
||||||
@ -0,0 +1,39 @@ |
|||||||
|
package database |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"fmt" |
||||||
|
"strings" |
||||||
|
) |
||||||
|
|
||||||
|
// NewDatabase creates a database instance based on the specified type.
|
||||||
|
// Supported types: "badger", "dgraph"
|
||||||
|
func NewDatabase( |
||||||
|
ctx context.Context, |
||||||
|
cancel context.CancelFunc, |
||||||
|
dbType string, |
||||||
|
dataDir string, |
||||||
|
logLevel string, |
||||||
|
) (Database, error) { |
||||||
|
switch strings.ToLower(dbType) { |
||||||
|
case "badger", "": |
||||||
|
// Use the existing badger implementation
|
||||||
|
return New(ctx, cancel, dataDir, logLevel) |
||||||
|
case "dgraph": |
||||||
|
// Use the new dgraph implementation
|
||||||
|
// Import dynamically to avoid import cycles
|
||||||
|
return newDgraphDatabase(ctx, cancel, dataDir, logLevel) |
||||||
|
default: |
||||||
|
return nil, fmt.Errorf("unsupported database type: %s (supported: badger, dgraph)", dbType) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// newDgraphDatabase creates a dgraph database instance
|
||||||
|
// This is defined here to avoid import cycles
|
||||||
|
var newDgraphDatabase func(context.Context, context.CancelFunc, string, string) (Database, error) |
||||||
|
|
||||||
|
// RegisterDgraphFactory registers the dgraph database factory
|
||||||
|
// This is called from the dgraph package's init() function
|
||||||
|
func RegisterDgraphFactory(factory func(context.Context, context.CancelFunc, string, string) (Database, error)) { |
||||||
|
newDgraphDatabase = factory |
||||||
|
} |
||||||
@ -0,0 +1,102 @@ |
|||||||
|
package database |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"io" |
||||||
|
"time" |
||||||
|
|
||||||
|
"next.orly.dev/pkg/database/indexes/types" |
||||||
|
"next.orly.dev/pkg/encoders/event" |
||||||
|
"next.orly.dev/pkg/encoders/filter" |
||||||
|
"next.orly.dev/pkg/encoders/tag" |
||||||
|
"next.orly.dev/pkg/interfaces/store" |
||||||
|
) |
||||||
|
|
||||||
|
// Database defines the interface that all database implementations must satisfy.
|
||||||
|
// This allows switching between different storage backends (badger, dgraph, etc.)
|
||||||
|
type Database interface { |
||||||
|
// Core lifecycle methods
|
||||||
|
Path() string |
||||||
|
Init(path string) error |
||||||
|
Sync() error |
||||||
|
Close() error |
||||||
|
Wipe() error |
||||||
|
SetLogLevel(level string) |
||||||
|
Ready() <-chan struct{} // Returns a channel that closes when database is ready to serve requests
|
||||||
|
|
||||||
|
// Event storage and retrieval
|
||||||
|
SaveEvent(c context.Context, ev *event.E) (exists bool, err error) |
||||||
|
GetSerialsFromFilter(f *filter.F) (serials types.Uint40s, err error) |
||||||
|
WouldReplaceEvent(ev *event.E) (bool, types.Uint40s, error) |
||||||
|
|
||||||
|
QueryEvents(c context.Context, f *filter.F) (evs event.S, err error) |
||||||
|
QueryAllVersions(c context.Context, f *filter.F) (evs event.S, err error) |
||||||
|
QueryEventsWithOptions(c context.Context, f *filter.F, includeDeleteEvents bool, showAllVersions bool) (evs event.S, err error) |
||||||
|
QueryDeleteEventsByTargetId(c context.Context, targetEventId []byte) (evs event.S, err error) |
||||||
|
QueryForSerials(c context.Context, f *filter.F) (serials types.Uint40s, err error) |
||||||
|
QueryForIds(c context.Context, f *filter.F) (idPkTs []*store.IdPkTs, err error) |
||||||
|
|
||||||
|
CountEvents(c context.Context, f *filter.F) (count int, approximate bool, err error) |
||||||
|
|
||||||
|
FetchEventBySerial(ser *types.Uint40) (ev *event.E, err error) |
||||||
|
FetchEventsBySerials(serials []*types.Uint40) (events map[uint64]*event.E, err error) |
||||||
|
|
||||||
|
GetSerialById(id []byte) (ser *types.Uint40, err error) |
||||||
|
GetSerialsByIds(ids *tag.T) (serials map[string]*types.Uint40, err error) |
||||||
|
GetSerialsByIdsWithFilter(ids *tag.T, fn func(ev *event.E, ser *types.Uint40) bool) (serials map[string]*types.Uint40, err error) |
||||||
|
GetSerialsByRange(idx Range) (serials types.Uint40s, err error) |
||||||
|
|
||||||
|
GetFullIdPubkeyBySerial(ser *types.Uint40) (fidpk *store.IdPkTs, err error) |
||||||
|
GetFullIdPubkeyBySerials(sers []*types.Uint40) (fidpks []*store.IdPkTs, err error) |
||||||
|
|
||||||
|
// Event deletion
|
||||||
|
DeleteEvent(c context.Context, eid []byte) error |
||||||
|
DeleteEventBySerial(c context.Context, ser *types.Uint40, ev *event.E) error |
||||||
|
DeleteExpired() |
||||||
|
ProcessDelete(ev *event.E, admins [][]byte) error |
||||||
|
CheckForDeleted(ev *event.E, admins [][]byte) error |
||||||
|
|
||||||
|
// Import/Export
|
||||||
|
Import(rr io.Reader) |
||||||
|
Export(c context.Context, w io.Writer, pubkeys ...[]byte) |
||||||
|
ImportEventsFromReader(ctx context.Context, rr io.Reader) error |
||||||
|
ImportEventsFromStrings(ctx context.Context, eventJSONs []string, policyManager interface{ CheckPolicy(action string, ev *event.E, pubkey []byte, remote string) (bool, error) }) error |
||||||
|
|
||||||
|
// Relay identity
|
||||||
|
GetRelayIdentitySecret() (skb []byte, err error) |
||||||
|
SetRelayIdentitySecret(skb []byte) error |
||||||
|
GetOrCreateRelayIdentitySecret() (skb []byte, err error) |
||||||
|
|
||||||
|
// Markers (metadata key-value storage)
|
||||||
|
SetMarker(key string, value []byte) error |
||||||
|
GetMarker(key string) (value []byte, err error) |
||||||
|
HasMarker(key string) bool |
||||||
|
DeleteMarker(key string) error |
||||||
|
|
||||||
|
// Subscriptions (payment-based access control)
|
||||||
|
GetSubscription(pubkey []byte) (*Subscription, error) |
||||||
|
IsSubscriptionActive(pubkey []byte) (bool, error) |
||||||
|
ExtendSubscription(pubkey []byte, days int) error |
||||||
|
RecordPayment(pubkey []byte, amount int64, invoice, preimage string) error |
||||||
|
GetPaymentHistory(pubkey []byte) ([]Payment, error) |
||||||
|
ExtendBlossomSubscription(pubkey []byte, tier string, storageMB int64, daysExtended int) error |
||||||
|
GetBlossomStorageQuota(pubkey []byte) (quotaMB int64, err error) |
||||||
|
IsFirstTimeUser(pubkey []byte) (bool, error) |
||||||
|
|
||||||
|
// NIP-43 Invite-based ACL
|
||||||
|
AddNIP43Member(pubkey []byte, inviteCode string) error |
||||||
|
RemoveNIP43Member(pubkey []byte) error |
||||||
|
IsNIP43Member(pubkey []byte) (isMember bool, err error) |
||||||
|
GetNIP43Membership(pubkey []byte) (*NIP43Membership, error) |
||||||
|
GetAllNIP43Members() ([][]byte, error) |
||||||
|
StoreInviteCode(code string, expiresAt time.Time) error |
||||||
|
ValidateInviteCode(code string) (valid bool, err error) |
||||||
|
DeleteInviteCode(code string) error |
||||||
|
PublishNIP43MembershipEvent(kind int, pubkey []byte) error |
||||||
|
|
||||||
|
// Migrations (version tracking for schema updates)
|
||||||
|
RunMigrations() |
||||||
|
|
||||||
|
// Utility methods
|
||||||
|
EventIdsBySerial(start uint64, count int) (evs []uint64, err error) |
||||||
|
} |
||||||
@ -0,0 +1,280 @@ |
|||||||
|
# Dgraph Database Implementation for ORLY |
||||||
|
|
||||||
|
This package provides a Dgraph-based implementation of the ORLY database interface, enabling graph-based storage for Nostr events with powerful relationship querying capabilities. |
||||||
|
|
||||||
|
## Status: Step 1 Complete ✅ |
||||||
|
|
||||||
|
**Current State:** Dgraph server integration is complete and functional |
||||||
|
**Next Step:** DQL query/mutation implementation in save-event.go and query-events.go |
||||||
|
|
||||||
|
## Architecture |
||||||
|
|
||||||
|
### Client-Server Model |
||||||
|
|
||||||
|
The implementation uses a **client-server architecture**: |
||||||
|
|
||||||
|
``` |
||||||
|
┌─────────────────────────────────────────────┐ |
||||||
|
│ ORLY Relay Process │ |
||||||
|
│ │ |
||||||
|
│ ┌────────────────────────────────────┐ │ |
||||||
|
│ │ Dgraph Client (pkg/dgraph) │ │ |
||||||
|
│ │ - dgo library (gRPC) │ │ |
||||||
|
│ │ - Schema management │────┼───► Dgraph Server |
||||||
|
│ │ - Query/Mutate methods │ │ (localhost:9080) |
||||||
|
│ └────────────────────────────────────┘ │ - Event graph |
||||||
|
│ │ - Authors, tags |
||||||
|
│ ┌────────────────────────────────────┐ │ - Relationships |
||||||
|
│ │ Badger Metadata Store │ │ |
||||||
|
│ │ - Markers (key-value) │ │ |
||||||
|
│ │ - Serial counters │ │ |
||||||
|
│ │ - Relay identity │ │ |
||||||
|
│ └────────────────────────────────────┘ │ |
||||||
|
└─────────────────────────────────────────────┘ |
||||||
|
``` |
||||||
|
|
||||||
|
### Dual Storage Strategy |
||||||
|
|
||||||
|
1. **Dgraph** (Graph Database) |
||||||
|
- Nostr events and their content |
||||||
|
- Author relationships |
||||||
|
- Tag relationships |
||||||
|
- Event references and mentions |
||||||
|
- Optimized for graph traversals and complex queries |
||||||
|
|
||||||
|
2. **Badger** (Key-Value Store) |
||||||
|
- Metadata markers |
||||||
|
- Serial number counters |
||||||
|
- Relay identity keys |
||||||
|
- Fast key-value operations |
||||||
|
|
||||||
|
## Setup |
||||||
|
|
||||||
|
### 1. Start Dgraph Server |
||||||
|
|
||||||
|
Using Docker (recommended): |
||||||
|
|
||||||
|
```bash |
||||||
|
docker run -d \ |
||||||
|
--name dgraph \ |
||||||
|
-p 8080:8080 \ |
||||||
|
-p 9080:9080 \ |
||||||
|
-p 8000:8000 \ |
||||||
|
-v ~/dgraph:/dgraph \ |
||||||
|
dgraph/standalone:latest |
||||||
|
``` |
||||||
|
|
||||||
|
### 2. Configure ORLY |
||||||
|
|
||||||
|
```bash |
||||||
|
export ORLY_DB_TYPE=dgraph |
||||||
|
export ORLY_DGRAPH_URL=localhost:9080 # Optional, this is the default |
||||||
|
``` |
||||||
|
|
||||||
|
### 3. Run ORLY |
||||||
|
|
||||||
|
```bash |
||||||
|
./orly |
||||||
|
``` |
||||||
|
|
||||||
|
On startup, ORLY will: |
||||||
|
1. Connect to dgraph server via gRPC |
||||||
|
2. Apply the Nostr schema automatically |
||||||
|
3. Initialize badger metadata store |
||||||
|
4. Initialize serial number counter |
||||||
|
5. Start accepting events |
||||||
|
|
||||||
|
## Schema |
||||||
|
|
||||||
|
The Nostr schema defines the following types: |
||||||
|
|
||||||
|
### Event Nodes |
||||||
|
```dql |
||||||
|
type Event { |
||||||
|
event.id # Event ID (string, indexed) |
||||||
|
event.serial # Sequential number (int, indexed) |
||||||
|
event.kind # Event kind (int, indexed) |
||||||
|
event.created_at # Timestamp (int, indexed) |
||||||
|
event.content # Event content (string) |
||||||
|
event.sig # Signature (string, indexed) |
||||||
|
event.pubkey # Author pubkey (string, indexed) |
||||||
|
event.authored_by # -> Author (uid) |
||||||
|
event.references # -> Events (uid list) |
||||||
|
event.mentions # -> Events (uid list) |
||||||
|
event.tagged_with # -> Tags (uid list) |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
### Author Nodes |
||||||
|
```dql |
||||||
|
type Author { |
||||||
|
author.pubkey # Pubkey (string, indexed, unique) |
||||||
|
author.events # -> Events (uid list, reverse) |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
### Tag Nodes |
||||||
|
```dql |
||||||
|
type Tag { |
||||||
|
tag.type # Tag type (string, indexed) |
||||||
|
tag.value # Tag value (string, indexed + fulltext) |
||||||
|
tag.events # -> Events (uid list, reverse) |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
### Marker Nodes (Metadata) |
||||||
|
```dql |
||||||
|
type Marker { |
||||||
|
marker.key # Key (string, indexed, unique) |
||||||
|
marker.value # Value (string) |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
## Configuration |
||||||
|
|
||||||
|
### Environment Variables |
||||||
|
|
||||||
|
- `ORLY_DB_TYPE=dgraph` - Enable dgraph database (default: badger) |
||||||
|
- `ORLY_DGRAPH_URL=host:port` - Dgraph gRPC endpoint (default: localhost:9080) |
||||||
|
- `ORLY_DATA_DIR=/path` - Data directory for metadata storage |
||||||
|
|
||||||
|
### Connection Details |
||||||
|
|
||||||
|
The dgraph client uses **insecure gRPC** by default for local development. For production deployments: |
||||||
|
|
||||||
|
1. Set up TLS certificates for dgraph |
||||||
|
2. Modify `pkg/dgraph/dgraph.go` to use `grpc.WithTransportCredentials()` with your certs |
||||||
|
|
||||||
|
## Implementation Details |
||||||
|
|
||||||
|
### Files |
||||||
|
|
||||||
|
- `dgraph.go` - Main implementation, initialization, lifecycle |
||||||
|
- `schema.go` - Schema definition and application |
||||||
|
- `save-event.go` - Event storage (TODO: update to use Mutate) |
||||||
|
- `query-events.go` - Event queries (TODO: update to parse DQL responses) |
||||||
|
- `fetch-event.go` - Event retrieval methods |
||||||
|
- `delete.go` - Event deletion |
||||||
|
- `markers.go` - Key-value metadata storage (uses badger) |
||||||
|
- `serial.go` - Serial number generation (uses badger) |
||||||
|
- `subscriptions.go` - Subscription/payment tracking (uses markers) |
||||||
|
- `nip43.go` - NIP-43 invite system (uses markers) |
||||||
|
- `import-export.go` - Import/export operations |
||||||
|
- `logger.go` - Logging adapter |
||||||
|
|
||||||
|
### Key Methods |
||||||
|
|
||||||
|
#### Initialization |
||||||
|
```go |
||||||
|
d, err := dgraph.New(ctx, cancel, dataDir, logLevel) |
||||||
|
``` |
||||||
|
|
||||||
|
#### Querying (DQL) |
||||||
|
```go |
||||||
|
resp, err := d.Query(ctx, dqlQuery) |
||||||
|
``` |
||||||
|
|
||||||
|
#### Mutations (RDF N-Quads) |
||||||
|
```go |
||||||
|
mutation := &api.Mutation{SetNquads: []byte(nquads)} |
||||||
|
resp, err := d.Mutate(ctx, mutation) |
||||||
|
``` |
||||||
|
|
||||||
|
## Development Status |
||||||
|
|
||||||
|
### ✅ Step 1: Dgraph Server Integration (COMPLETE) |
||||||
|
|
||||||
|
- [x] dgo client library integration |
||||||
|
- [x] gRPC connection to external dgraph |
||||||
|
- [x] Schema definition and auto-application |
||||||
|
- [x] Query() and Mutate() method stubs |
||||||
|
- [x] ORLY_DGRAPH_URL configuration |
||||||
|
- [x] Dual-storage architecture |
||||||
|
- [x] Proper lifecycle management |
||||||
|
|
||||||
|
### 📝 Step 2: DQL Implementation (NEXT) |
||||||
|
|
||||||
|
Priority tasks: |
||||||
|
|
||||||
|
1. **save-event.go** - Replace RDF string building with actual Mutate() calls |
||||||
|
2. **query-events.go** - Parse actual JSON responses from Query() |
||||||
|
3. **fetch-event.go** - Implement DQL queries for event retrieval |
||||||
|
4. **delete.go** - Implement deletion mutations |
||||||
|
|
||||||
|
### 📝 Step 3: Testing (FUTURE) |
||||||
|
|
||||||
|
- Integration testing with relay-tester |
||||||
|
- Performance benchmarks vs badger |
||||||
|
- Memory profiling |
||||||
|
- Production deployment testing |
||||||
|
|
||||||
|
## Troubleshooting |
||||||
|
|
||||||
|
### Connection Refused |
||||||
|
|
||||||
|
``` |
||||||
|
failed to connect to dgraph at localhost:9080: connection refused |
||||||
|
``` |
||||||
|
|
||||||
|
**Solution:** Ensure dgraph server is running: |
||||||
|
```bash |
||||||
|
docker ps | grep dgraph |
||||||
|
docker logs dgraph |
||||||
|
``` |
||||||
|
|
||||||
|
### Schema Application Failed |
||||||
|
|
||||||
|
``` |
||||||
|
failed to apply schema: ... |
||||||
|
``` |
||||||
|
|
||||||
|
**Solution:** Check dgraph server logs and ensure no schema conflicts: |
||||||
|
```bash |
||||||
|
docker logs dgraph |
||||||
|
``` |
||||||
|
|
||||||
|
### Binary Not Finding libsecp256k1.so |
||||||
|
|
||||||
|
This is unrelated to dgraph. Ensure: |
||||||
|
```bash |
||||||
|
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}$(pwd)/pkg/crypto/p8k" |
||||||
|
``` |
||||||
|
|
||||||
|
## Performance Considerations |
||||||
|
|
||||||
|
### When to Use Dgraph |
||||||
|
|
||||||
|
**Good fit:** |
||||||
|
- Complex graph queries (follows-of-follows, social graphs) |
||||||
|
- Full-text search requirements |
||||||
|
- Advanced filtering and aggregations |
||||||
|
- Multi-hop relationship traversals |
||||||
|
|
||||||
|
**Not ideal for:** |
||||||
|
- Simple key-value lookups (badger is faster) |
||||||
|
- Very high write throughput (badger has lower latency) |
||||||
|
- Single-node deployments with simple queries |
||||||
|
|
||||||
|
### Optimization Tips |
||||||
|
|
||||||
|
1. **Indexing**: Ensure frequently queried fields have appropriate indexes |
||||||
|
2. **Pagination**: Use offset/limit in DQL queries for large result sets |
||||||
|
3. **Caching**: Consider adding an LRU cache for hot events |
||||||
|
4. **Schema Design**: Use reverse edges for efficient relationship traversal |
||||||
|
|
||||||
|
## Resources |
||||||
|
|
||||||
|
- [Dgraph Documentation](https://dgraph.io/docs/) |
||||||
|
- [DQL Query Language](https://dgraph.io/docs/query-language/) |
||||||
|
- [dgo Client Library](https://github.com/dgraph-io/dgo) |
||||||
|
- [ORLY Implementation Status](../../DGRAPH_IMPLEMENTATION_STATUS.md) |
||||||
|
|
||||||
|
## Contributing |
||||||
|
|
||||||
|
When working on dgraph implementation: |
||||||
|
|
||||||
|
1. Test changes against a local dgraph instance |
||||||
|
2. Update schema.go if adding new node types or predicates |
||||||
|
3. Ensure dual-storage strategy is maintained (dgraph for events, badger for metadata) |
||||||
|
4. Add integration tests for new features |
||||||
|
5. Update DGRAPH_IMPLEMENTATION_STATUS.md with progress |
||||||
@ -0,0 +1,330 @@ |
|||||||
|
# Dgraph Test Suite |
||||||
|
|
||||||
|
This directory contains a comprehensive test suite for the dgraph database implementation, mirroring all tests from the badger implementation to ensure feature parity. |
||||||
|
|
||||||
|
## Test Files |
||||||
|
|
||||||
|
- **testmain_test.go** - Test configuration (logging, setup) |
||||||
|
- **helpers_test.go** - Helper functions for test database setup/teardown |
||||||
|
- **save-event_test.go** - Event storage tests |
||||||
|
- **query-events_test.go** - Event query tests |
||||||
|
|
||||||
|
## Quick Start |
||||||
|
|
||||||
|
### 1. Start Dgraph Server |
||||||
|
|
||||||
|
```bash |
||||||
|
# From project root |
||||||
|
./scripts/dgraph-start.sh |
||||||
|
|
||||||
|
# Verify it's running |
||||||
|
curl http://localhost:8080/health |
||||||
|
``` |
||||||
|
|
||||||
|
### 2. Run Tests |
||||||
|
|
||||||
|
```bash |
||||||
|
# Run all dgraph tests |
||||||
|
./scripts/test-dgraph.sh |
||||||
|
|
||||||
|
# Or run manually |
||||||
|
export ORLY_DGRAPH_URL=localhost:9080 |
||||||
|
CGO_ENABLED=0 go test -v ./pkg/dgraph/... |
||||||
|
|
||||||
|
# Run specific test |
||||||
|
CGO_ENABLED=0 go test -v -run TestSaveEvents ./pkg/dgraph |
||||||
|
``` |
||||||
|
|
||||||
|
## Test Coverage |
||||||
|
|
||||||
|
### Event Storage Tests (`save-event_test.go`) |
||||||
|
|
||||||
|
✅ **TestSaveEvents** |
||||||
|
- Loads ~100 events from examples.Cache |
||||||
|
- Saves all events chronologically |
||||||
|
- Verifies no errors during save |
||||||
|
- Reports performance metrics |
||||||
|
|
||||||
|
✅ **TestDeletionEventWithETagRejection** |
||||||
|
- Creates a regular event |
||||||
|
- Attempts to save deletion event with e-tag |
||||||
|
- Verifies deletion events with e-tags are rejected |
||||||
|
|
||||||
|
✅ **TestSaveExistingEvent** |
||||||
|
- Saves an event |
||||||
|
- Attempts to save same event again |
||||||
|
- Verifies duplicate events are rejected |
||||||
|
|
||||||
|
### Event Query Tests (`query-events_test.go`) |
||||||
|
|
||||||
|
✅ **TestQueryEventsByID** |
||||||
|
- Queries event by exact ID match |
||||||
|
- Verifies single result returned |
||||||
|
- Verifies correct event retrieved |
||||||
|
|
||||||
|
✅ **TestQueryEventsByKind** |
||||||
|
- Queries events by kind (e.g., kind 1) |
||||||
|
- Verifies all results have correct kind |
||||||
|
- Tests filtering logic |
||||||
|
|
||||||
|
✅ **TestQueryEventsByAuthor** |
||||||
|
- Queries events by author pubkey |
||||||
|
- Verifies all results from correct author |
||||||
|
- Tests author filtering |
||||||
|
|
||||||
|
✅ **TestReplaceableEventsAndDeletion** |
||||||
|
- Creates replaceable event (kind 0) |
||||||
|
- Creates newer version |
||||||
|
- Verifies only newer version returned in general queries |
||||||
|
- Creates deletion event |
||||||
|
- Verifies deleted event not returned |
||||||
|
- Tests replaceable event logic and deletion |
||||||
|
|
||||||
|
✅ **TestParameterizedReplaceableEventsAndDeletion** |
||||||
|
- Creates parameterized replaceable event (kind 30000+) |
||||||
|
- Adds d-tag |
||||||
|
- Creates deletion event with e-tag |
||||||
|
- Verifies deleted event not returned |
||||||
|
- Tests parameterized replaceable logic |
||||||
|
|
||||||
|
✅ **TestQueryEventsByTimeRange** |
||||||
|
- Queries events by since/until timestamps |
||||||
|
- Verifies all results within time range |
||||||
|
- Tests temporal filtering |
||||||
|
|
||||||
|
✅ **TestQueryEventsByTag** |
||||||
|
- Finds event with tags |
||||||
|
- Queries by tag key/value |
||||||
|
- Verifies all results have the tag |
||||||
|
- Tests tag filtering logic |
||||||
|
|
||||||
|
✅ **TestCountEvents** |
||||||
|
- Counts all events |
||||||
|
- Counts events by kind filter |
||||||
|
- Verifies correct counts returned |
||||||
|
- Tests counting functionality |
||||||
|
|
||||||
|
## Test Helpers |
||||||
|
|
||||||
|
### setupTestDB(t *testing.T) |
||||||
|
|
||||||
|
Creates a test dgraph database: |
||||||
|
|
||||||
|
1. **Checks dgraph availability** - Skips test if server not running |
||||||
|
2. **Creates temp directory** - For metadata storage |
||||||
|
3. **Initializes dgraph client** - Connects to server |
||||||
|
4. **Drops all data** - Starts with clean slate |
||||||
|
5. **Loads test events** - From examples.Cache (~100 events) |
||||||
|
6. **Sorts chronologically** - Ensures addressable events processed in order |
||||||
|
7. **Saves all events** - Populates test database |
||||||
|
|
||||||
|
**Returns:** `(*D, []*event.E, context.Context, context.CancelFunc, string)` |
||||||
|
|
||||||
|
### cleanupTestDB(t, db, cancel, tempDir) |
||||||
|
|
||||||
|
Cleans up after tests: |
||||||
|
- Closes database connection |
||||||
|
- Cancels context |
||||||
|
- Removes temp directory |
||||||
|
|
||||||
|
### skipIfDgraphNotAvailable(t *testing.T) |
||||||
|
|
||||||
|
Checks if dgraph is running and skips test if not available. |
||||||
|
|
||||||
|
## Running Tests |
||||||
|
|
||||||
|
### Prerequisites |
||||||
|
|
||||||
|
1. **Dgraph Server** - Must be running before tests |
||||||
|
2. **Go 1.21+** - For running tests |
||||||
|
3. **CGO_ENABLED=0** - For pure Go build |
||||||
|
|
||||||
|
### Test Execution |
||||||
|
|
||||||
|
#### All Tests |
||||||
|
|
||||||
|
```bash |
||||||
|
./scripts/test-dgraph.sh |
||||||
|
``` |
||||||
|
|
||||||
|
#### Specific Test File |
||||||
|
|
||||||
|
```bash |
||||||
|
CGO_ENABLED=0 go test -v ./pkg/dgraph -run TestSaveEvents |
||||||
|
``` |
||||||
|
|
||||||
|
#### With Logging |
||||||
|
|
||||||
|
```bash |
||||||
|
export TEST_LOG=1 |
||||||
|
CGO_ENABLED=0 go test -v ./pkg/dgraph/... |
||||||
|
``` |
||||||
|
|
||||||
|
#### With Timeout |
||||||
|
|
||||||
|
```bash |
||||||
|
CGO_ENABLED=0 go test -v -timeout 10m ./pkg/dgraph/... |
||||||
|
``` |
||||||
|
|
||||||
|
### Integration Testing |
||||||
|
|
||||||
|
Run tests + relay-tester: |
||||||
|
|
||||||
|
```bash |
||||||
|
./scripts/test-dgraph.sh --relay-tester |
||||||
|
``` |
||||||
|
|
||||||
|
This will: |
||||||
|
1. Run all dgraph package tests |
||||||
|
2. Start ORLY with dgraph backend |
||||||
|
3. Run relay-tester against ORLY |
||||||
|
4. Report results |
||||||
|
|
||||||
|
## Test Data |
||||||
|
|
||||||
|
Tests use `pkg/encoders/event/examples.Cache` which contains: |
||||||
|
- ~100 real Nostr events |
||||||
|
- Text notes (kind 1) |
||||||
|
- Profile metadata (kind 0) |
||||||
|
- Various other kinds |
||||||
|
- Events with tags, references, mentions |
||||||
|
- Multiple authors and timestamps |
||||||
|
|
||||||
|
This ensures tests cover realistic scenarios. |
||||||
|
|
||||||
|
## Debugging Tests |
||||||
|
|
||||||
|
### View Test Output |
||||||
|
|
||||||
|
```bash |
||||||
|
CGO_ENABLED=0 go test -v ./pkg/dgraph/... 2>&1 | tee test-output.log |
||||||
|
``` |
||||||
|
|
||||||
|
### Check Dgraph State |
||||||
|
|
||||||
|
```bash |
||||||
|
# View data via Ratel UI |
||||||
|
open http://localhost:8000 |
||||||
|
|
||||||
|
# Query via HTTP |
||||||
|
curl -X POST localhost:8080/query -d '{ |
||||||
|
events(func: type(Event), first: 10) { |
||||||
|
uid |
||||||
|
event.id |
||||||
|
event.kind |
||||||
|
event.created_at |
||||||
|
} |
||||||
|
}' |
||||||
|
``` |
||||||
|
|
||||||
|
### Enable Dgraph Logging |
||||||
|
|
||||||
|
```bash |
||||||
|
docker logs dgraph-orly-test -f |
||||||
|
``` |
||||||
|
|
||||||
|
## Test Failures |
||||||
|
|
||||||
|
### "Dgraph server not available" |
||||||
|
|
||||||
|
**Cause:** Dgraph is not running |
||||||
|
|
||||||
|
**Fix:** |
||||||
|
```bash |
||||||
|
./scripts/dgraph-start.sh |
||||||
|
``` |
||||||
|
|
||||||
|
### Connection Timeouts |
||||||
|
|
||||||
|
**Cause:** Dgraph server overloaded or network issues |
||||||
|
|
||||||
|
**Fix:** |
||||||
|
- Increase test timeout: `go test -timeout 20m` |
||||||
|
- Check dgraph resources: `docker stats dgraph-orly-test` |
||||||
|
- Restart dgraph: `docker restart dgraph-orly-test` |
||||||
|
|
||||||
|
### Schema Errors |
||||||
|
|
||||||
|
**Cause:** Schema conflicts or version mismatch |
||||||
|
|
||||||
|
**Fix:** |
||||||
|
- Drop all data: Tests call `dropAll()` automatically |
||||||
|
- Check dgraph version: `docker exec dgraph-orly-test dgraph version` |
||||||
|
|
||||||
|
### Test Hangs |
||||||
|
|
||||||
|
**Cause:** Deadlock or infinite loop |
||||||
|
|
||||||
|
**Fix:** |
||||||
|
- Send SIGQUIT: `kill -QUIT <test-pid>` |
||||||
|
- View goroutine dump |
||||||
|
- Check dgraph logs |
||||||
|
|
||||||
|
## Continuous Integration |
||||||
|
|
||||||
|
### GitHub Actions Example |
||||||
|
|
||||||
|
```yaml |
||||||
|
name: Dgraph Tests |
||||||
|
|
||||||
|
on: [push, pull_request] |
||||||
|
|
||||||
|
jobs: |
||||||
|
test: |
||||||
|
runs-on: ubuntu-latest |
||||||
|
|
||||||
|
services: |
||||||
|
dgraph: |
||||||
|
image: dgraph/standalone:latest |
||||||
|
ports: |
||||||
|
- 8080:8080 |
||||||
|
- 9080:9080 |
||||||
|
options: >- |
||||||
|
--health-cmd "curl -f http://localhost:8080/health" |
||||||
|
--health-interval 10s |
||||||
|
--health-timeout 5s |
||||||
|
--health-retries 5 |
||||||
|
|
||||||
|
steps: |
||||||
|
- uses: actions/checkout@v3 |
||||||
|
|
||||||
|
- name: Set up Go |
||||||
|
uses: actions/setup-go@v4 |
||||||
|
with: |
||||||
|
go-version: '1.21' |
||||||
|
|
||||||
|
- name: Run dgraph tests |
||||||
|
env: |
||||||
|
ORLY_DGRAPH_URL: localhost:9080 |
||||||
|
run: | |
||||||
|
CGO_ENABLED=0 go test -v -timeout 10m ./pkg/dgraph/... |
||||||
|
``` |
||||||
|
|
||||||
|
## Performance Benchmarks |
||||||
|
|
||||||
|
Compare with badger: |
||||||
|
|
||||||
|
```bash |
||||||
|
# Badger benchmarks |
||||||
|
go test -bench=. -benchmem ./pkg/database/... |
||||||
|
|
||||||
|
# Dgraph benchmarks |
||||||
|
go test -bench=. -benchmem ./pkg/dgraph/... |
||||||
|
``` |
||||||
|
|
||||||
|
## Related Documentation |
||||||
|
|
||||||
|
- [Main Testing Guide](../../scripts/DGRAPH_TESTING.md) |
||||||
|
- [Implementation Status](../../DGRAPH_IMPLEMENTATION_STATUS.md) |
||||||
|
- [Package README](README.md) |
||||||
|
|
||||||
|
## Contributing |
||||||
|
|
||||||
|
When adding new tests: |
||||||
|
|
||||||
|
1. **Mirror badger tests** - Ensure feature parity |
||||||
|
2. **Use test helpers** - setupTestDB() and cleanupTestDB() |
||||||
|
3. **Skip if unavailable** - Call skipIfDgraphNotAvailable(t) |
||||||
|
4. **Clean up resources** - Always defer cleanupTestDB() |
||||||
|
5. **Test chronologically** - Sort events by timestamp for addressable events |
||||||
|
6. **Verify behavior** - Don't just check for no errors, verify correctness |
||||||
@ -0,0 +1,190 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"fmt" |
||||||
|
|
||||||
|
"github.com/dgraph-io/dgo/v230/protos/api" |
||||||
|
"next.orly.dev/pkg/database/indexes/types" |
||||||
|
"next.orly.dev/pkg/encoders/event" |
||||||
|
"next.orly.dev/pkg/encoders/hex" |
||||||
|
) |
||||||
|
|
||||||
|
// DeleteEvent deletes an event by its ID
|
||||||
|
func (d *D) DeleteEvent(c context.Context, eid []byte) error { |
||||||
|
idStr := hex.Enc(eid) |
||||||
|
|
||||||
|
// Find the event's UID
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
event(func: eq(event.id, %q)) { |
||||||
|
uid |
||||||
|
} |
||||||
|
}`, idStr) |
||||||
|
|
||||||
|
resp, err := d.Query(c, query) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to find event for deletion: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Parse UID
|
||||||
|
var result struct { |
||||||
|
Event []struct { |
||||||
|
UID string `json:"uid"` |
||||||
|
} `json:"event"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = unmarshalJSON(resp.Json, &result); err != nil { |
||||||
|
return err |
||||||
|
} |
||||||
|
|
||||||
|
if len(result.Event) == 0 { |
||||||
|
return nil // Event doesn't exist
|
||||||
|
} |
||||||
|
|
||||||
|
// Delete the event node
|
||||||
|
mutation := &api.Mutation{ |
||||||
|
DelNquads: []byte(fmt.Sprintf("<%s> * * .", result.Event[0].UID)), |
||||||
|
CommitNow: true, |
||||||
|
} |
||||||
|
|
||||||
|
if _, err = d.Mutate(c, mutation); err != nil { |
||||||
|
return fmt.Errorf("failed to delete event: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// DeleteEventBySerial deletes an event by its serial number
|
||||||
|
func (d *D) DeleteEventBySerial(c context.Context, ser *types.Uint40, ev *event.E) error { |
||||||
|
serial := ser.Get() |
||||||
|
|
||||||
|
// Find the event's UID
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
event(func: eq(event.serial, %d)) { |
||||||
|
uid |
||||||
|
} |
||||||
|
}`, serial) |
||||||
|
|
||||||
|
resp, err := d.Query(c, query) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to find event for deletion: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Parse UID
|
||||||
|
var result struct { |
||||||
|
Event []struct { |
||||||
|
UID string `json:"uid"` |
||||||
|
} `json:"event"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = unmarshalJSON(resp.Json, &result); err != nil { |
||||||
|
return err |
||||||
|
} |
||||||
|
|
||||||
|
if len(result.Event) == 0 { |
||||||
|
return nil // Event doesn't exist
|
||||||
|
} |
||||||
|
|
||||||
|
// Delete the event node
|
||||||
|
mutation := &api.Mutation{ |
||||||
|
DelNquads: []byte(fmt.Sprintf("<%s> * * .", result.Event[0].UID)), |
||||||
|
CommitNow: true, |
||||||
|
} |
||||||
|
|
||||||
|
if _, err = d.Mutate(c, mutation); err != nil { |
||||||
|
return fmt.Errorf("failed to delete event: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// DeleteExpired removes events that have passed their expiration time
|
||||||
|
func (d *D) DeleteExpired() { |
||||||
|
// Query for events with expiration tags
|
||||||
|
// This is a stub - full implementation would:
|
||||||
|
// 1. Find events with "expiration" tag
|
||||||
|
// 2. Check if current time > expiration time
|
||||||
|
// 3. Delete those events
|
||||||
|
} |
||||||
|
|
||||||
|
// ProcessDelete processes a kind 5 deletion event
|
||||||
|
func (d *D) ProcessDelete(ev *event.E, admins [][]byte) (err error) { |
||||||
|
if ev.Kind != 5 { |
||||||
|
return fmt.Errorf("event is not a deletion event (kind 5)") |
||||||
|
} |
||||||
|
|
||||||
|
// Extract event IDs to delete from tags
|
||||||
|
for _, tag := range *ev.Tags { |
||||||
|
if len(tag.T) >= 2 && string(tag.T[0]) == "e" { |
||||||
|
eventID := tag.T[1] |
||||||
|
|
||||||
|
// Verify the deletion is authorized (author must match or be admin)
|
||||||
|
if err = d.CheckForDeleted(ev, admins); err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
|
||||||
|
// Delete the event
|
||||||
|
if err = d.DeleteEvent(context.Background(), eventID); err != nil { |
||||||
|
// Log error but continue with other deletions
|
||||||
|
d.Logger.Errorf("failed to delete event %s: %v", hex.Enc(eventID), err) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// CheckForDeleted checks if an event has been deleted
|
||||||
|
func (d *D) CheckForDeleted(ev *event.E, admins [][]byte) (err error) { |
||||||
|
// Query for delete events (kind 5) that reference this event
|
||||||
|
evID := hex.Enc(ev.ID[:]) |
||||||
|
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
deletes(func: eq(event.kind, 5)) @filter(eq(event.pubkey, %q)) { |
||||||
|
uid |
||||||
|
event.pubkey |
||||||
|
references @filter(eq(event.id, %q)) { |
||||||
|
event.id |
||||||
|
} |
||||||
|
} |
||||||
|
}`, hex.Enc(ev.Pubkey), evID) |
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to check for deletions: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Deletes []struct { |
||||||
|
UID string `json:"uid"` |
||||||
|
Pubkey string `json:"event.pubkey"` |
||||||
|
References []struct { |
||||||
|
ID string `json:"event.id"` |
||||||
|
} `json:"references"` |
||||||
|
} `json:"deletes"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = unmarshalJSON(resp.Json, &result); err != nil { |
||||||
|
return err |
||||||
|
} |
||||||
|
|
||||||
|
// Check if any delete events reference this event
|
||||||
|
for _, del := range result.Deletes { |
||||||
|
if len(del.References) > 0 { |
||||||
|
// Check if deletion is from the author or an admin
|
||||||
|
delPubkey, _ := hex.Dec(del.Pubkey) |
||||||
|
if string(delPubkey) == string(ev.Pubkey) { |
||||||
|
return fmt.Errorf("event has been deleted by author") |
||||||
|
} |
||||||
|
|
||||||
|
// Check admins
|
||||||
|
for _, admin := range admins { |
||||||
|
if string(delPubkey) == string(admin) { |
||||||
|
return fmt.Errorf("event has been deleted by admin") |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return nil |
||||||
|
} |
||||||
@ -0,0 +1,285 @@ |
|||||||
|
// Package dgraph provides a Dgraph-based implementation of the database interface.
|
||||||
|
// This is a simplified implementation for testing - full dgraph integration to be completed later.
|
||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"fmt" |
||||||
|
"os" |
||||||
|
"path/filepath" |
||||||
|
|
||||||
|
"github.com/dgraph-io/badger/v4" |
||||||
|
"github.com/dgraph-io/dgo/v230" |
||||||
|
"github.com/dgraph-io/dgo/v230/protos/api" |
||||||
|
"google.golang.org/grpc" |
||||||
|
"google.golang.org/grpc/credentials/insecure" |
||||||
|
"lol.mleku.dev" |
||||||
|
"lol.mleku.dev/chk" |
||||||
|
"next.orly.dev/pkg/database" |
||||||
|
"next.orly.dev/pkg/utils/apputil" |
||||||
|
) |
||||||
|
|
||||||
|
// D implements the database.Database interface using Dgraph as the storage backend
|
||||||
|
type D struct { |
||||||
|
ctx context.Context |
||||||
|
cancel context.CancelFunc |
||||||
|
dataDir string |
||||||
|
Logger *logger |
||||||
|
|
||||||
|
// Dgraph client connection
|
||||||
|
client *dgo.Dgraph |
||||||
|
conn *grpc.ClientConn |
||||||
|
|
||||||
|
// Fallback badger storage for metadata
|
||||||
|
pstore *badger.DB |
||||||
|
|
||||||
|
// Configuration
|
||||||
|
dgraphURL string |
||||||
|
enableGraphQL bool |
||||||
|
enableIntrospection bool |
||||||
|
|
||||||
|
ready chan struct{} // Closed when database is ready to serve requests
|
||||||
|
} |
||||||
|
|
||||||
|
// Ensure D implements database.Database interface at compile time
|
||||||
|
var _ database.Database = (*D)(nil) |
||||||
|
|
||||||
|
// init registers the dgraph database factory
|
||||||
|
func init() { |
||||||
|
database.RegisterDgraphFactory(func( |
||||||
|
ctx context.Context, |
||||||
|
cancel context.CancelFunc, |
||||||
|
dataDir string, |
||||||
|
logLevel string, |
||||||
|
) (database.Database, error) { |
||||||
|
return New(ctx, cancel, dataDir, logLevel) |
||||||
|
}) |
||||||
|
} |
||||||
|
|
||||||
|
// Config holds configuration options for the Dgraph database
|
||||||
|
type Config struct { |
||||||
|
DataDir string |
||||||
|
LogLevel string |
||||||
|
DgraphURL string // Dgraph gRPC endpoint (e.g., "localhost:9080")
|
||||||
|
EnableGraphQL bool |
||||||
|
EnableIntrospection bool |
||||||
|
} |
||||||
|
|
||||||
|
// New creates a new Dgraph-based database instance
|
||||||
|
func New( |
||||||
|
ctx context.Context, cancel context.CancelFunc, dataDir, logLevel string, |
||||||
|
) ( |
||||||
|
d *D, err error, |
||||||
|
) { |
||||||
|
// Get dgraph URL from environment, default to localhost
|
||||||
|
dgraphURL := os.Getenv("ORLY_DGRAPH_URL") |
||||||
|
if dgraphURL == "" { |
||||||
|
dgraphURL = "localhost:9080" |
||||||
|
} |
||||||
|
|
||||||
|
d = &D{ |
||||||
|
ctx: ctx, |
||||||
|
cancel: cancel, |
||||||
|
dataDir: dataDir, |
||||||
|
Logger: NewLogger(lol.GetLogLevel(logLevel), dataDir), |
||||||
|
dgraphURL: dgraphURL, |
||||||
|
enableGraphQL: false, |
||||||
|
enableIntrospection: false, |
||||||
|
ready: make(chan struct{}), |
||||||
|
} |
||||||
|
|
||||||
|
// Ensure the data directory exists
|
||||||
|
if err = os.MkdirAll(dataDir, 0755); chk.E(err) { |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// Ensure directory structure
|
||||||
|
dummyFile := filepath.Join(dataDir, "dummy.sst") |
||||||
|
if err = apputil.EnsureDir(dummyFile); chk.E(err) { |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// Initialize dgraph client connection
|
||||||
|
if err = d.initDgraphClient(); chk.E(err) { |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// Initialize badger for metadata storage
|
||||||
|
if err = d.initStorage(); chk.E(err) { |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// Apply Nostr schema to dgraph
|
||||||
|
if err = d.applySchema(ctx); chk.E(err) { |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// Initialize serial counter
|
||||||
|
if err = d.initSerialCounter(); chk.E(err) { |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// Start warmup goroutine to signal when database is ready
|
||||||
|
go d.warmup() |
||||||
|
|
||||||
|
// Setup shutdown handler
|
||||||
|
go func() { |
||||||
|
<-d.ctx.Done() |
||||||
|
d.cancel() |
||||||
|
if d.conn != nil { |
||||||
|
d.conn.Close() |
||||||
|
} |
||||||
|
if d.pstore != nil { |
||||||
|
d.pstore.Close() |
||||||
|
} |
||||||
|
}() |
||||||
|
|
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// initDgraphClient establishes connection to dgraph server
|
||||||
|
func (d *D) initDgraphClient() error { |
||||||
|
d.Logger.Infof("connecting to dgraph at %s", d.dgraphURL) |
||||||
|
|
||||||
|
// Establish gRPC connection
|
||||||
|
conn, err := grpc.Dial(d.dgraphURL, grpc.WithTransportCredentials(insecure.NewCredentials())) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to connect to dgraph at %s: %w", d.dgraphURL, err) |
||||||
|
} |
||||||
|
|
||||||
|
d.conn = conn |
||||||
|
d.client = dgo.NewDgraphClient(api.NewDgraphClient(conn)) |
||||||
|
|
||||||
|
d.Logger.Infof("successfully connected to dgraph") |
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// initStorage opens Badger database for metadata storage
|
||||||
|
func (d *D) initStorage() error { |
||||||
|
metadataDir := filepath.Join(d.dataDir, "metadata") |
||||||
|
|
||||||
|
if err := os.MkdirAll(metadataDir, 0755); err != nil { |
||||||
|
return fmt.Errorf("failed to create metadata directory: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
opts := badger.DefaultOptions(metadataDir) |
||||||
|
|
||||||
|
var err error |
||||||
|
d.pstore, err = badger.Open(opts) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to open badger metadata store: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
d.Logger.Infof("metadata storage initialized") |
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// Query executes a DQL query against dgraph
|
||||||
|
func (d *D) Query(ctx context.Context, query string) (*api.Response, error) { |
||||||
|
txn := d.client.NewReadOnlyTxn() |
||||||
|
defer txn.Discard(ctx) |
||||||
|
|
||||||
|
resp, err := txn.Query(ctx, query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("dgraph query failed: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return resp, nil |
||||||
|
} |
||||||
|
|
||||||
|
// Mutate executes a mutation against dgraph
|
||||||
|
func (d *D) Mutate(ctx context.Context, mutation *api.Mutation) (*api.Response, error) { |
||||||
|
txn := d.client.NewTxn() |
||||||
|
defer txn.Discard(ctx) |
||||||
|
|
||||||
|
resp, err := txn.Mutate(ctx, mutation) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("dgraph mutation failed: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
if err := txn.Commit(ctx); err != nil { |
||||||
|
return nil, fmt.Errorf("dgraph commit failed: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return resp, nil |
||||||
|
} |
||||||
|
|
||||||
|
// Path returns the data directory path
|
||||||
|
func (d *D) Path() string { return d.dataDir } |
||||||
|
|
||||||
|
// Init initializes the database with a given path (no-op, path set in New)
|
||||||
|
func (d *D) Init(path string) (err error) { |
||||||
|
// Path already set in New()
|
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// Sync flushes pending writes
|
||||||
|
func (d *D) Sync() (err error) { |
||||||
|
if d.pstore != nil { |
||||||
|
return d.pstore.Sync() |
||||||
|
} |
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// Close closes the database
|
||||||
|
func (d *D) Close() (err error) { |
||||||
|
d.cancel() |
||||||
|
if d.conn != nil { |
||||||
|
if e := d.conn.Close(); e != nil { |
||||||
|
err = e |
||||||
|
} |
||||||
|
} |
||||||
|
if d.pstore != nil { |
||||||
|
if e := d.pstore.Close(); e != nil && err == nil { |
||||||
|
err = e |
||||||
|
} |
||||||
|
} |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// Wipe removes all data
|
||||||
|
func (d *D) Wipe() (err error) { |
||||||
|
if d.pstore != nil { |
||||||
|
if err = d.pstore.Close(); chk.E(err) { |
||||||
|
return |
||||||
|
} |
||||||
|
} |
||||||
|
if err = os.RemoveAll(d.dataDir); chk.E(err) { |
||||||
|
return |
||||||
|
} |
||||||
|
return d.initStorage() |
||||||
|
} |
||||||
|
|
||||||
|
// SetLogLevel sets the logging level
|
||||||
|
func (d *D) SetLogLevel(level string) { |
||||||
|
// d.Logger.SetLevel(lol.GetLogLevel(level))
|
||||||
|
} |
||||||
|
|
||||||
|
// EventIdsBySerial retrieves event IDs by serial range (stub)
|
||||||
|
func (d *D) EventIdsBySerial(start uint64, count int) ( |
||||||
|
evs []uint64, err error, |
||||||
|
) { |
||||||
|
err = fmt.Errorf("not implemented") |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// RunMigrations runs database migrations (no-op for dgraph)
|
||||||
|
func (d *D) RunMigrations() { |
||||||
|
// No-op for dgraph
|
||||||
|
} |
||||||
|
|
||||||
|
// Ready returns a channel that closes when the database is ready to serve requests.
|
||||||
|
// This allows callers to wait for database warmup to complete.
|
||||||
|
func (d *D) Ready() <-chan struct{} { |
||||||
|
return d.ready |
||||||
|
} |
||||||
|
|
||||||
|
// warmup performs database warmup operations and closes the ready channel when complete.
|
||||||
|
// For Dgraph, warmup ensures the connection is healthy and schema is applied.
|
||||||
|
func (d *D) warmup() { |
||||||
|
defer close(d.ready) |
||||||
|
|
||||||
|
// Dgraph connection and schema are already verified during initialization
|
||||||
|
// Just give a brief moment for any background processes to settle
|
||||||
|
d.Logger.Infof("dgraph database warmup complete, ready to serve requests") |
||||||
|
} |
||||||
@ -0,0 +1,270 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"encoding/json" |
||||||
|
"fmt" |
||||||
|
|
||||||
|
"next.orly.dev/pkg/database" |
||||||
|
"next.orly.dev/pkg/database/indexes/types" |
||||||
|
"next.orly.dev/pkg/encoders/event" |
||||||
|
"next.orly.dev/pkg/encoders/hex" |
||||||
|
"next.orly.dev/pkg/encoders/tag" |
||||||
|
"next.orly.dev/pkg/interfaces/store" |
||||||
|
) |
||||||
|
|
||||||
|
// FetchEventBySerial retrieves an event by its serial number
|
||||||
|
func (d *D) FetchEventBySerial(ser *types.Uint40) (ev *event.E, err error) { |
||||||
|
serial := ser.Get() |
||||||
|
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
event(func: eq(event.serial, %d)) { |
||||||
|
event.id |
||||||
|
event.kind |
||||||
|
event.created_at |
||||||
|
event.content |
||||||
|
event.sig |
||||||
|
event.pubkey |
||||||
|
event.tags |
||||||
|
} |
||||||
|
}`, serial) |
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to fetch event by serial: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
evs, err := d.parseEventsFromResponse(resp.Json) |
||||||
|
if err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
if len(evs) == 0 { |
||||||
|
return nil, fmt.Errorf("event not found") |
||||||
|
} |
||||||
|
|
||||||
|
return evs[0], nil |
||||||
|
} |
||||||
|
|
||||||
|
// FetchEventsBySerials retrieves multiple events by their serial numbers
|
||||||
|
func (d *D) FetchEventsBySerials(serials []*types.Uint40) ( |
||||||
|
events map[uint64]*event.E, err error, |
||||||
|
) { |
||||||
|
if len(serials) == 0 { |
||||||
|
return make(map[uint64]*event.E), nil |
||||||
|
} |
||||||
|
|
||||||
|
// Build query for multiple serials
|
||||||
|
serialStrs := make([]string, len(serials)) |
||||||
|
for i, ser := range serials { |
||||||
|
serialStrs[i] = fmt.Sprintf("%d", ser.Get()) |
||||||
|
} |
||||||
|
|
||||||
|
// Use uid() function for efficient multi-get
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
events(func: uid(%s)) { |
||||||
|
event.id |
||||||
|
event.kind |
||||||
|
event.created_at |
||||||
|
event.content |
||||||
|
event.sig |
||||||
|
event.pubkey |
||||||
|
event.tags |
||||||
|
event.serial |
||||||
|
} |
||||||
|
}`, serialStrs[0]) // Simplified - in production you'd handle multiple UIDs properly
|
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to fetch events by serials: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
evs, err := d.parseEventsFromResponse(resp.Json) |
||||||
|
if err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
// Map events by serial
|
||||||
|
events = make(map[uint64]*event.E) |
||||||
|
for i, ser := range serials { |
||||||
|
if i < len(evs) { |
||||||
|
events[ser.Get()] = evs[i] |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return events, nil |
||||||
|
} |
||||||
|
|
||||||
|
// GetSerialById retrieves the serial number for an event ID
|
||||||
|
func (d *D) GetSerialById(id []byte) (ser *types.Uint40, err error) { |
||||||
|
idStr := hex.Enc(id) |
||||||
|
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
event(func: eq(event.id, %q)) { |
||||||
|
event.serial |
||||||
|
} |
||||||
|
}`, idStr) |
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to get serial by ID: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Event []struct { |
||||||
|
Serial int64 `json:"event.serial"` |
||||||
|
} `json:"event"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
if len(result.Event) == 0 { |
||||||
|
return nil, fmt.Errorf("event not found") |
||||||
|
} |
||||||
|
|
||||||
|
ser = &types.Uint40{} |
||||||
|
ser.Set(uint64(result.Event[0].Serial)) |
||||||
|
|
||||||
|
return ser, nil |
||||||
|
} |
||||||
|
|
||||||
|
// GetSerialsByIds retrieves serial numbers for multiple event IDs
|
||||||
|
func (d *D) GetSerialsByIds(ids *tag.T) ( |
||||||
|
serials map[string]*types.Uint40, err error, |
||||||
|
) { |
||||||
|
serials = make(map[string]*types.Uint40) |
||||||
|
|
||||||
|
if len(ids.T) == 0 { |
||||||
|
return serials, nil |
||||||
|
} |
||||||
|
|
||||||
|
// Query each ID individually (simplified implementation)
|
||||||
|
for _, id := range ids.T { |
||||||
|
if len(id) >= 2 { |
||||||
|
idStr := string(id[1]) |
||||||
|
serial, err := d.GetSerialById([]byte(idStr)) |
||||||
|
if err == nil { |
||||||
|
serials[idStr] = serial |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return serials, nil |
||||||
|
} |
||||||
|
|
||||||
|
// GetSerialsByIdsWithFilter retrieves serials with a filter function
|
||||||
|
func (d *D) GetSerialsByIdsWithFilter( |
||||||
|
ids *tag.T, fn func(ev *event.E, ser *types.Uint40) bool, |
||||||
|
) (serials map[string]*types.Uint40, err error) { |
||||||
|
serials = make(map[string]*types.Uint40) |
||||||
|
|
||||||
|
if fn == nil { |
||||||
|
// No filter, just return all
|
||||||
|
return d.GetSerialsByIds(ids) |
||||||
|
} |
||||||
|
|
||||||
|
// With filter, need to fetch events
|
||||||
|
for _, id := range ids.T { |
||||||
|
if len(id) > 0 { |
||||||
|
serial, err := d.GetSerialById(id) |
||||||
|
if err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
|
||||||
|
ev, err := d.FetchEventBySerial(serial) |
||||||
|
if err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
|
||||||
|
if fn(ev, serial) { |
||||||
|
serials[string(id)] = serial |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return serials, nil |
||||||
|
} |
||||||
|
|
||||||
|
// GetSerialsByRange retrieves serials within a range
|
||||||
|
func (d *D) GetSerialsByRange(idx database.Range) ( |
||||||
|
serials types.Uint40s, err error, |
||||||
|
) { |
||||||
|
// This would need to be implemented based on how ranges are defined
|
||||||
|
// For now, returning not implemented
|
||||||
|
err = fmt.Errorf("not implemented") |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// GetFullIdPubkeyBySerial retrieves ID and pubkey for a serial number
|
||||||
|
func (d *D) GetFullIdPubkeyBySerial(ser *types.Uint40) ( |
||||||
|
fidpk *store.IdPkTs, err error, |
||||||
|
) { |
||||||
|
serial := ser.Get() |
||||||
|
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
event(func: eq(event.serial, %d)) { |
||||||
|
event.id |
||||||
|
event.pubkey |
||||||
|
event.created_at |
||||||
|
} |
||||||
|
}`, serial) |
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to get ID and pubkey by serial: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Event []struct { |
||||||
|
ID string `json:"event.id"` |
||||||
|
Pubkey string `json:"event.pubkey"` |
||||||
|
CreatedAt int64 `json:"event.created_at"` |
||||||
|
} `json:"event"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
if len(result.Event) == 0 { |
||||||
|
return nil, fmt.Errorf("event not found") |
||||||
|
} |
||||||
|
|
||||||
|
id, err := hex.Dec(result.Event[0].ID) |
||||||
|
if err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
pubkey, err := hex.Dec(result.Event[0].Pubkey) |
||||||
|
if err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
fidpk = &store.IdPkTs{ |
||||||
|
Id: id, |
||||||
|
Pub: pubkey, |
||||||
|
Ts: result.Event[0].CreatedAt, |
||||||
|
Ser: serial, |
||||||
|
} |
||||||
|
|
||||||
|
return fidpk, nil |
||||||
|
} |
||||||
|
|
||||||
|
// GetFullIdPubkeyBySerials retrieves IDs and pubkeys for multiple serials
|
||||||
|
func (d *D) GetFullIdPubkeyBySerials(sers []*types.Uint40) ( |
||||||
|
fidpks []*store.IdPkTs, err error, |
||||||
|
) { |
||||||
|
fidpks = make([]*store.IdPkTs, 0, len(sers)) |
||||||
|
|
||||||
|
for _, ser := range sers { |
||||||
|
fidpk, err := d.GetFullIdPubkeyBySerial(ser) |
||||||
|
if err != nil { |
||||||
|
continue // Skip errors, continue with others
|
||||||
|
} |
||||||
|
fidpks = append(fidpks, fidpk) |
||||||
|
} |
||||||
|
|
||||||
|
return fidpks, nil |
||||||
|
} |
||||||
@ -0,0 +1,144 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"bufio" |
||||||
|
"bytes" |
||||||
|
"context" |
||||||
|
"net" |
||||||
|
"os" |
||||||
|
"sort" |
||||||
|
"testing" |
||||||
|
"time" |
||||||
|
|
||||||
|
"lol.mleku.dev/chk" |
||||||
|
"next.orly.dev/pkg/encoders/event" |
||||||
|
"next.orly.dev/pkg/encoders/event/examples" |
||||||
|
) |
||||||
|
|
||||||
|
// isDgraphAvailable checks if a dgraph server is running
|
||||||
|
func isDgraphAvailable() bool { |
||||||
|
dgraphURL := os.Getenv("ORLY_DGRAPH_URL") |
||||||
|
if dgraphURL == "" { |
||||||
|
dgraphURL = "localhost:9080" |
||||||
|
} |
||||||
|
|
||||||
|
conn, err := net.DialTimeout("tcp", dgraphURL, 2*time.Second) |
||||||
|
if err != nil { |
||||||
|
return false |
||||||
|
} |
||||||
|
conn.Close() |
||||||
|
return true |
||||||
|
} |
||||||
|
|
||||||
|
// skipIfDgraphNotAvailable skips the test if dgraph is not available
|
||||||
|
func skipIfDgraphNotAvailable(t *testing.T) { |
||||||
|
if !isDgraphAvailable() { |
||||||
|
dgraphURL := os.Getenv("ORLY_DGRAPH_URL") |
||||||
|
if dgraphURL == "" { |
||||||
|
dgraphURL = "localhost:9080" |
||||||
|
} |
||||||
|
t.Skipf("Dgraph server not available at %s. Start with: docker run -p 9080:9080 dgraph/standalone:latest", dgraphURL) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// setupTestDB creates a new test dgraph database and loads example events
|
||||||
|
func setupTestDB(t *testing.T) ( |
||||||
|
*D, []*event.E, context.Context, context.CancelFunc, string, |
||||||
|
) { |
||||||
|
skipIfDgraphNotAvailable(t) |
||||||
|
|
||||||
|
// Create a temporary directory for metadata storage
|
||||||
|
tempDir, err := os.MkdirTemp("", "test-dgraph-*") |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to create temporary directory: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a context and cancel function for the database
|
||||||
|
ctx, cancel := context.WithCancel(context.Background()) |
||||||
|
|
||||||
|
// Initialize the dgraph database
|
||||||
|
db, err := New(ctx, cancel, tempDir, "info") |
||||||
|
if err != nil { |
||||||
|
cancel() |
||||||
|
os.RemoveAll(tempDir) |
||||||
|
t.Fatalf("Failed to create dgraph database: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Drop all data to start fresh
|
||||||
|
if err := db.dropAll(ctx); err != nil { |
||||||
|
db.Close() |
||||||
|
cancel() |
||||||
|
os.RemoveAll(tempDir) |
||||||
|
t.Fatalf("Failed to drop all data: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a scanner to read events from examples.Cache
|
||||||
|
scanner := bufio.NewScanner(bytes.NewBuffer(examples.Cache)) |
||||||
|
scanner.Buffer(make([]byte, 0, 1_000_000_000), 1_000_000_000) |
||||||
|
|
||||||
|
var events []*event.E |
||||||
|
|
||||||
|
// First, collect all events from examples.Cache
|
||||||
|
for scanner.Scan() { |
||||||
|
chk.E(scanner.Err()) |
||||||
|
b := scanner.Bytes() |
||||||
|
ev := event.New() |
||||||
|
|
||||||
|
// Unmarshal the event
|
||||||
|
if _, err = ev.Unmarshal(b); chk.E(err) { |
||||||
|
ev.Free() |
||||||
|
db.Close() |
||||||
|
cancel() |
||||||
|
os.RemoveAll(tempDir) |
||||||
|
t.Fatal(err) |
||||||
|
} |
||||||
|
|
||||||
|
events = append(events, ev) |
||||||
|
} |
||||||
|
|
||||||
|
// Check for scanner errors
|
||||||
|
if err = scanner.Err(); err != nil { |
||||||
|
db.Close() |
||||||
|
cancel() |
||||||
|
os.RemoveAll(tempDir) |
||||||
|
t.Fatalf("Scanner error: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Sort events by CreatedAt to ensure addressable events are processed in chronological order
|
||||||
|
sort.Slice(events, func(i, j int) bool { |
||||||
|
return events[i].CreatedAt < events[j].CreatedAt |
||||||
|
}) |
||||||
|
|
||||||
|
// Count the number of events processed
|
||||||
|
eventCount := 0 |
||||||
|
|
||||||
|
// Now process each event in chronological order
|
||||||
|
for _, ev := range events { |
||||||
|
// Save the event to the database
|
||||||
|
if _, err = db.SaveEvent(ctx, ev); err != nil { |
||||||
|
db.Close() |
||||||
|
cancel() |
||||||
|
os.RemoveAll(tempDir) |
||||||
|
t.Fatalf("Failed to save event #%d: %v", eventCount+1, err) |
||||||
|
} |
||||||
|
|
||||||
|
eventCount++ |
||||||
|
} |
||||||
|
|
||||||
|
t.Logf("Successfully saved %d events to dgraph database", eventCount) |
||||||
|
|
||||||
|
return db, events, ctx, cancel, tempDir |
||||||
|
} |
||||||
|
|
||||||
|
// cleanupTestDB cleans up the test database
|
||||||
|
func cleanupTestDB(t *testing.T, db *D, cancel context.CancelFunc, tempDir string) { |
||||||
|
if db != nil { |
||||||
|
db.Close() |
||||||
|
} |
||||||
|
if cancel != nil { |
||||||
|
cancel() |
||||||
|
} |
||||||
|
if tempDir != "" { |
||||||
|
os.RemoveAll(tempDir) |
||||||
|
} |
||||||
|
} |
||||||
@ -0,0 +1,44 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"fmt" |
||||||
|
|
||||||
|
"next.orly.dev/pkg/crypto/keys" |
||||||
|
) |
||||||
|
|
||||||
|
// Relay identity methods
|
||||||
|
// We use the marker system to store the relay's private key
|
||||||
|
|
||||||
|
const relayIdentityMarkerKey = "relay_identity_secret" |
||||||
|
|
||||||
|
// GetRelayIdentitySecret retrieves the relay's identity secret key
|
||||||
|
func (d *D) GetRelayIdentitySecret() (skb []byte, err error) { |
||||||
|
return d.GetMarker(relayIdentityMarkerKey) |
||||||
|
} |
||||||
|
|
||||||
|
// SetRelayIdentitySecret sets the relay's identity secret key
|
||||||
|
func (d *D) SetRelayIdentitySecret(skb []byte) error { |
||||||
|
return d.SetMarker(relayIdentityMarkerKey, skb) |
||||||
|
} |
||||||
|
|
||||||
|
// GetOrCreateRelayIdentitySecret retrieves or creates the relay identity
|
||||||
|
func (d *D) GetOrCreateRelayIdentitySecret() (skb []byte, err error) { |
||||||
|
skb, err = d.GetRelayIdentitySecret() |
||||||
|
if err == nil { |
||||||
|
return skb, nil |
||||||
|
} |
||||||
|
|
||||||
|
// Generate new identity
|
||||||
|
skb, err = keys.GenerateSecretKey() |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to generate identity: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Store it
|
||||||
|
if err = d.SetRelayIdentitySecret(skb); err != nil { |
||||||
|
return nil, fmt.Errorf("failed to store identity: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
d.Logger.Infof("generated new relay identity") |
||||||
|
return skb, nil |
||||||
|
} |
||||||
@ -0,0 +1,97 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"bufio" |
||||||
|
"context" |
||||||
|
"encoding/json" |
||||||
|
"fmt" |
||||||
|
"io" |
||||||
|
|
||||||
|
"next.orly.dev/pkg/encoders/event" |
||||||
|
) |
||||||
|
|
||||||
|
// Import imports events from a reader (JSONL format)
|
||||||
|
func (d *D) Import(rr io.Reader) { |
||||||
|
d.ImportEventsFromReader(context.Background(), rr) |
||||||
|
} |
||||||
|
|
||||||
|
// Export exports events to a writer (JSONL format)
|
||||||
|
func (d *D) Export(c context.Context, w io.Writer, pubkeys ...[]byte) { |
||||||
|
// Query all events or events for specific pubkeys
|
||||||
|
// Write as JSONL
|
||||||
|
|
||||||
|
// Stub implementation
|
||||||
|
fmt.Fprintf(w, "# Export not yet implemented for dgraph\n") |
||||||
|
} |
||||||
|
|
||||||
|
// ImportEventsFromReader imports events from a reader
|
||||||
|
func (d *D) ImportEventsFromReader(ctx context.Context, rr io.Reader) error { |
||||||
|
scanner := bufio.NewScanner(rr) |
||||||
|
scanner.Buffer(make([]byte, 1024*1024), 10*1024*1024) // 10MB max line size
|
||||||
|
|
||||||
|
count := 0 |
||||||
|
for scanner.Scan() { |
||||||
|
line := scanner.Bytes() |
||||||
|
if len(line) == 0 { |
||||||
|
continue |
||||||
|
} |
||||||
|
|
||||||
|
// Skip comments
|
||||||
|
if line[0] == '#' { |
||||||
|
continue |
||||||
|
} |
||||||
|
|
||||||
|
// Parse event
|
||||||
|
ev := &event.E{} |
||||||
|
if err := json.Unmarshal(line, ev); err != nil { |
||||||
|
d.Logger.Warningf("failed to parse event: %v", err) |
||||||
|
continue |
||||||
|
} |
||||||
|
|
||||||
|
// Save event
|
||||||
|
if _, err := d.SaveEvent(ctx, ev); err != nil { |
||||||
|
d.Logger.Warningf("failed to import event: %v", err) |
||||||
|
continue |
||||||
|
} |
||||||
|
|
||||||
|
count++ |
||||||
|
if count%1000 == 0 { |
||||||
|
d.Logger.Infof("imported %d events", count) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
if err := scanner.Err(); err != nil { |
||||||
|
return fmt.Errorf("scanner error: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
d.Logger.Infof("import complete: %d events", count) |
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// ImportEventsFromStrings imports events from JSON strings
|
||||||
|
func (d *D) ImportEventsFromStrings( |
||||||
|
ctx context.Context, |
||||||
|
eventJSONs []string, |
||||||
|
policyManager interface{ CheckPolicy(action string, ev *event.E, pubkey []byte, remote string) (bool, error) }, |
||||||
|
) error { |
||||||
|
for _, eventJSON := range eventJSONs { |
||||||
|
ev := &event.E{} |
||||||
|
if err := json.Unmarshal([]byte(eventJSON), ev); err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
|
||||||
|
// Check policy if manager is provided
|
||||||
|
if policyManager != nil { |
||||||
|
if allowed, err := policyManager.CheckPolicy("write", ev, ev.Pubkey[:], "import"); err != nil || !allowed { |
||||||
|
continue |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Save event
|
||||||
|
if _, err := d.SaveEvent(ctx, ev); err != nil { |
||||||
|
d.Logger.Warningf("failed to import event: %v", err) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return nil |
||||||
|
} |
||||||
@ -0,0 +1,68 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"fmt" |
||||||
|
"runtime" |
||||||
|
"strings" |
||||||
|
|
||||||
|
"go.uber.org/atomic" |
||||||
|
"lol.mleku.dev" |
||||||
|
"lol.mleku.dev/log" |
||||||
|
) |
||||||
|
|
||||||
|
// NewLogger creates a new dgraph logger.
|
||||||
|
func NewLogger(logLevel int, label string) (l *logger) { |
||||||
|
l = &logger{Label: label} |
||||||
|
l.Level.Store(int32(logLevel)) |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
type logger struct { |
||||||
|
Level atomic.Int32 |
||||||
|
Label string |
||||||
|
} |
||||||
|
|
||||||
|
// SetLogLevel atomically adjusts the log level to the given log level code.
|
||||||
|
func (l *logger) SetLogLevel(level int) { |
||||||
|
l.Level.Store(int32(level)) |
||||||
|
} |
||||||
|
|
||||||
|
// Errorf is a log printer for this level of message.
|
||||||
|
func (l *logger) Errorf(s string, i ...interface{}) { |
||||||
|
if l.Level.Load() >= lol.Error { |
||||||
|
s = l.Label + ": " + s |
||||||
|
txt := fmt.Sprintf(s, i...) |
||||||
|
_, file, line, _ := runtime.Caller(2) |
||||||
|
log.E.F("%s\n%s:%d", strings.TrimSpace(txt), file, line) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Warningf is a log printer for this level of message.
|
||||||
|
func (l *logger) Warningf(s string, i ...interface{}) { |
||||||
|
if l.Level.Load() >= lol.Warn { |
||||||
|
s = l.Label + ": " + s |
||||||
|
txt := fmt.Sprintf(s, i...) |
||||||
|
_, file, line, _ := runtime.Caller(2) |
||||||
|
log.W.F("%s\n%s:%d", strings.TrimSpace(txt), file, line) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Infof is a log printer for this level of message.
|
||||||
|
func (l *logger) Infof(s string, i ...interface{}) { |
||||||
|
if l.Level.Load() >= lol.Info { |
||||||
|
s = l.Label + ": " + s |
||||||
|
txt := fmt.Sprintf(s, i...) |
||||||
|
_, file, line, _ := runtime.Caller(2) |
||||||
|
log.I.F("%s\n%s:%d", strings.TrimSpace(txt), file, line) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Debugf is a log printer for this level of message.
|
||||||
|
func (l *logger) Debugf(s string, i ...interface{}) { |
||||||
|
if l.Level.Load() >= lol.Debug { |
||||||
|
s = l.Label + ": " + s |
||||||
|
txt := fmt.Sprintf(s, i...) |
||||||
|
_, file, line, _ := runtime.Caller(2) |
||||||
|
log.D.F("%s\n%s:%d", strings.TrimSpace(txt), file, line) |
||||||
|
} |
||||||
|
} |
||||||
@ -0,0 +1,120 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"encoding/json" |
||||||
|
"fmt" |
||||||
|
|
||||||
|
"github.com/dgraph-io/dgo/v230/protos/api" |
||||||
|
"next.orly.dev/pkg/encoders/hex" |
||||||
|
) |
||||||
|
|
||||||
|
// Markers provide metadata key-value storage using Dgraph predicates
|
||||||
|
// We store markers as special nodes with type "Marker"
|
||||||
|
|
||||||
|
// SetMarker sets a metadata marker
|
||||||
|
func (d *D) SetMarker(key string, value []byte) error { |
||||||
|
// Create or update a marker node
|
||||||
|
markerID := "marker_" + key |
||||||
|
valueHex := hex.Enc(value) |
||||||
|
|
||||||
|
nquads := fmt.Sprintf(` |
||||||
|
_:%s <dgraph.type> "Marker" . |
||||||
|
_:%s <marker.key> %q . |
||||||
|
_:%s <marker.value> %q . |
||||||
|
`, markerID, markerID, key, markerID, valueHex) |
||||||
|
|
||||||
|
mutation := &api.Mutation{ |
||||||
|
SetNquads: []byte(nquads), |
||||||
|
CommitNow: true, |
||||||
|
} |
||||||
|
|
||||||
|
if _, err := d.Mutate(context.Background(), mutation); err != nil { |
||||||
|
return fmt.Errorf("failed to set marker: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// GetMarker retrieves a metadata marker
|
||||||
|
func (d *D) GetMarker(key string) (value []byte, err error) { |
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
marker(func: eq(marker.key, %q)) { |
||||||
|
marker.value |
||||||
|
} |
||||||
|
}`, key) |
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to get marker: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Marker []struct { |
||||||
|
Value string `json:"marker.value"` |
||||||
|
} `json:"marker"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return nil, fmt.Errorf("failed to parse marker response: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
if len(result.Marker) == 0 { |
||||||
|
return nil, fmt.Errorf("marker not found: %s", key) |
||||||
|
} |
||||||
|
|
||||||
|
// Decode hex value
|
||||||
|
value, err = hex.Dec(result.Marker[0].Value) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to decode marker value: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return value, nil |
||||||
|
} |
||||||
|
|
||||||
|
// HasMarker checks if a marker exists
|
||||||
|
func (d *D) HasMarker(key string) bool { |
||||||
|
_, err := d.GetMarker(key) |
||||||
|
return err == nil |
||||||
|
} |
||||||
|
|
||||||
|
// DeleteMarker removes a metadata marker
|
||||||
|
func (d *D) DeleteMarker(key string) error { |
||||||
|
// Find the marker's UID
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
marker(func: eq(marker.key, %q)) { |
||||||
|
uid |
||||||
|
} |
||||||
|
}`, key) |
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to find marker: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Marker []struct { |
||||||
|
UID string `json:"uid"` |
||||||
|
} `json:"marker"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return fmt.Errorf("failed to parse marker query: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
if len(result.Marker) == 0 { |
||||||
|
return nil // Marker doesn't exist
|
||||||
|
} |
||||||
|
|
||||||
|
// Delete the marker node
|
||||||
|
mutation := &api.Mutation{ |
||||||
|
DelNquads: []byte(fmt.Sprintf("<%s> * * .", result.Marker[0].UID)), |
||||||
|
CommitNow: true, |
||||||
|
} |
||||||
|
|
||||||
|
if _, err = d.Mutate(context.Background(), mutation); err != nil { |
||||||
|
return fmt.Errorf("failed to delete marker: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return nil |
||||||
|
} |
||||||
@ -0,0 +1,211 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"encoding/json" |
||||||
|
"fmt" |
||||||
|
"time" |
||||||
|
|
||||||
|
"next.orly.dev/pkg/database" |
||||||
|
"next.orly.dev/pkg/encoders/hex" |
||||||
|
) |
||||||
|
|
||||||
|
// NIP-43 Invite-based ACL methods
|
||||||
|
// Simplified implementation using marker-based storage
|
||||||
|
|
||||||
|
// AddNIP43Member adds a member using an invite code
|
||||||
|
func (d *D) AddNIP43Member(pubkey []byte, inviteCode string) error { |
||||||
|
key := "nip43_" + hex.Enc(pubkey) |
||||||
|
|
||||||
|
member := database.NIP43Membership{ |
||||||
|
InviteCode: inviteCode, |
||||||
|
AddedAt: time.Now(), |
||||||
|
} |
||||||
|
copy(member.Pubkey[:], pubkey) |
||||||
|
|
||||||
|
data, err := json.Marshal(member) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to marshal membership: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Also add to members list
|
||||||
|
if err := d.addToMembersList(pubkey); err != nil { |
||||||
|
return err |
||||||
|
} |
||||||
|
|
||||||
|
return d.SetMarker(key, data) |
||||||
|
} |
||||||
|
|
||||||
|
// RemoveNIP43Member removes a member
|
||||||
|
func (d *D) RemoveNIP43Member(pubkey []byte) error { |
||||||
|
key := "nip43_" + hex.Enc(pubkey) |
||||||
|
|
||||||
|
// Remove from members list
|
||||||
|
if err := d.removeFromMembersList(pubkey); err != nil { |
||||||
|
return err |
||||||
|
} |
||||||
|
|
||||||
|
return d.DeleteMarker(key) |
||||||
|
} |
||||||
|
|
||||||
|
// IsNIP43Member checks if a pubkey is a member
|
||||||
|
func (d *D) IsNIP43Member(pubkey []byte) (isMember bool, err error) { |
||||||
|
_, err = d.GetNIP43Membership(pubkey) |
||||||
|
return err == nil, nil |
||||||
|
} |
||||||
|
|
||||||
|
// GetNIP43Membership retrieves membership information
|
||||||
|
func (d *D) GetNIP43Membership(pubkey []byte) (*database.NIP43Membership, error) { |
||||||
|
key := "nip43_" + hex.Enc(pubkey) |
||||||
|
|
||||||
|
data, err := d.GetMarker(key) |
||||||
|
if err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
var member database.NIP43Membership |
||||||
|
if err := json.Unmarshal(data, &member); err != nil { |
||||||
|
return nil, fmt.Errorf("failed to unmarshal membership: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return &member, nil |
||||||
|
} |
||||||
|
|
||||||
|
// GetAllNIP43Members retrieves all member pubkeys
|
||||||
|
func (d *D) GetAllNIP43Members() ([][]byte, error) { |
||||||
|
data, err := d.GetMarker("nip43_members_list") |
||||||
|
if err != nil { |
||||||
|
return nil, nil // No members = empty list
|
||||||
|
} |
||||||
|
|
||||||
|
var members []string |
||||||
|
if err := json.Unmarshal(data, &members); err != nil { |
||||||
|
return nil, fmt.Errorf("failed to unmarshal members list: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
result := make([][]byte, 0, len(members)) |
||||||
|
for _, hexPubkey := range members { |
||||||
|
pubkey, err := hex.Dec(hexPubkey) |
||||||
|
if err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
result = append(result, pubkey) |
||||||
|
} |
||||||
|
|
||||||
|
return result, nil |
||||||
|
} |
||||||
|
|
||||||
|
// StoreInviteCode stores an invite code with expiration
|
||||||
|
func (d *D) StoreInviteCode(code string, expiresAt time.Time) error { |
||||||
|
key := "invite_" + code |
||||||
|
|
||||||
|
inviteData := map[string]interface{}{ |
||||||
|
"code": code, |
||||||
|
"expiresAt": expiresAt, |
||||||
|
} |
||||||
|
|
||||||
|
data, err := json.Marshal(inviteData) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to marshal invite: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return d.SetMarker(key, data) |
||||||
|
} |
||||||
|
|
||||||
|
// ValidateInviteCode checks if an invite code is valid
|
||||||
|
func (d *D) ValidateInviteCode(code string) (valid bool, err error) { |
||||||
|
key := "invite_" + code |
||||||
|
|
||||||
|
data, err := d.GetMarker(key) |
||||||
|
if err != nil { |
||||||
|
return false, nil // Code doesn't exist
|
||||||
|
} |
||||||
|
|
||||||
|
var inviteData map[string]interface{} |
||||||
|
if err := json.Unmarshal(data, &inviteData); err != nil { |
||||||
|
return false, fmt.Errorf("failed to unmarshal invite: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Check expiration
|
||||||
|
if expiresStr, ok := inviteData["expiresAt"].(string); ok { |
||||||
|
expiresAt, err := time.Parse(time.RFC3339, expiresStr) |
||||||
|
if err == nil && time.Now().After(expiresAt) { |
||||||
|
return false, nil // Expired
|
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return true, nil |
||||||
|
} |
||||||
|
|
||||||
|
// DeleteInviteCode removes an invite code
|
||||||
|
func (d *D) DeleteInviteCode(code string) error { |
||||||
|
key := "invite_" + code |
||||||
|
return d.DeleteMarker(key) |
||||||
|
} |
||||||
|
|
||||||
|
// PublishNIP43MembershipEvent publishes a membership event
|
||||||
|
func (d *D) PublishNIP43MembershipEvent(kind int, pubkey []byte) error { |
||||||
|
// This would require publishing an actual Nostr event
|
||||||
|
// For now, just log it
|
||||||
|
d.Logger.Infof("would publish NIP-43 event kind %d for %s", kind, hex.Enc(pubkey)) |
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// Helper functions
|
||||||
|
|
||||||
|
func (d *D) addToMembersList(pubkey []byte) error { |
||||||
|
data, err := d.GetMarker("nip43_members_list") |
||||||
|
|
||||||
|
var members []string |
||||||
|
if err == nil { |
||||||
|
if err := json.Unmarshal(data, &members); err != nil { |
||||||
|
return fmt.Errorf("failed to unmarshal members list: %w", err) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
hexPubkey := hex.Enc(pubkey) |
||||||
|
|
||||||
|
// Check if already in list
|
||||||
|
for _, member := range members { |
||||||
|
if member == hexPubkey { |
||||||
|
return nil // Already in list
|
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
members = append(members, hexPubkey) |
||||||
|
|
||||||
|
data, err = json.Marshal(members) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to marshal members list: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return d.SetMarker("nip43_members_list", data) |
||||||
|
} |
||||||
|
|
||||||
|
func (d *D) removeFromMembersList(pubkey []byte) error { |
||||||
|
data, err := d.GetMarker("nip43_members_list") |
||||||
|
if err != nil { |
||||||
|
return nil // List doesn't exist
|
||||||
|
} |
||||||
|
|
||||||
|
var members []string |
||||||
|
if err := json.Unmarshal(data, &members); err != nil { |
||||||
|
return fmt.Errorf("failed to unmarshal members list: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
hexPubkey := hex.Enc(pubkey) |
||||||
|
|
||||||
|
// Remove from list
|
||||||
|
newMembers := make([]string, 0, len(members)) |
||||||
|
for _, member := range members { |
||||||
|
if member != hexPubkey { |
||||||
|
newMembers = append(newMembers, member) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
data, err = json.Marshal(newMembers) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to marshal members list: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return d.SetMarker("nip43_members_list", data) |
||||||
|
} |
||||||
@ -0,0 +1,371 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"encoding/json" |
||||||
|
"fmt" |
||||||
|
"strings" |
||||||
|
|
||||||
|
"next.orly.dev/pkg/database/indexes/types" |
||||||
|
"next.orly.dev/pkg/encoders/event" |
||||||
|
"next.orly.dev/pkg/encoders/filter" |
||||||
|
"next.orly.dev/pkg/encoders/hex" |
||||||
|
"next.orly.dev/pkg/encoders/tag" |
||||||
|
"next.orly.dev/pkg/interfaces/store" |
||||||
|
) |
||||||
|
|
||||||
|
// QueryEvents retrieves events matching the given filter
|
||||||
|
func (d *D) QueryEvents(c context.Context, f *filter.F) (evs event.S, err error) { |
||||||
|
return d.QueryEventsWithOptions(c, f, false, false) |
||||||
|
} |
||||||
|
|
||||||
|
// QueryAllVersions retrieves all versions of events matching the filter
|
||||||
|
func (d *D) QueryAllVersions(c context.Context, f *filter.F) (evs event.S, err error) { |
||||||
|
return d.QueryEventsWithOptions(c, f, false, true) |
||||||
|
} |
||||||
|
|
||||||
|
// QueryEventsWithOptions retrieves events with specific options
|
||||||
|
func (d *D) QueryEventsWithOptions( |
||||||
|
c context.Context, f *filter.F, includeDeleteEvents bool, showAllVersions bool, |
||||||
|
) (evs event.S, err error) { |
||||||
|
// Build DQL query from Nostr filter
|
||||||
|
query := d.buildDQLQuery(f, includeDeleteEvents) |
||||||
|
|
||||||
|
// Execute query
|
||||||
|
resp, err := d.Query(c, query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to execute query: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Parse response
|
||||||
|
evs, err = d.parseEventsFromResponse(resp.Json) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to parse events: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return evs, nil |
||||||
|
} |
||||||
|
|
||||||
|
// buildDQLQuery constructs a DQL query from a Nostr filter
|
||||||
|
func (d *D) buildDQLQuery(f *filter.F, includeDeleteEvents bool) string { |
||||||
|
var conditions []string |
||||||
|
var funcQuery string |
||||||
|
|
||||||
|
// IDs filter
|
||||||
|
if len(f.Ids.T) > 0 { |
||||||
|
idConditions := make([]string, len(f.Ids.T)) |
||||||
|
for i, id := range f.Ids.T { |
||||||
|
// Handle prefix matching
|
||||||
|
if len(id) < 64 { |
||||||
|
// Prefix search
|
||||||
|
idConditions[i] = fmt.Sprintf("regexp(event.id, /^%s/)", hex.Enc(id)) |
||||||
|
} else { |
||||||
|
idConditions[i] = fmt.Sprintf("eq(event.id, %q)", hex.Enc(id)) |
||||||
|
} |
||||||
|
} |
||||||
|
if len(idConditions) == 1 { |
||||||
|
funcQuery = idConditions[0] |
||||||
|
} else { |
||||||
|
conditions = append(conditions, "("+strings.Join(idConditions, " OR ")+")") |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Authors filter
|
||||||
|
if len(f.Authors.T) > 0 { |
||||||
|
authorConditions := make([]string, len(f.Authors.T)) |
||||||
|
for i, author := range f.Authors.T { |
||||||
|
// Handle prefix matching
|
||||||
|
if len(author) < 64 { |
||||||
|
authorConditions[i] = fmt.Sprintf("regexp(event.pubkey, /^%s/)", hex.Enc(author)) |
||||||
|
} else { |
||||||
|
authorConditions[i] = fmt.Sprintf("eq(event.pubkey, %q)", hex.Enc(author)) |
||||||
|
} |
||||||
|
} |
||||||
|
if funcQuery == "" && len(authorConditions) == 1 { |
||||||
|
funcQuery = authorConditions[0] |
||||||
|
} else { |
||||||
|
conditions = append(conditions, "("+strings.Join(authorConditions, " OR ")+")") |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Kinds filter
|
||||||
|
if len(f.Kinds.K) > 0 { |
||||||
|
kindConditions := make([]string, len(f.Kinds.K)) |
||||||
|
for i, kind := range f.Kinds.K { |
||||||
|
kindConditions[i] = fmt.Sprintf("eq(event.kind, %d)", kind) |
||||||
|
} |
||||||
|
conditions = append(conditions, "("+strings.Join(kindConditions, " OR ")+")") |
||||||
|
} |
||||||
|
|
||||||
|
// Time range filters
|
||||||
|
if f.Since != nil { |
||||||
|
conditions = append(conditions, fmt.Sprintf("ge(event.created_at, %d)", f.Since.V)) |
||||||
|
} |
||||||
|
if f.Until != nil { |
||||||
|
conditions = append(conditions, fmt.Sprintf("le(event.created_at, %d)", f.Until.V)) |
||||||
|
} |
||||||
|
|
||||||
|
// Tag filters
|
||||||
|
for _, tagValues := range *f.Tags { |
||||||
|
if len(tagValues.T) > 0 { |
||||||
|
tagConditions := make([]string, len(tagValues.T)) |
||||||
|
for i, tagValue := range tagValues.T { |
||||||
|
// This is a simplified tag query - in production you'd want to use facets
|
||||||
|
tagConditions[i] = fmt.Sprintf("eq(tag.value, %q)", string(tagValue)) |
||||||
|
} |
||||||
|
conditions = append(conditions, "("+strings.Join(tagConditions, " OR ")+")") |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Exclude delete events unless requested
|
||||||
|
if !includeDeleteEvents { |
||||||
|
conditions = append(conditions, "NOT eq(event.kind, 5)") |
||||||
|
} |
||||||
|
|
||||||
|
// Build the final query
|
||||||
|
if funcQuery == "" { |
||||||
|
funcQuery = "has(event.id)" |
||||||
|
} |
||||||
|
|
||||||
|
filterStr := "" |
||||||
|
if len(conditions) > 0 { |
||||||
|
filterStr = " @filter(" + strings.Join(conditions, " AND ") + ")" |
||||||
|
} |
||||||
|
|
||||||
|
// Add ordering and limit
|
||||||
|
orderBy := ", orderdesc: event.created_at" |
||||||
|
limitStr := "" |
||||||
|
if *f.Limit > 0 { |
||||||
|
limitStr = fmt.Sprintf(", first: %d", f.Limit) |
||||||
|
} |
||||||
|
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
events(func: %s%s%s%s) { |
||||||
|
uid |
||||||
|
event.id |
||||||
|
event.kind |
||||||
|
event.created_at |
||||||
|
event.content |
||||||
|
event.sig |
||||||
|
event.pubkey |
||||||
|
event.tags |
||||||
|
} |
||||||
|
}`, funcQuery, filterStr, orderBy, limitStr) |
||||||
|
|
||||||
|
return query |
||||||
|
} |
||||||
|
|
||||||
|
// parseEventsFromResponse converts Dgraph JSON response to Nostr events
|
||||||
|
func (d *D) parseEventsFromResponse(jsonData []byte) ([]*event.E, error) { |
||||||
|
var result struct { |
||||||
|
Events []struct { |
||||||
|
UID string `json:"uid"` |
||||||
|
ID string `json:"event.id"` |
||||||
|
Kind int `json:"event.kind"` |
||||||
|
CreatedAt int64 `json:"event.created_at"` |
||||||
|
Content string `json:"event.content"` |
||||||
|
Sig string `json:"event.sig"` |
||||||
|
Pubkey string `json:"event.pubkey"` |
||||||
|
Tags string `json:"event.tags"` |
||||||
|
} `json:"events"` |
||||||
|
} |
||||||
|
|
||||||
|
if err := json.Unmarshal(jsonData, &result); err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
events := make([]*event.E, 0, len(result.Events)) |
||||||
|
for _, ev := range result.Events { |
||||||
|
// Decode hex strings
|
||||||
|
id, err := hex.Dec(ev.ID) |
||||||
|
if err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
sig, err := hex.Dec(ev.Sig) |
||||||
|
if err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
pubkey, err := hex.Dec(ev.Pubkey) |
||||||
|
if err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
|
||||||
|
// Parse tags from JSON
|
||||||
|
var tags tag.S |
||||||
|
if ev.Tags != "" { |
||||||
|
if err := json.Unmarshal([]byte(ev.Tags), &tags); err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Create event
|
||||||
|
e := &event.E{ |
||||||
|
Kind: uint16(ev.Kind), |
||||||
|
CreatedAt: ev.CreatedAt, |
||||||
|
Content: []byte(ev.Content), |
||||||
|
Tags: &tags, |
||||||
|
} |
||||||
|
|
||||||
|
// Copy fixed-size arrays
|
||||||
|
copy(e.ID[:], id) |
||||||
|
copy(e.Sig[:], sig) |
||||||
|
copy(e.Pubkey[:], pubkey) |
||||||
|
|
||||||
|
events = append(events, e) |
||||||
|
} |
||||||
|
|
||||||
|
return events, nil |
||||||
|
} |
||||||
|
|
||||||
|
// QueryDeleteEventsByTargetId retrieves delete events targeting a specific event ID
|
||||||
|
func (d *D) QueryDeleteEventsByTargetId(c context.Context, targetEventId []byte) ( |
||||||
|
evs event.S, err error, |
||||||
|
) { |
||||||
|
targetIDStr := hex.Enc(targetEventId) |
||||||
|
|
||||||
|
// Query for kind 5 events that reference this event
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
events(func: eq(event.kind, 5)) { |
||||||
|
uid |
||||||
|
event.id |
||||||
|
event.kind |
||||||
|
event.created_at |
||||||
|
event.content |
||||||
|
event.sig |
||||||
|
event.pubkey |
||||||
|
event.tags |
||||||
|
references @filter(eq(event.id, %q)) { |
||||||
|
event.id |
||||||
|
} |
||||||
|
} |
||||||
|
}`, targetIDStr) |
||||||
|
|
||||||
|
resp, err := d.Query(c, query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to query delete events: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
evs, err = d.parseEventsFromResponse(resp.Json) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to parse delete events: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return evs, nil |
||||||
|
} |
||||||
|
|
||||||
|
// QueryForSerials retrieves event serials matching a filter
|
||||||
|
func (d *D) QueryForSerials(c context.Context, f *filter.F) ( |
||||||
|
serials types.Uint40s, err error, |
||||||
|
) { |
||||||
|
// Build query
|
||||||
|
query := d.buildDQLQuery(f, false) |
||||||
|
|
||||||
|
// Modify query to only return serial numbers
|
||||||
|
query = strings.Replace(query, "event.id\n\t\t\tevent.kind", "event.serial", 1) |
||||||
|
query = strings.Replace(query, "\t\t\tevent.created_at\n\t\t\tevent.content\n\t\t\tevent.sig\n\t\t\tevent.pubkey\n\t\t\tevent.tags", "", 1) |
||||||
|
|
||||||
|
resp, err := d.Query(c, query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to query serials: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Events []struct { |
||||||
|
Serial int64 `json:"event.serial"` |
||||||
|
} `json:"events"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
serials = make([]*types.Uint40, 0, len(result.Events)) |
||||||
|
for _, ev := range result.Events { |
||||||
|
serial := types.Uint40{} |
||||||
|
serial.Set(uint64(ev.Serial)) |
||||||
|
serials = append(serials, &serial) |
||||||
|
} |
||||||
|
|
||||||
|
return serials, nil |
||||||
|
} |
||||||
|
|
||||||
|
// QueryForIds retrieves event IDs matching a filter
|
||||||
|
func (d *D) QueryForIds(c context.Context, f *filter.F) ( |
||||||
|
idPkTs []*store.IdPkTs, err error, |
||||||
|
) { |
||||||
|
// Build query
|
||||||
|
query := d.buildDQLQuery(f, false) |
||||||
|
|
||||||
|
// Modify query to only return ID, pubkey, created_at, serial
|
||||||
|
query = strings.Replace(query, "event.kind\n\t\t\tevent.created_at\n\t\t\tevent.content\n\t\t\tevent.sig\n\t\t\tevent.pubkey\n\t\t\tevent.tags", "event.id\n\t\t\tevent.pubkey\n\t\t\tevent.created_at\n\t\t\tevent.serial", 1) |
||||||
|
|
||||||
|
resp, err := d.Query(c, query) |
||||||
|
if err != nil { |
||||||
|
return nil, fmt.Errorf("failed to query IDs: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Events []struct { |
||||||
|
ID string `json:"event.id"` |
||||||
|
Pubkey string `json:"event.pubkey"` |
||||||
|
CreatedAt int64 `json:"event.created_at"` |
||||||
|
Serial int64 `json:"event.serial"` |
||||||
|
} `json:"events"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
idPkTs = make([]*store.IdPkTs, 0, len(result.Events)) |
||||||
|
for _, ev := range result.Events { |
||||||
|
id, err := hex.Dec(ev.ID) |
||||||
|
if err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
pubkey, err := hex.Dec(ev.Pubkey) |
||||||
|
if err != nil { |
||||||
|
continue |
||||||
|
} |
||||||
|
idPkTs = append(idPkTs, &store.IdPkTs{ |
||||||
|
Id: id, |
||||||
|
Pub: pubkey, |
||||||
|
Ts: ev.CreatedAt, |
||||||
|
Ser: uint64(ev.Serial), |
||||||
|
}) |
||||||
|
} |
||||||
|
|
||||||
|
return idPkTs, nil |
||||||
|
} |
||||||
|
|
||||||
|
// CountEvents counts events matching a filter
|
||||||
|
func (d *D) CountEvents(c context.Context, f *filter.F) ( |
||||||
|
count int, approximate bool, err error, |
||||||
|
) { |
||||||
|
// Build query with count
|
||||||
|
query := d.buildDQLQuery(f, false) |
||||||
|
|
||||||
|
// Modify to count instead of returning full data
|
||||||
|
query = strings.Replace(query, "uid\n\t\t\tevent.id\n\t\t\tevent.kind\n\t\t\tevent.created_at\n\t\t\tevent.content\n\t\t\tevent.sig\n\t\t\tevent.pubkey\n\t\t\tevent.tags", "count(uid)", 1) |
||||||
|
|
||||||
|
resp, err := d.Query(c, query) |
||||||
|
if err != nil { |
||||||
|
return 0, false, fmt.Errorf("failed to count events: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Events []struct { |
||||||
|
Count int `json:"count"` |
||||||
|
} `json:"events"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return 0, false, err |
||||||
|
} |
||||||
|
|
||||||
|
if len(result.Events) > 0 { |
||||||
|
count = result.Events[0].Count |
||||||
|
} |
||||||
|
|
||||||
|
return count, false, nil |
||||||
|
} |
||||||
@ -0,0 +1,517 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"fmt" |
||||||
|
"testing" |
||||||
|
|
||||||
|
"lol.mleku.dev/chk" |
||||||
|
"next.orly.dev/pkg/encoders/event" |
||||||
|
"next.orly.dev/pkg/encoders/filter" |
||||||
|
"next.orly.dev/pkg/encoders/hex" |
||||||
|
"next.orly.dev/pkg/encoders/kind" |
||||||
|
"next.orly.dev/pkg/encoders/tag" |
||||||
|
"next.orly.dev/pkg/encoders/timestamp" |
||||||
|
"next.orly.dev/pkg/interfaces/signer/p8k" |
||||||
|
"next.orly.dev/pkg/utils" |
||||||
|
) |
||||||
|
|
||||||
|
func TestQueryEventsByID(t *testing.T) { |
||||||
|
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||||
|
defer cleanupTestDB(t, db, cancel, tempDir) |
||||||
|
|
||||||
|
// Test QueryEvents with an ID filter
|
||||||
|
testEvent := events[3] |
||||||
|
|
||||||
|
evs, err := db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Ids: tag.NewFromBytesSlice(testEvent.ID), |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to query events by ID: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify we got exactly one event
|
||||||
|
if len(evs) != 1 { |
||||||
|
t.Fatalf("Expected 1 event, got %d", len(evs)) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify it's the correct event
|
||||||
|
if !utils.FastEqual(evs[0].ID, testEvent.ID) { |
||||||
|
t.Fatalf( |
||||||
|
"Event ID doesn't match. Got %x, expected %x", evs[0].ID, |
||||||
|
testEvent.ID, |
||||||
|
) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
func TestQueryEventsByKind(t *testing.T) { |
||||||
|
db, _, ctx, cancel, tempDir := setupTestDB(t) |
||||||
|
defer cleanupTestDB(t, db, cancel, tempDir) |
||||||
|
|
||||||
|
// Test querying by kind
|
||||||
|
testKind := kind.New(1) // Kind 1 is typically text notes
|
||||||
|
kindFilter := kind.NewS(testKind) |
||||||
|
|
||||||
|
evs, err := db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Kinds: kindFilter, |
||||||
|
Tags: tag.NewS(), |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to query events by kind: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify we got results
|
||||||
|
if len(evs) == 0 { |
||||||
|
t.Fatal("Expected events with kind 1, but got none") |
||||||
|
} |
||||||
|
|
||||||
|
// Verify all events have the correct kind
|
||||||
|
for i, ev := range evs { |
||||||
|
if ev.Kind != testKind.K { |
||||||
|
t.Fatalf( |
||||||
|
"Event %d has incorrect kind. Got %d, expected %d", i, |
||||||
|
ev.Kind, testKind.K, |
||||||
|
) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
func TestQueryEventsByAuthor(t *testing.T) { |
||||||
|
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||||
|
defer cleanupTestDB(t, db, cancel, tempDir) |
||||||
|
|
||||||
|
// Test querying by author
|
||||||
|
authorFilter := tag.NewFromBytesSlice(events[1].Pubkey) |
||||||
|
|
||||||
|
evs, err := db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Authors: authorFilter, |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to query events by author: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify we got results
|
||||||
|
if len(evs) == 0 { |
||||||
|
t.Fatal("Expected events from author, but got none") |
||||||
|
} |
||||||
|
|
||||||
|
// Verify all events have the correct author
|
||||||
|
for i, ev := range evs { |
||||||
|
if !utils.FastEqual(ev.Pubkey, events[1].Pubkey) { |
||||||
|
t.Fatalf( |
||||||
|
"Event %d has incorrect author. Got %x, expected %x", |
||||||
|
i, ev.Pubkey, events[1].Pubkey, |
||||||
|
) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
func TestReplaceableEventsAndDeletion(t *testing.T) { |
||||||
|
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||||
|
defer cleanupTestDB(t, db, cancel, tempDir) |
||||||
|
|
||||||
|
// Create a signer
|
||||||
|
sign := p8k.MustNew() |
||||||
|
if err := sign.Generate(); chk.E(err) { |
||||||
|
t.Fatal(err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a replaceable event
|
||||||
|
replaceableEvent := event.New() |
||||||
|
replaceableEvent.Kind = kind.ProfileMetadata.K // Kind 0 is replaceable
|
||||||
|
replaceableEvent.Pubkey = events[0].Pubkey // Use the same pubkey as an existing event
|
||||||
|
replaceableEvent.CreatedAt = timestamp.Now().V - 7200 // 2 hours ago
|
||||||
|
replaceableEvent.Content = []byte("Original profile") |
||||||
|
replaceableEvent.Tags = tag.NewS() |
||||||
|
replaceableEvent.Sign(sign) |
||||||
|
|
||||||
|
// Save the replaceable event
|
||||||
|
if _, err := db.SaveEvent(ctx, replaceableEvent); err != nil { |
||||||
|
t.Fatalf("Failed to save replaceable event: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a newer version of the replaceable event
|
||||||
|
newerEvent := event.New() |
||||||
|
newerEvent.Kind = kind.ProfileMetadata.K // Same kind
|
||||||
|
newerEvent.Pubkey = replaceableEvent.Pubkey // Same pubkey
|
||||||
|
newerEvent.CreatedAt = timestamp.Now().V - 3600 // 1 hour ago (newer than the original)
|
||||||
|
newerEvent.Content = []byte("Updated profile") |
||||||
|
newerEvent.Tags = tag.NewS() |
||||||
|
newerEvent.Sign(sign) |
||||||
|
|
||||||
|
// Save the newer event
|
||||||
|
if _, err := db.SaveEvent(ctx, newerEvent); err != nil { |
||||||
|
t.Fatalf("Failed to save newer event: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Query for the original event by ID
|
||||||
|
evs, err := db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Ids: tag.NewFromAny(replaceableEvent.ID), |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to query for replaced event by ID: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify the original event is still found (it's kept but not returned in general queries)
|
||||||
|
if len(evs) != 1 { |
||||||
|
t.Fatalf("Expected 1 event when querying for replaced event by ID, got %d", len(evs)) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify it's the original event
|
||||||
|
if !utils.FastEqual(evs[0].ID, replaceableEvent.ID) { |
||||||
|
t.Fatalf( |
||||||
|
"Event ID doesn't match when querying for replaced event. Got %x, expected %x", |
||||||
|
evs[0].ID, replaceableEvent.ID, |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Query for all events of this kind and pubkey
|
||||||
|
kindFilter := kind.NewS(kind.ProfileMetadata) |
||||||
|
authorFilter := tag.NewFromAny(replaceableEvent.Pubkey) |
||||||
|
|
||||||
|
evs, err = db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Kinds: kindFilter, |
||||||
|
Authors: authorFilter, |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to query for replaceable events: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify we got only one event (the latest one)
|
||||||
|
if len(evs) != 1 { |
||||||
|
t.Fatalf( |
||||||
|
"Expected 1 event when querying for replaceable events, got %d", |
||||||
|
len(evs), |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify it's the newer event
|
||||||
|
if !utils.FastEqual(evs[0].ID, newerEvent.ID) { |
||||||
|
t.Fatalf( |
||||||
|
"Event ID doesn't match when querying for replaceable events. Got %x, expected %x", |
||||||
|
evs[0].ID, newerEvent.ID, |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Test deletion events
|
||||||
|
// Create a deletion event that references the replaceable event
|
||||||
|
deletionEvent := event.New() |
||||||
|
deletionEvent.Kind = kind.Deletion.K // Kind 5 is deletion
|
||||||
|
deletionEvent.Pubkey = replaceableEvent.Pubkey // Same pubkey as the event being deleted
|
||||||
|
deletionEvent.CreatedAt = timestamp.Now().V // Current time
|
||||||
|
deletionEvent.Content = []byte("Deleting the replaceable event") |
||||||
|
deletionEvent.Tags = tag.NewS() |
||||||
|
deletionEvent.Sign(sign) |
||||||
|
|
||||||
|
// Add an e-tag referencing the replaceable event
|
||||||
|
*deletionEvent.Tags = append( |
||||||
|
*deletionEvent.Tags, |
||||||
|
tag.NewFromAny("e", hex.Enc(replaceableEvent.ID)), |
||||||
|
) |
||||||
|
|
||||||
|
// Save the deletion event
|
||||||
|
if _, err = db.SaveEvent(ctx, deletionEvent); err != nil { |
||||||
|
t.Fatalf("Failed to save deletion event: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Query for all events of this kind and pubkey again
|
||||||
|
evs, err = db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Kinds: kindFilter, |
||||||
|
Authors: authorFilter, |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf( |
||||||
|
"Failed to query for replaceable events after deletion: %v", err, |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify we still get the newer event (deletion should only affect the original event)
|
||||||
|
if len(evs) != 1 { |
||||||
|
t.Fatalf( |
||||||
|
"Expected 1 event when querying for replaceable events after deletion, got %d", |
||||||
|
len(evs), |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify it's still the newer event
|
||||||
|
if !utils.FastEqual(evs[0].ID, newerEvent.ID) { |
||||||
|
t.Fatalf( |
||||||
|
"Event ID doesn't match after deletion. Got %x, expected %x", |
||||||
|
evs[0].ID, newerEvent.ID, |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Query for the original event by ID
|
||||||
|
evs, err = db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Ids: tag.NewFromBytesSlice(replaceableEvent.ID), |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to query for deleted event by ID: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify the original event is not found (it was deleted)
|
||||||
|
if len(evs) != 0 { |
||||||
|
t.Fatalf("Expected 0 events when querying for deleted event by ID, got %d", len(evs)) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
func TestParameterizedReplaceableEventsAndDeletion(t *testing.T) { |
||||||
|
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||||
|
defer cleanupTestDB(t, db, cancel, tempDir) |
||||||
|
|
||||||
|
sign := p8k.MustNew() |
||||||
|
if err := sign.Generate(); chk.E(err) { |
||||||
|
t.Fatal(err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a parameterized replaceable event
|
||||||
|
paramEvent := event.New() |
||||||
|
paramEvent.Kind = 30000 // Kind 30000+ is parameterized replaceable
|
||||||
|
paramEvent.Pubkey = events[0].Pubkey // Use the same pubkey as an existing event
|
||||||
|
paramEvent.CreatedAt = timestamp.Now().V - 7200 // 2 hours ago
|
||||||
|
paramEvent.Content = []byte("Original parameterized event") |
||||||
|
paramEvent.Tags = tag.NewS() |
||||||
|
// Add a d-tag
|
||||||
|
*paramEvent.Tags = append( |
||||||
|
*paramEvent.Tags, tag.NewFromAny([]byte{'d'}, []byte("test-d-tag")), |
||||||
|
) |
||||||
|
paramEvent.Sign(sign) |
||||||
|
|
||||||
|
// Save the parameterized replaceable event
|
||||||
|
if _, err := db.SaveEvent(ctx, paramEvent); err != nil { |
||||||
|
t.Fatalf("Failed to save parameterized replaceable event: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a deletion event using e-tag
|
||||||
|
paramDeletionEvent := event.New() |
||||||
|
paramDeletionEvent.Kind = kind.Deletion.K // Kind 5 is deletion
|
||||||
|
paramDeletionEvent.Pubkey = paramEvent.Pubkey // Same pubkey as the event being deleted
|
||||||
|
paramDeletionEvent.CreatedAt = timestamp.Now().V // Current time
|
||||||
|
paramDeletionEvent.Content = []byte("Deleting the parameterized replaceable event with e-tag") |
||||||
|
paramDeletionEvent.Tags = tag.NewS() |
||||||
|
// Add an e-tag referencing the parameterized replaceable event
|
||||||
|
*paramDeletionEvent.Tags = append( |
||||||
|
*paramDeletionEvent.Tags, |
||||||
|
tag.NewFromAny("e", []byte(hex.Enc(paramEvent.ID))), |
||||||
|
) |
||||||
|
paramDeletionEvent.Sign(sign) |
||||||
|
|
||||||
|
// Save the parameterized deletion event with e-tag
|
||||||
|
if _, err := db.SaveEvent(ctx, paramDeletionEvent); err != nil { |
||||||
|
t.Fatalf( |
||||||
|
"Failed to save parameterized deletion event with e-tag: %v", err, |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Query for parameterized events
|
||||||
|
paramKindFilter := kind.NewS(kind.New(paramEvent.Kind)) |
||||||
|
paramAuthorFilter := tag.NewFromBytesSlice(paramEvent.Pubkey) |
||||||
|
|
||||||
|
evs, err := db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Kinds: paramKindFilter, |
||||||
|
Authors: paramAuthorFilter, |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf( |
||||||
|
"Failed to query for parameterized replaceable events after deletion: %v", |
||||||
|
err, |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Debug output
|
||||||
|
fmt.Printf("Got %d events after deletion\n", len(evs)) |
||||||
|
for i, ev := range evs { |
||||||
|
fmt.Printf( |
||||||
|
"Event %d: kind=%d, pubkey=%s\n", |
||||||
|
i, ev.Kind, hex.Enc(ev.Pubkey), |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify we get no events (since the only one was deleted)
|
||||||
|
if len(evs) != 0 { |
||||||
|
t.Fatalf( |
||||||
|
"Expected 0 events when querying for deleted parameterized replaceable events, got %d", |
||||||
|
len(evs), |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Query for the parameterized event by ID
|
||||||
|
evs, err = db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Ids: tag.NewFromBytesSlice(paramEvent.ID), |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf( |
||||||
|
"Failed to query for deleted parameterized event by ID: %v", err, |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify the deleted event is not found when querying by ID
|
||||||
|
if len(evs) != 0 { |
||||||
|
t.Fatalf( |
||||||
|
"Expected 0 events when querying for deleted parameterized event by ID, got %d", |
||||||
|
len(evs), |
||||||
|
) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
func TestQueryEventsByTimeRange(t *testing.T) { |
||||||
|
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||||
|
defer cleanupTestDB(t, db, cancel, tempDir) |
||||||
|
|
||||||
|
// Test querying by time range
|
||||||
|
// Use the timestamp from the middle event as a reference
|
||||||
|
middleIndex := len(events) / 2 |
||||||
|
middleEvent := events[middleIndex] |
||||||
|
|
||||||
|
// Create a timestamp range that includes events before and after the middle event
|
||||||
|
sinceTime := new(timestamp.T) |
||||||
|
sinceTime.V = middleEvent.CreatedAt - 3600 // 1 hour before middle event
|
||||||
|
|
||||||
|
untilTime := new(timestamp.T) |
||||||
|
untilTime.V = middleEvent.CreatedAt + 3600 // 1 hour after middle event
|
||||||
|
|
||||||
|
evs, err := db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Since: sinceTime, |
||||||
|
Until: untilTime, |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to query events by time range: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify we got results
|
||||||
|
if len(evs) == 0 { |
||||||
|
t.Fatal("Expected events in time range, but got none") |
||||||
|
} |
||||||
|
|
||||||
|
// Verify all events are within the time range
|
||||||
|
for i, ev := range evs { |
||||||
|
if ev.CreatedAt < sinceTime.V || ev.CreatedAt > untilTime.V { |
||||||
|
t.Fatalf( |
||||||
|
"Event %d is outside the time range. Got %d, expected between %d and %d", |
||||||
|
i, ev.CreatedAt, sinceTime.V, untilTime.V, |
||||||
|
) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
func TestQueryEventsByTag(t *testing.T) { |
||||||
|
db, events, ctx, cancel, tempDir := setupTestDB(t) |
||||||
|
defer cleanupTestDB(t, db, cancel, tempDir) |
||||||
|
|
||||||
|
// Find an event with tags to use for testing
|
||||||
|
var testTagEvent *event.E |
||||||
|
for _, ev := range events { |
||||||
|
if ev.Tags != nil && ev.Tags.Len() > 0 { |
||||||
|
// Find a tag with at least 2 elements and first element of length 1
|
||||||
|
for _, tag := range *ev.Tags { |
||||||
|
if tag.Len() >= 2 && len(tag.Key()) == 1 { |
||||||
|
testTagEvent = ev |
||||||
|
break |
||||||
|
} |
||||||
|
} |
||||||
|
if testTagEvent != nil { |
||||||
|
break |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
if testTagEvent == nil { |
||||||
|
t.Skip("No suitable event with tags found for testing") |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// Get the first tag with at least 2 elements and first element of length 1
|
||||||
|
var testTag *tag.T |
||||||
|
for _, tag := range *testTagEvent.Tags { |
||||||
|
if tag.Len() >= 2 && len(tag.Key()) == 1 { |
||||||
|
testTag = tag |
||||||
|
break |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Create a tags filter with the test tag
|
||||||
|
tagsFilter := tag.NewS(testTag) |
||||||
|
|
||||||
|
evs, err := db.QueryEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Tags: tagsFilter, |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to query events by tag: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify we got results
|
||||||
|
if len(evs) == 0 { |
||||||
|
t.Fatal("Expected events with tag, but got none") |
||||||
|
} |
||||||
|
|
||||||
|
// Verify all events have the tag
|
||||||
|
for i, ev := range evs { |
||||||
|
var hasTag bool |
||||||
|
for _, tag := range *ev.Tags { |
||||||
|
if tag.Len() >= 2 && len(tag.Key()) == 1 { |
||||||
|
if utils.FastEqual(tag.Key(), testTag.Key()) && |
||||||
|
utils.FastEqual(tag.Value(), testTag.Value()) { |
||||||
|
hasTag = true |
||||||
|
break |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
if !hasTag { |
||||||
|
t.Fatalf("Event %d does not have the expected tag", i) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
func TestCountEvents(t *testing.T) { |
||||||
|
db, _, ctx, cancel, tempDir := setupTestDB(t) |
||||||
|
defer cleanupTestDB(t, db, cancel, tempDir) |
||||||
|
|
||||||
|
// Test counting all events
|
||||||
|
count, _, err := db.CountEvents(ctx, &filter.F{}) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to count events: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Verify we got a non-zero count
|
||||||
|
if count == 0 { |
||||||
|
t.Fatal("Expected non-zero event count, but got 0") |
||||||
|
} |
||||||
|
|
||||||
|
t.Logf("Total events in database: %d", count) |
||||||
|
|
||||||
|
// Test counting events by kind
|
||||||
|
testKind := kind.New(1) |
||||||
|
kindFilter := kind.NewS(testKind) |
||||||
|
|
||||||
|
count, _, err = db.CountEvents( |
||||||
|
ctx, &filter.F{ |
||||||
|
Kinds: kindFilter, |
||||||
|
}, |
||||||
|
) |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to count events by kind: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
t.Logf("Events with kind 1: %d", count) |
||||||
|
} |
||||||
@ -0,0 +1,185 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"encoding/json" |
||||||
|
"fmt" |
||||||
|
"strings" |
||||||
|
|
||||||
|
"github.com/dgraph-io/dgo/v230/protos/api" |
||||||
|
"next.orly.dev/pkg/database/indexes/types" |
||||||
|
"next.orly.dev/pkg/encoders/event" |
||||||
|
"next.orly.dev/pkg/encoders/filter" |
||||||
|
"next.orly.dev/pkg/encoders/hex" |
||||||
|
) |
||||||
|
|
||||||
|
// SaveEvent stores a Nostr event in the Dgraph database.
|
||||||
|
// It creates event nodes and relationships for authors, tags, and references.
|
||||||
|
func (d *D) SaveEvent(c context.Context, ev *event.E) (exists bool, err error) { |
||||||
|
eventID := hex.Enc(ev.ID[:]) |
||||||
|
|
||||||
|
// Check if event already exists
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
event(func: eq(event.id, %q)) { |
||||||
|
uid |
||||||
|
event.id |
||||||
|
} |
||||||
|
}`, eventID) |
||||||
|
|
||||||
|
resp, err := d.Query(c, query) |
||||||
|
if err != nil { |
||||||
|
return false, fmt.Errorf("failed to check event existence: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Parse response to check if event exists
|
||||||
|
var result struct { |
||||||
|
Event []map[string]interface{} `json:"event"` |
||||||
|
} |
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return false, fmt.Errorf("failed to parse query response: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
if len(result.Event) > 0 { |
||||||
|
return true, nil // Event already exists
|
||||||
|
} |
||||||
|
|
||||||
|
// Get next serial number
|
||||||
|
serial, err := d.getNextSerial() |
||||||
|
if err != nil { |
||||||
|
return false, fmt.Errorf("failed to get serial number: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Build N-Quads for the event with serial number
|
||||||
|
nquads := d.buildEventNQuads(ev, serial) |
||||||
|
|
||||||
|
// Store the event
|
||||||
|
mutation := &api.Mutation{ |
||||||
|
SetNquads: []byte(nquads), |
||||||
|
CommitNow: true, |
||||||
|
} |
||||||
|
|
||||||
|
if _, err = d.Mutate(c, mutation); err != nil { |
||||||
|
return false, fmt.Errorf("failed to save event: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return false, nil |
||||||
|
} |
||||||
|
|
||||||
|
// buildEventNQuads constructs RDF triples for a Nostr event
|
||||||
|
func (d *D) buildEventNQuads(ev *event.E, serial uint64) string { |
||||||
|
var nquads strings.Builder |
||||||
|
|
||||||
|
eventID := hex.Enc(ev.ID[:]) |
||||||
|
authorPubkey := hex.Enc(ev.Pubkey) |
||||||
|
|
||||||
|
// Event node
|
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Event\" .\n", eventID)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <event.id> %q .\n", eventID, eventID)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <event.serial> \"%d\"^^<xs:int> .\n", eventID, serial)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <event.kind> \"%d\"^^<xs:int> .\n", eventID, ev.Kind)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <event.created_at> \"%d\"^^<xs:int> .\n", eventID, int64(ev.CreatedAt))) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <event.content> %q .\n", eventID, ev.Content)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <event.sig> %q .\n", eventID, hex.Enc(ev.Sig[:]))) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <event.pubkey> %q .\n", eventID, authorPubkey)) |
||||||
|
|
||||||
|
// Serialize tags as JSON string for storage
|
||||||
|
tagsJSON, _ := json.Marshal(ev.Tags) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <event.tags> %q .\n", eventID, string(tagsJSON))) |
||||||
|
|
||||||
|
// Author relationship
|
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <authored_by> _:%s .\n", eventID, authorPubkey)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Author\" .\n", authorPubkey)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <author.pubkey> %q .\n", authorPubkey, authorPubkey)) |
||||||
|
|
||||||
|
// Tag relationships
|
||||||
|
for _, tag := range *ev.Tags { |
||||||
|
if len(tag.T) >= 2 { |
||||||
|
tagType := string(tag.T[0]) |
||||||
|
tagValue := string(tag.T[1]) |
||||||
|
|
||||||
|
switch tagType { |
||||||
|
case "e": // Event reference
|
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <references> _:%s .\n", eventID, tagValue)) |
||||||
|
case "p": // Pubkey mention
|
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <mentions> _:%s .\n", eventID, tagValue)) |
||||||
|
// Ensure mentioned author exists
|
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Author\" .\n", tagValue)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <author.pubkey> %q .\n", tagValue, tagValue)) |
||||||
|
case "t": // Hashtag
|
||||||
|
tagID := "tag_" + tagType + "_" + tagValue |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <tagged_with> _:%s .\n", eventID, tagID)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Tag\" .\n", tagID)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <tag.type> %q .\n", tagID, tagType)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <tag.value> %q .\n", tagID, tagValue)) |
||||||
|
default: |
||||||
|
// Store other tag types
|
||||||
|
tagID := "tag_" + tagType + "_" + tagValue |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <tagged_with> _:%s .\n", eventID, tagID)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <dgraph.type> \"Tag\" .\n", tagID)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <tag.type> %q .\n", tagID, tagType)) |
||||||
|
nquads.WriteString(fmt.Sprintf("_:%s <tag.value> %q .\n", tagID, tagValue)) |
||||||
|
} |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return nquads.String() |
||||||
|
} |
||||||
|
|
||||||
|
// GetSerialsFromFilter returns event serials matching a filter
|
||||||
|
func (d *D) GetSerialsFromFilter(f *filter.F) (serials types.Uint40s, err error) { |
||||||
|
// For dgraph, we'll use the event.serial field
|
||||||
|
// This is a stub implementation
|
||||||
|
err = fmt.Errorf("not implemented") |
||||||
|
return |
||||||
|
} |
||||||
|
|
||||||
|
// WouldReplaceEvent checks if an event would replace existing events
|
||||||
|
func (d *D) WouldReplaceEvent(ev *event.E) (bool, types.Uint40s, error) { |
||||||
|
// Check for replaceable events (kinds 0, 3, and 10000-19999)
|
||||||
|
isReplaceable := ev.Kind == 0 || ev.Kind == 3 || (ev.Kind >= 10000 && ev.Kind < 20000) |
||||||
|
if !isReplaceable { |
||||||
|
return false, nil, nil |
||||||
|
} |
||||||
|
|
||||||
|
// Query for existing events with same kind and pubkey
|
||||||
|
authorPubkey := hex.Enc(ev.Pubkey) |
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
events(func: eq(event.pubkey, %q)) @filter(eq(event.kind, %d)) { |
||||||
|
uid |
||||||
|
event.serial |
||||||
|
event.created_at |
||||||
|
} |
||||||
|
}`, authorPubkey, ev.Kind) |
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return false, nil, fmt.Errorf("failed to query replaceable events: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Events []struct { |
||||||
|
UID string `json:"uid"` |
||||||
|
Serial int64 `json:"event.serial"` |
||||||
|
CreatedAt int64 `json:"event.created_at"` |
||||||
|
} `json:"events"` |
||||||
|
} |
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return false, nil, fmt.Errorf("failed to parse query response: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Check if our event is newer
|
||||||
|
evTime := int64(ev.CreatedAt) |
||||||
|
var serials types.Uint40s |
||||||
|
wouldReplace := false |
||||||
|
|
||||||
|
for _, existing := range result.Events { |
||||||
|
if existing.CreatedAt < evTime { |
||||||
|
wouldReplace = true |
||||||
|
serial := types.Uint40{} |
||||||
|
serial.Set(uint64(existing.Serial)) |
||||||
|
serials = append(serials, &serial) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return wouldReplace, serials, nil |
||||||
|
} |
||||||
@ -0,0 +1,253 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"bufio" |
||||||
|
"bytes" |
||||||
|
"context" |
||||||
|
"os" |
||||||
|
"sort" |
||||||
|
"testing" |
||||||
|
"time" |
||||||
|
|
||||||
|
"lol.mleku.dev/chk" |
||||||
|
"lol.mleku.dev/errorf" |
||||||
|
"next.orly.dev/pkg/encoders/event" |
||||||
|
"next.orly.dev/pkg/encoders/event/examples" |
||||||
|
"next.orly.dev/pkg/encoders/hex" |
||||||
|
"next.orly.dev/pkg/encoders/kind" |
||||||
|
"next.orly.dev/pkg/encoders/tag" |
||||||
|
"next.orly.dev/pkg/encoders/timestamp" |
||||||
|
"next.orly.dev/pkg/interfaces/signer/p8k" |
||||||
|
) |
||||||
|
|
||||||
|
// TestSaveEvents tests saving all events from examples.Cache to the dgraph database
|
||||||
|
// to verify there are no errors during the saving process.
|
||||||
|
func TestSaveEvents(t *testing.T) { |
||||||
|
skipIfDgraphNotAvailable(t) |
||||||
|
|
||||||
|
// Create a temporary directory for metadata
|
||||||
|
tempDir, err := os.MkdirTemp("", "test-dgraph-*") |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to create temporary directory: %v", err) |
||||||
|
} |
||||||
|
defer os.RemoveAll(tempDir) |
||||||
|
|
||||||
|
// Create a context and cancel function for the database
|
||||||
|
ctx, cancel := context.WithCancel(context.Background()) |
||||||
|
defer cancel() |
||||||
|
|
||||||
|
// Initialize the dgraph database
|
||||||
|
db, err := New(ctx, cancel, tempDir, "info") |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to create dgraph database: %v", err) |
||||||
|
} |
||||||
|
defer db.Close() |
||||||
|
|
||||||
|
// Drop all data to start fresh
|
||||||
|
if err := db.dropAll(ctx); err != nil { |
||||||
|
t.Fatalf("Failed to drop all data: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a scanner to read events from examples.Cache
|
||||||
|
scanner := bufio.NewScanner(bytes.NewBuffer(examples.Cache)) |
||||||
|
scanner.Buffer(make([]byte, 0, 1_000_000_000), 1_000_000_000) |
||||||
|
|
||||||
|
// Collect all events first
|
||||||
|
var events []*event.E |
||||||
|
var original int |
||||||
|
for scanner.Scan() { |
||||||
|
chk.E(scanner.Err()) |
||||||
|
b := scanner.Bytes() |
||||||
|
original += len(b) |
||||||
|
ev := event.New() |
||||||
|
|
||||||
|
// Unmarshal the event
|
||||||
|
if _, err = ev.Unmarshal(b); chk.E(err) { |
||||||
|
t.Fatal(err) |
||||||
|
} |
||||||
|
|
||||||
|
events = append(events, ev) |
||||||
|
} |
||||||
|
|
||||||
|
// Sort events by timestamp to ensure addressable events are processed in order
|
||||||
|
sort.Slice(events, func(i, j int) bool { |
||||||
|
return events[i].CreatedAt < events[j].CreatedAt |
||||||
|
}) |
||||||
|
|
||||||
|
// Count the number of events processed
|
||||||
|
eventCount := 0 |
||||||
|
now := time.Now() |
||||||
|
|
||||||
|
// Process each event in chronological order
|
||||||
|
for _, ev := range events { |
||||||
|
// Save the event to the database
|
||||||
|
if _, err = db.SaveEvent(ctx, ev); err != nil { |
||||||
|
t.Fatalf("Failed to save event #%d: %v", eventCount+1, err) |
||||||
|
} |
||||||
|
eventCount++ |
||||||
|
} |
||||||
|
|
||||||
|
// Check for scanner errors
|
||||||
|
if err = scanner.Err(); err != nil { |
||||||
|
t.Fatalf("Scanner error: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
dur := time.Since(now) |
||||||
|
t.Logf( |
||||||
|
"Successfully saved %d events (%d bytes) to dgraph in %v (%v/ev; %.2f ev/s)", |
||||||
|
eventCount, |
||||||
|
original, |
||||||
|
dur, |
||||||
|
dur/time.Duration(eventCount), |
||||||
|
float64(time.Second)/float64(dur/time.Duration(eventCount)), |
||||||
|
) |
||||||
|
} |
||||||
|
|
||||||
|
// TestDeletionEventWithETagRejection tests that a deletion event with an "e" tag is rejected.
|
||||||
|
func TestDeletionEventWithETagRejection(t *testing.T) { |
||||||
|
skipIfDgraphNotAvailable(t) |
||||||
|
|
||||||
|
// Create a temporary directory for metadata
|
||||||
|
tempDir, err := os.MkdirTemp("", "test-dgraph-*") |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to create temporary directory: %v", err) |
||||||
|
} |
||||||
|
defer os.RemoveAll(tempDir) |
||||||
|
|
||||||
|
// Create a context and cancel function for the database
|
||||||
|
ctx, cancel := context.WithCancel(context.Background()) |
||||||
|
defer cancel() |
||||||
|
|
||||||
|
// Initialize the dgraph database
|
||||||
|
db, err := New(ctx, cancel, tempDir, "info") |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to create dgraph database: %v", err) |
||||||
|
} |
||||||
|
defer db.Close() |
||||||
|
|
||||||
|
// Drop all data to start fresh
|
||||||
|
if err := db.dropAll(ctx); err != nil { |
||||||
|
t.Fatalf("Failed to drop all data: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a signer
|
||||||
|
sign := p8k.MustNew() |
||||||
|
if err := sign.Generate(); chk.E(err) { |
||||||
|
t.Fatal(err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a regular event
|
||||||
|
regularEvent := event.New() |
||||||
|
regularEvent.Kind = kind.TextNote.K |
||||||
|
regularEvent.Pubkey = sign.Pub() |
||||||
|
regularEvent.CreatedAt = timestamp.Now().V - 3600 // 1 hour ago
|
||||||
|
regularEvent.Content = []byte("Regular event") |
||||||
|
regularEvent.Tags = tag.NewS() |
||||||
|
regularEvent.Sign(sign) |
||||||
|
|
||||||
|
// Save the regular event
|
||||||
|
if _, err := db.SaveEvent(ctx, regularEvent); err != nil { |
||||||
|
t.Fatalf("Failed to save regular event: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a deletion event with an "e" tag referencing the regular event
|
||||||
|
deletionEvent := event.New() |
||||||
|
deletionEvent.Kind = kind.Deletion.K |
||||||
|
deletionEvent.Pubkey = sign.Pub() |
||||||
|
deletionEvent.CreatedAt = timestamp.Now().V // Current time
|
||||||
|
deletionEvent.Content = []byte("Deleting the regular event") |
||||||
|
deletionEvent.Tags = tag.NewS() |
||||||
|
|
||||||
|
// Add an e-tag referencing the regular event
|
||||||
|
*deletionEvent.Tags = append( |
||||||
|
*deletionEvent.Tags, |
||||||
|
tag.NewFromAny("e", hex.Enc(regularEvent.ID)), |
||||||
|
) |
||||||
|
|
||||||
|
deletionEvent.Sign(sign) |
||||||
|
|
||||||
|
// Check if this is a deletion event with "e" tags
|
||||||
|
if deletionEvent.Kind == kind.Deletion.K && deletionEvent.Tags.GetFirst([]byte{'e'}) != nil { |
||||||
|
// In this test, we want to reject deletion events with "e" tags
|
||||||
|
err = errorf.E("deletion events referencing other events with 'e' tag are not allowed") |
||||||
|
} else { |
||||||
|
// Try to save the deletion event
|
||||||
|
_, err = db.SaveEvent(ctx, deletionEvent) |
||||||
|
} |
||||||
|
|
||||||
|
if err == nil { |
||||||
|
t.Fatal("Expected deletion event with e-tag to be rejected, but it was accepted") |
||||||
|
} |
||||||
|
|
||||||
|
// Verify the error message
|
||||||
|
expectedError := "deletion events referencing other events with 'e' tag are not allowed" |
||||||
|
if err.Error() != expectedError { |
||||||
|
t.Fatalf( |
||||||
|
"Expected error message '%s', got '%s'", expectedError, err.Error(), |
||||||
|
) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// TestSaveExistingEvent tests that attempting to save an event that already exists
|
||||||
|
// returns an error.
|
||||||
|
func TestSaveExistingEvent(t *testing.T) { |
||||||
|
skipIfDgraphNotAvailable(t) |
||||||
|
|
||||||
|
// Create a temporary directory for metadata
|
||||||
|
tempDir, err := os.MkdirTemp("", "test-dgraph-*") |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to create temporary directory: %v", err) |
||||||
|
} |
||||||
|
defer os.RemoveAll(tempDir) |
||||||
|
|
||||||
|
// Create a context and cancel function for the database
|
||||||
|
ctx, cancel := context.WithCancel(context.Background()) |
||||||
|
defer cancel() |
||||||
|
|
||||||
|
// Initialize the dgraph database
|
||||||
|
db, err := New(ctx, cancel, tempDir, "info") |
||||||
|
if err != nil { |
||||||
|
t.Fatalf("Failed to create dgraph database: %v", err) |
||||||
|
} |
||||||
|
defer db.Close() |
||||||
|
|
||||||
|
// Drop all data to start fresh
|
||||||
|
if err := db.dropAll(ctx); err != nil { |
||||||
|
t.Fatalf("Failed to drop all data: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create a signer
|
||||||
|
sign := p8k.MustNew() |
||||||
|
if err := sign.Generate(); chk.E(err) { |
||||||
|
t.Fatal(err) |
||||||
|
} |
||||||
|
|
||||||
|
// Create an event
|
||||||
|
ev := event.New() |
||||||
|
ev.Kind = kind.TextNote.K |
||||||
|
ev.Pubkey = sign.Pub() |
||||||
|
ev.CreatedAt = timestamp.Now().V |
||||||
|
ev.Content = []byte("Test event") |
||||||
|
ev.Tags = tag.NewS() |
||||||
|
ev.Sign(sign) |
||||||
|
|
||||||
|
// Save the event for the first time
|
||||||
|
if _, err := db.SaveEvent(ctx, ev); err != nil { |
||||||
|
t.Fatalf("Failed to save event: %v", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Try to save the same event again, it should be rejected
|
||||||
|
_, err = db.SaveEvent(ctx, ev) |
||||||
|
if err == nil { |
||||||
|
t.Fatal("Expected error when saving an existing event, but got nil") |
||||||
|
} |
||||||
|
|
||||||
|
// Verify the error message contains indication of duplicate
|
||||||
|
expectedErrorPrefix := "blocked: event already exists" |
||||||
|
if !bytes.Contains([]byte(err.Error()), []byte(expectedErrorPrefix)) { |
||||||
|
t.Fatalf( |
||||||
|
"Expected error message to contain '%s', got '%s'", |
||||||
|
expectedErrorPrefix, err.Error(), |
||||||
|
) |
||||||
|
} |
||||||
|
} |
||||||
@ -0,0 +1,105 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"fmt" |
||||||
|
|
||||||
|
"github.com/dgraph-io/dgo/v230/protos/api" |
||||||
|
) |
||||||
|
|
||||||
|
// NostrSchema defines the Dgraph schema for Nostr events
|
||||||
|
const NostrSchema = ` |
||||||
|
# Event node type |
||||||
|
type Event { |
||||||
|
event.id |
||||||
|
event.serial |
||||||
|
event.kind |
||||||
|
event.created_at |
||||||
|
event.content |
||||||
|
event.sig |
||||||
|
event.pubkey |
||||||
|
event.authored_by |
||||||
|
event.references |
||||||
|
event.mentions |
||||||
|
event.tagged_with |
||||||
|
} |
||||||
|
|
||||||
|
# Author node type |
||||||
|
type Author { |
||||||
|
author.pubkey |
||||||
|
author.events |
||||||
|
} |
||||||
|
|
||||||
|
# Tag node type |
||||||
|
type Tag { |
||||||
|
tag.type |
||||||
|
tag.value |
||||||
|
tag.events |
||||||
|
} |
||||||
|
|
||||||
|
# Marker node type (for key-value metadata) |
||||||
|
type Marker { |
||||||
|
marker.key |
||||||
|
marker.value |
||||||
|
} |
||||||
|
|
||||||
|
# Event fields |
||||||
|
event.id: string @index(exact) @upsert . |
||||||
|
event.serial: int @index(int) . |
||||||
|
event.kind: int @index(int) . |
||||||
|
event.created_at: int @index(int) . |
||||||
|
event.content: string . |
||||||
|
event.sig: string @index(exact) . |
||||||
|
event.pubkey: string @index(exact) . |
||||||
|
|
||||||
|
# Event relationships |
||||||
|
event.authored_by: uid @reverse . |
||||||
|
event.references: [uid] @reverse . |
||||||
|
event.mentions: [uid] @reverse . |
||||||
|
event.tagged_with: [uid] @reverse . |
||||||
|
|
||||||
|
# Author fields |
||||||
|
author.pubkey: string @index(exact) @upsert . |
||||||
|
author.events: [uid] @count @reverse . |
||||||
|
|
||||||
|
# Tag fields |
||||||
|
tag.type: string @index(exact) . |
||||||
|
tag.value: string @index(exact, fulltext) . |
||||||
|
tag.events: [uid] @count @reverse . |
||||||
|
|
||||||
|
# Marker fields (key-value storage) |
||||||
|
marker.key: string @index(exact) @upsert . |
||||||
|
marker.value: string . |
||||||
|
` |
||||||
|
|
||||||
|
// applySchema applies the Nostr schema to the connected Dgraph instance
|
||||||
|
func (d *D) applySchema(ctx context.Context) error { |
||||||
|
d.Logger.Infof("applying Nostr schema to dgraph") |
||||||
|
|
||||||
|
op := &api.Operation{ |
||||||
|
Schema: NostrSchema, |
||||||
|
} |
||||||
|
|
||||||
|
if err := d.client.Alter(ctx, op); err != nil { |
||||||
|
return fmt.Errorf("failed to apply schema: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
d.Logger.Infof("schema applied successfully") |
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// dropAll drops all data from dgraph (useful for testing)
|
||||||
|
func (d *D) dropAll(ctx context.Context) error { |
||||||
|
d.Logger.Warningf("dropping all data from dgraph") |
||||||
|
|
||||||
|
op := &api.Operation{ |
||||||
|
DropAll: true, |
||||||
|
} |
||||||
|
|
||||||
|
if err := d.client.Alter(ctx, op); err != nil { |
||||||
|
return fmt.Errorf("failed to drop all data: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Reapply schema after dropping
|
||||||
|
return d.applySchema(ctx) |
||||||
|
} |
||||||
@ -0,0 +1,136 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"encoding/json" |
||||||
|
"fmt" |
||||||
|
"sync" |
||||||
|
|
||||||
|
"github.com/dgraph-io/dgo/v230/protos/api" |
||||||
|
) |
||||||
|
|
||||||
|
// Serial number management
|
||||||
|
// We use a special counter node to track the next available serial number
|
||||||
|
|
||||||
|
const serialCounterKey = "serial_counter" |
||||||
|
|
||||||
|
var ( |
||||||
|
serialMutex sync.Mutex |
||||||
|
) |
||||||
|
|
||||||
|
// getNextSerial atomically increments and returns the next serial number
|
||||||
|
func (d *D) getNextSerial() (uint64, error) { |
||||||
|
serialMutex.Lock() |
||||||
|
defer serialMutex.Unlock() |
||||||
|
|
||||||
|
// Query current serial value
|
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
counter(func: eq(marker.key, %q)) { |
||||||
|
uid |
||||||
|
marker.value |
||||||
|
} |
||||||
|
}`, serialCounterKey) |
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return 0, fmt.Errorf("failed to query serial counter: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Counter []struct { |
||||||
|
UID string `json:"uid"` |
||||||
|
Value string `json:"marker.value"` |
||||||
|
} `json:"counter"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return 0, fmt.Errorf("failed to parse serial counter: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var currentSerial uint64 = 1 |
||||||
|
var uid string |
||||||
|
|
||||||
|
if len(result.Counter) > 0 { |
||||||
|
// Parse current serial
|
||||||
|
uid = result.Counter[0].UID |
||||||
|
if result.Counter[0].Value != "" { |
||||||
|
fmt.Sscanf(result.Counter[0].Value, "%d", ¤tSerial) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
// Increment serial
|
||||||
|
nextSerial := currentSerial + 1 |
||||||
|
|
||||||
|
// Update or create counter
|
||||||
|
var nquads string |
||||||
|
if uid != "" { |
||||||
|
// Update existing counter
|
||||||
|
nquads = fmt.Sprintf(`<%s> <marker.value> "%d" .`, uid, nextSerial) |
||||||
|
} else { |
||||||
|
// Create new counter
|
||||||
|
nquads = fmt.Sprintf(` |
||||||
|
_:counter <dgraph.type> "Marker" . |
||||||
|
_:counter <marker.key> %q . |
||||||
|
_:counter <marker.value> "%d" . |
||||||
|
`, serialCounterKey, nextSerial) |
||||||
|
} |
||||||
|
|
||||||
|
mutation := &api.Mutation{ |
||||||
|
SetNquads: []byte(nquads), |
||||||
|
CommitNow: true, |
||||||
|
} |
||||||
|
|
||||||
|
if _, err = d.Mutate(context.Background(), mutation); err != nil { |
||||||
|
return 0, fmt.Errorf("failed to update serial counter: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return currentSerial, nil |
||||||
|
} |
||||||
|
|
||||||
|
// initSerialCounter initializes the serial counter if it doesn't exist
|
||||||
|
func (d *D) initSerialCounter() error { |
||||||
|
query := fmt.Sprintf(`{ |
||||||
|
counter(func: eq(marker.key, %q)) { |
||||||
|
uid |
||||||
|
} |
||||||
|
}`, serialCounterKey) |
||||||
|
|
||||||
|
resp, err := d.Query(context.Background(), query) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to check serial counter: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
var result struct { |
||||||
|
Counter []struct { |
||||||
|
UID string `json:"uid"` |
||||||
|
} `json:"counter"` |
||||||
|
} |
||||||
|
|
||||||
|
if err = json.Unmarshal(resp.Json, &result); err != nil { |
||||||
|
return fmt.Errorf("failed to parse counter check: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Counter already exists
|
||||||
|
if len(result.Counter) > 0 { |
||||||
|
return nil |
||||||
|
} |
||||||
|
|
||||||
|
// Initialize counter at 1
|
||||||
|
nquads := fmt.Sprintf(` |
||||||
|
_:counter <dgraph.type> "Marker" . |
||||||
|
_:counter <marker.key> %q . |
||||||
|
_:counter <marker.value> "1" . |
||||||
|
`, serialCounterKey) |
||||||
|
|
||||||
|
mutation := &api.Mutation{ |
||||||
|
SetNquads: []byte(nquads), |
||||||
|
CommitNow: true, |
||||||
|
} |
||||||
|
|
||||||
|
if _, err = d.Mutate(context.Background(), mutation); err != nil { |
||||||
|
return fmt.Errorf("failed to initialize serial counter: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
d.Logger.Infof("initialized serial counter") |
||||||
|
return nil |
||||||
|
} |
||||||
@ -0,0 +1,188 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"encoding/json" |
||||||
|
"fmt" |
||||||
|
"time" |
||||||
|
|
||||||
|
"next.orly.dev/pkg/database" |
||||||
|
"next.orly.dev/pkg/encoders/hex" |
||||||
|
) |
||||||
|
|
||||||
|
// Subscription and payment methods
|
||||||
|
// Simplified implementation using marker-based storage
|
||||||
|
// For production, these should use proper graph nodes with relationships
|
||||||
|
|
||||||
|
// GetSubscription retrieves subscription information for a pubkey
|
||||||
|
func (d *D) GetSubscription(pubkey []byte) (*database.Subscription, error) { |
||||||
|
key := "sub_" + hex.Enc(pubkey) |
||||||
|
data, err := d.GetMarker(key) |
||||||
|
if err != nil { |
||||||
|
return nil, err |
||||||
|
} |
||||||
|
|
||||||
|
var sub database.Subscription |
||||||
|
if err := json.Unmarshal(data, &sub); err != nil { |
||||||
|
return nil, fmt.Errorf("failed to unmarshal subscription: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return &sub, nil |
||||||
|
} |
||||||
|
|
||||||
|
// IsSubscriptionActive checks if a pubkey has an active subscription
|
||||||
|
func (d *D) IsSubscriptionActive(pubkey []byte) (bool, error) { |
||||||
|
sub, err := d.GetSubscription(pubkey) |
||||||
|
if err != nil { |
||||||
|
return false, nil // No subscription = not active
|
||||||
|
} |
||||||
|
|
||||||
|
return sub.PaidUntil.After(time.Now()), nil |
||||||
|
} |
||||||
|
|
||||||
|
// ExtendSubscription extends a subscription by the specified number of days
|
||||||
|
func (d *D) ExtendSubscription(pubkey []byte, days int) error { |
||||||
|
key := "sub_" + hex.Enc(pubkey) |
||||||
|
|
||||||
|
// Get existing subscription or create new
|
||||||
|
var sub database.Subscription |
||||||
|
data, err := d.GetMarker(key) |
||||||
|
if err == nil { |
||||||
|
if err := json.Unmarshal(data, &sub); err != nil { |
||||||
|
return fmt.Errorf("failed to unmarshal subscription: %w", err) |
||||||
|
} |
||||||
|
} else { |
||||||
|
// New subscription - set trial period
|
||||||
|
sub.TrialEnd = time.Now() |
||||||
|
sub.PaidUntil = time.Now() |
||||||
|
} |
||||||
|
|
||||||
|
// Extend expiration
|
||||||
|
if sub.PaidUntil.Before(time.Now()) { |
||||||
|
sub.PaidUntil = time.Now() |
||||||
|
} |
||||||
|
sub.PaidUntil = sub.PaidUntil.Add(time.Duration(days) * 24 * time.Hour) |
||||||
|
|
||||||
|
// Save
|
||||||
|
data, err = json.Marshal(sub) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to marshal subscription: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return d.SetMarker(key, data) |
||||||
|
} |
||||||
|
|
||||||
|
// RecordPayment records a payment for subscription extension
|
||||||
|
func (d *D) RecordPayment( |
||||||
|
pubkey []byte, amount int64, invoice, preimage string, |
||||||
|
) error { |
||||||
|
// Store payment in payments list
|
||||||
|
key := "payments_" + hex.Enc(pubkey) |
||||||
|
|
||||||
|
var payments []database.Payment |
||||||
|
data, err := d.GetMarker(key) |
||||||
|
if err == nil { |
||||||
|
if err := json.Unmarshal(data, &payments); err != nil { |
||||||
|
return fmt.Errorf("failed to unmarshal payments: %w", err) |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
payment := database.Payment{ |
||||||
|
Amount: amount, |
||||||
|
Timestamp: time.Now(), |
||||||
|
Invoice: invoice, |
||||||
|
Preimage: preimage, |
||||||
|
} |
||||||
|
|
||||||
|
payments = append(payments, payment) |
||||||
|
|
||||||
|
data, err = json.Marshal(payments) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to marshal payments: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return d.SetMarker(key, data) |
||||||
|
} |
||||||
|
|
||||||
|
// GetPaymentHistory retrieves payment history for a pubkey
|
||||||
|
func (d *D) GetPaymentHistory(pubkey []byte) ([]database.Payment, error) { |
||||||
|
key := "payments_" + hex.Enc(pubkey) |
||||||
|
|
||||||
|
data, err := d.GetMarker(key) |
||||||
|
if err != nil { |
||||||
|
return nil, nil // No payments = empty list
|
||||||
|
} |
||||||
|
|
||||||
|
var payments []database.Payment |
||||||
|
if err := json.Unmarshal(data, &payments); err != nil { |
||||||
|
return nil, fmt.Errorf("failed to unmarshal payments: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return payments, nil |
||||||
|
} |
||||||
|
|
||||||
|
// ExtendBlossomSubscription extends a Blossom storage subscription
|
||||||
|
func (d *D) ExtendBlossomSubscription( |
||||||
|
pubkey []byte, tier string, storageMB int64, daysExtended int, |
||||||
|
) error { |
||||||
|
key := "blossom_" + hex.Enc(pubkey) |
||||||
|
|
||||||
|
// Simple implementation - just store tier and expiry
|
||||||
|
data := map[string]interface{}{ |
||||||
|
"tier": tier, |
||||||
|
"storageMB": storageMB, |
||||||
|
"extended": daysExtended, |
||||||
|
"updated": time.Now(), |
||||||
|
} |
||||||
|
|
||||||
|
jsonData, err := json.Marshal(data) |
||||||
|
if err != nil { |
||||||
|
return fmt.Errorf("failed to marshal blossom subscription: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
return d.SetMarker(key, jsonData) |
||||||
|
} |
||||||
|
|
||||||
|
// GetBlossomStorageQuota retrieves the storage quota for a pubkey
|
||||||
|
func (d *D) GetBlossomStorageQuota(pubkey []byte) (quotaMB int64, err error) { |
||||||
|
key := "blossom_" + hex.Enc(pubkey) |
||||||
|
|
||||||
|
data, err := d.GetMarker(key) |
||||||
|
if err != nil { |
||||||
|
return 0, nil // No subscription = 0 quota
|
||||||
|
} |
||||||
|
|
||||||
|
var result map[string]interface{} |
||||||
|
if err := json.Unmarshal(data, &result); err != nil { |
||||||
|
return 0, fmt.Errorf("failed to unmarshal blossom data: %w", err) |
||||||
|
} |
||||||
|
|
||||||
|
// Default quota based on tier - simplified
|
||||||
|
if tier, ok := result["tier"].(string); ok { |
||||||
|
switch tier { |
||||||
|
case "basic": |
||||||
|
return 100, nil |
||||||
|
case "premium": |
||||||
|
return 1000, nil |
||||||
|
default: |
||||||
|
return 10, nil |
||||||
|
} |
||||||
|
} |
||||||
|
|
||||||
|
return 0, nil |
||||||
|
} |
||||||
|
|
||||||
|
// IsFirstTimeUser checks if a pubkey is a first-time user
|
||||||
|
func (d *D) IsFirstTimeUser(pubkey []byte) (bool, error) { |
||||||
|
// Check if they have any subscription or payment history
|
||||||
|
sub, _ := d.GetSubscription(pubkey) |
||||||
|
if sub != nil { |
||||||
|
return false, nil |
||||||
|
} |
||||||
|
|
||||||
|
payments, _ := d.GetPaymentHistory(pubkey) |
||||||
|
if len(payments) > 0 { |
||||||
|
return false, nil |
||||||
|
} |
||||||
|
|
||||||
|
return true, nil |
||||||
|
} |
||||||
@ -0,0 +1,30 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"io" |
||||||
|
"os" |
||||||
|
"testing" |
||||||
|
|
||||||
|
"lol.mleku.dev" |
||||||
|
"lol.mleku.dev/log" |
||||||
|
) |
||||||
|
|
||||||
|
func TestMain(m *testing.M) { |
||||||
|
// Disable all logging during tests unless explicitly enabled
|
||||||
|
if os.Getenv("TEST_LOG") == "" { |
||||||
|
// Set log level to Off to suppress all logs
|
||||||
|
lol.SetLogLevel("off") |
||||||
|
// Also redirect output to discard
|
||||||
|
lol.Writer = io.Discard |
||||||
|
// Disable all log printers
|
||||||
|
log.T = lol.GetNullPrinter() |
||||||
|
log.D = lol.GetNullPrinter() |
||||||
|
log.I = lol.GetNullPrinter() |
||||||
|
log.W = lol.GetNullPrinter() |
||||||
|
log.E = lol.GetNullPrinter() |
||||||
|
log.F = lol.GetNullPrinter() |
||||||
|
} |
||||||
|
|
||||||
|
// Run tests
|
||||||
|
os.Exit(m.Run()) |
||||||
|
} |
||||||
@ -0,0 +1,10 @@ |
|||||||
|
package dgraph |
||||||
|
|
||||||
|
import ( |
||||||
|
"encoding/json" |
||||||
|
) |
||||||
|
|
||||||
|
// unmarshalJSON is a helper to unmarshal JSON with error handling
|
||||||
|
func unmarshalJSON(data []byte, v interface{}) error { |
||||||
|
return json.Unmarshal(data, v) |
||||||
|
} |
||||||
@ -0,0 +1,276 @@ |
|||||||
|
# Dgraph Integration Testing |
||||||
|
|
||||||
|
This directory contains scripts and configuration for testing the ORLY dgraph integration. |
||||||
|
|
||||||
|
## Quick Start |
||||||
|
|
||||||
|
### 1. Start Dgraph Server |
||||||
|
|
||||||
|
```bash |
||||||
|
# Using the convenience script |
||||||
|
./scripts/dgraph-start.sh |
||||||
|
|
||||||
|
# Or manually with docker-compose |
||||||
|
cd scripts |
||||||
|
docker-compose -f dgraph-docker-compose.yml up -d |
||||||
|
|
||||||
|
# Or directly with docker |
||||||
|
docker run -d \ |
||||||
|
-p 8080:8080 \ |
||||||
|
-p 9080:9080 \ |
||||||
|
-p 8000:8000 \ |
||||||
|
--name dgraph-orly \ |
||||||
|
dgraph/standalone:latest |
||||||
|
``` |
||||||
|
|
||||||
|
### 2. Run Dgraph Tests |
||||||
|
|
||||||
|
```bash |
||||||
|
# Run all dgraph package tests |
||||||
|
./scripts/test-dgraph.sh |
||||||
|
|
||||||
|
# Run tests with relay-tester |
||||||
|
./scripts/test-dgraph.sh --relay-tester |
||||||
|
``` |
||||||
|
|
||||||
|
### 3. Manual Testing |
||||||
|
|
||||||
|
```bash |
||||||
|
# Start ORLY with dgraph backend |
||||||
|
export ORLY_DB_TYPE=dgraph |
||||||
|
export ORLY_DGRAPH_URL=localhost:9080 |
||||||
|
./orly |
||||||
|
|
||||||
|
# In another terminal, run relay-tester |
||||||
|
go run cmd/relay-tester/main.go -url ws://localhost:3334 |
||||||
|
``` |
||||||
|
|
||||||
|
## Test Files |
||||||
|
|
||||||
|
The dgraph package includes comprehensive tests: |
||||||
|
|
||||||
|
- **testmain_test.go** - Test configuration and logging setup |
||||||
|
- **helpers_test.go** - Helper functions for test setup/teardown |
||||||
|
- **save-event_test.go** - Event storage tests |
||||||
|
- **query-events_test.go** - Event query tests |
||||||
|
|
||||||
|
All tests mirror the existing badger tests to ensure feature parity. |
||||||
|
|
||||||
|
## Test Coverage |
||||||
|
|
||||||
|
The dgraph tests cover: |
||||||
|
|
||||||
|
✅ **Event Storage** |
||||||
|
- Saving events from examples.Cache |
||||||
|
- Duplicate event rejection |
||||||
|
- Deletion event validation |
||||||
|
|
||||||
|
✅ **Event Queries** |
||||||
|
- Query by ID |
||||||
|
- Query by kind |
||||||
|
- Query by author |
||||||
|
- Query by time range |
||||||
|
- Query by tags |
||||||
|
- Event counting |
||||||
|
|
||||||
|
✅ **Advanced Features** |
||||||
|
- Replaceable events (kind 0) |
||||||
|
- Parameterized replaceable events (kind 30000+) |
||||||
|
- Event deletion (kind 5) |
||||||
|
- Event replacement logic |
||||||
|
|
||||||
|
## Requirements |
||||||
|
|
||||||
|
### Dgraph Server |
||||||
|
|
||||||
|
The tests require a running dgraph server. Tests will be skipped if dgraph is not available. |
||||||
|
|
||||||
|
**Endpoints:** |
||||||
|
- gRPC: `localhost:9080` (required for ORLY) |
||||||
|
- HTTP: `localhost:8080` (for health checks) |
||||||
|
- Ratel UI: `localhost:8000` (optional, for debugging) |
||||||
|
|
||||||
|
**Custom Endpoint:** |
||||||
|
```bash |
||||||
|
export ORLY_DGRAPH_URL=remote.server.com:9080 |
||||||
|
./scripts/test-dgraph.sh |
||||||
|
``` |
||||||
|
|
||||||
|
### Docker |
||||||
|
|
||||||
|
The docker-compose setup requires: |
||||||
|
- Docker Engine 20.10+ |
||||||
|
- Docker Compose 1.29+ (or docker-compose plugin) |
||||||
|
|
||||||
|
## Test Workflow |
||||||
|
|
||||||
|
### Running Tests Locally |
||||||
|
|
||||||
|
```bash |
||||||
|
# 1. Start dgraph |
||||||
|
./scripts/dgraph-start.sh |
||||||
|
|
||||||
|
# 2. Run tests |
||||||
|
./scripts/test-dgraph.sh |
||||||
|
|
||||||
|
# 3. Clean up when done |
||||||
|
cd scripts && docker-compose -f dgraph-docker-compose.yml down |
||||||
|
``` |
||||||
|
|
||||||
|
### CI/CD Integration |
||||||
|
|
||||||
|
For CI pipelines, use the docker-compose file: |
||||||
|
|
||||||
|
```yaml |
||||||
|
# Example GitHub Actions workflow |
||||||
|
services: |
||||||
|
dgraph: |
||||||
|
image: dgraph/standalone:latest |
||||||
|
ports: |
||||||
|
- 8080:8080 |
||||||
|
- 9080:9080 |
||||||
|
|
||||||
|
steps: |
||||||
|
- name: Run dgraph tests |
||||||
|
run: | |
||||||
|
export ORLY_DGRAPH_URL=localhost:9080 |
||||||
|
CGO_ENABLED=0 go test -v ./pkg/dgraph/... |
||||||
|
``` |
||||||
|
|
||||||
|
## Debugging |
||||||
|
|
||||||
|
### View Dgraph Logs |
||||||
|
|
||||||
|
```bash |
||||||
|
docker logs dgraph-orly-test -f |
||||||
|
``` |
||||||
|
|
||||||
|
### Access Ratel UI |
||||||
|
|
||||||
|
Open http://localhost:8000 in your browser to: |
||||||
|
- View schema |
||||||
|
- Run DQL queries |
||||||
|
- Inspect data |
||||||
|
|
||||||
|
### Enable Test Logging |
||||||
|
|
||||||
|
```bash |
||||||
|
export TEST_LOG=1 |
||||||
|
./scripts/test-dgraph.sh |
||||||
|
``` |
||||||
|
|
||||||
|
### Manual DQL Queries |
||||||
|
|
||||||
|
```bash |
||||||
|
# Using curl |
||||||
|
curl -X POST localhost:8080/query -d '{ |
||||||
|
q(func: type(Event)) { |
||||||
|
uid |
||||||
|
event.id |
||||||
|
event.kind |
||||||
|
event.created_at |
||||||
|
} |
||||||
|
}' |
||||||
|
|
||||||
|
# Using grpcurl (if installed) |
||||||
|
grpcurl -plaintext -d '{ |
||||||
|
"query": "{ q(func: type(Event)) { uid event.id } }" |
||||||
|
}' localhost:9080 api.Dgraph/Query |
||||||
|
``` |
||||||
|
|
||||||
|
## Troubleshooting |
||||||
|
|
||||||
|
### Tests Skip with "Dgraph server not available" |
||||||
|
|
||||||
|
**Solution:** Ensure dgraph is running: |
||||||
|
```bash |
||||||
|
docker ps | grep dgraph |
||||||
|
./scripts/dgraph-start.sh |
||||||
|
``` |
||||||
|
|
||||||
|
### Connection Refused Errors |
||||||
|
|
||||||
|
**Symptoms:** |
||||||
|
``` |
||||||
|
failed to connect to dgraph at localhost:9080: connection refused |
||||||
|
``` |
||||||
|
|
||||||
|
**Solutions:** |
||||||
|
1. Check dgraph is running: `docker ps` |
||||||
|
2. Check port mapping: `docker port dgraph-orly-test` |
||||||
|
3. Check firewall rules |
||||||
|
4. Verify ORLY_DGRAPH_URL is correct |
||||||
|
|
||||||
|
### Schema Application Failed |
||||||
|
|
||||||
|
**Symptoms:** |
||||||
|
``` |
||||||
|
failed to apply schema: ... |
||||||
|
``` |
||||||
|
|
||||||
|
**Solutions:** |
||||||
|
1. Check dgraph logs: `docker logs dgraph-orly-test` |
||||||
|
2. Drop all data and retry: Use `dropAll` in test setup |
||||||
|
3. Verify dgraph version compatibility |
||||||
|
|
||||||
|
### Tests Timeout |
||||||
|
|
||||||
|
**Symptoms:** |
||||||
|
``` |
||||||
|
panic: test timed out after 10m |
||||||
|
``` |
||||||
|
|
||||||
|
**Solutions:** |
||||||
|
1. Increase timeout: `go test -timeout 20m ./pkg/dgraph/...` |
||||||
|
2. Check dgraph performance: May need more resources |
||||||
|
3. Reduce test dataset size |
||||||
|
|
||||||
|
## Performance Benchmarks |
||||||
|
|
||||||
|
Compare dgraph vs badger performance: |
||||||
|
|
||||||
|
```bash |
||||||
|
# Run badger benchmarks |
||||||
|
go test -bench=. ./pkg/database/... |
||||||
|
|
||||||
|
# Run dgraph benchmarks |
||||||
|
go test -bench=. ./pkg/dgraph/... |
||||||
|
``` |
||||||
|
|
||||||
|
## Test Data |
||||||
|
|
||||||
|
Tests use `pkg/encoders/event/examples.Cache` which contains: |
||||||
|
- ~100 real Nostr events |
||||||
|
- Various kinds (text notes, metadata, etc.) |
||||||
|
- Different authors and timestamps |
||||||
|
- Events with tags and relationships |
||||||
|
|
||||||
|
## Cleanup |
||||||
|
|
||||||
|
### Remove Test Data |
||||||
|
|
||||||
|
```bash |
||||||
|
# Stop and remove containers |
||||||
|
cd scripts |
||||||
|
docker-compose -f dgraph-docker-compose.yml down |
||||||
|
|
||||||
|
# Remove volumes |
||||||
|
docker volume rm scripts_dgraph-data |
||||||
|
``` |
||||||
|
|
||||||
|
### Reset Dgraph |
||||||
|
|
||||||
|
```bash |
||||||
|
# Drop all data (via test helper) |
||||||
|
# The dropAll() function is called in test setup |
||||||
|
|
||||||
|
# Or manually via HTTP |
||||||
|
curl -X POST localhost:8080/alter -d '{"drop_all": true}' |
||||||
|
``` |
||||||
|
|
||||||
|
## Related Documentation |
||||||
|
|
||||||
|
- [Dgraph Implementation Status](../DGRAPH_IMPLEMENTATION_STATUS.md) |
||||||
|
- [Package README](../pkg/dgraph/README.md) |
||||||
|
- [Dgraph Documentation](https://dgraph.io/docs/) |
||||||
|
- [DQL Query Language](https://dgraph.io/docs/query-language/) |
||||||
@ -0,0 +1,546 @@ |
|||||||
|
# Docker-Based Integration Testing |
||||||
|
|
||||||
|
This guide covers running ORLY and Dgraph together in Docker containers for integration testing. |
||||||
|
|
||||||
|
## Overview |
||||||
|
|
||||||
|
The Docker setup provides: |
||||||
|
- **Isolated Environment**: Dgraph + ORLY in containers |
||||||
|
- **Automated Testing**: Health checks and dependency management |
||||||
|
- **Reproducible Tests**: Consistent environment across systems |
||||||
|
- **Easy Cleanup**: Remove everything with one command |
||||||
|
|
||||||
|
## Architecture |
||||||
|
|
||||||
|
``` |
||||||
|
┌─────────────────────────────────────────────┐ |
||||||
|
│ Docker Network (orly-network) │ |
||||||
|
│ │ |
||||||
|
│ ┌──────────────────┐ ┌─────────────────┐ │ |
||||||
|
│ │ Dgraph │ │ ORLY Relay │ │ |
||||||
|
│ │ standalone │◄─┤ (dgraph mode) │ │ |
||||||
|
│ │ │ │ │ │ |
||||||
|
│ │ :8080 (HTTP) │ │ :3334 (WS) │ │ |
||||||
|
│ │ :9080 (gRPC) │ │ │ │ |
||||||
|
│ │ :8000 (Ratel) │ │ │ │ |
||||||
|
│ └──────────────────┘ └─────────────────┘ │ |
||||||
|
│ │ │ │ |
||||||
|
└─────────┼───────────────────────┼───────────┘ |
||||||
|
│ │ |
||||||
|
Published Published |
||||||
|
to host to host |
||||||
|
``` |
||||||
|
|
||||||
|
## Quick Start |
||||||
|
|
||||||
|
### 1. Build Images |
||||||
|
|
||||||
|
```bash |
||||||
|
# Build ORLY image only |
||||||
|
./scripts/docker-build.sh |
||||||
|
|
||||||
|
# Build ORLY + relay-tester |
||||||
|
./scripts/docker-build.sh --with-tester |
||||||
|
``` |
||||||
|
|
||||||
|
### 2. Run Integration Tests |
||||||
|
|
||||||
|
```bash |
||||||
|
# Basic test (start containers, verify connectivity) |
||||||
|
./scripts/test-docker.sh |
||||||
|
|
||||||
|
# Run with relay-tester |
||||||
|
./scripts/test-docker.sh --relay-tester |
||||||
|
|
||||||
|
# Keep containers running after test |
||||||
|
./scripts/test-docker.sh --keep-running |
||||||
|
|
||||||
|
# Skip rebuild (use existing images) |
||||||
|
./scripts/test-docker.sh --skip-build |
||||||
|
``` |
||||||
|
|
||||||
|
### 3. Manual Container Management |
||||||
|
|
||||||
|
```bash |
||||||
|
# Start containers |
||||||
|
cd scripts |
||||||
|
docker-compose -f docker-compose-test.yml up -d |
||||||
|
|
||||||
|
# View logs |
||||||
|
docker-compose -f docker-compose-test.yml logs -f |
||||||
|
|
||||||
|
# Stop containers |
||||||
|
docker-compose -f docker-compose-test.yml down |
||||||
|
|
||||||
|
# Stop and remove volumes |
||||||
|
docker-compose -f docker-compose-test.yml down -v |
||||||
|
``` |
||||||
|
|
||||||
|
## Docker Files |
||||||
|
|
||||||
|
### Dockerfile |
||||||
|
|
||||||
|
Multi-stage build for ORLY: |
||||||
|
|
||||||
|
**Stage 1: Builder** |
||||||
|
- Based on golang:1.21-alpine |
||||||
|
- Downloads dependencies |
||||||
|
- Builds static binary with `CGO_ENABLED=0` |
||||||
|
- Copies libsecp256k1.so for crypto operations |
||||||
|
|
||||||
|
**Stage 2: Runtime** |
||||||
|
- Based on alpine:latest (minimal) |
||||||
|
- Copies binary and shared library |
||||||
|
- Creates non-root user |
||||||
|
- Sets up health checks |
||||||
|
- ~50MB final image size |
||||||
|
|
||||||
|
### Dockerfile.relay-tester |
||||||
|
|
||||||
|
Builds relay-tester for automated testing: |
||||||
|
- Static binary from cmd/relay-tester |
||||||
|
- Configurable RELAY_URL |
||||||
|
- Runs as part of test profile |
||||||
|
|
||||||
|
### docker-compose-test.yml |
||||||
|
|
||||||
|
Orchestrates the full stack: |
||||||
|
|
||||||
|
**Services:** |
||||||
|
1. **dgraph** - Database backend |
||||||
|
- Health check via HTTP |
||||||
|
- Persistent volume for data |
||||||
|
- Exposed ports for debugging |
||||||
|
|
||||||
|
2. **orly** - Relay server |
||||||
|
- Depends on dgraph (waits for healthy) |
||||||
|
- Configured with ORLY_DB_TYPE=dgraph |
||||||
|
- Health check via HTTP |
||||||
|
- Auto-restart on failure |
||||||
|
|
||||||
|
3. **relay-tester** - Test runner |
||||||
|
- Profile: test (optional) |
||||||
|
- Runs tests against ORLY |
||||||
|
- Exits after completion |
||||||
|
|
||||||
|
## Configuration |
||||||
|
|
||||||
|
### Environment Variables |
||||||
|
|
||||||
|
The docker-compose file sets: |
||||||
|
|
||||||
|
```yaml |
||||||
|
# Database |
||||||
|
ORLY_DB_TYPE: dgraph |
||||||
|
ORLY_DGRAPH_URL: dgraph:9080 # Internal network name |
||||||
|
|
||||||
|
# Server |
||||||
|
ORLY_LISTEN: 0.0.0.0 |
||||||
|
ORLY_PORT: 3334 |
||||||
|
ORLY_DATA_DIR: /data |
||||||
|
|
||||||
|
# Application |
||||||
|
ORLY_LOG_LEVEL: info |
||||||
|
ORLY_APP_NAME: ORLY-Dgraph-Test |
||||||
|
ORLY_ACL_MODE: none |
||||||
|
``` |
||||||
|
|
||||||
|
Override via environment or .env file: |
||||||
|
|
||||||
|
```bash |
||||||
|
# Create .env file in scripts/ |
||||||
|
cat > scripts/.env << EOF |
||||||
|
ORLY_LOG_LEVEL=debug |
||||||
|
ORLY_ADMINS=npub1... |
||||||
|
EOF |
||||||
|
``` |
||||||
|
|
||||||
|
### Volumes |
||||||
|
|
||||||
|
**Persistent Data:** |
||||||
|
- `dgraph-data:/dgraph` - Dgraph database |
||||||
|
- `orly-data:/data` - ORLY metadata |
||||||
|
|
||||||
|
**Inspect Volumes:** |
||||||
|
```bash |
||||||
|
docker volume ls |
||||||
|
docker volume inspect scripts_dgraph-data |
||||||
|
``` |
||||||
|
|
||||||
|
### Networks |
||||||
|
|
||||||
|
**Custom Bridge Network:** |
||||||
|
- Name: orly-network |
||||||
|
- Subnet: 172.28.0.0/16 |
||||||
|
- Allows container-to-container communication |
||||||
|
- DNS resolution by service name |
||||||
|
|
||||||
|
## Testing Workflows |
||||||
|
|
||||||
|
### Basic Integration Test |
||||||
|
|
||||||
|
```bash |
||||||
|
./scripts/test-docker.sh |
||||||
|
``` |
||||||
|
|
||||||
|
**What it does:** |
||||||
|
1. Stops any existing containers |
||||||
|
2. Starts dgraph and waits for health |
||||||
|
3. Starts ORLY and waits for health |
||||||
|
4. Verifies HTTP connectivity |
||||||
|
5. Tests WebSocket (if websocat installed) |
||||||
|
6. Shows container status |
||||||
|
7. Cleans up (unless --keep-running) |
||||||
|
|
||||||
|
### With Relay-Tester |
||||||
|
|
||||||
|
```bash |
||||||
|
./scripts/test-docker.sh --relay-tester |
||||||
|
``` |
||||||
|
|
||||||
|
**Additional steps:** |
||||||
|
1. Builds relay-tester image |
||||||
|
2. Runs comprehensive protocol tests |
||||||
|
3. Reports pass/fail |
||||||
|
4. Shows ORLY logs on failure |
||||||
|
|
||||||
|
### Development Workflow |
||||||
|
|
||||||
|
```bash |
||||||
|
# Start and keep running |
||||||
|
./scripts/test-docker.sh --keep-running |
||||||
|
|
||||||
|
# Make changes to code |
||||||
|
vim pkg/dgraph/save-event.go |
||||||
|
|
||||||
|
# Rebuild and restart |
||||||
|
docker-compose -f scripts/docker-compose-test.yml up -d --build orly |
||||||
|
|
||||||
|
# View logs |
||||||
|
docker logs orly-relay -f |
||||||
|
|
||||||
|
# Test changes |
||||||
|
go run cmd/relay-tester/main.go -url ws://localhost:3334 |
||||||
|
|
||||||
|
# Stop when done |
||||||
|
cd scripts && docker-compose -f docker-compose-test.yml down |
||||||
|
``` |
||||||
|
|
||||||
|
## Debugging |
||||||
|
|
||||||
|
### View Container Logs |
||||||
|
|
||||||
|
```bash |
||||||
|
# All services |
||||||
|
docker-compose -f scripts/docker-compose-test.yml logs -f |
||||||
|
|
||||||
|
# Specific service |
||||||
|
docker logs orly-relay -f |
||||||
|
docker logs orly-dgraph -f |
||||||
|
|
||||||
|
# Last N lines |
||||||
|
docker logs orly-relay --tail 50 |
||||||
|
``` |
||||||
|
|
||||||
|
### Execute Commands in Container |
||||||
|
|
||||||
|
```bash |
||||||
|
# ORLY version |
||||||
|
docker exec orly-relay /app/orly version |
||||||
|
|
||||||
|
# Check ORLY processes |
||||||
|
docker exec orly-relay ps aux |
||||||
|
|
||||||
|
# Inspect data directory |
||||||
|
docker exec orly-relay ls -la /data |
||||||
|
|
||||||
|
# Query dgraph |
||||||
|
docker exec orly-dgraph curl http://localhost:8080/health |
||||||
|
``` |
||||||
|
|
||||||
|
### Access Ratel UI |
||||||
|
|
||||||
|
Open http://localhost:8000 in browser: |
||||||
|
- View dgraph schema |
||||||
|
- Run DQL queries |
||||||
|
- Inspect stored data |
||||||
|
- Monitor performance |
||||||
|
|
||||||
|
### Network Inspection |
||||||
|
|
||||||
|
```bash |
||||||
|
# List networks |
||||||
|
docker network ls |
||||||
|
|
||||||
|
# Inspect orly network |
||||||
|
docker network inspect scripts_orly-network |
||||||
|
|
||||||
|
# Test connectivity |
||||||
|
docker exec orly-relay ping dgraph |
||||||
|
docker exec orly-relay nc -zv dgraph 9080 |
||||||
|
``` |
||||||
|
|
||||||
|
### Health Check Status |
||||||
|
|
||||||
|
```bash |
||||||
|
# Check health |
||||||
|
docker inspect orly-relay | grep -A 10 Health |
||||||
|
|
||||||
|
# View health check logs |
||||||
|
docker inspect --format='{{json .State.Health}}' orly-relay | jq |
||||||
|
``` |
||||||
|
|
||||||
|
## Troubleshooting |
||||||
|
|
||||||
|
### Build Failures |
||||||
|
|
||||||
|
**Error: Cannot find libsecp256k1.so** |
||||||
|
|
||||||
|
```bash |
||||||
|
# Ensure library exists |
||||||
|
ls -l pkg/crypto/p8k/libsecp256k1.so |
||||||
|
|
||||||
|
# Rebuild if needed |
||||||
|
cd pkg/crypto/p8k && make |
||||||
|
``` |
||||||
|
|
||||||
|
**Error: Go module download fails** |
||||||
|
|
||||||
|
```bash |
||||||
|
# Clear module cache |
||||||
|
go clean -modcache |
||||||
|
|
||||||
|
# Try building locally first |
||||||
|
CGO_ENABLED=0 go build |
||||||
|
``` |
||||||
|
|
||||||
|
### Runtime Failures |
||||||
|
|
||||||
|
**ORLY fails health check** |
||||||
|
|
||||||
|
```bash |
||||||
|
# Check logs |
||||||
|
docker logs orly-relay |
||||||
|
|
||||||
|
# Common issues: |
||||||
|
# - Port already in use: docker ps (check for conflicts) |
||||||
|
# - Dgraph not ready: docker logs orly-dgraph |
||||||
|
# - Bad configuration: docker exec orly-relay env |
||||||
|
``` |
||||||
|
|
||||||
|
**Cannot connect to dgraph** |
||||||
|
|
||||||
|
```bash |
||||||
|
# Verify dgraph is healthy |
||||||
|
docker inspect orly-dgraph | grep Health |
||||||
|
|
||||||
|
# Check network connectivity |
||||||
|
docker exec orly-relay ping dgraph |
||||||
|
docker exec orly-relay nc -zv dgraph 9080 |
||||||
|
|
||||||
|
# Verify dgraph is listening |
||||||
|
docker exec orly-dgraph netstat -tlnp | grep 9080 |
||||||
|
``` |
||||||
|
|
||||||
|
**WebSocket connection fails** |
||||||
|
|
||||||
|
```bash |
||||||
|
# Test from host |
||||||
|
websocat ws://localhost:3334 |
||||||
|
|
||||||
|
# Test from container |
||||||
|
docker exec orly-relay curl -v http://localhost:3334 |
||||||
|
|
||||||
|
# Check firewall |
||||||
|
sudo iptables -L | grep 3334 |
||||||
|
``` |
||||||
|
|
||||||
|
### Performance Issues |
||||||
|
|
||||||
|
**Slow startup** |
||||||
|
|
||||||
|
```bash |
||||||
|
# Increase health check timeouts in docker-compose-test.yml |
||||||
|
start_period: 60s # Default is 20-30s |
||||||
|
|
||||||
|
# Pre-pull images |
||||||
|
docker pull dgraph/standalone:latest |
||||||
|
docker pull golang:1.21-alpine |
||||||
|
``` |
||||||
|
|
||||||
|
**High memory usage** |
||||||
|
|
||||||
|
```bash |
||||||
|
# Check resource usage |
||||||
|
docker stats |
||||||
|
|
||||||
|
# Limit container resources |
||||||
|
# Add to docker-compose-test.yml: |
||||||
|
deploy: |
||||||
|
resources: |
||||||
|
limits: |
||||||
|
memory: 2G |
||||||
|
cpus: '2' |
||||||
|
``` |
||||||
|
|
||||||
|
## CI/CD Integration |
||||||
|
|
||||||
|
### GitHub Actions Example |
||||||
|
|
||||||
|
```yaml |
||||||
|
name: Docker Integration Tests |
||||||
|
|
||||||
|
on: [push, pull_request] |
||||||
|
|
||||||
|
jobs: |
||||||
|
docker-test: |
||||||
|
runs-on: ubuntu-latest |
||||||
|
|
||||||
|
steps: |
||||||
|
- uses: actions/checkout@v3 |
||||||
|
|
||||||
|
- name: Set up Docker Buildx |
||||||
|
uses: docker/setup-buildx-action@v2 |
||||||
|
|
||||||
|
- name: Build images |
||||||
|
run: ./scripts/docker-build.sh --with-tester |
||||||
|
|
||||||
|
- name: Run integration tests |
||||||
|
run: ./scripts/test-docker.sh --relay-tester |
||||||
|
|
||||||
|
- name: Upload logs on failure |
||||||
|
if: failure() |
||||||
|
uses: actions/upload-artifact@v3 |
||||||
|
with: |
||||||
|
name: container-logs |
||||||
|
path: | |
||||||
|
scripts/orly-relay.log |
||||||
|
scripts/dgraph.log |
||||||
|
``` |
||||||
|
|
||||||
|
### GitLab CI Example |
||||||
|
|
||||||
|
```yaml |
||||||
|
docker-test: |
||||||
|
image: docker:latest |
||||||
|
services: |
||||||
|
- docker:dind |
||||||
|
script: |
||||||
|
- ./scripts/docker-build.sh --with-tester |
||||||
|
- ./scripts/test-docker.sh --relay-tester |
||||||
|
artifacts: |
||||||
|
when: on_failure |
||||||
|
paths: |
||||||
|
- scripts/*.log |
||||||
|
``` |
||||||
|
|
||||||
|
## Advanced Usage |
||||||
|
|
||||||
|
### Custom Configuration |
||||||
|
|
||||||
|
Create a custom docker-compose override: |
||||||
|
|
||||||
|
```yaml |
||||||
|
# docker-compose.override.yml |
||||||
|
version: '3.8' |
||||||
|
|
||||||
|
services: |
||||||
|
orly: |
||||||
|
environment: |
||||||
|
- ORLY_LOG_LEVEL=debug |
||||||
|
- ORLY_ADMINS=npub1... |
||||||
|
- ORLY_ACL_MODE=follows |
||||||
|
ports: |
||||||
|
- "3335:3334" # Different host port |
||||||
|
``` |
||||||
|
|
||||||
|
### Multi-Instance Testing |
||||||
|
|
||||||
|
Test multiple ORLY instances: |
||||||
|
|
||||||
|
```yaml |
||||||
|
# docker-compose-multi.yml |
||||||
|
services: |
||||||
|
orly-1: |
||||||
|
extends: |
||||||
|
file: docker-compose-test.yml |
||||||
|
service: orly |
||||||
|
container_name: orly-relay-1 |
||||||
|
ports: |
||||||
|
- "3334:3334" |
||||||
|
|
||||||
|
orly-2: |
||||||
|
extends: |
||||||
|
file: docker-compose-test.yml |
||||||
|
service: orly |
||||||
|
container_name: orly-relay-2 |
||||||
|
ports: |
||||||
|
- "3335:3334" |
||||||
|
``` |
||||||
|
|
||||||
|
### Performance Benchmarking |
||||||
|
|
||||||
|
```bash |
||||||
|
# Start with --keep-running |
||||||
|
./scripts/test-docker.sh --keep-running |
||||||
|
|
||||||
|
# Run stress test |
||||||
|
go run cmd/stresstest/main.go -url ws://localhost:3334 -connections 100 |
||||||
|
|
||||||
|
# Monitor resources |
||||||
|
docker stats |
||||||
|
|
||||||
|
# Profile ORLY |
||||||
|
docker exec orly-relay sh -c 'curl http://localhost:6060/debug/pprof/profile?seconds=30 > /tmp/cpu.prof' |
||||||
|
``` |
||||||
|
|
||||||
|
## Cleanup |
||||||
|
|
||||||
|
### Remove Everything |
||||||
|
|
||||||
|
```bash |
||||||
|
# Stop and remove containers |
||||||
|
cd scripts && docker-compose -f docker-compose-test.yml down |
||||||
|
|
||||||
|
# Remove volumes (data) |
||||||
|
docker-compose -f docker-compose-test.yml down -v |
||||||
|
|
||||||
|
# Remove images |
||||||
|
docker rmi orly:latest orly-relay-tester:latest |
||||||
|
|
||||||
|
# Remove networks |
||||||
|
docker network rm scripts_orly-network |
||||||
|
|
||||||
|
# Prune everything (careful!) |
||||||
|
docker system prune -a --volumes |
||||||
|
``` |
||||||
|
|
||||||
|
### Selective Cleanup |
||||||
|
|
||||||
|
```bash |
||||||
|
# Just stop containers (keep data) |
||||||
|
docker-compose -f docker-compose-test.yml stop |
||||||
|
|
||||||
|
# Remove only one service |
||||||
|
docker-compose -f docker-compose-test.yml rm -s -f orly |
||||||
|
|
||||||
|
# Clear dgraph data |
||||||
|
docker volume rm scripts_dgraph-data |
||||||
|
``` |
||||||
|
|
||||||
|
## Related Documentation |
||||||
|
|
||||||
|
- [Main Testing Guide](DGRAPH_TESTING.md) |
||||||
|
- [Package Tests](../pkg/dgraph/TESTING.md) |
||||||
|
- [Docker Documentation](https://docs.docker.com/) |
||||||
|
- [Docker Compose](https://docs.docker.com/compose/) |
||||||
|
|
||||||
|
## Best Practices |
||||||
|
|
||||||
|
1. **Always use health checks** - Ensure services are ready |
||||||
|
2. **Use specific tags** - Don't rely on :latest in production |
||||||
|
3. **Limit resources** - Prevent container resource exhaustion |
||||||
|
4. **Volume backups** - Backup dgraph-data volume before updates |
||||||
|
5. **Network isolation** - Use custom networks for security |
||||||
|
6. **Read-only root** - Run as non-root user |
||||||
|
7. **Clean up regularly** - Remove unused containers/volumes |
||||||
@ -0,0 +1,424 @@ |
|||||||
|
# ORLY Scripts Directory |
||||||
|
|
||||||
|
This directory contains automation scripts for building, testing, and deploying ORLY. |
||||||
|
|
||||||
|
## Quick Reference |
||||||
|
|
||||||
|
### Dgraph Integration Testing |
||||||
|
|
||||||
|
```bash |
||||||
|
# Local testing (requires dgraph server) |
||||||
|
./dgraph-start.sh # Start dgraph server |
||||||
|
./test-dgraph.sh # Run dgraph package tests |
||||||
|
./test-dgraph.sh --relay-tester # Run tests + relay-tester |
||||||
|
|
||||||
|
# Docker testing (containers for everything) |
||||||
|
./docker-build.sh # Build ORLY docker image |
||||||
|
./test-docker.sh # Run integration tests in containers |
||||||
|
./test-docker.sh --relay-tester --keep-running # Full test, keep running |
||||||
|
``` |
||||||
|
|
||||||
|
### Build & Deploy |
||||||
|
|
||||||
|
```bash |
||||||
|
./build-all-platforms.sh # Build for multiple platforms |
||||||
|
./deploy.sh # Deploy to systemd |
||||||
|
./update-embedded-web.sh # Build and embed web UI |
||||||
|
``` |
||||||
|
|
||||||
|
## Script Descriptions |
||||||
|
|
||||||
|
### Dgraph Testing Scripts |
||||||
|
|
||||||
|
#### dgraph-start.sh |
||||||
|
Starts dgraph server using docker-compose for local testing. |
||||||
|
|
||||||
|
**Usage:** |
||||||
|
```bash |
||||||
|
./dgraph-start.sh |
||||||
|
``` |
||||||
|
|
||||||
|
**What it does:** |
||||||
|
- Checks if dgraph is already running |
||||||
|
- Starts dgraph via docker-compose |
||||||
|
- Waits for health check |
||||||
|
- Shows endpoints and commands |
||||||
|
|
||||||
|
#### dgraph-docker-compose.yml |
||||||
|
Docker Compose configuration for standalone dgraph server. |
||||||
|
|
||||||
|
**Ports:** |
||||||
|
- 8080: HTTP API |
||||||
|
- 9080: gRPC (ORLY connects here) |
||||||
|
- 8000: Ratel UI |
||||||
|
|
||||||
|
#### test-dgraph.sh |
||||||
|
Runs dgraph package tests against a running dgraph server. |
||||||
|
|
||||||
|
**Usage:** |
||||||
|
```bash |
||||||
|
./test-dgraph.sh # Just tests |
||||||
|
./test-dgraph.sh --relay-tester # Tests + relay-tester |
||||||
|
``` |
||||||
|
|
||||||
|
**Requirements:** |
||||||
|
- Dgraph server running at ORLY_DGRAPH_URL (default: localhost:9080) |
||||||
|
- Go 1.21+ |
||||||
|
|
||||||
|
### Docker Integration Scripts |
||||||
|
|
||||||
|
#### docker-build.sh |
||||||
|
Builds Docker images for ORLY and optionally relay-tester. |
||||||
|
|
||||||
|
**Usage:** |
||||||
|
```bash |
||||||
|
./docker-build.sh # ORLY only |
||||||
|
./docker-build.sh --with-tester # ORLY + relay-tester |
||||||
|
``` |
||||||
|
|
||||||
|
**Output:** |
||||||
|
- orly:latest |
||||||
|
- orly-relay-tester:latest (if --with-tester) |
||||||
|
|
||||||
|
#### docker-compose-test.yml |
||||||
|
Full-stack docker-compose with dgraph, ORLY, and relay-tester. |
||||||
|
|
||||||
|
**Services:** |
||||||
|
- dgraph: Database backend |
||||||
|
- orly: Relay with dgraph backend |
||||||
|
- relay-tester: Protocol tests (optional, profile: test) |
||||||
|
|
||||||
|
**Features:** |
||||||
|
- Health checks for all services |
||||||
|
- Dependency management (ORLY waits for dgraph) |
||||||
|
- Custom network with DNS |
||||||
|
- Persistent volumes |
||||||
|
|
||||||
|
#### test-docker.sh |
||||||
|
Comprehensive integration testing in Docker containers. |
||||||
|
|
||||||
|
**Usage:** |
||||||
|
```bash |
||||||
|
./test-docker.sh # Basic test |
||||||
|
./test-docker.sh --relay-tester # Run relay-tester |
||||||
|
./test-docker.sh --keep-running # Keep containers running |
||||||
|
./test-docker.sh --skip-build # Use existing images |
||||||
|
./test-docker.sh --relay-tester --keep-running # Full test + keep running |
||||||
|
``` |
||||||
|
|
||||||
|
**What it does:** |
||||||
|
1. Stops any existing containers |
||||||
|
2. Optionally rebuilds images |
||||||
|
3. Starts dgraph and waits for health |
||||||
|
4. Starts ORLY and waits for health |
||||||
|
5. Verifies connectivity |
||||||
|
6. Optionally runs relay-tester |
||||||
|
7. Shows status and endpoints |
||||||
|
8. Cleanup (unless --keep-running) |
||||||
|
|
||||||
|
### Build Scripts |
||||||
|
|
||||||
|
#### build-all-platforms.sh |
||||||
|
Cross-compiles ORLY for multiple platforms. |
||||||
|
|
||||||
|
**Platforms:** |
||||||
|
- linux/amd64 |
||||||
|
- linux/arm64 |
||||||
|
- darwin/amd64 |
||||||
|
- darwin/arm64 |
||||||
|
|
||||||
|
**Output:** `dist/` directory with platform-specific binaries |
||||||
|
|
||||||
|
#### update-embedded-web.sh |
||||||
|
Builds the Svelte web UI and embeds it in ORLY binary. |
||||||
|
|
||||||
|
**Steps:** |
||||||
|
1. Builds web UI with bun |
||||||
|
2. Generates embedded assets |
||||||
|
3. Rebuilds ORLY with embedded UI |
||||||
|
|
||||||
|
### Deployment Scripts |
||||||
|
|
||||||
|
#### deploy.sh |
||||||
|
Automated deployment with systemd service. |
||||||
|
|
||||||
|
**What it does:** |
||||||
|
1. Installs Go if needed |
||||||
|
2. Builds ORLY with embedded web UI |
||||||
|
3. Installs to ~/.local/bin/orly |
||||||
|
4. Creates systemd service |
||||||
|
5. Enables and starts service |
||||||
|
6. Sets up port binding capabilities |
||||||
|
|
||||||
|
### Test Scripts |
||||||
|
|
||||||
|
#### test.sh |
||||||
|
Runs all Go tests in the project. |
||||||
|
|
||||||
|
**Usage:** |
||||||
|
```bash |
||||||
|
./test.sh # All tests |
||||||
|
TEST_LOG=1 ./test.sh # With logging |
||||||
|
``` |
||||||
|
|
||||||
|
## Environment Variables |
||||||
|
|
||||||
|
### Common Variables |
||||||
|
|
||||||
|
```bash |
||||||
|
# Dgraph |
||||||
|
export ORLY_DGRAPH_URL=localhost:9080 # Dgraph endpoint |
||||||
|
export ORLY_DB_TYPE=dgraph # Use dgraph backend |
||||||
|
|
||||||
|
# Logging |
||||||
|
export ORLY_LOG_LEVEL=debug # Log verbosity |
||||||
|
export TEST_LOG=1 # Enable test logging |
||||||
|
|
||||||
|
# Server |
||||||
|
export ORLY_PORT=3334 # HTTP/WebSocket port |
||||||
|
export ORLY_LISTEN=0.0.0.0 # Listen address |
||||||
|
|
||||||
|
# Data |
||||||
|
export ORLY_DATA_DIR=/path/to/data # Data directory |
||||||
|
``` |
||||||
|
|
||||||
|
### Script-Specific Variables |
||||||
|
|
||||||
|
```bash |
||||||
|
# Docker scripts |
||||||
|
export SKIP_BUILD=true # Skip image rebuild |
||||||
|
export KEEP_RUNNING=true # Don't cleanup containers |
||||||
|
|
||||||
|
# Dgraph scripts |
||||||
|
export DGRAPH_VERSION=latest # Dgraph image tag |
||||||
|
``` |
||||||
|
|
||||||
|
## File Organization |
||||||
|
|
||||||
|
``` |
||||||
|
scripts/ |
||||||
|
├── README.md # This file |
||||||
|
├── DGRAPH_TESTING.md # Dgraph testing guide |
||||||
|
├── DOCKER_TESTING.md # Docker testing guide |
||||||
|
│ |
||||||
|
├── dgraph-start.sh # Start dgraph server |
||||||
|
├── dgraph-docker-compose.yml # Dgraph docker config |
||||||
|
├── test-dgraph.sh # Run dgraph tests |
||||||
|
│ |
||||||
|
├── docker-build.sh # Build docker images |
||||||
|
├── docker-compose-test.yml # Full stack docker config |
||||||
|
├── test-docker.sh # Run docker integration tests |
||||||
|
│ |
||||||
|
├── build-all-platforms.sh # Cross-compile |
||||||
|
├── deploy.sh # Deploy to systemd |
||||||
|
├── update-embedded-web.sh # Build web UI |
||||||
|
└── test.sh # Run Go tests |
||||||
|
``` |
||||||
|
|
||||||
|
## Workflows |
||||||
|
|
||||||
|
### Local Development with Dgraph |
||||||
|
|
||||||
|
```bash |
||||||
|
# 1. Start dgraph |
||||||
|
./scripts/dgraph-start.sh |
||||||
|
|
||||||
|
# 2. Run ORLY locally with dgraph |
||||||
|
export ORLY_DB_TYPE=dgraph |
||||||
|
export ORLY_DGRAPH_URL=localhost:9080 |
||||||
|
./orly |
||||||
|
|
||||||
|
# 3. Test changes |
||||||
|
go run cmd/relay-tester/main.go -url ws://localhost:3334 |
||||||
|
|
||||||
|
# 4. Run unit tests |
||||||
|
./scripts/test-dgraph.sh |
||||||
|
``` |
||||||
|
|
||||||
|
### Docker Development |
||||||
|
|
||||||
|
```bash |
||||||
|
# 1. Make changes |
||||||
|
vim pkg/dgraph/save-event.go |
||||||
|
|
||||||
|
# 2. Build and test in containers |
||||||
|
./scripts/test-docker.sh --relay-tester --keep-running |
||||||
|
|
||||||
|
# 3. Make more changes |
||||||
|
|
||||||
|
# 4. Rebuild just ORLY |
||||||
|
cd scripts |
||||||
|
docker-compose -f docker-compose-test.yml up -d --build orly |
||||||
|
|
||||||
|
# 5. View logs |
||||||
|
docker logs orly-relay -f |
||||||
|
|
||||||
|
# 6. Stop when done |
||||||
|
docker-compose -f docker-compose-test.yml down |
||||||
|
``` |
||||||
|
|
||||||
|
### CI/CD Testing |
||||||
|
|
||||||
|
```bash |
||||||
|
# Quick test (no containers) |
||||||
|
./scripts/test.sh |
||||||
|
|
||||||
|
# Full integration test |
||||||
|
./scripts/test-docker.sh --relay-tester |
||||||
|
|
||||||
|
# Build for deployment |
||||||
|
./scripts/build-all-platforms.sh |
||||||
|
``` |
||||||
|
|
||||||
|
### Production Deployment |
||||||
|
|
||||||
|
```bash |
||||||
|
# Deploy with systemd |
||||||
|
./scripts/deploy.sh |
||||||
|
|
||||||
|
# Check status |
||||||
|
systemctl status orly |
||||||
|
|
||||||
|
# View logs |
||||||
|
journalctl -u orly -f |
||||||
|
|
||||||
|
# Update |
||||||
|
./scripts/deploy.sh # Rebuilds and restarts |
||||||
|
``` |
||||||
|
|
||||||
|
## Troubleshooting |
||||||
|
|
||||||
|
### Dgraph Not Available |
||||||
|
|
||||||
|
```bash |
||||||
|
# Check if running |
||||||
|
docker ps | grep dgraph |
||||||
|
|
||||||
|
# Start it |
||||||
|
./scripts/dgraph-start.sh |
||||||
|
|
||||||
|
# Check logs |
||||||
|
docker logs dgraph-orly-test -f |
||||||
|
``` |
||||||
|
|
||||||
|
### Port Conflicts |
||||||
|
|
||||||
|
```bash |
||||||
|
# Find what's using port 3334 |
||||||
|
lsof -i :3334 |
||||||
|
netstat -tlnp | grep 3334 |
||||||
|
|
||||||
|
# Kill process |
||||||
|
kill $(lsof -t -i :3334) |
||||||
|
|
||||||
|
# Or use different port |
||||||
|
export ORLY_PORT=3335 |
||||||
|
``` |
||||||
|
|
||||||
|
### Docker Build Failures |
||||||
|
|
||||||
|
```bash |
||||||
|
# Clear docker cache |
||||||
|
docker builder prune |
||||||
|
|
||||||
|
# Rebuild from scratch |
||||||
|
docker build --no-cache -t orly:latest -f Dockerfile . |
||||||
|
|
||||||
|
# Check Dockerfile syntax |
||||||
|
docker build --dry-run -f Dockerfile . |
||||||
|
``` |
||||||
|
|
||||||
|
### Permission Issues |
||||||
|
|
||||||
|
```bash |
||||||
|
# Fix script permissions |
||||||
|
chmod +x scripts/*.sh |
||||||
|
|
||||||
|
# Fix docker socket |
||||||
|
sudo usermod -aG docker $USER |
||||||
|
newgrp docker |
||||||
|
``` |
||||||
|
|
||||||
|
## Best Practices |
||||||
|
|
||||||
|
1. **Always use scripts from project root** |
||||||
|
```bash |
||||||
|
./scripts/test-docker.sh # Good |
||||||
|
cd scripts && ./test-docker.sh # May have path issues |
||||||
|
``` |
||||||
|
|
||||||
|
2. **Check prerequisites before running** |
||||||
|
```bash |
||||||
|
# Check docker |
||||||
|
docker --version |
||||||
|
docker-compose --version |
||||||
|
|
||||||
|
# Check dgraph |
||||||
|
curl http://localhost:9080/health |
||||||
|
``` |
||||||
|
|
||||||
|
3. **Clean up after testing** |
||||||
|
```bash |
||||||
|
# Stop containers |
||||||
|
cd scripts && docker-compose -f docker-compose-test.yml down |
||||||
|
|
||||||
|
# Remove volumes if needed |
||||||
|
docker-compose -f docker-compose-test.yml down -v |
||||||
|
``` |
||||||
|
|
||||||
|
4. **Use --keep-running for debugging** |
||||||
|
```bash |
||||||
|
./scripts/test-docker.sh --keep-running |
||||||
|
# Inspect, debug, make changes |
||||||
|
docker-compose -f scripts/docker-compose-test.yml down |
||||||
|
``` |
||||||
|
|
||||||
|
5. **Check logs on failures** |
||||||
|
```bash |
||||||
|
# Container logs |
||||||
|
docker logs orly-relay --tail 100 |
||||||
|
|
||||||
|
# Test output |
||||||
|
./scripts/test-dgraph.sh 2>&1 | tee test.log |
||||||
|
``` |
||||||
|
|
||||||
|
## Related Documentation |
||||||
|
|
||||||
|
- [Dgraph Testing Guide](DGRAPH_TESTING.md) |
||||||
|
- [Docker Testing Guide](DOCKER_TESTING.md) |
||||||
|
- [Package Tests](../pkg/dgraph/TESTING.md) |
||||||
|
- [Main Implementation Status](../DGRAPH_IMPLEMENTATION_STATUS.md) |
||||||
|
|
||||||
|
## Contributing |
||||||
|
|
||||||
|
When adding new scripts: |
||||||
|
|
||||||
|
1. **Add executable permission** |
||||||
|
```bash |
||||||
|
chmod +x scripts/new-script.sh |
||||||
|
``` |
||||||
|
|
||||||
|
2. **Use bash strict mode** |
||||||
|
```bash |
||||||
|
#!/bin/bash |
||||||
|
set -e # Exit on error |
||||||
|
``` |
||||||
|
|
||||||
|
3. **Add help text** |
||||||
|
```bash |
||||||
|
if [ "$1" == "--help" ]; then |
||||||
|
echo "Usage: $0 [options]" |
||||||
|
exit 0 |
||||||
|
fi |
||||||
|
``` |
||||||
|
|
||||||
|
4. **Document in this README** |
||||||
|
- Add to appropriate section |
||||||
|
- Include usage examples |
||||||
|
- Note any requirements |
||||||
|
|
||||||
|
5. **Test on fresh system** |
||||||
|
```bash |
||||||
|
# Use Docker to test |
||||||
|
docker run --rm -v $(pwd):/app -w /app ubuntu:latest ./scripts/new-script.sh |
||||||
|
``` |
||||||
@ -0,0 +1,25 @@ |
|||||||
|
version: '3.8' |
||||||
|
|
||||||
|
services: |
||||||
|
dgraph: |
||||||
|
image: dgraph/standalone:latest |
||||||
|
container_name: dgraph-orly-test |
||||||
|
ports: |
||||||
|
- "8080:8080" # HTTP API |
||||||
|
- "9080:9080" # gRPC |
||||||
|
- "8000:8000" # Ratel UI |
||||||
|
volumes: |
||||||
|
- dgraph-data:/dgraph |
||||||
|
environment: |
||||||
|
- DGRAPH_ALPHA_JAEGER_COLLECTOR=false |
||||||
|
restart: unless-stopped |
||||||
|
healthcheck: |
||||||
|
test: ["CMD", "curl", "-f", "http://localhost:8080/health"] |
||||||
|
interval: 10s |
||||||
|
timeout: 5s |
||||||
|
retries: 5 |
||||||
|
start_period: 20s |
||||||
|
|
||||||
|
volumes: |
||||||
|
dgraph-data: |
||||||
|
driver: local |
||||||
@ -0,0 +1,50 @@ |
|||||||
|
#!/bin/bash |
||||||
|
# Quick script to start dgraph for testing |
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
||||||
|
|
||||||
|
echo "Starting dgraph server for ORLY testing..." |
||||||
|
cd "$SCRIPT_DIR" |
||||||
|
|
||||||
|
# Check if already running |
||||||
|
if docker ps | grep -q dgraph-orly-test; then |
||||||
|
echo "✅ Dgraph is already running" |
||||||
|
echo "" |
||||||
|
echo "Dgraph endpoints:" |
||||||
|
echo " gRPC: localhost:9080" |
||||||
|
echo " HTTP: http://localhost:8080" |
||||||
|
echo " Ratel UI: http://localhost:8000" |
||||||
|
exit 0 |
||||||
|
fi |
||||||
|
|
||||||
|
# Determine docker-compose command |
||||||
|
if docker compose version &> /dev/null 2>&1; then |
||||||
|
DOCKER_COMPOSE="docker compose" |
||||||
|
else |
||||||
|
DOCKER_COMPOSE="docker-compose" |
||||||
|
fi |
||||||
|
|
||||||
|
# Start using docker compose |
||||||
|
$DOCKER_COMPOSE -f dgraph-docker-compose.yml up -d |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo "Waiting for dgraph to be healthy..." |
||||||
|
for i in {1..30}; do |
||||||
|
if docker exec dgraph-orly-test curl -sf http://localhost:8080/health > /dev/null 2>&1; then |
||||||
|
echo "✅ Dgraph is healthy and ready" |
||||||
|
echo "" |
||||||
|
echo "Dgraph endpoints:" |
||||||
|
echo " gRPC: localhost:9080" |
||||||
|
echo " HTTP: http://localhost:8080" |
||||||
|
echo " Ratel UI: http://localhost:8000" |
||||||
|
echo "" |
||||||
|
echo "To stop: $DOCKER_COMPOSE -f dgraph-docker-compose.yml down" |
||||||
|
echo "To view logs: docker logs dgraph-orly-test -f" |
||||||
|
exit 0 |
||||||
|
fi |
||||||
|
sleep 1 |
||||||
|
done |
||||||
|
|
||||||
|
echo "❌ Dgraph failed to become healthy" |
||||||
|
docker logs dgraph-orly-test |
||||||
|
exit 1 |
||||||
@ -0,0 +1,45 @@ |
|||||||
|
#!/bin/bash |
||||||
|
set -e |
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
||||||
|
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")" |
||||||
|
|
||||||
|
echo "=== Building ORLY Docker Images ===" |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Change to project root |
||||||
|
cd "$PROJECT_ROOT" |
||||||
|
|
||||||
|
# Determine docker-compose command |
||||||
|
if docker compose version &> /dev/null 2>&1; then |
||||||
|
DOCKER_COMPOSE="docker compose" |
||||||
|
else |
||||||
|
DOCKER_COMPOSE="docker-compose" |
||||||
|
fi |
||||||
|
|
||||||
|
# Build ORLY image |
||||||
|
echo "Building ORLY relay image..." |
||||||
|
docker build -t orly:latest -f Dockerfile . |
||||||
|
echo "✅ ORLY image built successfully" |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Build relay-tester image (optional) |
||||||
|
if [ "$1" == "--with-tester" ]; then |
||||||
|
echo "Building relay-tester image..." |
||||||
|
docker build -t orly-relay-tester:latest -f Dockerfile.relay-tester . |
||||||
|
echo "✅ Relay-tester image built successfully" |
||||||
|
echo "" |
||||||
|
fi |
||||||
|
|
||||||
|
# Show images |
||||||
|
echo "Built images:" |
||||||
|
docker images | grep -E "orly|REPOSITORY" |
||||||
|
echo "" |
||||||
|
|
||||||
|
echo "=== Build Complete ===" |
||||||
|
echo "" |
||||||
|
echo "To run:" |
||||||
|
echo " cd scripts && $DOCKER_COMPOSE -f docker-compose-test.yml up -d" |
||||||
|
echo "" |
||||||
|
echo "To test:" |
||||||
|
echo " ./scripts/test-docker.sh" |
||||||
@ -0,0 +1,93 @@ |
|||||||
|
version: '3.8' |
||||||
|
|
||||||
|
services: |
||||||
|
# Dgraph database |
||||||
|
dgraph: |
||||||
|
image: dgraph/standalone:latest |
||||||
|
container_name: orly-dgraph |
||||||
|
ports: |
||||||
|
- "8080:8080" # HTTP API |
||||||
|
- "9080:9080" # gRPC (ORLY connects here) |
||||||
|
- "8000:8000" # Ratel UI |
||||||
|
volumes: |
||||||
|
- dgraph-data:/dgraph |
||||||
|
environment: |
||||||
|
- DGRAPH_ALPHA_JAEGER_COLLECTOR=false |
||||||
|
networks: |
||||||
|
- orly-network |
||||||
|
healthcheck: |
||||||
|
test: ["CMD", "curl", "-f", "http://localhost:8080/health"] |
||||||
|
interval: 10s |
||||||
|
timeout: 5s |
||||||
|
retries: 5 |
||||||
|
start_period: 20s |
||||||
|
|
||||||
|
# ORLY relay with dgraph backend |
||||||
|
orly: |
||||||
|
build: |
||||||
|
context: .. |
||||||
|
dockerfile: Dockerfile |
||||||
|
container_name: orly-relay |
||||||
|
ports: |
||||||
|
- "3334:3334" # WebSocket/HTTP |
||||||
|
depends_on: |
||||||
|
dgraph: |
||||||
|
condition: service_healthy |
||||||
|
environment: |
||||||
|
# Database configuration |
||||||
|
- ORLY_DB_TYPE=dgraph |
||||||
|
- ORLY_DGRAPH_URL=dgraph:9080 |
||||||
|
- ORLY_DATA_DIR=/data |
||||||
|
|
||||||
|
# Server configuration |
||||||
|
- ORLY_LISTEN=0.0.0.0 |
||||||
|
- ORLY_PORT=3334 |
||||||
|
- ORLY_LOG_LEVEL=info |
||||||
|
- ORLY_APP_NAME=ORLY-Dgraph-Test |
||||||
|
|
||||||
|
# Admin configuration (example) |
||||||
|
- ORLY_ADMINS=npub1fjqqy4a93z5zsjwsfxqhc2764kvykfdyttvldkkkdera8dr78vhsmmleku |
||||||
|
- ORLY_OWNERS=npub1fjqqy4a93z5zsjwsfxqhc2764kvykfdyttvldkkkdera8dr78vhsmmleku |
||||||
|
|
||||||
|
# ACL mode |
||||||
|
- ORLY_ACL_MODE=none |
||||||
|
volumes: |
||||||
|
- orly-data:/data |
||||||
|
networks: |
||||||
|
- orly-network |
||||||
|
healthcheck: |
||||||
|
test: ["CMD", "curl", "-f", "http://localhost:3334/"] |
||||||
|
interval: 10s |
||||||
|
timeout: 5s |
||||||
|
retries: 5 |
||||||
|
start_period: 30s |
||||||
|
restart: unless-stopped |
||||||
|
|
||||||
|
# Relay tester (optional, for automated testing) |
||||||
|
relay-tester: |
||||||
|
build: |
||||||
|
context: .. |
||||||
|
dockerfile: Dockerfile.relay-tester |
||||||
|
container_name: orly-tester |
||||||
|
depends_on: |
||||||
|
orly: |
||||||
|
condition: service_healthy |
||||||
|
environment: |
||||||
|
- RELAY_URL=ws://orly:3334 |
||||||
|
networks: |
||||||
|
- orly-network |
||||||
|
profiles: |
||||||
|
- test |
||||||
|
|
||||||
|
networks: |
||||||
|
orly-network: |
||||||
|
driver: bridge |
||||||
|
ipam: |
||||||
|
config: |
||||||
|
- subnet: 172.28.0.0/16 |
||||||
|
|
||||||
|
volumes: |
||||||
|
dgraph-data: |
||||||
|
driver: local |
||||||
|
orly-data: |
||||||
|
driver: local |
||||||
@ -0,0 +1,308 @@ |
|||||||
|
#!/bin/bash |
||||||
|
# Badger Database Migration Script |
||||||
|
# Migrates ORLY database to new Badger configuration with VLogPercentile optimization |
||||||
|
|
||||||
|
set -e # Exit on error |
||||||
|
|
||||||
|
# Colors for output |
||||||
|
RED='\033[0;31m' |
||||||
|
GREEN='\033[0;32m' |
||||||
|
YELLOW='\033[1;33m' |
||||||
|
NC='\033[0m' # No Color |
||||||
|
|
||||||
|
echo -e "${GREEN}=== ORLY Badger Database Migration ===${NC}" |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Configuration |
||||||
|
DATA_DIR="${ORLY_DATA_DIR:-$HOME/.local/share/ORLY}" |
||||||
|
BACKUP_DIR="${DATA_DIR}-backup-$(date +%Y%m%d-%H%M%S)" |
||||||
|
EXPORT_FILE="${DATA_DIR}/events-export.jsonl" |
||||||
|
RELAY_BIN="${RELAY_BIN:-./orly}" |
||||||
|
|
||||||
|
# Check if relay binary exists |
||||||
|
if [ ! -f "$RELAY_BIN" ]; then |
||||||
|
echo -e "${RED}Error: ORLY binary not found at $RELAY_BIN${NC}" |
||||||
|
echo "Please build the relay first: go build -o orly" |
||||||
|
echo "Or set RELAY_BIN environment variable to the binary location" |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
|
||||||
|
# Check if database exists |
||||||
|
if [ ! -d "$DATA_DIR" ]; then |
||||||
|
echo -e "${YELLOW}Warning: Database directory not found at $DATA_DIR${NC}" |
||||||
|
echo "Nothing to migrate. If this is a fresh install, you can skip migration." |
||||||
|
exit 0 |
||||||
|
fi |
||||||
|
|
||||||
|
# Check disk space |
||||||
|
DB_SIZE=$(du -sb "$DATA_DIR" | cut -f1) |
||||||
|
AVAILABLE_SPACE=$(df "$HOME" | tail -1 | awk '{print $4}') |
||||||
|
AVAILABLE_SPACE=$((AVAILABLE_SPACE * 1024)) # Convert to bytes |
||||||
|
REQUIRED_SPACE=$((DB_SIZE * 3)) # 3x for safety (export + backup + new DB) |
||||||
|
|
||||||
|
echo "Database size: $(numfmt --to=iec-i --suffix=B $DB_SIZE)" |
||||||
|
echo "Available space: $(numfmt --to=iec-i --suffix=B $AVAILABLE_SPACE)" |
||||||
|
echo "Required space: $(numfmt --to=iec-i --suffix=B $REQUIRED_SPACE)" |
||||||
|
echo "" |
||||||
|
|
||||||
|
if [ $AVAILABLE_SPACE -lt $REQUIRED_SPACE ]; then |
||||||
|
echo -e "${RED}Error: Not enough disk space!${NC}" |
||||||
|
echo "Required: $(numfmt --to=iec-i --suffix=B $REQUIRED_SPACE)" |
||||||
|
echo "Available: $(numfmt --to=iec-i --suffix=B $AVAILABLE_SPACE)" |
||||||
|
echo "" |
||||||
|
echo "Options:" |
||||||
|
echo " 1. Free up disk space" |
||||||
|
echo " 2. Use natural compaction (no migration needed)" |
||||||
|
echo " 3. Export to external drive and import back" |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
|
||||||
|
# Check if relay is running |
||||||
|
if pgrep -x "orly" > /dev/null; then |
||||||
|
echo -e "${YELLOW}Warning: ORLY relay is currently running${NC}" |
||||||
|
echo "The relay should be stopped before migration." |
||||||
|
echo "" |
||||||
|
read -p "Stop the relay now? (y/N) " -n 1 -r |
||||||
|
echo |
||||||
|
if [[ $REPLY =~ ^[Yy]$ ]]; then |
||||||
|
echo "Attempting to stop relay..." |
||||||
|
if systemctl is-active --quiet orly; then |
||||||
|
sudo systemctl stop orly |
||||||
|
echo -e "${GREEN}Relay stopped via systemd${NC}" |
||||||
|
else |
||||||
|
pkill orly |
||||||
|
sleep 2 |
||||||
|
if pgrep -x "orly" > /dev/null; then |
||||||
|
echo -e "${RED}Failed to stop relay. Please stop it manually and try again.${NC}" |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
echo -e "${GREEN}Relay stopped${NC}" |
||||||
|
fi |
||||||
|
else |
||||||
|
echo "Please stop the relay and run this script again." |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
fi |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo -e "${YELLOW}=== Migration Plan ===${NC}" |
||||||
|
echo "1. Export all events to JSONL: $EXPORT_FILE" |
||||||
|
echo "2. Backup current database to: $BACKUP_DIR" |
||||||
|
echo "3. Create new database with optimized configuration" |
||||||
|
echo "4. Import all events (rebuilds indexes)" |
||||||
|
echo "5. Verify event counts match" |
||||||
|
echo "" |
||||||
|
echo "Estimated time: $(( (DB_SIZE / 1024 / 1024 / 100) + 1 )) - $(( (DB_SIZE / 1024 / 1024 / 50) + 1 )) minutes" |
||||||
|
echo "" |
||||||
|
read -p "Proceed with migration? (y/N) " -n 1 -r |
||||||
|
echo |
||||||
|
if [[ ! $REPLY =~ ^[Yy]$ ]]; then |
||||||
|
echo "Migration cancelled." |
||||||
|
exit 0 |
||||||
|
fi |
||||||
|
|
||||||
|
# Step 1: Export events |
||||||
|
echo "" |
||||||
|
echo -e "${GREEN}=== Step 1: Exporting Events ===${NC}" |
||||||
|
echo "This may take several minutes for large databases..." |
||||||
|
echo "" |
||||||
|
|
||||||
|
# We'll use a Go program to export since the binary doesn't have a CLI export command |
||||||
|
# Create temporary export program |
||||||
|
EXPORT_PROG=$(mktemp -d)/export-db.go |
||||||
|
cat > "$EXPORT_PROG" << 'EOF' |
||||||
|
package main |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"fmt" |
||||||
|
"os" |
||||||
|
"next.orly.dev/pkg/database" |
||||||
|
) |
||||||
|
|
||||||
|
func main() { |
||||||
|
if len(os.Args) < 3 { |
||||||
|
fmt.Fprintf(os.Stderr, "Usage: %s <data-dir> <output-file>\n", os.Args[0]) |
||||||
|
os.Exit(1) |
||||||
|
} |
||||||
|
|
||||||
|
dataDir := os.Args[1] |
||||||
|
outFile := os.Args[2] |
||||||
|
|
||||||
|
ctx := context.Background() |
||||||
|
cancel := func() {} |
||||||
|
|
||||||
|
db, err := database.New(ctx, cancel, dataDir, "error") |
||||||
|
if err != nil { |
||||||
|
fmt.Fprintf(os.Stderr, "Failed to open database: %v\n", err) |
||||||
|
os.Exit(1) |
||||||
|
} |
||||||
|
defer db.Close() |
||||||
|
|
||||||
|
f, err := os.Create(outFile) |
||||||
|
if err != nil { |
||||||
|
fmt.Fprintf(os.Stderr, "Failed to create output file: %v\n", err) |
||||||
|
os.Exit(1) |
||||||
|
} |
||||||
|
defer f.Close() |
||||||
|
|
||||||
|
fmt.Println("Exporting events...") |
||||||
|
db.Export(ctx, f) |
||||||
|
fmt.Println("Export complete!") |
||||||
|
} |
||||||
|
EOF |
||||||
|
|
||||||
|
# Build and run export program |
||||||
|
echo "Building export tool..." |
||||||
|
EXPORT_BIN=$(mktemp) |
||||||
|
if ! go build -o "$EXPORT_BIN" "$EXPORT_PROG" 2>&1; then |
||||||
|
echo -e "${RED}Failed to build export tool${NC}" |
||||||
|
rm -f "$EXPORT_PROG" "$EXPORT_BIN" |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
|
||||||
|
echo "Running export..." |
||||||
|
if ! "$EXPORT_BIN" "$DATA_DIR" "$EXPORT_FILE"; then |
||||||
|
echo -e "${RED}Export failed!${NC}" |
||||||
|
rm -f "$EXPORT_PROG" "$EXPORT_BIN" |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
|
||||||
|
rm -f "$EXPORT_PROG" "$EXPORT_BIN" |
||||||
|
|
||||||
|
# Count exported events |
||||||
|
EXPORT_COUNT=$(wc -l < "$EXPORT_FILE") |
||||||
|
echo -e "${GREEN}Exported $EXPORT_COUNT events${NC}" |
||||||
|
echo "Export size: $(du -h "$EXPORT_FILE" | cut -f1)" |
||||||
|
|
||||||
|
# Step 2: Backup current database |
||||||
|
echo "" |
||||||
|
echo -e "${GREEN}=== Step 2: Backing Up Current Database ===${NC}" |
||||||
|
echo "Moving $DATA_DIR to $BACKUP_DIR" |
||||||
|
mv "$DATA_DIR" "$BACKUP_DIR" |
||||||
|
echo -e "${GREEN}Backup complete${NC}" |
||||||
|
|
||||||
|
# Step 3 & 4: Create new database and import |
||||||
|
echo "" |
||||||
|
echo -e "${GREEN}=== Step 3 & 4: Creating New Database and Importing ===${NC}" |
||||||
|
echo "This will take longer as indexes are rebuilt..." |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Create temporary import program |
||||||
|
IMPORT_PROG=$(mktemp -d)/import-db.go |
||||||
|
cat > "$IMPORT_PROG" << 'EOF' |
||||||
|
package main |
||||||
|
|
||||||
|
import ( |
||||||
|
"context" |
||||||
|
"fmt" |
||||||
|
"os" |
||||||
|
"next.orly.dev/pkg/database" |
||||||
|
) |
||||||
|
|
||||||
|
func main() { |
||||||
|
if len(os.Args) < 3 { |
||||||
|
fmt.Fprintf(os.Stderr, "Usage: %s <data-dir> <import-file>\n", os.Args[0]) |
||||||
|
os.Exit(1) |
||||||
|
} |
||||||
|
|
||||||
|
dataDir := os.Args[1] |
||||||
|
importFile := os.Args[2] |
||||||
|
|
||||||
|
ctx := context.Background() |
||||||
|
cancel := func() {} |
||||||
|
|
||||||
|
// This will create new database with updated configuration from database.go |
||||||
|
db, err := database.New(ctx, cancel, dataDir, "info") |
||||||
|
if err != nil { |
||||||
|
fmt.Fprintf(os.Stderr, "Failed to create database: %v\n", err) |
||||||
|
os.Exit(1) |
||||||
|
} |
||||||
|
defer db.Close() |
||||||
|
|
||||||
|
f, err := os.Open(importFile) |
||||||
|
if err != nil { |
||||||
|
fmt.Fprintf(os.Stderr, "Failed to open import file: %v\n", err) |
||||||
|
os.Exit(1) |
||||||
|
} |
||||||
|
defer f.Close() |
||||||
|
|
||||||
|
fmt.Println("Importing events (this may take a while)...") |
||||||
|
db.Import(f) |
||||||
|
|
||||||
|
// Wait for import to complete |
||||||
|
fmt.Println("Import started. Waiting for completion...") |
||||||
|
fmt.Println("Check the log output above for progress (logged every 100 events)") |
||||||
|
} |
||||||
|
EOF |
||||||
|
|
||||||
|
# Build and run import program |
||||||
|
echo "Building import tool..." |
||||||
|
IMPORT_BIN=$(mktemp) |
||||||
|
if ! go build -o "$IMPORT_BIN" "$IMPORT_PROG" 2>&1; then |
||||||
|
echo -e "${RED}Failed to build import tool${NC}" |
||||||
|
echo "Rolling back..." |
||||||
|
mv "$BACKUP_DIR" "$DATA_DIR" |
||||||
|
rm -f "$IMPORT_PROG" "$IMPORT_BIN" |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
|
||||||
|
echo "Running import..." |
||||||
|
if ! "$IMPORT_BIN" "$DATA_DIR" "$EXPORT_FILE"; then |
||||||
|
echo -e "${RED}Import failed!${NC}" |
||||||
|
echo "Rolling back..." |
||||||
|
rm -rf "$DATA_DIR" |
||||||
|
mv "$BACKUP_DIR" "$DATA_DIR" |
||||||
|
rm -f "$IMPORT_PROG" "$IMPORT_BIN" |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
|
||||||
|
rm -f "$IMPORT_PROG" "$IMPORT_BIN" |
||||||
|
|
||||||
|
# Give import goroutine time to process |
||||||
|
echo "Waiting for import to complete..." |
||||||
|
sleep 10 |
||||||
|
|
||||||
|
# Step 5: Verify |
||||||
|
echo "" |
||||||
|
echo -e "${GREEN}=== Step 5: Verification ===${NC}" |
||||||
|
|
||||||
|
NEW_DB_SIZE=$(du -sb "$DATA_DIR" | cut -f1) |
||||||
|
echo "Old database size: $(numfmt --to=iec-i --suffix=B $DB_SIZE)" |
||||||
|
echo "New database size: $(numfmt --to=iec-i --suffix=B $NEW_DB_SIZE)" |
||||||
|
echo "" |
||||||
|
|
||||||
|
if [ $NEW_DB_SIZE -lt $((DB_SIZE / 10)) ]; then |
||||||
|
echo -e "${YELLOW}Warning: New database is suspiciously small${NC}" |
||||||
|
echo "This may indicate an incomplete import." |
||||||
|
echo "Check the logs in $DATA_DIR/migration.log" |
||||||
|
echo "" |
||||||
|
read -p "Continue anyway? (y/N) " -n 1 -r |
||||||
|
echo |
||||||
|
if [[ ! $REPLY =~ ^[Yy]$ ]]; then |
||||||
|
echo "Rolling back..." |
||||||
|
rm -rf "$DATA_DIR" |
||||||
|
mv "$BACKUP_DIR" "$DATA_DIR" |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
fi |
||||||
|
|
||||||
|
echo -e "${GREEN}=== Migration Complete! ===${NC}" |
||||||
|
echo "" |
||||||
|
echo "Summary:" |
||||||
|
echo " - Exported: $EXPORT_COUNT events" |
||||||
|
echo " - Old DB size: $(numfmt --to=iec-i --suffix=B $DB_SIZE)" |
||||||
|
echo " - New DB size: $(numfmt --to=iec-i --suffix=B $NEW_DB_SIZE)" |
||||||
|
echo " - Space saved: $(numfmt --to=iec-i --suffix=B $((DB_SIZE - NEW_DB_SIZE)))" |
||||||
|
echo " - Backup location: $BACKUP_DIR" |
||||||
|
echo "" |
||||||
|
echo "Next steps:" |
||||||
|
echo " 1. Start the relay: sudo systemctl start orly (or ./orly)" |
||||||
|
echo " 2. Monitor performance for 24-48 hours" |
||||||
|
echo " 3. Watch for cache hit ratio >85% in logs" |
||||||
|
echo " 4. Verify event count and queries work correctly" |
||||||
|
echo " 5. After verification, remove backup: rm -rf $BACKUP_DIR" |
||||||
|
echo "" |
||||||
|
echo "Rollback (if needed):" |
||||||
|
echo " Stop relay, then: rm -rf $DATA_DIR && mv $BACKUP_DIR $DATA_DIR" |
||||||
|
echo "" |
||||||
@ -0,0 +1,88 @@ |
|||||||
|
#!/bin/bash |
||||||
|
set -e |
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
||||||
|
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")" |
||||||
|
|
||||||
|
echo "=== ORLY Dgraph Integration Test Suite ===" |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Check if dgraph is running |
||||||
|
echo "Checking for dgraph server..." |
||||||
|
DGRAPH_URL="${ORLY_DGRAPH_URL:-localhost:9080}" |
||||||
|
|
||||||
|
if ! timeout 2 bash -c "echo > /dev/tcp/${DGRAPH_URL%:*}/${DGRAPH_URL#*:}" 2>/dev/null; then |
||||||
|
echo "❌ Dgraph server not available at $DGRAPH_URL" |
||||||
|
echo "" |
||||||
|
echo "To start dgraph using docker-compose:" |
||||||
|
echo " cd $SCRIPT_DIR && docker-compose -f dgraph-docker-compose.yml up -d" |
||||||
|
echo "" |
||||||
|
echo "Or using docker directly:" |
||||||
|
echo " docker run -d -p 8080:8080 -p 9080:9080 -p 8000:8000 --name dgraph-orly dgraph/standalone:latest" |
||||||
|
echo "" |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
|
||||||
|
echo "✅ Dgraph server is running at $DGRAPH_URL" |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Run dgraph tests |
||||||
|
echo "Running dgraph package tests..." |
||||||
|
cd "$PROJECT_ROOT" |
||||||
|
CGO_ENABLED=0 go test -v -timeout 10m ./pkg/dgraph/... || { |
||||||
|
echo "❌ Dgraph tests failed" |
||||||
|
exit 1 |
||||||
|
} |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo "✅ All dgraph tests passed!" |
||||||
|
echo "" |
||||||
|
|
||||||
|
# Optional: Run relay-tester if requested |
||||||
|
if [ "$1" == "--relay-tester" ]; then |
||||||
|
echo "Starting ORLY with dgraph backend..." |
||||||
|
export ORLY_DB_TYPE=dgraph |
||||||
|
export ORLY_DGRAPH_URL="$DGRAPH_URL" |
||||||
|
export ORLY_LOG_LEVEL=info |
||||||
|
export ORLY_PORT=3334 |
||||||
|
|
||||||
|
# Kill any existing ORLY instance |
||||||
|
pkill -f "./orly" || true |
||||||
|
sleep 1 |
||||||
|
|
||||||
|
# Start ORLY in background |
||||||
|
./orly & |
||||||
|
ORLY_PID=$! |
||||||
|
|
||||||
|
# Wait for ORLY to start |
||||||
|
echo "Waiting for ORLY to start..." |
||||||
|
for i in {1..30}; do |
||||||
|
if curl -s http://localhost:3334 > /dev/null 2>&1; then |
||||||
|
echo "✅ ORLY started successfully" |
||||||
|
break |
||||||
|
fi |
||||||
|
sleep 1 |
||||||
|
if [ $i -eq 30 ]; then |
||||||
|
echo "❌ ORLY failed to start" |
||||||
|
kill $ORLY_PID 2>/dev/null || true |
||||||
|
exit 1 |
||||||
|
fi |
||||||
|
done |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo "Running relay-tester against dgraph backend..." |
||||||
|
go run cmd/relay-tester/main.go -url ws://localhost:3334 || { |
||||||
|
echo "❌ Relay-tester failed" |
||||||
|
kill $ORLY_PID 2>/dev/null || true |
||||||
|
exit 1 |
||||||
|
} |
||||||
|
|
||||||
|
# Clean up |
||||||
|
kill $ORLY_PID 2>/dev/null || true |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo "✅ Relay-tester passed!" |
||||||
|
fi |
||||||
|
|
||||||
|
echo "" |
||||||
|
echo "=== All tests completed successfully! ===" |
||||||
Loading…
Reference in new issue