Add optimization plan for NIP-33 query performance (issue #29)

Problem: Queries with kinds + authors + d-tags take 10+ seconds Root cause: Using TagKindPubkey index requires timestamp iteration Solution: Use AddressableEvent index for direct O(1) lookup Plan covers: - Phase 1: Enable AddressableEvent index writes on save - Phase 2: Add fast path query for NIP-33 lookups - Phase 3: Integrate into QueryEvents flow - Phase 4: Optimize WouldReplaceEvent - Phase 5: Handle index updates on replace - Migration strategy for existing data Expected improvement: 10+ seconds → <1ms Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
4 months ago · 3c0c3aa744
1 changed files with 324 additions and 0 deletions
--- a/docs/plans/NIP33_QUERY_OPTIMIZATION.md
+++ b/docs/plans/NIP33_QUERY_OPTIMIZATION.md
@ -0,0 +1,324 @@
 # Plan: NIP-33 Parameterized Replaceable Event Query Optimization
 ## Issue Reference
 https://git.nostrdev.com/mleku/next.orly.dev/issues/29
 ## Problem Statement
 Queries filtering by kinds, authors, and d-tags for NIP-33 parameterized replaceable events (kinds 30000-39999) take 10+ seconds when they should be nearly instantaneous.
 **Example slow query:**
 ```json
 ["REQ","UserCards",{"kinds":[30382],"authors":["89330ec6f7fa08717d36d203e74644d7cccbda280547e5a3ce81ff389110d2e5"],"#d":["55f04590674f3648f4cdc9dc8ce32da2a282074cd0b020596ee033d12d385185"]}]
 ```
 This query uniquely identifies a single event by its NIP-33 address (kind + pubkey + d-tag).
 ## Root Cause Analysis
 ### Current Query Path
 1. Query enters `GetIndexesFromFilter()` (get-indexes-from-filter.go:199)
 2. Detects all three criteria (kinds, authors, tags) → Uses `TagKindPubkey` index
 3. `TagKindPubkey` index format: `tkp | tag_letter | value_hash | kind | pubkey_hash | timestamp | serial`
 4. Query iterates **backwards through all timestamps** to find matching events
 5. With large databases, this timestamp iteration is extremely slow
 ### Unused Optimization
 An `AddressableEvent` index exists but is **never used for queries**:
 ```go
 // indexes/keys.go:300-315
 // Key format: aev | pubkey_hash | kind | dtag_hash
 var AddressableEvent = next()
 ```
 This index is:
 - ✅ Defined in indexes/keys.go
 - ✅ Created during migrations (migrations.go:519)
 - ❌ **NOT written during save-event**
 - ❌ **NOT used in GetIndexesFromFilter**
 ## Solution Design
 ### Phase 1: Enable AddressableEvent Index Writes
 **File:** `pkg/database/get-indexes-for-event.go`
 Add AddressableEvent index writes for parameterized replaceable events:
 ```go
 // In getIndexesForEvent(), after existing index writes:
 // For parameterized replaceable events (kinds 30000-39999), write AddressableEvent index
 if kind.IsParameterizedReplaceable(ev.Kind) {
    dTag := ev.Tags.GetFirst([]byte("d"))
    if dTag != nil {
        dValue := dTag.Value()
        dTagHash := new(types.Ident)
        dTagHash.FromIdent(dValue)
        aevKey := indexes.AddressableEventEnc(pubHash, kindVal, dTagHash)
        aevKeyBuf := new(bytes.Buffer)
        if err = aevKey.MarshalWrite(aevKeyBuf); chk.E(err) {
            return
        }
        // Store serial as value for direct lookup
        serialBytes := make([]byte, 5)
        serial.Put(serialBytes)
        keys = append(keys, KeyValue{
            Key:   aevKeyBuf.Bytes(),
            Value: serialBytes,
        })
    }
 }
 ```
 ### Phase 2: Fast Path for NIP-33 Queries
 **File:** `pkg/database/query-for-ids.go` or new file `query-addressable.go`
 Add detection and fast path for NIP-33 queries:
 ```go
 // QueryForAddressableEvent performs a direct lookup for NIP-33 events
 func (d *D) QueryForAddressableEvent(f *filter.F) (serial types.Uint40, found bool, err error) {
    // Check if this is a NIP-33 query pattern
    if !isAddressableEventQuery(f) {
        return 0, false, nil
    }
    // Extract components
    kindVal := f.Kinds.T[0]
    if !kind.IsParameterizedReplaceable(kindVal) {
        return 0, false, nil
    }
    author := f.Authors.T[0]
    pubHash, err := CreatePubHashFromData(author)
    if err != nil {
        return 0, false, err
    }
    dTag := f.Tags.GetFirst([]byte("#d"))
    if dTag == nil || dTag.Len() < 2 {
        return 0, false, nil
    }
    dValue := dTag.T[1]
    dTagHash := new(types.Ident)
    dTagHash.FromIdent(dValue)
    // Build direct lookup key
    kindType := new(types.Uint16)
    kindType.Set(kindVal)
    aevKey := indexes.AddressableEventEnc(pubHash, kindType, dTagHash)
    keyBuf := new(bytes.Buffer)
    if err = aevKey.MarshalWrite(keyBuf); chk.E(err) {
        return 0, false, err
    }
    // Direct key lookup - O(1)
    err = d.db.View(func(txn *badger.Txn) error {
        item, err := txn.Get(keyBuf.Bytes())
        if err == badger.ErrKeyNotFound {
            return nil
        }
        if err != nil {
            return err
        }
        return item.Value(func(val []byte) error {
            if len(val) >= 5 {
                serial.Get(val)
                found = true
            }
            return nil
        })
    })
    return serial, found, err
 }
 func isAddressableEventQuery(f *filter.F) bool {
    // Must have exactly one kind, one author, and one d-tag
    if f.Kinds == nil || f.Kinds.Len() != 1 {
        return false
    }
    if f.Authors == nil || f.Authors.Len() != 1 {
        return false
    }
    if f.Tags == nil {
        return false
    }
    dTag := f.Tags.GetFirst([]byte("#d"))
    if dTag == nil || dTag.Len() != 2 {
        return false
    }
    // Must be a parameterized replaceable kind
    return kind.IsParameterizedReplaceable(f.Kinds.T[0])
 }
 ```
 ### Phase 3: Integrate Fast Path into Query Flow
 **File:** `pkg/database/query-events.go`
 Add fast path check at the beginning of QueryEvents:
 ```go
 func (d *D) QueryEvents(f *filter.F, ...) (res iter.I, err error) {
    // Fast path for NIP-33 addressable event lookups
    if serial, found, err := d.QueryForAddressableEvent(f); err == nil && found {
        ev, err := d.FetchEventBySerial(serial)
        if err == nil && ev != nil {
            // Apply any remaining filters (since, until) and return
            if f.Since != nil && ev.CreatedAt < *f.Since {
                return iter.Empty(), nil
            }
            if f.Until != nil && ev.CreatedAt > *f.Until {
                return iter.Empty(), nil
            }
            return iter.Single(ev), nil
        }
    }
    // Fall through to existing query logic
    ...
 }
 ```
 ### Phase 4: Optimize WouldReplaceEvent
 **File:** `pkg/database/save-event.go`
 The `WouldReplaceEvent` function (line 70) also benefits from this optimization since it constructs the same filter pattern:
 ```go
 func (d *D) WouldReplaceEvent(ev *event.E) (bool, types.Uint40s, error) {
    if kind.IsParameterizedReplaceable(ev.Kind) {
        dTag := ev.Tags.GetFirst([]byte("d"))
        if dTag == nil {
            return false, nil, ErrMissingDTag
        }
        // Use fast path for NIP-33 lookup
        f := &filter.F{
            Authors: tag.NewFromBytesSlice(ev.Pubkey),
            Kinds:   kind.NewS(kind.New(ev.Kind)),
            Tags:    tag.NewS(tag.NewFromAny("d", dTag.Value())),
        }
        if serial, found, err := d.QueryForAddressableEvent(f); err == nil && found {
            oldEv, err := d.FetchEventBySerial(serial)
            if err == nil && oldEv != nil {
                if ev.CreatedAt < oldEv.CreatedAt {
                    return false, types.Uint40s{serial}, nil
                }
                return true, nil, nil
            }
        }
    }
    // Fall through to existing logic for regular replaceables
    ...
 }
 ```
 ### Phase 5: Handle Index Updates on Replace
 When a parameterized replaceable event is replaced, the old AddressableEvent index entry must be deleted:
 **File:** `pkg/database/save-event.go`
 In the replacement logic, delete the old AddressableEvent entry:
 ```go
 // When replacing a parameterized replaceable event:
 if kind.IsParameterizedReplaceable(ev.Kind) && len(serialsToDelete) > 0 {
    // Delete old AddressableEvent index entries
    for _, oldSerial := range serialsToDelete {
        oldEv, _ := d.FetchEventBySerial(oldSerial)
        if oldEv != nil {
            dTag := oldEv.Tags.GetFirst([]byte("d"))
            if dTag != nil {
                // Build and delete old index key
                ...
            }
        }
    }
 }
 ```
 ## Migration Strategy
 ### For Existing Data
 Events stored before this optimization won't have AddressableEvent index entries. Options:
 1. **Background rebuild** (recommended): Add a migration that scans existing events and builds AddressableEvent indexes
 2. **Lazy rebuild**: Fall back to existing query path if AddressableEvent lookup fails
 3. **Full reindex**: Trigger a full database reindex on upgrade
 **Recommended approach:** Implement lazy rebuild with background migration:
 ```go
 func (d *D) QueryForAddressableEvent(f *filter.F) (serial types.Uint40, found bool, err error) {
    // Try direct lookup first
    serial, found, err = d.directAddressableLookup(f)
    if found || err != nil {
        return
    }
    // Fall back to existing query path (slow but correct)
    sers, err := d.GetSerialsFromFilter(f)
    if err != nil || len(sers) == 0 {
        return 0, false, err
    }
    // Opportunistically build the index for next time
    go d.buildAddressableIndex(f, sers[0])
    return sers[0], true, nil
 }
 ```
 ## Performance Impact
 | Operation | Before | After |
 |-----------|--------|-------|
 | NIP-33 query (kind+author+d-tag) | 10+ seconds (index scan) | <1ms (direct lookup) |
 | WouldReplaceEvent check | 10+ seconds | <1ms |
 | Event save (NIP-33) | N/A | +1 index write (~µs) |
 | Storage overhead | N/A | ~20 bytes per NIP-33 event |
 ## Files to Modify
 1. `pkg/database/indexes/keys.go` - Already defined, no changes needed
 2. `pkg/database/get-indexes-for-event.go` - Add AddressableEvent index writes
 3. `pkg/database/query-addressable.go` - New file for fast path logic
 4. `pkg/database/query-events.go` - Integrate fast path
 5. `pkg/database/save-event.go` - Optimize WouldReplaceEvent, handle index updates
 6. `pkg/database/migrations.go` - Add migration for existing data
 ## Testing
 1. **Unit tests:**
   - `TestQueryAddressableEvent` - Direct lookup works
   - `TestAddressableEventIndexWrite` - Index written on save
   - `TestAddressableEventReplace` - Index updated on replace
 2. **Benchmark tests:**
   - Compare query time before/after for NIP-33 patterns
   - Measure with varying database sizes (100k, 1M, 10M events)
 3. **Integration tests:**
   - Verify compatibility with existing queries
   - Test migration on real database
 ## Rollout Plan
 1. Implement Phase 1-2 with lazy rebuild (safe, no migration needed)
 2. Add background migration for existing databases
 3. Monitor performance metrics
 4. Remove lazy rebuild fallback once migration complete