next.orly.dev/docs/plans/NIP33_QUERY_OPTIMIZATION.md

Plan: NIP-33 Parameterized Replaceable Event Query Optimization

Issue Reference

https://git.nostrdev.com/mleku/next.orly.dev/issues/29

Problem Statement

Queries filtering by kinds, authors, and d-tags for NIP-33 parameterized replaceable events (kinds 30000-39999) take 10+ seconds when they should be nearly instantaneous.

Example slow query:

["REQ","UserCards",{"kinds":[30382],"authors":["89330ec6f7fa08717d36d203e74644d7cccbda280547e5a3ce81ff389110d2e5"],"#d":["55f04590674f3648f4cdc9dc8ce32da2a282074cd0b020596ee033d12d385185"]}]

This query uniquely identifies a single event by its NIP-33 address (kind + pubkey + d-tag).

Root Cause Analysis

Current Query Path

1. Query enters GetIndexesFromFilter() (get-indexes-from-filter.go:199)
2. Detects all three criteria (kinds, authors, tags) → Uses TagKindPubkey index
3. TagKindPubkey index format: tkp | tag_letter | value_hash | kind | pubkey_hash | timestamp | serial
4. Query iterates backwards through all timestamps to find matching events
5. With large databases, this timestamp iteration is extremely slow

Unused Optimization

An AddressableEvent index exists but is never used for queries:

// indexes/keys.go:300-315
// Key format: aev | pubkey_hash | kind | dtag_hash
var AddressableEvent = next()

This index is:

• ✅ Defined in indexes/keys.go
• ✅ Created during migrations (migrations.go:519)
• ❌ NOT written during save-event
• ❌ NOT used in GetIndexesFromFilter

Solution Design

Phase 1: Enable AddressableEvent Index Writes

File: pkg/database/get-indexes-for-event.go

Add AddressableEvent index writes for parameterized replaceable events:

// In getIndexesForEvent(), after existing index writes:

// For parameterized replaceable events (kinds 30000-39999), write AddressableEvent index
if kind.IsParameterizedReplaceable(ev.Kind) {
    dTag := ev.Tags.GetFirst([]byte("d"))
    if dTag != nil {
        dValue := dTag.Value()
        dTagHash := new(types.Ident)
        dTagHash.FromIdent(dValue)

        aevKey := indexes.AddressableEventEnc(pubHash, kindVal, dTagHash)
        aevKeyBuf := new(bytes.Buffer)
        if err = aevKey.MarshalWrite(aevKeyBuf); chk.E(err) {
            return
        }
        // Store serial as value for direct lookup
        serialBytes := make([]byte, 5)
        serial.Put(serialBytes)
        keys = append(keys, KeyValue{
            Key:   aevKeyBuf.Bytes(),
            Value: serialBytes,
        })
    }
}

Phase 2: Fast Path for NIP-33 Queries

File: pkg/database/query-for-ids.go or new file query-addressable.go

Add detection and fast path for NIP-33 queries:

// QueryForAddressableEvent performs a direct lookup for NIP-33 events
func (d *D) QueryForAddressableEvent(f *filter.F) (serial types.Uint40, found bool, err error) {
    // Check if this is a NIP-33 query pattern
    if !isAddressableEventQuery(f) {
        return 0, false, nil
    }

    // Extract components
    kindVal := f.Kinds.T[0]
    if !kind.IsParameterizedReplaceable(kindVal) {
        return 0, false, nil
    }

    author := f.Authors.T[0]
    pubHash, err := CreatePubHashFromData(author)
    if err != nil {
        return 0, false, err
    }

    dTag := f.Tags.GetFirst([]byte("#d"))
    if dTag == nil || dTag.Len() < 2 {
        return 0, false, nil
    }
    dValue := dTag.T[1]
    dTagHash := new(types.Ident)
    dTagHash.FromIdent(dValue)

    // Build direct lookup key
    kindType := new(types.Uint16)
    kindType.Set(kindVal)

    aevKey := indexes.AddressableEventEnc(pubHash, kindType, dTagHash)
    keyBuf := new(bytes.Buffer)
    if err = aevKey.MarshalWrite(keyBuf); chk.E(err) {
        return 0, false, err
    }

    // Direct key lookup - O(1)
    err = d.db.View(func(txn *badger.Txn) error {
        item, err := txn.Get(keyBuf.Bytes())
        if err == badger.ErrKeyNotFound {
            return nil
        }
        if err != nil {
            return err
        }
        return item.Value(func(val []byte) error {
            if len(val) >= 5 {
                serial.Get(val)
                found = true
            }
            return nil
        })
    })

    return serial, found, err
}

func isAddressableEventQuery(f *filter.F) bool {
    // Must have exactly one kind, one author, and one d-tag
    if f.Kinds == nil || f.Kinds.Len() != 1 {
        return false
    }
    if f.Authors == nil || f.Authors.Len() != 1 {
        return false
    }
    if f.Tags == nil {
        return false
    }
    dTag := f.Tags.GetFirst([]byte("#d"))
    if dTag == nil || dTag.Len() != 2 {
        return false
    }
    // Must be a parameterized replaceable kind
    return kind.IsParameterizedReplaceable(f.Kinds.T[0])
}

Phase 3: Integrate Fast Path into Query Flow

File: pkg/database/query-events.go

Add fast path check at the beginning of QueryEvents:

func (d *D) QueryEvents(f *filter.F, ...) (res iter.I, err error) {
    // Fast path for NIP-33 addressable event lookups
    if serial, found, err := d.QueryForAddressableEvent(f); err == nil && found {
        ev, err := d.FetchEventBySerial(serial)
        if err == nil && ev != nil {
            // Apply any remaining filters (since, until) and return
            if f.Since != nil && ev.CreatedAt < *f.Since {
                return iter.Empty(), nil
            }
            if f.Until != nil && ev.CreatedAt > *f.Until {
                return iter.Empty(), nil
            }
            return iter.Single(ev), nil
        }
    }

    // Fall through to existing query logic
    ...
}

Phase 4: Optimize WouldReplaceEvent

File: pkg/database/save-event.go

The WouldReplaceEvent function (line 70) also benefits from this optimization since it constructs the same filter pattern:

func (d *D) WouldReplaceEvent(ev *event.E) (bool, types.Uint40s, error) {
    if kind.IsParameterizedReplaceable(ev.Kind) {
        dTag := ev.Tags.GetFirst([]byte("d"))
        if dTag == nil {
            return false, nil, ErrMissingDTag
        }

        // Use fast path for NIP-33 lookup
        f := &filter.F{
            Authors: tag.NewFromBytesSlice(ev.Pubkey),
            Kinds:   kind.NewS(kind.New(ev.Kind)),
            Tags:    tag.NewS(tag.NewFromAny("d", dTag.Value())),
        }

        if serial, found, err := d.QueryForAddressableEvent(f); err == nil && found {
            oldEv, err := d.FetchEventBySerial(serial)
            if err == nil && oldEv != nil {
                if ev.CreatedAt < oldEv.CreatedAt {
                    return false, types.Uint40s{serial}, nil
                }
                return true, nil, nil
            }
        }
    }

    // Fall through to existing logic for regular replaceables
    ...
}

Phase 5: Handle Index Updates on Replace

When a parameterized replaceable event is replaced, the old AddressableEvent index entry must be deleted:

File: pkg/database/save-event.go

In the replacement logic, delete the old AddressableEvent entry:

// When replacing a parameterized replaceable event:
if kind.IsParameterizedReplaceable(ev.Kind) && len(serialsToDelete) > 0 {
    // Delete old AddressableEvent index entries
    for _, oldSerial := range serialsToDelete {
        oldEv, _ := d.FetchEventBySerial(oldSerial)
        if oldEv != nil {
            dTag := oldEv.Tags.GetFirst([]byte("d"))
            if dTag != nil {
                // Build and delete old index key
                ...
            }
        }
    }
}

Migration Strategy

For Existing Data

Events stored before this optimization won't have AddressableEvent index entries. Options:

1. Background rebuild (recommended): Add a migration that scans existing events and builds AddressableEvent indexes
2. Lazy rebuild: Fall back to existing query path if AddressableEvent lookup fails
3. Full reindex: Trigger a full database reindex on upgrade

Recommended approach: Implement lazy rebuild with background migration:

func (d *D) QueryForAddressableEvent(f *filter.F) (serial types.Uint40, found bool, err error) {
    // Try direct lookup first
    serial, found, err = d.directAddressableLookup(f)
    if found || err != nil {
        return
    }

    // Fall back to existing query path (slow but correct)
    sers, err := d.GetSerialsFromFilter(f)
    if err != nil || len(sers) == 0 {
        return 0, false, err
    }

    // Opportunistically build the index for next time
    go d.buildAddressableIndex(f, sers[0])

    return sers[0], true, nil
}

Performance Impact

Operation	Before	After
NIP-33 query (kind+author+d-tag)	10+ seconds (index scan)	<1ms (direct lookup)
WouldReplaceEvent check	10+ seconds	<1ms
Event save (NIP-33)	N/A	+1 index write (~µs)
Storage overhead	N/A	~20 bytes per NIP-33 event

Files to Modify

1. pkg/database/indexes/keys.go - Already defined, no changes needed
2. pkg/database/get-indexes-for-event.go - Add AddressableEvent index writes
3. pkg/database/query-addressable.go - New file for fast path logic
4. pkg/database/query-events.go - Integrate fast path
5. pkg/database/save-event.go - Optimize WouldReplaceEvent, handle index updates
6. pkg/database/migrations.go - Add migration for existing data

Testing

1. Unit tests:

   • TestQueryAddressableEvent - Direct lookup works
   • TestAddressableEventIndexWrite - Index written on save
   • TestAddressableEventReplace - Index updated on replace

2. Benchmark tests:

   • Compare query time before/after for NIP-33 patterns
   • Measure with varying database sizes (100k, 1M, 10M events)

3. Integration tests:

   • Verify compatibility with existing queries
   • Test migration on real database

Rollout Plan

1. Implement Phase 1-2 with lazy rebuild (safe, no migration needed)
2. Add background migration for existing databases
3. Monitor performance metrics
4. Remove lazy rebuild fallback once migration complete