Legacy events stored before compact format (v6 migration) may have IDs
that start with byte 0x01. The v6 migration incorrectly skipped these
events because it checked `eventData[0] == CompactFormatVersion` before
attempting to decode, causing them to never receive SerialEventId (sei)
mappings.
This fix:
1. Changes compact format detection to try sei lookup first - if the
mapping doesn't exist, fall back to legacy binary format instead
of returning an error
2. Adds v9 migration (BackfillMissingSerialEventIdMappings) that finds
legacy events with IDs starting with 0x01 that were skipped by the
v6 migration and creates their sei mappings
3. Quiets the "Key not found" logging for sei lookups since missing
mappings are now expected and handled gracefully
Files modified:
- pkg/database/fetch-events-by-serials.go: Fallback to legacy format
- pkg/database/fetch-event-by-serial.go: Fallback to legacy format
- pkg/database/export.go: Fallback to legacy format
- pkg/database/serial_cache.go: Quiet sei lookup logging
- pkg/database/migrations.go: Add v9 sei backfill migration
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add query semaphore limiting concurrent database queries to 3
- Reuse single iterator in FetchEventsBySerials instead of one per serial
- Disable expensive e-tag fallback lookup in deletion processing
- Add empty ID validation in access tracker loop
- Reduce log spam from GetSerialById empty ID errors
These changes reduce memory usage from ~5GB to ~150MB under load by
limiting concurrent Badger iterators which consume significant memory.
Files modified:
- pkg/database/database.go: Add query semaphore with acquire/release
- pkg/database/fetch-events-by-serials.go: Reuse iterator for sev lookups
- pkg/database/query-events.go: Add semaphore, disable e-tag fallback
- pkg/database/get-serial-by-id.go: Don't log empty ID errors
- app/handle-req.go: Validate event ID before GetSerialById
- pkg/version/version: Bump to v0.52.17
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add validation in GetEventIdBySerial to ensure sei value is 32 bytes
- Fix fallback-to-legacy bug: return error instead of attempting legacy
unmarshal on compact format data when event ID lookup fails
- Add upfront validation in UnmarshalCompactEvent for eventId length
- Prevents events with all-zero IDs from being returned to clients
Files modified:
- pkg/database/serial_cache.go: Validate sei value is exactly 32 bytes
- pkg/database/fetch-events-by-serials.go: Return error for compact format
when eventId missing instead of falling back to legacy unmarshal
- pkg/database/fetch-event-by-serial.go: Same fix for single event fetch
- pkg/database/compact_event.go: Validate eventId is 32 bytes upfront
- pkg/version/version: Bump to v0.47.1
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Introduced benchmark tests for various database operations, including event saving, querying, and fetching by serials, to assess performance.
- Implemented optimizations to reduce memory allocations and improve efficiency by pre-allocating slices and maps in critical functions.
- Enhanced the `FetchEventsBySerials`, `GetFullIdPubkeyBySerials`, and `QueryForIds` methods with pre-allocation strategies to minimize reallocations.
- Documented performance improvements in the new PERFORMANCE_REPORT.md file, highlighting significant reductions in execution time and memory usage.
- Bumped version to v0.23.1 to reflect these changes.
- Included results for `relayer-basic`, `strfry`, and `nostr-rs-relay` relay benchmarks.
- Comprehensive performance metrics added for throughput, latency, query, and concurrent operations.
- Reports saved as plain text and AsciiDoc formats.
- Updated `publishCacheEvents` to utilize multiple concurrent connections for event publishing.
- Introduced worker-based architecture leveraging `runtime.NumCPU` for parallel uploads.
- Optimized database fetch logic in `FetchEventsBySerials` for improved maintainability and performance.
- Bumped version to `v0.4.8`.
Removed old benchmark reports and detailed logs from the repository to clean up unnecessary files. These reports appear to be auto-generated and no longer relevant for ongoing development.
woikos
mleku