Legacy events stored before compact format (v6 migration) may have IDs
that start with byte 0x01. The v6 migration incorrectly skipped these
events because it checked `eventData[0] == CompactFormatVersion` before
attempting to decode, causing them to never receive SerialEventId (sei)
mappings.
This fix:
1. Changes compact format detection to try sei lookup first - if the
mapping doesn't exist, fall back to legacy binary format instead
of returning an error
2. Adds v9 migration (BackfillMissingSerialEventIdMappings) that finds
legacy events with IDs starting with 0x01 that were skipped by the
v6 migration and creates their sei mappings
3. Quiets the "Key not found" logging for sei lookups since missing
mappings are now expected and handled gracefully
Files modified:
- pkg/database/fetch-events-by-serials.go: Fallback to legacy format
- pkg/database/fetch-event-by-serial.go: Fallback to legacy format
- pkg/database/export.go: Fallback to legacy format
- pkg/database/serial_cache.go: Quiet sei lookup logging
- pkg/database/migrations.go: Add v9 sei backfill migration
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change IdPkTs Id/Pub fields from []byte to [32]byte (ntypes.EventID/Pubkey)
- Add NewIdPkTs() constructor for copying byte slices into fixed arrays
- Add IDSlice() and PubSlice() methods for slice access when needed
- Update protobuf converters to handle array-to-slice conversion
- Update all database and neo4j creation sites to use NewIdPkTs()
- Fix test files to use bytes.Equal() for array comparisons
- Benchmark confirms 0 B/op, 0 allocs/op for copy operations
Files modified:
- pkg/interfaces/store/store_interface.go: Core type change to fixed arrays
- pkg/interfaces/store/store_interface_test.go: Tests for new API
- pkg/proto/orlydb/v1/converters.go: Protobuf conversion with ntypes
- pkg/database/get-fullidpubkey-by-serial.go: Use NewIdPkTs constructor
- pkg/database/get-fullidpubkey-by-serials.go: Use NewIdPkTs constructor
- pkg/database/process-delete.go: Slice arrays for DeleteEvent calls
- pkg/neo4j/fetch-event.go: Use NewIdPkTs constructor
- pkg/neo4j/query-events.go: Use NewIdPkTs constructor
- pkg/database/query-for-*_test.go: Use bytes.Equal for comparisons
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Add O(1) direct lookup for parameterized replaceable events (kinds 30000-39999)
- Reduces query time from 10+ seconds to <1ms for kind+author+d-tag queries
- Optimizes WouldReplaceEvent() for faster event replacement checks
Fixes#29
## Changes
- `pkg/database/query-addressable.go` - New file with fast path lookup logic
- `pkg/database/query-addressable_test.go` - Tests for the optimization
- `pkg/database/save-event.go` - Writes AddressableEvent index, optimizes WouldReplaceEvent
- `pkg/database/query-events.go` - Integrates fast path at start of QueryEventsWithOptions
- `pkg/database/get-indexes-for-event.go` - Minor refactor (variable rename)
## Test plan
- [x] All existing database tests pass
- [x] New tests added for addressable event queries
- [x] Tests cover: valid/invalid query patterns, not found case, event replacement
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: woikos <woikos@users.noreply.github.com>
Reviewed-on: https://git.nostrdev.com/mleku/next.orly.dev/pulls/30
- Remove debug 100 iteration limit from GetSerialsByRange
The limit was a debug safety check that should have been removed.
This was capping all range queries to 100 results, causing negentropy
sync to only see 100 events instead of all events.
- Fix ACL service startup to listen immediately before Configure
The ACL server now starts the gRPC listener immediately after database
ready, then runs Configure in the background. This prevents the
launcher from timing out while waiting for ACL to load follow lists.
- Add SetReady method to ACL service for proper ready signaling
Files modified:
- pkg/database/get-serials-by-range.go: Remove debug iteration limit
- cmd/orly-acl/main.go: Start gRPC server before Configure
- cmd/orly-acl/service.go: Add ready field and SetReady method
- pkg/version/version: Bump to v0.55.1
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add query semaphore limiting concurrent database queries to 3
- Reuse single iterator in FetchEventsBySerials instead of one per serial
- Disable expensive e-tag fallback lookup in deletion processing
- Add empty ID validation in access tracker loop
- Reduce log spam from GetSerialById empty ID errors
These changes reduce memory usage from ~5GB to ~150MB under load by
limiting concurrent Badger iterators which consume significant memory.
Files modified:
- pkg/database/database.go: Add query semaphore with acquire/release
- pkg/database/fetch-events-by-serials.go: Reuse iterator for sev lookups
- pkg/database/query-events.go: Add semaphore, disable e-tag fallback
- pkg/database/get-serial-by-id.go: Don't log empty ID errors
- app/handle-req.go: Validate event ID before GetSerialById
- pkg/version/version: Bump to v0.52.17
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Disable goroutine that deleted expired events during query processing
- This was causing "assignment to entry in nil map" Badger panics under load
- Expired events will remain until proper background cleanup is implemented
- Added TODO comments explaining the issue and fix needed
Files modified:
- pkg/database/query-events.go: Disable inline expired event deletion
- pkg/version/version: Bump to v0.52.9
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add geteventsforpubkey API method for viewing user events with pagination
- Add deleteeventsforpubkey API method to purge blacklisted user events
- Add clickable user detail view in curation UI showing all events
- Add event content expansion/truncation for long content
- Add kind name display for common Nostr event types
- Implement safety check requiring blacklist before event deletion
Files modified:
- app/handle-nip86-curating.go: Add event fetch/delete handlers
- pkg/database/curating-acl.go: Add GetEventsForPubkey, DeleteEventsForPubkey
- app/web/src/CurationView.svelte: Add user detail view with event listing
- pkg/version/version: Bump to v0.50.0
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix countEventsForPubkey to use SHA256 hash of pubkey (first 8 bytes)
matching the PubHash type used in the Pubkey index
- Fix UI to use event_count field instead of total_events
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ScanAllPubkeys method to scan SerialPubkey index for all pubkeys
- Count events for each pubkey using the Pubkey index
- Store event counts in CURATING_ACL_EVENT_COUNT_ prefix
- Add NIP-86 "scanpubkeys" API endpoint to trigger the scan
This allows the curation UI to show all existing users in the unclassified
list, even if they had events before curating mode was enabled.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add sanity bounds to prevent memory exhaustion when decoding corrupt
events with garbage varint values. Previously, corrupt data could cause
massive allocations (e.g., make([]byte, 2^60)) leading to OOM crashes.
- Add MaxTagsPerEvent (10000), MaxTagElements (100), MaxContentLength (10MB),
MaxTagElementLength (1MB) limits
- Return sentinel errors for corrupt data instead of logging
- Silently skip corrupt events (caller handles gracefully)
This fixes crash loops on archive.orly.dev where OOM during writes
left corrupt events in bbolt database.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add validation in GetEventIdBySerial to ensure sei value is 32 bytes
- Fix fallback-to-legacy bug: return error instead of attempting legacy
unmarshal on compact format data when event ID lookup fails
- Add upfront validation in UnmarshalCompactEvent for eventId length
- Prevents events with all-zero IDs from being returned to clients
Files modified:
- pkg/database/serial_cache.go: Validate sei value is exactly 32 bytes
- pkg/database/fetch-events-by-serials.go: Return error for compact format
when eventId missing instead of falling back to legacy unmarshal
- pkg/database/fetch-event-by-serial.go: Same fix for single event fetch
- pkg/database/compact_event.go: Validate eventId is 32 bytes upfront
- pkg/version/version: Bump to v0.47.1
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix Content-Type header being set on request instead of response
- Add Vary: Accept header to prevent browser/CDN caching NIP-11 for HTML
- Add periodic flushing during export for HTTP streaming (every 100 events)
- Add initial flush after headers to start streaming immediately
- Add X-Content-Type-Options: nosniff to prevent browser buffering
Files modified:
- app/handle-relayinfo.go: Fix header and add Vary: Accept
- app/server.go: Add initial flush and nosniff header for export
- pkg/database/export.go: Add periodic and final flushing for streaming
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix binary pubkey/event ID values not being detected by tag.Marshal
- Compact event decoder now returns 33-byte values with null terminator
- This allows tag.Marshal to detect and hex-encode binary values correctly
- Fixes "Could not decode a text frame as UTF-8" WebSocket errors
Files modified:
- pkg/database/compact_event.go: Return 33-byte binary with null terminator
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add embedded WireGuard VPN server using wireguard-go + netstack
- Implement deterministic /31 subnet allocation from seed + sequence
- Use Badger's built-in Sequence for atomic counter allocation
- Add NIP-46 bunker server for remote signing over VPN
- Add revoked key tracking and access audit logging for users
- Add Bunker tab to web UI with WireGuard/bunker QR codes
- Support key regeneration with old keypair archiving
New environment variables:
- ORLY_WG_ENABLED: Enable WireGuard VPN server
- ORLY_WG_PORT: UDP port for WireGuard (default 51820)
- ORLY_WG_ENDPOINT: Public endpoint for WireGuard
- ORLY_WG_NETWORK: Base network for subnet pool (default 10.0.0.0/8)
- ORLY_BUNKER_ENABLED: Enable NIP-46 bunker
- ORLY_BUNKER_PORT: WebSocket port for bunker (default 3335)
Files added:
- pkg/wireguard/: WireGuard server, keygen, subnet pool, errors
- pkg/bunker/: NIP-46 bunker server and session handling
- pkg/database/wireguard.go: Peer storage with audit logging
- app/handle-wireguard.go: API endpoints for config/regenerate/audit
- app/wireguard-helpers.go: Key derivation helpers
- app/web/src/BunkerView.svelte: Bunker UI with QR codes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add generic LRUCache[K, V] implementation using container/list for O(1) ops
- Replace random 50% eviction with proper LRU eviction in SerialCache
- Cache now starts empty and grows on demand up to configured limits
- Use [32]byte keys instead of string([]byte) to avoid allocation overhead
- Single-entry eviction at capacity instead of 50% bulk clearing
- Add comprehensive unit tests and benchmarks for LRUCache
- Benchmarks show ~32-34 ns/op with 0 allocations for Get/Put
Files modified:
- pkg/database/lrucache.go: New generic LRU cache implementation
- pkg/database/lrucache_test.go: Unit tests and benchmarks
- pkg/database/serial_cache.go: Refactored to use LRUCache
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add buffer pool (pkg/database/bufpool) with SmallPool (64B) and MediumPool (1KB)
for reusing bytes.Buffer instances on hot paths
- Fix escape analysis in index types (uint40, letter, word) by using fixed-size
arrays instead of make() calls that escape to heap
- Add handler concurrency limiter (ORLY_MAX_HANDLERS_PER_CONN, default 100) to
prevent unbounded goroutine growth under WebSocket load
- Add pre-allocation hints to Uint40s.Union/Intersection/Difference methods
- Update compact_event.go, save-event.go, serial_cache.go, and
get-indexes-for-event.go to use pooled buffers
Files modified:
- app/config/config.go: Add MaxHandlersPerConnection config
- app/handle-websocket.go: Initialize handler semaphore
- app/listener.go: Add semaphore acquire/release in messageProcessor
- pkg/database/bufpool/pool.go: New buffer pool package
- pkg/database/compact_event.go: Use buffer pool, fix escape analysis
- pkg/database/get-indexes-for-event.go: Reuse single buffer for all indexes
- pkg/database/indexes/types/letter.go: Fixed array in UnmarshalRead
- pkg/database/indexes/types/uint40.go: Fixed arrays, pre-allocation hints
- pkg/database/indexes/types/word.go: Fixed array in UnmarshalRead
- pkg/database/save-event.go: Use buffer pool for key encoding
- pkg/database/serial_cache.go: Use buffer pool for lookups
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add unicode_normalize.go with mappings for small caps and fraktur
- Map 77 decorative unicode characters to ASCII equivalents:
- Small caps (25 chars): ᴅᴇᴀᴛʜ → death
- Fraktur lowercase (26 chars): 𝔡𝔢𝔞𝔱𝔥 → death
- Fraktur uppercase (26 chars): 𝔇𝔈𝔄𝔗ℌ → death
- Fix broken utf8DecodeRuneInString() that failed on multi-byte UTF-8
- Add migration v7 to rebuild word indexes with normalization
- Add comprehensive unit tests for all character mappings
Files modified:
- pkg/database/unicode_normalize.go: New - character mapping tables
- pkg/database/unicode_normalize_test.go: New - unit tests
- pkg/database/tokenize.go: Integrate normalizeRune(), fix UTF-8 decoder
- pkg/database/migrations.go: Add version 7 migration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove broken submodule reference for pkg/protocol/blossom/blossom
and track blossom spec files as regular files instead
- Add IMPORT_MEMORY_OPTIMIZATION_PLAN.md documenting strategies to
constrain import memory usage to ≤1.5GB through cache reduction,
batched syncs, batch transactions, and adaptive rate limiting
- Based on test results: 2.1M events imported in 48min at 736 events/sec
with peak memory of 6.4GB (target is 1.5GB)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Simplified event import to run synchronously, ensuring proper resource handling and accurate feedback. Enhanced frontend with real-time import status messages and error handling. Adjusted migrations to handle events individually, improving reliability and granular progress tracking.
This commit allows skipping authentication, permission checks, and certain filters (e.g., deletions, expirations) when the ACL mode is set to "none" (open relay mode). It also introduces a configuration option to disable query caching to reduce memory usage. These changes improve operational flexibility for open relay setups and resource-constrained environments.
Dgraph-related functionality, configuration, and benchmarks have been removed from the project. This streamlines the codebase to focus on supported backends, specifically eliminating Dgraph references in favor of Neo4j and other implementations. Version bumped to reflect the changes.
Replaced individual environment variable access with a unified `DatabaseConfig` struct for all database backends. This centralizes configuration management, reduces redundant code, and ensures all options are documented in `app/config/config.go`. Backward compatibility is maintained with default values and retained constructors.
Introduce comprehensive tests for policy validation logic, including owner and policy admin scenarios. Update `HandlePolicyConfigUpdate` to differentiate permissions for owners and policy admins, enforcing stricter field restrictions and validation flows.
Deleted `subscription_stability_test.go` to clean up unused or redundant code. Updated naming in test files for improved readability, replacing `tag` with `tg` for consistency. Also updated the `github.com/klauspost/compress` dependency to v1.18.2.
Introduce tests to validate functionality for new policy fields, including `max_expiry_duration`, `protected_required`, `identifier_regex`, and `follows_whitelist_admins`. Also, cover combinations of new and existing fields to ensure compatibility and precedence rules are correctly enforced.
bump to v0.31.2
- Increase WebSocket message size limit from 500KB to 10MB to prevent
truncation of large kind 3 follow list events (8000+ follows)
- Add validation in SaveEvent to reject kind 3 events without p tags
before storage, preventing malformed events from buggy relays
- Implement CleanupKind3WithoutPTags() to remove existing malformed
kind 3 events at startup
- Add enhanced logging showing tag count and event ID when rejecting
invalid kind 3 events for better observability
- Create round-trip test proving binary tag encoding preserves p tags
correctly through JSON→binary→JSON cycle
- Root cause: 500KB limit was truncating large follow lists during
WebSocket receive, causing tags to be lost or incomplete
- Three-layer defense: prevent at gate (size), validate (save time),
and cleanup (startup)
Files modified:
- app/handle-websocket.go: Increase DefaultMaxMessageSize to 10MB
- pkg/database/save-event.go: Add kind 3 validation with logging
- pkg/database/cleanup-kind3.go: New cleanup function
- pkg/database/cleanup-kind3_test.go: Round-trip test
- app/main.go: Invoke cleanup at startup
- Update nostr library to v1.0.3 with improved binary tag support
- Replace tag.Value() calls with tag.ValueHex() to handle both binary and hex formats
- Add NormalizeTagValueForHash() for consistent filter tag normalization
- Update QueryPTagGraph to handle binary-encoded and hex-encoded pubkeys
- Fix tag matching in query-events.go using TagValuesMatchUsingTagMethods
- Add filter_utils.go with tag normalization helper functions
- Update delete operations in process-delete.go and neo4j/delete.go
- Fix ACL follows extraction to use ValueHex() for consistent decoding
- Add binary_tag_filter_test.go for testing tag value normalization
- Bump version to v0.30.3
woikos
mleku