next.orly.dev

623 Commits

4 Branches

314 Tags

32 MiB

Tree: e8c50abe1b

Commit Graph

Author	SHA1	Message	Date
woikos	34ab56815e	Add database IPC split mode and blob storage abstraction (v0.53.0) Major refactoring to support running the database as a separate gRPC service: - Add gRPC database server (cmd/orly-db/) and launcher (cmd/orly-launcher/) - Add proto definitions for all database operations (proto/orlydb/v1/) - Add gRPC client implementing database.Database interface - Abstract Blossom blob storage into database interface (9 methods) - Remove BBolt database backend (pkg/bbolt/ deleted) - Hide Blossom tab in UI when blob storage unavailable - Update documentation for new architecture Database backends: badger (default), neo4j, wasmdb, grpc Files modified: - cmd/orly-db/: New gRPC database server - cmd/orly-launcher/: Process supervisor for split mode - proto/orlydb/v1/: Protocol buffer definitions - pkg/database/grpc/: gRPC client implementation - pkg/database/blob.go: Badger blob storage implementation - pkg/database/interface.go: Added blob storage methods - pkg/database/types.go: Added BlobMetadata, BlobDescriptor types - pkg/database/factory.go: Removed BBolt, added gRPC backend - pkg/neo4j/blob.go: Blob storage stubs - pkg/wasmdb/blob.go: Blob storage stubs - pkg/blossom/: Refactored to use database.Database interface - app/config/config.go: Removed BBolt config, added gRPC settings - CLAUDE.md: Updated documentation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	4 months ago
woikos	be72b694eb	Add BBolt rate limiting and tune Badger defaults for large archives (v0.48.12) - Increase Badger cache defaults: block 512→1024MB, index 256→512MB - Increase serial cache defaults: pubkeys 100k→250k, event IDs 500k→1M - Change ZSTD default from level 1 (fast) to level 3 (balanced) - Add memory-only rate limiter for BBolt backend - Add BBolt to database backend docs with scaling recommendations - Document migration between Badger and BBolt backends Files modified: - app/config/config.go: Tuned defaults for large-scale deployments - main.go: Add BBolt rate limiter support - pkg/ratelimit/factory.go: Add NewMemoryOnlyLimiter factory - pkg/ratelimit/memory_monitor.go: New memory-only load monitor - CLAUDE.md: Add BBolt docs and scaling guide 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	4 months ago
mleku	88b0509ad8	Implement PID-controlled adaptive rate limiting for database operations - Add LoadMonitor interface in pkg/interfaces/loadmonitor/ for database load metrics - Implement PIDController with filtered derivative to suppress high-frequency noise - Proportional (P): immediate response to current error - Integral (I): eliminates steady-state offset with anti-windup clamping - Derivative (D): rate-of-change prediction with low-pass filtering - Create BadgerLoadMonitor tracking L0 tables, compaction score, and cache hit ratio - Create Neo4jLoadMonitor tracking query semaphore usage and latencies - Add AdaptiveRateLimiter combining PID controllers for reads and writes - Configure via environment variables: - ORLY_RATE_LIMIT_ENABLED: enable/disable rate limiting - ORLY_RATE_LIMIT_TARGET_MB: target memory limit (default 1500MB) - ORLY_RATE_LIMIT__K[PID]: PID gains for reads/writes - ORLY_RATE_LIMIT_MAX__MS: maximum delays - ORLY_RATE_LIMIT_*_TARGET: setpoints for reads/writes - Integrate rate limiter into Server struct and lifecycle management - Add comprehensive unit tests for PID controller behavior Files modified: - app/config/config.go: Add rate limiting configuration options - app/main.go: Initialize and start/stop rate limiter - app/server.go: Add rateLimiter field to Server struct - main.go: Create rate limiter with appropriate monitor - pkg/run/run.go: Pass disabled limiter for test instances - pkg/interfaces/loadmonitor/: New LoadMonitor interface - pkg/ratelimit/: New PID controller and limiter implementation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>	5 months ago

Author

SHA1

Message

Date

woikos

34ab56815e

Add database IPC split mode and blob storage abstraction (v0.53.0)

Major refactoring to support running the database as a separate gRPC service:

- Add gRPC database server (cmd/orly-db/) and launcher (cmd/orly-launcher/)
- Add proto definitions for all database operations (proto/orlydb/v1/)
- Add gRPC client implementing database.Database interface
- Abstract Blossom blob storage into database interface (9 methods)
- Remove BBolt database backend (pkg/bbolt/ deleted)
- Hide Blossom tab in UI when blob storage unavailable
- Update documentation for new architecture

Database backends: badger (default), neo4j, wasmdb, grpc

Files modified:
- cmd/orly-db/: New gRPC database server
- cmd/orly-launcher/: Process supervisor for split mode
- proto/orlydb/v1/: Protocol buffer definitions
- pkg/database/grpc/: gRPC client implementation
- pkg/database/blob.go: Badger blob storage implementation
- pkg/database/interface.go: Added blob storage methods
- pkg/database/types.go: Added BlobMetadata, BlobDescriptor types
- pkg/database/factory.go: Removed BBolt, added gRPC backend
- pkg/neo4j/blob.go: Blob storage stubs
- pkg/wasmdb/blob.go: Blob storage stubs
- pkg/blossom/: Refactored to use database.Database interface
- app/config/config.go: Removed BBolt config, added gRPC settings
- CLAUDE.md: Updated documentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

4 months ago

woikos

be72b694eb

Add BBolt rate limiting and tune Badger defaults for large archives (v0.48.12)

- Increase Badger cache defaults: block 512→1024MB, index 256→512MB
- Increase serial cache defaults: pubkeys 100k→250k, event IDs 500k→1M
- Change ZSTD default from level 1 (fast) to level 3 (balanced)
- Add memory-only rate limiter for BBolt backend
- Add BBolt to database backend docs with scaling recommendations
- Document migration between Badger and BBolt backends

Files modified:
- app/config/config.go: Tuned defaults for large-scale deployments
- main.go: Add BBolt rate limiter support
- pkg/ratelimit/factory.go: Add NewMemoryOnlyLimiter factory
- pkg/ratelimit/memory_monitor.go: New memory-only load monitor
- CLAUDE.md: Add BBolt docs and scaling guide

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

4 months ago

mleku

88b0509ad8

Implement PID-controlled adaptive rate limiting for database operations

- Add LoadMonitor interface in pkg/interfaces/loadmonitor/ for database load metrics
- Implement PIDController with filtered derivative to suppress high-frequency noise
  - Proportional (P): immediate response to current error
  - Integral (I): eliminates steady-state offset with anti-windup clamping
  - Derivative (D): rate-of-change prediction with low-pass filtering
- Create BadgerLoadMonitor tracking L0 tables, compaction score, and cache hit ratio
- Create Neo4jLoadMonitor tracking query semaphore usage and latencies
- Add AdaptiveRateLimiter combining PID controllers for reads and writes
- Configure via environment variables:
  - ORLY_RATE_LIMIT_ENABLED: enable/disable rate limiting
  - ORLY_RATE_LIMIT_TARGET_MB: target memory limit (default 1500MB)
  - ORLY_RATE_LIMIT_*_K[PID]: PID gains for reads/writes
  - ORLY_RATE_LIMIT_MAX_*_MS: maximum delays
  - ORLY_RATE_LIMIT_*_TARGET: setpoints for reads/writes
- Integrate rate limiter into Server struct and lifecycle management
- Add comprehensive unit tests for PID controller behavior

Files modified:
- app/config/config.go: Add rate limiting configuration options
- app/main.go: Initialize and start/stop rate limiter
- app/server.go: Add rateLimiter field to Server struct
- main.go: Create rate limiter with appropriate monitor
- pkg/run/run.go: Pass disabled limiter for test instances
- pkg/interfaces/loadmonitor/: New LoadMonitor interface
- pkg/ratelimit/: New PID controller and limiter implementation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

5 months ago

3 Commits (e8c50abe1bbaca7e7f4d690f9fd0219f47dbd1f8)