Browse Source
- Add README.md table of contents for easier navigation - Add Curation ACL documentation section to README.md - Create detailed Curation Mode Guide (docs/CURATION_MODE_GUIDE.md) - Fix OOM during BBolt index building by closing temp file before build - Add GC calls before index building to reclaim batch buffer memory - Improve import-export.go with processJSONLEventsReturningCount - Add policy-aware import path for sync operations Files modified: - README.md: Added TOC and curation ACL documentation - docs/CURATION_MODE_GUIDE.md: New comprehensive curation mode guide - pkg/bbolt/import-export.go: Memory-safe import with deferred cleanup - pkg/bbolt/import-minimal.go: Added GC before index build - pkg/version/version: Bump to v0.48.8 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>main
woikos
5 days ago
5 changed files with 598 additions and 19 deletions
@ -0,0 +1,411 @@ |
|||||||
|
# Curation Mode Guide |
||||||
|
|
||||||
|
Curation mode is a sophisticated access control system for Nostr relays that provides three-tier publisher classification, rate limiting, IP-based flood protection, and event kind whitelisting. |
||||||
|
|
||||||
|
## Overview |
||||||
|
|
||||||
|
Unlike simple allow/deny lists, curation mode classifies publishers into three tiers: |
||||||
|
|
||||||
|
| Tier | Rate Limited | Daily Limit | Visibility | |
||||||
|
|------|--------------|-------------|------------| |
||||||
|
| **Trusted** | No | Unlimited | Full | |
||||||
|
| **Blacklisted** | N/A (blocked) | 0 | Hidden from regular users | |
||||||
|
| **Unclassified** | Yes | 50 events/day (default) | Full | |
||||||
|
|
||||||
|
This allows relay operators to: |
||||||
|
- Reward quality contributors with unlimited publishing |
||||||
|
- Block bad actors while preserving their events for admin review |
||||||
|
- Allow new users to participate with reasonable rate limits |
||||||
|
- Prevent spam floods through automatic IP-based protections |
||||||
|
|
||||||
|
## Quick Start |
||||||
|
|
||||||
|
### 1. Start the Relay |
||||||
|
|
||||||
|
```bash |
||||||
|
export ORLY_ACL_MODE=curating |
||||||
|
export ORLY_OWNERS=npub1your_owner_pubkey |
||||||
|
./orly |
||||||
|
``` |
||||||
|
|
||||||
|
### 2. Publish Configuration |
||||||
|
|
||||||
|
The relay will not accept events until you publish a configuration event. Use the web UI at `http://your-relay/#curation` or publish a kind 30078 event: |
||||||
|
|
||||||
|
```json |
||||||
|
{ |
||||||
|
"kind": 30078, |
||||||
|
"tags": [["d", "curating-config"]], |
||||||
|
"content": "{\"dailyLimit\":50,\"ipDailyLimit\":500,\"firstBanHours\":1,\"secondBanHours\":168,\"kindCategories\":[\"social\"]}" |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
### 3. Manage Publishers |
||||||
|
|
||||||
|
Use the web UI or NIP-86 API to: |
||||||
|
- Trust quality publishers |
||||||
|
- Blacklist spammers |
||||||
|
- Review unclassified users by activity |
||||||
|
- Unblock IPs if needed |
||||||
|
|
||||||
|
## Configuration |
||||||
|
|
||||||
|
### Environment Variables |
||||||
|
|
||||||
|
| Variable | Default | Description | |
||||||
|
|----------|---------|-------------| |
||||||
|
| `ORLY_ACL_MODE` | `none` | Set to `curating` to enable | |
||||||
|
| `ORLY_OWNERS` | | Owner pubkeys (can configure relay) | |
||||||
|
| `ORLY_ADMINS` | | Admin pubkeys (can manage publishers) | |
||||||
|
|
||||||
|
### Configuration Event (Kind 30078) |
||||||
|
|
||||||
|
Configuration is stored as a replaceable Nostr event (kind 30078) with d-tag `curating-config`. Only owners and admins can publish configuration. |
||||||
|
|
||||||
|
```typescript |
||||||
|
interface CuratingConfig { |
||||||
|
// Rate Limiting |
||||||
|
dailyLimit: number; // Max events/day for unclassified users (default: 50) |
||||||
|
ipDailyLimit: number; // Max events/day from single IP (default: 500) |
||||||
|
|
||||||
|
// IP Ban Durations |
||||||
|
firstBanHours: number; // First offense ban duration (default: 1 hour) |
||||||
|
secondBanHours: number; // Subsequent offense ban duration (default: 168 hours / 1 week) |
||||||
|
|
||||||
|
// Kind Filtering (choose one or combine) |
||||||
|
allowedKinds: number[]; // Explicit kind numbers: [0, 1, 3, 7] |
||||||
|
allowedRanges: string[]; // Kind ranges: ["1000-1999", "30000-39999"] |
||||||
|
kindCategories: string[]; // Pre-defined categories: ["social", "dm"] |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
### Event Kind Categories |
||||||
|
|
||||||
|
Pre-defined categories for convenient kind whitelisting: |
||||||
|
|
||||||
|
| Category | Kinds | Description | |
||||||
|
|----------|-------|-------------| |
||||||
|
| `social` | 0, 1, 3, 6, 7, 10002 | Profiles, notes, contacts, reposts, reactions | |
||||||
|
| `dm` | 4, 14, 1059 | Direct messages (NIP-04, NIP-17, gift wraps) | |
||||||
|
| `longform` | 30023, 30024 | Long-form articles and drafts | |
||||||
|
| `media` | 1063, 20, 21, 22 | File metadata, picture/video/audio events | |
||||||
|
| `marketplace` | 30017-30020, 1021, 1022 | Products, stalls, auctions, bids | |
||||||
|
| `groups_nip29` | 9-12, 9000-9002, 39000-39002 | NIP-29 relay-based groups | |
||||||
|
| `groups_nip72` | 34550, 1111, 4550 | NIP-72 moderated communities | |
||||||
|
| `lists` | 10000, 10001, 10003, 30000, 30001, 30003 | Mute, pin, bookmark lists | |
||||||
|
|
||||||
|
Example configuration allowing social interactions and DMs: |
||||||
|
|
||||||
|
```json |
||||||
|
{ |
||||||
|
"kindCategories": ["social", "dm"], |
||||||
|
"dailyLimit": 100, |
||||||
|
"ipDailyLimit": 1000 |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
## Three-Tier Classification |
||||||
|
|
||||||
|
### Trusted Publishers |
||||||
|
|
||||||
|
Trusted publishers have unlimited publishing rights: |
||||||
|
- Bypass all rate limiting |
||||||
|
- Can publish any allowed kind |
||||||
|
- Events always visible to all users |
||||||
|
|
||||||
|
**Use case**: Known quality contributors, verified community members, partner relays. |
||||||
|
|
||||||
|
### Blacklisted Publishers |
||||||
|
|
||||||
|
Blacklisted publishers are blocked from publishing: |
||||||
|
- All events rejected with `"pubkey is blacklisted"` error |
||||||
|
- Existing events become invisible to regular users |
||||||
|
- Admins and owners can still see blacklisted events (for review) |
||||||
|
|
||||||
|
**Use case**: Spammers, abusive users, bad actors. |
||||||
|
|
||||||
|
### Unclassified Publishers |
||||||
|
|
||||||
|
Everyone else falls into the unclassified tier: |
||||||
|
- Subject to daily event limit (default: 50 events/day) |
||||||
|
- Subject to IP-based flood protection |
||||||
|
- Events visible to all users |
||||||
|
- Can be promoted to trusted or demoted to blacklisted |
||||||
|
|
||||||
|
**Use case**: New users, general public. |
||||||
|
|
||||||
|
## Rate Limiting & Flood Protection |
||||||
|
|
||||||
|
### Per-Pubkey Limits |
||||||
|
|
||||||
|
Unclassified publishers are limited to a configurable number of events per day (default: 50). The count resets at midnight UTC. |
||||||
|
|
||||||
|
When a user exceeds their limit: |
||||||
|
1. Event is rejected with `"daily event limit exceeded"` error |
||||||
|
2. Their IP is flagged for potential abuse |
||||||
|
|
||||||
|
### Per-IP Limits |
||||||
|
|
||||||
|
To prevent Sybil attacks (creating many pubkeys from one IP), there's also an IP-based daily limit (default: 500 events). |
||||||
|
|
||||||
|
When an IP exceeds its limit: |
||||||
|
1. All events from that IP are rejected |
||||||
|
2. The IP is temporarily banned |
||||||
|
|
||||||
|
### Automatic IP Banning |
||||||
|
|
||||||
|
When rate limits are exceeded: |
||||||
|
|
||||||
|
| Offense | Ban Duration | Description | |
||||||
|
|---------|--------------|-------------| |
||||||
|
| First | 1 hour | Quick timeout for accidental over-posting | |
||||||
|
| Second+ | 1 week | Extended ban for repeated abuse | |
||||||
|
|
||||||
|
Ban durations are configurable via `firstBanHours` and `secondBanHours`. |
||||||
|
|
||||||
|
### Offense Tracking |
||||||
|
|
||||||
|
The system tracks which pubkeys triggered rate limits from each IP: |
||||||
|
|
||||||
|
``` |
||||||
|
IP 192.168.1.100: |
||||||
|
- npub1abc... exceeded limit at 2024-01-15 10:30:00 |
||||||
|
- npub1xyz... exceeded limit at 2024-01-15 10:45:00 |
||||||
|
Offense count: 2 |
||||||
|
Status: Banned until 2024-01-22 10:45:00 |
||||||
|
``` |
||||||
|
|
||||||
|
This helps identify coordinated spam attacks. |
||||||
|
|
||||||
|
## Spam Flagging |
||||||
|
|
||||||
|
Events can be flagged as spam without deletion: |
||||||
|
|
||||||
|
- Flagged events are hidden from regular users |
||||||
|
- Admins can review flagged events |
||||||
|
- Events can be unflagged if incorrectly marked |
||||||
|
- Original event data is preserved |
||||||
|
|
||||||
|
This is useful for: |
||||||
|
- Moderation review queues |
||||||
|
- Training spam detection systems |
||||||
|
- Preserving evidence of abuse |
||||||
|
|
||||||
|
## NIP-86 Management API |
||||||
|
|
||||||
|
All management operations use NIP-98 HTTP authentication. |
||||||
|
|
||||||
|
### Trust Management |
||||||
|
|
||||||
|
```bash |
||||||
|
# Trust a pubkey |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"trustpubkey","params":["<pubkey_hex>"]}' |
||||||
|
|
||||||
|
# Untrust a pubkey |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"untrustpubkey","params":["<pubkey_hex>"]}' |
||||||
|
|
||||||
|
# List trusted pubkeys |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"listtrustedpubkeys","params":[]}' |
||||||
|
``` |
||||||
|
|
||||||
|
### Blacklist Management |
||||||
|
|
||||||
|
```bash |
||||||
|
# Blacklist a pubkey |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"blacklistpubkey","params":["<pubkey_hex>"]}' |
||||||
|
|
||||||
|
# Remove from blacklist |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"unblacklistpubkey","params":["<pubkey_hex>"]}' |
||||||
|
|
||||||
|
# List blacklisted pubkeys |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"listblacklistedpubkeys","params":[]}' |
||||||
|
``` |
||||||
|
|
||||||
|
### Unclassified User Management |
||||||
|
|
||||||
|
```bash |
||||||
|
# List unclassified users sorted by event count |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"listunclassifiedusers","params":[]}' |
||||||
|
``` |
||||||
|
|
||||||
|
Response includes pubkey, event count, and last activity for each user. |
||||||
|
|
||||||
|
### Spam Management |
||||||
|
|
||||||
|
```bash |
||||||
|
# Mark event as spam |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"markspam","params":["<event_id_hex>"]}' |
||||||
|
|
||||||
|
# Unmark spam |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"unmarkspam","params":["<event_id_hex>"]}' |
||||||
|
|
||||||
|
# List spam events |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"listspamevents","params":[]}' |
||||||
|
``` |
||||||
|
|
||||||
|
### IP Block Management |
||||||
|
|
||||||
|
```bash |
||||||
|
# List blocked IPs |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"listblockedips","params":[]}' |
||||||
|
|
||||||
|
# Unblock an IP |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"unblockip","params":["<ip_address>"]}' |
||||||
|
``` |
||||||
|
|
||||||
|
### Configuration Management |
||||||
|
|
||||||
|
```bash |
||||||
|
# Get current configuration |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"getcuratingconfig","params":[]}' |
||||||
|
|
||||||
|
# Set allowed kind categories |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"setallowedkindcategories","params":[["social","dm","longform"]]}' |
||||||
|
|
||||||
|
# Get allowed kind categories |
||||||
|
curl -X POST https://relay.example.com \ |
||||||
|
-H "Authorization: Nostr <nip98_token>" \ |
||||||
|
-d '{"method":"getallowedkindcategories","params":[]}' |
||||||
|
``` |
||||||
|
|
||||||
|
## Web UI |
||||||
|
|
||||||
|
The curation web UI is available at `/#curation` and provides: |
||||||
|
|
||||||
|
- **Configuration Panel**: Set rate limits, ban durations, and allowed kinds |
||||||
|
- **Publisher Management**: Trust/blacklist pubkeys with one click |
||||||
|
- **Unclassified Users**: View users sorted by activity, promote or blacklist |
||||||
|
- **IP Blocks**: View and unblock banned IP addresses |
||||||
|
- **Spam Queue**: Review flagged events, confirm or unflag |
||||||
|
|
||||||
|
## Database Storage |
||||||
|
|
||||||
|
Curation data is stored in the relay database with the following key prefixes: |
||||||
|
|
||||||
|
| Prefix | Purpose | |
||||||
|
|--------|---------| |
||||||
|
| `CURATING_ACL_CONFIG` | Current configuration | |
||||||
|
| `CURATING_ACL_TRUSTED_PUBKEY_{pubkey}` | Trusted publisher list | |
||||||
|
| `CURATING_ACL_BLACKLISTED_PUBKEY_{pubkey}` | Blacklisted publisher list | |
||||||
|
| `CURATING_ACL_EVENT_COUNT_{pubkey}_{date}` | Daily event counts per pubkey | |
||||||
|
| `CURATING_ACL_IP_EVENT_COUNT_{ip}_{date}` | Daily event counts per IP | |
||||||
|
| `CURATING_ACL_IP_OFFENSE_{ip}` | Offense tracking per IP | |
||||||
|
| `CURATING_ACL_BLOCKED_IP_{ip}` | Active IP blocks | |
||||||
|
| `CURATING_ACL_SPAM_EVENT_{eventID}` | Spam-flagged events | |
||||||
|
|
||||||
|
## Caching |
||||||
|
|
||||||
|
For performance, the following data is cached in memory: |
||||||
|
|
||||||
|
- Trusted pubkey set |
||||||
|
- Blacklisted pubkey set |
||||||
|
- Allowed kinds set |
||||||
|
- Current configuration |
||||||
|
|
||||||
|
Caches are refreshed every hour by the background cleanup goroutine. |
||||||
|
|
||||||
|
## Background Maintenance |
||||||
|
|
||||||
|
A background goroutine runs hourly to: |
||||||
|
|
||||||
|
1. Remove expired IP blocks |
||||||
|
2. Clean up old event count entries (older than 2 days) |
||||||
|
3. Refresh in-memory caches |
||||||
|
4. Log maintenance statistics |
||||||
|
|
||||||
|
## Best Practices |
||||||
|
|
||||||
|
### Starting a New Curated Relay |
||||||
|
|
||||||
|
1. Start with permissive settings: |
||||||
|
```json |
||||||
|
{"dailyLimit": 100, "ipDailyLimit": 1000, "kindCategories": ["social"]} |
||||||
|
``` |
||||||
|
|
||||||
|
2. Monitor unclassified users for a few days |
||||||
|
|
||||||
|
3. Trust active, quality contributors |
||||||
|
|
||||||
|
4. Blacklist obvious spammers |
||||||
|
|
||||||
|
5. Adjust rate limits based on observed patterns |
||||||
|
|
||||||
|
### Handling Spam Waves |
||||||
|
|
||||||
|
During spam attacks: |
||||||
|
|
||||||
|
1. The IP-based flood protection will auto-ban attack sources |
||||||
|
2. Review blocked IPs via web UI or API |
||||||
|
3. Blacklist any pubkeys that got through |
||||||
|
4. Consider temporarily lowering `ipDailyLimit` |
||||||
|
|
||||||
|
### Recovering from Mistakes |
||||||
|
|
||||||
|
- **Accidentally blacklisted someone**: Use `unblacklistpubkey` - their events become visible again |
||||||
|
- **Wrongly flagged spam**: Use `unmarkspam` - event becomes visible again |
||||||
|
- **Blocked legitimate IP**: Use `unblockip` - IP can publish again immediately |
||||||
|
|
||||||
|
## Comparison with Other ACL Modes |
||||||
|
|
||||||
|
| Feature | None | Follows | Managed | Curating | |
||||||
|
|---------|------|---------|---------|----------| |
||||||
|
| Default Access | Write | Write if followed | Explicit allow | Rate-limited | |
||||||
|
| Rate Limiting | No | No | No | Yes | |
||||||
|
| Kind Filtering | No | No | Optional | Yes | |
||||||
|
| IP Protection | No | No | No | Yes | |
||||||
|
| Spam Flagging | No | No | No | Yes | |
||||||
|
| Configuration | Env vars | Follow lists | NIP-86 | Kind 30078 events | |
||||||
|
| Web UI | Basic | Basic | Basic | Full curation panel | |
||||||
|
|
||||||
|
## Troubleshooting |
||||||
|
|
||||||
|
### "Relay not accepting events" |
||||||
|
|
||||||
|
The relay requires a configuration event before accepting any events. Publish a kind 30078 event with d-tag `curating-config`. |
||||||
|
|
||||||
|
### "daily event limit exceeded" |
||||||
|
|
||||||
|
The user has exceeded their daily limit. Options: |
||||||
|
1. Wait until midnight UTC for reset |
||||||
|
2. Trust the pubkey if they're a quality contributor |
||||||
|
3. Increase `dailyLimit` in configuration |
||||||
|
|
||||||
|
### "pubkey is blacklisted" |
||||||
|
|
||||||
|
The pubkey is on the blacklist. Use `unblacklistpubkey` if this was a mistake. |
||||||
|
|
||||||
|
### "IP is blocked" |
||||||
|
|
||||||
|
The IP has been auto-banned due to rate limit violations. Use `unblockip` if legitimate, or wait for the ban to expire. |
||||||
|
|
||||||
|
### Events disappearing for users |
||||||
|
|
||||||
|
Check if the event author has been blacklisted. Blacklisted authors' events are hidden from regular users but visible to admins. |
||||||
Loading…
Reference in new issue