@ -122,7 +124,7 @@ ORLY is a standard Go application that can be built using the Go toolchain.
- Go 1.25.3 or later
- Go 1.25.3 or later
- Git
- Git
- For web UI: [Bun](https://bun.sh/) JavaScript runtime
- For web UI: Node.js and npm (JavaScript runtime and package manager)
### Basic Build
### Basic Build
@ -141,8 +143,8 @@ To build with the embedded web interface:
```bash
```bash
# Build the Svelte web application
# Build the Svelte web application
cd app/web
cd app/web
bun install
npm install
bun run build
npm run build
# Build the Go binary from project root
# Build the Go binary from project root
cd ../../
cd ../../
@ -156,7 +158,7 @@ The recommended way to build and embed the web UI is using the provided script:
```
```
This script will:
This script will:
- Build the Svelte app in `app/web` to `app/web/dist` using Bun (preferred) or fall back to npm/yarn/pnpm
- Build the Svelte app in `app/web` to `app/web/dist` using npm (or fall back to yarn/pnpm if npm is not available)
- Run `go install` from the repository root so the binary picks up the new embedded assets
- Run `go install` from the repository root so the binary picks up the new embedded assets
- Automatically detect and use the best available JavaScript package manager
- Automatically detect and use the best available JavaScript package manager
@ -167,8 +169,8 @@ For manual builds, you can also use:
# build.sh
# build.sh
echo "Building Svelte app..."
echo "Building Svelte app..."
cd app/web
cd app/web
bun install
npm install
bun run build
npm run build
echo "Building Go binary..."
echo "Building Go binary..."
cd ../../
cd ../../
@ -210,8 +212,8 @@ For development with hot-reloading, ORLY can proxy web requests to a local dev s
```bash
```bash
cd app/web
cd app/web
bun install
npm install
bun run dev
npm run dev
```
```
Note the port sirv is listening on (e.g., `http://localhost:8080`).
Note the port sirv is listening on (e.g., `http://localhost:8080`).
@ -244,7 +246,7 @@ Example with the relay on port 3334 and sirv on port 8080:
```bash
```bash
# Terminal 1: Dev server
# Terminal 1: Dev server
cd app/web &&bun run dev
cd app/web && npm run dev
# Output: Your application is ready~!
# Output: Your application is ready~!
# Local: http://localhost:8080
# Local: http://localhost:8080
@ -259,6 +261,168 @@ export ORLY_PORT=3334
If you only want to disable the embedded web UI (without proxying to a dev server), just set `ORLY_WEB_DISABLE=true` without setting `ORLY_WEB_DEV_PROXY_URL`. The relay will return 404 for web UI requests while still handling WebSocket and API requests.
If you only want to disable the embedded web UI (without proxying to a dev server), just set `ORLY_WEB_DISABLE=true` without setting `ORLY_WEB_DEV_PROXY_URL`. The relay will return 404 for web UI requests while still handling WebSocket and API requests.
### Event Import
ORLY provides flexible event import capabilities for bulk data migration and synchronization. The import system supports both single files and entire directories, with efficient disk-based buffering for large datasets.
#### Features
- **Large File Support**: Handles multi-gigabyte files efficiently with disk buffering
- **Folder Import**: Import entire directories of JSONL files in one operation
- **Localhost Authentication**: Automatic authentication for localhost requests using `$NOSTR_PRIVATE_KEY`
- **Progress Tracking**: Real-time progress logging for long-running imports
- **Error Handling**: Continues processing even if individual events fail
#### HTTP API Import
Import events via the `/api/import` endpoint:
**Single File Upload:**
```bash
# Using multipart form data
curl -X POST http://localhost:3334/api/import \
-H "Authorization: Nostr <base64_nip98_auth>" \
-F "file=@events.jsonl"
# Using raw body
curl -X POST http://localhost:3334/api/import \
-H "Authorization: Nostr <base64_nip98_auth>" \
-H "Content-Type: application/x-ndjson" \
--data-binary @events.jsonl
```
**Folder Import:**
Import all `.jsonl` files from a directory:
```bash
curl -X POST "http://localhost:3334/api/import?folder=../scripts/exports" \
-H "Authorization: Nostr <base64_nip98_auth>"
```
The folder import will:
- Process all `.jsonl` files in the specified directory
- Return a summary with results for each file
- Continue processing even if individual files fail
**Localhost Authentication:**
For localhost requests, you can use the `$NOSTR_PRIVATE_KEY` environment variable instead of NIP-98 authentication:
**Security:** Localhost authentication only works for requests from `127.0.0.1`, `::1`, or `localhost`. Folder imports are restricted to allowed directories (e.g., `../scripts/exports`) for security.
#### Command-Line Import
For direct database access, use the `orly db import` command:
```bash
# Import a single file directly to the database
orly db import events.jsonl
# Or specify with flag
orly db import --file /path/to/large-export.jsonl
```
This bypasses the HTTP API and is faster for very large imports when you have direct database access.
#### Import Format
Events must be in JSONL (JSON Lines) format - one JSON event per line:
ORLY includes powerful event streaming capabilities to forward events from a local relay to remote relays, supporting both standard Nostr protocol and efficient negentropy-based synchronization.
#### Features
- **WebSocket Streaming**: Stream events to any Nostr-compatible relay
- **Negentropy Support**: Efficient set reconciliation using NIP-77 protocol
- **Batch Processing**: Processes events in batches for optimal performance
- **Progress Tracking**: Real-time logging of streaming progress
curl -X POST "http://localhost:3334/api/stream-to-relay?target=wss://remote-relay.com"
```
**How It Works:**
1. The endpoint accepts the request and returns immediately with a status message
2. Streaming happens asynchronously in the background
3. All events from the local database are queried and streamed to the target relay
4. Events are sent in batches using standard Nostr EVENT messages
5. Progress is logged to the relay's log output
**Example Workflow:**
```bash
# 1. Import events to local relay
curl -X POST http://localhost:3334/api/import \
-F "file=@book-events.jsonl"
# 2. Stream events to remote relay
curl -X POST "http://localhost:3334/api/stream-to-relay?target=wss://orly-relay.imwald.eu" \
-H "Authorization: Nostr <base64_nip98_auth>"
```
#### Negentropy Synchronization
For more efficient synchronization, you can use negentropy (NIP-77) set reconciliation:
```bash
curl -X POST "http://localhost:3334/api/stream-to-relay?target=wss://remote-relay.com&use_negentropy=true" \
-H "Authorization: Nostr <base64_nip98_auth>"
```
**Note:** Full negentropy support requires the target relay to be configured as a peer in the sync manager. For ad-hoc streaming, use the standard EVENT streaming method.
#### Use Cases
- **Data Migration**: Move events from one relay to another
- **Backup Synchronization**: Keep multiple relays in sync
- **Local to Remote**: Transfer events from a local development relay to production
- **Relay Consolidation**: Merge events from multiple sources
### Sprocket Event Processing
### Sprocket Event Processing
ORLY includes a powerful sprocket system for external event processing scripts. Sprocket scripts enable custom filtering, validation, and processing logic for Nostr events before storage.
ORLY includes a powerful sprocket system for external event processing scripts. Sprocket scripts enable custom filtering, validation, and processing logic for Nostr events before storage.
log.Printf("Connected to remote relay: %s",targetURL)
log.Printf("Connected to remote relay: %s",targetURL)
// Query all events from local database
// Stream events in batches to avoid memory exhaustion
f:=filter.New()// Empty filter = all events
// Query events in chunks using timestamp-based pagination
events,err:=s.DB.QueryEvents(ctx,f)
constbatchLimit=1000// Query 1000 events at a time
iferr!=nil{
varsince*timestamp.T
returnfmt.Errorf("failed to query events: %w",err)
vartotalSentint64
}
for{
// Create filter for this batch
f:=filter.New()
ifsince!=nil{
f.Since=since
}
limit:=uint(batchLimit)
f.Limit=&limit
log.Printf("Streaming %d events to %s",len(events),targetURL)
// Query batch of events
events,err:=s.DB.QueryEvents(ctx,f)
iferr!=nil{
returnfmt.Errorf("failed to query events: %w",err)
}
// Stream events in batches
iflen(events)==0{
batchSize:=100
// No more events
sent:=0
break
fori:=0;i<len(events);i+=batchSize{
end:=i+batchSize
ifend>len(events){
end=len(events)
}
}
forj:=i;j<end;j++{
// Stream this batch - ensure all events are freed
ev:=events[j]
processedCount:=0
fori,ev:=rangeevents{
ifev==nil{
ifev==nil{
continue
continue
}
}
// Capture timestamp before freeing
createdAt:=ev.CreatedAt
// Send EVENT message: ["EVENT", event]
// Send EVENT message: ["EVENT", event]
eventMsg:=[]interface{}{"EVENT",ev}
eventMsg:=[]interface{}{"EVENT",ev}
iferr:=conn.WriteJSON(eventMsg);err!=nil{
iferr:=conn.WriteJSON(eventMsg);err!=nil{
// Free this event and all remaining events in the batch
ev.Free()
forj:=i+1;j<len(events);j++{
ifevents[j]!=nil{
events[j].Free()
}
}
returnfmt.Errorf("failed to send event: %w",err)
returnfmt.Errorf("failed to send event: %w",err)
}
}
sent++
// Free the event to return it to the pool
ifsent%1000==0{
ev.Free()
log.Printf("Streamed %d/%d events to %s",sent,len(events),targetURL)
processedCount++
totalSent++
iftotalSent%1000==0{
log.Printf("Streamed %d events to %s",totalSent,targetURL)
}
}
// Update since timestamp for next batch (use created_at + 1 to avoid duplicates)
since=timestamp.FromUnix(createdAt+1)
}
// If we got fewer events than the limit, we've reached the end
iflen(events)<batchLimit{
break
}
}
// Small delay between batches to avoid overwhelming the remote relay
// Small delay between batches to avoid overwhelming the remote relay
time.Sleep(100*time.Millisecond)
time.Sleep(100*time.Millisecond)
}
}
log.Printf("Successfully streamed %d events to %s",sent,targetURL)
log.Printf("Successfully streamed %d events to %s",totalSent,targetURL)
returnnil
returnnil
}
}
@ -1492,7 +1538,7 @@ func (s *Server) streamWithNegentropy(ctx context.Context, targetURL string) (in
// 2. Creating negentropy instance
// 2. Creating negentropy instance
// 3. Performing NIP-77 protocol exchange
// 3. Performing NIP-77 protocol exchange
// 4. Fetching and pushing events
// 4. Fetching and pushing events
// This is a placeholder - the sync manager's negentropy functionality
// This is a placeholder - the sync manager's negentropy functionality
// should be used via the existing sync infrastructure
// should be used via the existing sync infrastructure
return0,fmt.Errorf("negentropy sync via HTTP endpoint not yet fully implemented. Use the sync manager's peer configuration or implement direct negentropy protocol")
return0,fmt.Errorf("negentropy sync via HTTP endpoint not yet fully implemented. Use the sync manager's peer configuration or implement direct negentropy protocol")
