Compare commits
5 Commits
d5313b2211
...
2e6b7e178a
| Author | SHA1 | Date |
|---|---|---|
 Silberengel
|
2e6b7e178a | 1 week ago |
|
Silberengel
|
e62e8604a4 | 1 week ago |
|
Silberengel
|
129fc4964b | 1 week ago |
|
Silberengel
|
a9ad84c186 | 1 week ago |
|
Silberengel
|
8353ae8c25 | 1 week ago |
53 changed files with 2568 additions and 626 deletions
@ -0,0 +1,7 @@
@@ -0,0 +1,7 @@
|
||||
# Default compose file from repo root; override with `make prewarm COMPOSE=...` if needed.
|
||||
COMPOSE ?= docker compose
|
||||
|
||||
.PHONY: prewarm |
||||
prewarm: |
||||
@chmod +x scripts/docker-prewarm.sh 2>/dev/null || true
|
||||
@./scripts/docker-prewarm.sh
|
||||
@ -1,79 +1,130 @@
@@ -1,79 +1,130 @@
|
||||
# Unfold |
||||
# Unfold: Imwald |
||||
|
||||
Unfold is a customizable framework for your Nostr-based magazine. |
||||
<p align="center"> |
||||
<img src="assets/laeserin_logo.png" alt="Imwald" width="150"> |
||||
</p> |
||||
|
||||
(This is the **Imwald** edition of Unfold.) |
||||
A Symfony + FrankenPHP site that **reads Nostr long-form articles (kind 30023)** and related data from relays, stores articles in **MySQL**, and serves pages with Twig. **Comments and profile metadata** are **cache-backed** (not the full source of truth in the DB). |
||||
|
||||
## Setup |
||||
--- |
||||
|
||||
### Clone the repository |
||||
## Requirements |
||||
|
||||
| Requirement | Version / notes | |
||||
|------------|-----------------| |
||||
| PHP | **≥ 8.3.13** (see `composer.json`) | |
||||
| Docker | Optional; recommended for local dev and production images | |
||||
| Database | MySQL **8.0** (configurable) | |
||||
|
||||
--- |
||||
|
||||
## Local development (Docker) |
||||
|
||||
1. **Env:** copy `.env.dist` to `.env` and adjust if needed (especially `APP_SECRET` outside dev). |
||||
2. **Start stack** |
||||
|
||||
```bash |
||||
docker compose up -d |
||||
``` |
||||
|
||||
3. **App URL (default):** [http://127.0.0.1:9080](http://127.0.0.1:9080) |
||||
Port comes from `HTTP_PORT` in `.env` and `compose.override.yaml` (loopback only). |
||||
|
||||
4. **First-time DB:** migrations run on **php** container start when `migrations/` contains PHP files (see `frankenphp/docker-entrypoint.sh`). |
||||
|
||||
| Service | Role | |
||||
|--------|------| |
||||
| `php` | FrankenPHP + Caddy, Symfony app, console | |
||||
| `database` | MySQL; dev exposes `127.0.0.1:3307 → 3306` for local clients | |
||||
| `cron` | Runs full **`app:prewarm` every 10 minutes**; repo bind-mounted at `/var/www/html` (see `docker/cron/`) | |
||||
|
||||
--- |
||||
|
||||
## Backfill articles + warm caches (recommended) |
||||
|
||||
To **migrate**, **import articles from Nostr** for a time window, then **prewarm** magazine indices, author metadata, and comment caches: |
||||
|
||||
```bash |
||||
git clone https://github.com/decent-newsroom/unfold.git |
||||
cd unfold |
||||
make prewarm |
||||
``` |
||||
|
||||
### Create the .env file |
||||
| Step (script order) | Command / effect | |
||||
|---------------------|------------------| |
||||
| 1 | `docker compose up -d --wait` — starts **php**, **database**, and **cron** (the `cron` image runs a full `app:prewarm` on a 10 min schedule) | |
||||
| 2 | `doctrine:migrations:migrate` | |
||||
| 3 | `articles:get -- '-2 month' 'now'` — sync long-form into MySQL for that window | |
||||
| 4 | `app:prewarm` — magazine **30040**, **kind-0** profiles, **comment** cache (default **`--comments-max=20`**, newest by `createdAt`) | |
||||
|
||||
Copy the example file `.env.dist` and replace placeholders with your actual configuration. |
||||
`make prewarm` brings the stack (including `cron`) up so scheduled prewarm is active. **Optional** extra arguments for the **cron**-scheduled `app:prewarm` go in **`.env`** as **`PREWARM_FLAGS`** (same as you might pass to `php bin/console app:prewarm …`); Compose passes them into the `cron` container. Example: `PREWARM_FLAGS="--metadata-limit=50 --no-magazine"`. **Restart** the `cron` service after changing `PREWARM_FLAGS` so the container reloads the env. Hub / `compose.hub.yaml` has no `cron` service; use a host timer or `exec` if you need the same there. |
||||
|
||||
If you have your own MySQL database, comment out the database service in `compose.yaml` and skip root password in `.env`. |
||||
There are additional comments to that effect in the files. |
||||
--- |
||||
|
||||
### Configure `config/unfold.yaml` |
||||
## Console commands (overview) |
||||
|
||||
Before running the application, review and update `config/unfold.yaml` to match your desired magazine settings, theme, and external links. This file controls: |
||||
- Magazine name, short name, and description |
||||
- Theme and color settings |
||||
- Community articles feature |
||||
- External footer links |
||||
- Other project-specific configuration |
||||
| Command | Purpose | |
||||
|---------|---------| |
||||
| `articles:get <from> <to>` | Pull long-form articles from Nostr for the time range, persist to DB | |
||||
| `app:prewarm` | Magazine relay refresh + metadata cache + comment cache warm | |
||||
| `doctrine:migrations:migrate` | Apply SQL migrations | |
||||
| `user:elevate` | (If used) user elevation helper | |
||||
|
||||
Edit the values in `config/unfold.yaml` as needed for your deployment. |
||||
`php bin/console list` and `… -h` for full options. |
||||
|
||||
### Customizing Theme and Icons |
||||
### `app:prewarm` (notable options) |
||||
|
||||
You can override the default theme and icons by adding your own files to `/assets/theme/local/`. To do this: |
||||
- Copy the structure and file names from `/assets/theme/default/`. |
||||
- Place your custom `theme.css` and icon files in your theme folder. |
||||
- Update your configuration in `config/unfold.yaml` to reference your custom theme if needed. |
||||
| Option | Default | Meaning | |
||||
|--------|---------|--------| |
||||
| `--no-magazine` | off | Skip magazine 30040 index | |
||||
| `--no-metadata` | off | Skip Nostr kind-0 / profile cache | |
||||
| `--no-comments` | off | Skip comment thread cache | |
||||
| `--metadata-limit` | `0` (all authors) | Cap distinct author pubkeys | |
||||
| `--metadata-batch` | `50` | Pubkeys per batched Nostr `REQ` | |
||||
| `--comments-max` | `20` | Newest **N** articles (by `createdAt` **DESC**); `0` = all (still bounded by budget) | |
||||
| `--comments-budget` | `120` | Max wall seconds for the comments phase | |
||||
| `--magazine-budget` | `30` | Max wall seconds for magazine refresh | |
||||
|
||||
This allows you to easily switch or update the look and feel of your magazine without modifying the default assets. |
||||
Prewarm clears the PHP **CLI** execution time limit for that run; relay work can be slow. |
||||
|
||||
### `PREWARM_ON_START` (optional) |
||||
|
||||
### Build the Docker containers |
||||
| Variable | Set where | Effect | |
||||
|----------|------------|--------| |
||||
| `PREWARM_ON_START=1` | **Compose `environment` on the `php` service** (not only Symfony `.env` inside the container) | After DB is up and migrations run, executes **`app:prewarm` once** on start. **Does not** run `articles:get`. | |
||||
|
||||
For development: |
||||
```bash |
||||
docker compose build |
||||
``` |
||||
For a full **Nostr backfill** + one-shot prewarm, use **`make prewarm`** (or a host **cron** / **systemd** timer) instead of relying on **`PREWARM_ON_START` alone**. |
||||
|
||||
For production (using production overrides), set `APP_ENV=prod` in your `.env` file and run: |
||||
```bash |
||||
docker compose -f compose.yaml -f compose.prod.yaml build |
||||
``` |
||||
--- |
||||
|
||||
## Configuration |
||||
|
||||
### Start the Docker containers |
||||
```bash |
||||
docker compose up -d |
||||
``` |
||||
| What | File | |
||||
|------|------| |
||||
| Site title, `npub`, `d_tag`, **relays** (`default_relay`, `article_relays`, `profile_relays`), theme | `config/unfold.yaml` (imported as Symfony parameters) | |
||||
| `DATABASE_URL`, `APP_SECRET`, `HTTP_PORT`, `MYSQL_*`, optional **`PREWARM_FLAGS`** (for the Docker `cron` service) | `.env` / `.env.local` (see `.env.dist`) | |
||||
| Service wiring (e.g. cache, `NostrClient` args) | `config/services.yaml` | |
||||
|
||||
**Relays (short):** `default_relay` and `article_relays` drive article sync and many queries; `profile_relays` are used **first** for kind-0 / profile fetches, then the merged default + article set (see `NostrClient`). |
||||
|
||||
### Run Database Migrations |
||||
--- |
||||
|
||||
Before fetching or displaying articles, make sure your database schema is up to date. Run: |
||||
## Production / Hub image |
||||
|
||||
```bash |
||||
docker compose exec php php bin/console doctrine:migrations:migrate --no-interaction |
||||
``` |
||||
| Topic | Notes | |
||||
|-------|--------| |
||||
| `compose.hub.yaml` | Runs a **pulled** image (default `silberengel/unfold:latest`), no local PHP app build. Override with `UNFOLD_DOCKER_IMAGE`. | |
||||
| HTTP publish | `HTTP_PUBLISH` in `.env` (default **9080** → container **80**). Set `TRUSTED_PROXIES` behind a reverse proxy. | |
||||
| Secrets | Set `APP_SECRET` and DB credentials in **real** env; do not commit production secrets. | |
||||
|
||||
### Fetching Articles |
||||
File header in `compose.hub.yaml` lists pull, migrate, and optional build/push one-liners. |
||||
|
||||
To fetch articles from the default relay for the last two months, run: |
||||
--- |
||||
|
||||
```bash |
||||
docker compose exec php php bin/console articles:get -- '-2 month' 'now' |
||||
``` |
||||
## License |
||||
|
||||
**MIT** — see [`LICENSE`](LICENSE). |
||||
|
||||
--- |
||||
|
||||
## Project links (example) |
||||
|
||||
You can adjust the date range as needed. This command will import articles into the local database. |
||||
Configurable under `parameters.external_links` in `config/unfold.yaml` (e.g. Unfold on GitHub, Decent Newsroom). Adjust for your deployment. |
||||
|
||||
@ -0,0 +1,49 @@
@@ -0,0 +1,49 @@
|
||||
import { Controller } from "@hotwired/stimulus"; |
||||
|
||||
/** |
||||
* After first paint, refreshes Nostr magazine indices (server-side, ≤5s) and swaps header/body HTML. |
||||
*/ |
||||
export default class extends Controller { |
||||
static targets = ["headerNav", "pageBody"]; |
||||
static values = { |
||||
page: String, |
||||
slug: String, |
||||
url: String, |
||||
}; |
||||
|
||||
connect() { |
||||
this.sync(); |
||||
} |
||||
|
||||
async sync() { |
||||
const base = this.urlValue || "/ux/magazine-sync"; |
||||
const params = new URLSearchParams(); |
||||
params.set("page", this.pageValue || "article"); |
||||
const slug = this.slugValue || ""; |
||||
if (slug !== "") { |
||||
params.set("slug", slug); |
||||
} |
||||
const url = `${base}?${params.toString()}`; |
||||
try { |
||||
const res = await fetch(url, { |
||||
headers: { Accept: "application/json" }, |
||||
credentials: "same-origin", |
||||
}); |
||||
if (!res.ok) { |
||||
return; |
||||
} |
||||
const data = await res.json(); |
||||
if (!data.ok) { |
||||
return; |
||||
} |
||||
if (this.hasHeaderNavTarget && data.header) { |
||||
this.headerNavTarget.outerHTML = data.header; |
||||
} |
||||
if (this.hasPageBodyTarget && data.body) { |
||||
this.pageBodyTarget.outerHTML = data.body; |
||||
} |
||||
} catch { |
||||
/* ignore network errors */ |
||||
} |
||||
} |
||||
} |
||||
@ -0,0 +1,61 @@
@@ -0,0 +1,61 @@
|
||||
# Run a pre-built production image from Docker Hub (no local PHP image build). |
||||
# |
||||
# Usage on the server (copy this file + your .env, no app source required): |
||||
# docker compose -f compose.hub.yaml pull |
||||
# docker compose -f compose.hub.yaml up -d |
||||
# docker compose -f compose.hub.yaml exec php php bin/console doctrine:migrations:migrate --no-interaction |
||||
# |
||||
# Required in .env: APP_SECRET. Set MYSQL_* (or replace DATABASE_URL after editing this file) if you |
||||
# use the bundled database. For TLS in front, set TRUSTED_PROXIES to include your reverse proxy CIDR. |
||||
# |
||||
# Host HTTP port defaults to 9080 (same idea as local dev) so Apache/nginx can keep :80. Override with |
||||
# HTTP_PUBLISH=80 or HTTP_PUBLISH=127.0.0.1:9080 in .env if needed. |
||||
# |
||||
# Build & push (on your machine or CI), e.g.: |
||||
# docker build --platform linux/amd64 --target frankenphp_prod -t silberengel/unfold:latest . |
||||
# docker push silberengel/unfold:latest |
||||
# |
||||
# Override image: UNFOLD_DOCKER_IMAGE=myregistry/unfold:1.0.0 docker compose -f compose.hub.yaml up -d |
||||
|
||||
name: unfold |
||||
|
||||
services: |
||||
php: |
||||
image: ${UNFOLD_DOCKER_IMAGE:-silberengel/unfold:latest} |
||||
pull_policy: always |
||||
restart: unless-stopped |
||||
environment: |
||||
APP_ENV: ${APP_ENV:-prod} |
||||
APP_SECRET: ${APP_SECRET} |
||||
TRUSTED_PROXIES: ${TRUSTED_PROXIES:-127.0.0.0/8,10.0.0.0/8} |
||||
SERVER_NAME: ${SERVER_NAME:-:80} |
||||
DATABASE_URL: mysql://${MYSQL_USER:-unfold_user}:${MYSQL_PASSWORD:-password}@database:3306/${MYSQL_DATABASE:-unfold_db}?serverVersion=${MYSQL_VERSION:-8.0}&charset=${MYSQL_CHARSET:-utf8mb4} |
||||
volumes: |
||||
- caddy_data:/data |
||||
- caddy_config:/config |
||||
ports: |
||||
- "${HTTP_PUBLISH:-9080}:80/tcp" |
||||
depends_on: |
||||
database: |
||||
condition: service_healthy |
||||
|
||||
database: |
||||
image: mysql:${MYSQL_VERSION:-8.0} |
||||
restart: unless-stopped |
||||
environment: |
||||
MYSQL_DATABASE: ${MYSQL_DATABASE:-unfold_db} |
||||
MYSQL_USER: ${MYSQL_USER:-unfold_user} |
||||
MYSQL_PASSWORD: ${MYSQL_PASSWORD:-password} |
||||
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-root_password} |
||||
healthcheck: |
||||
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"] |
||||
timeout: 5s |
||||
retries: 5 |
||||
start_period: 60s |
||||
volumes: |
||||
- database_data:/var/lib/mysql:rw |
||||
|
||||
volumes: |
||||
caddy_data: |
||||
caddy_config: |
||||
database_data: |
||||
@ -1,30 +1,21 @@
@@ -1,30 +1,21 @@
|
||||
FROM php:8.2-cli |
||||
FROM php:8.3-cli |
||||
|
||||
# Install cron and Redis PHP extension dependencies |
||||
RUN apt-get update && apt-get install -y \ |
||||
RUN apt-get update && apt-get install -y --no-install-recommends \ |
||||
bash \ |
||||
cron \ |
||||
libzip-dev \ |
||||
libicu-dev \ |
||||
libpq-dev \ |
||||
libonig-dev |
||||
&& rm -rf /var/lib/apt/lists/* |
||||
|
||||
ADD https://github.com/mlocati/docker-php-extension-installer/releases/latest/download/install-php-extensions /usr/local/bin/ |
||||
RUN chmod +x /usr/local/bin/install-php-extensions \ |
||||
&& install-php-extensions pdo_mysql intl opcache zip gmp |
||||
|
||||
# Set working directory |
||||
WORKDIR /var/www/html |
||||
|
||||
# Install Symfony CLI tools (optional) |
||||
# RUN curl -sS https://get.symfony.com/cli/installer | bash |
||||
COPY crontab /etc/cron.d/unfold-prewarm |
||||
COPY prewarm_cron.sh /prewarm_cron.sh |
||||
COPY entry-cron.sh /entry-cron.sh |
||||
|
||||
# Copy cron and script |
||||
COPY crontab /etc/cron.d/app-cron |
||||
COPY index_articles.sh /index_articles.sh |
||||
RUN chmod 0644 /etc/cron.d/unfold-prewarm \ |
||||
&& chmod +x /prewarm_cron.sh /entry-cron.sh |
||||
|
||||
# Set permissions |
||||
RUN chmod 0644 /etc/cron.d/app-cron && \ |
||||
chmod +x /index_articles.sh |
||||
|
||||
# Apply cron job |
||||
RUN crontab /etc/cron.d/app-cron |
||||
|
||||
# Run cron in the foreground |
||||
CMD ["cron", "-f"] |
||||
CMD ["/entry-cron.sh"] |
||||
|
||||
@ -1,94 +1,15 @@
@@ -1,94 +1,15 @@
|
||||
# `cron` service (Docker) |
||||
|
||||
# 🕒 Cron Job Container |
||||
The `cron` image runs a single job: **`php bin/console app:prewarm` every 10 minutes**, against the app tree bind-mounted at `/var/www/html`. |
||||
|
||||
This folder contains the Docker configuration to run scheduled Symfony commands via cron inside a separate container. |
||||
- **Flags:** set **`PREWARM_FLAGS`** in the project `.env` (Compose injects it). Example: `PREWARM_FLAGS="--metadata-limit=30 --no-magazine"`. After editing, run `docker compose up -d --force-recreate cron` (or `docker compose up -d cron`) so the container gets the new value. If unset, `app:prewarm` uses its **built-in defaults** (same idea as running the console with no args). |
||||
|
||||
- Run Symfony console commands periodically using a cron schedule (e.g. every 6 hours) |
||||
- Decouple scheduled jobs from the main PHP/FPM container |
||||
- Easily manage and test cron execution in a Dockerized Symfony project |
||||
- **How env reaches cron:** the entrypoint writes `PREWARM_FLAGS` to `/run/cron-prewarm.env` at boot, because the system `crond` does not pass the container environment into crontab jobs. |
||||
|
||||
--- |
||||
- **Logs inside the container:** `tail -f /var/log/cron.log` (e.g. `docker compose exec cron tail -f /var/log/cron.log`). |
||||
|
||||
## Build & Run |
||||
- **PHP 8.3** extensions in the image are limited to what `app:prewarm` needs; the host **vendor** tree is what you mount from the repo. |
||||
|
||||
1. **Build the cron image** |
||||
From the project root: |
||||
```bash |
||||
docker-compose build cron |
||||
``` |
||||
- **Not included in** `compose.hub.yaml` (no app source mount). For production images, use host **cron** / **systemd** to `exec` the same command. |
||||
|
||||
2. **Start the cron container** |
||||
```bash |
||||
docker-compose up -d cron |
||||
``` |
||||
|
||||
--- |
||||
|
||||
## Cron Schedule |
||||
|
||||
The default cron schedule is set to run **every 6 hours**: |
||||
|
||||
```cron |
||||
0 */6 * * * root /run_commands.sh >> /var/log/cron.log 2>&1 |
||||
``` |
||||
|
||||
To customize the schedule, edit the `crontab` file and rebuild the container. |
||||
|
||||
--- |
||||
|
||||
## Testing & Debugging |
||||
|
||||
### Manually test the command runner |
||||
|
||||
You can run the script manually to check behavior without waiting for the cron trigger: |
||||
|
||||
```bash |
||||
docker-compose exec cron /run_commands.sh |
||||
``` |
||||
|
||||
### Check the cron output log |
||||
|
||||
```bash |
||||
docker-compose exec cron tail -f /var/log/cron.log |
||||
``` |
||||
|
||||
### Shell into the cron container |
||||
|
||||
```bash |
||||
docker-compose exec cron bash |
||||
``` |
||||
|
||||
Once inside, you can: |
||||
- Check crontab entries: `crontab -l` |
||||
- Manually trigger cron: `cron` or `cron -f` (in another session) |
||||
|
||||
--- |
||||
|
||||
## Customization |
||||
|
||||
- **Add/Remove Symfony Commands:** |
||||
Edit `run_commands.sh` to include the commands you want to run. |
||||
|
||||
- **Change Schedule:** |
||||
Edit `crontab` using standard cron syntax. |
||||
|
||||
- **Logging:** |
||||
Logs are sent to `/var/log/cron.log` inside the container. |
||||
|
||||
--- |
||||
|
||||
## Rebuilding After Changes |
||||
|
||||
If you modify the `crontab` or `run_commands.sh`, make sure to rebuild: |
||||
|
||||
```bash |
||||
docker-compose build cron |
||||
docker-compose up -d cron |
||||
``` |
||||
|
||||
--- |
||||
|
||||
## Notes |
||||
|
||||
- Symfony project source is mounted at `/var/www/html` via volume. |
||||
- Make sure your commands do **not rely on services** (like `php-fpm`) that are not running in this container. |
||||
Change the schedule: edit `docker/cron/crontab`, then `docker compose build cron && docker compose up -d cron`. |
||||
|
||||
@ -1 +1 @@
@@ -1 +1 @@
|
||||
*/5 * * * * /index_articles.sh >> /var/log/cron.log 2>&1 |
||||
*/10 * * * * root /prewarm_cron.sh >>/var/log/cron.log 2>&1 |
||||
|
||||
@ -0,0 +1,6 @@
@@ -0,0 +1,6 @@
|
||||
#!/bin/bash |
||||
set -euo pipefail |
||||
# crond does not pass Compose env to job children; write once at boot for prewarm_cron.sh. |
||||
export PREWARM_FLAGS="${PREWARM_FLAGS:-}" |
||||
declare -p PREWARM_FLAGS >/run/cron-prewarm.env |
||||
exec cron -f |
||||
@ -1,5 +0,0 @@
@@ -1,5 +0,0 @@
|
||||
#!/bin/bash |
||||
set -e |
||||
export PATH="/usr/local/bin:/usr/bin:/bin" |
||||
|
||||
php /var/www/html/bin/console articles:get -- '-6 min' 'now' |
||||
@ -0,0 +1,9 @@
@@ -0,0 +1,9 @@
|
||||
#!/bin/bash |
||||
set -euo pipefail |
||||
# shellcheck source=/dev/null |
||||
if [[ -f /run/cron-prewarm.env ]]; then |
||||
source /run/cron-prewarm.env |
||||
fi |
||||
cd /var/www/html |
||||
# shellcheck disable=SC2086 |
||||
exec php bin/console app:prewarm ${PREWARM_FLAGS:-} |
||||
@ -0,0 +1,22 @@
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env sh |
||||
# After `docker compose up`, run migrations, backfill long-form articles, and app:prewarm. |
||||
# Usage: from repository root: ./scripts/docker-prewarm.sh |
||||
# Or: make prewarm |
||||
set -eu |
||||
|
||||
ROOT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd) |
||||
cd "$ROOT_DIR" |
||||
|
||||
echo "==> docker compose up -d --wait (php, database, cron: full app:prewarm every 10 min; set PREWARM_FLAGS in .env to match your CLI)" |
||||
docker compose up -d --wait |
||||
|
||||
echo "==> doctrine:migrations:migrate" |
||||
docker compose exec -T php php bin/console doctrine:migrations:migrate --no-interaction |
||||
|
||||
echo "==> articles:get (last 2 months → now)" |
||||
docker compose exec -T php php bin/console articles:get -- '-2 month' 'now' |
||||
|
||||
echo "==> app:prewarm" |
||||
docker compose exec -T php php bin/console app:prewarm |
||||
|
||||
echo "Done." |
||||
@ -0,0 +1,250 @@
@@ -0,0 +1,250 @@
|
||||
<?php |
||||
|
||||
declare(strict_types=1); |
||||
|
||||
namespace App\Command; |
||||
|
||||
use App\Entity\Article; |
||||
use App\Repository\ArticleRepository; |
||||
use App\Service\ArticleCommentThreadLoader; |
||||
use App\Service\CacheService; |
||||
use App\Service\MagazineRefresher; |
||||
use App\Service\Nip09DeletionApplier; |
||||
use App\Service\NostrClient; |
||||
use Psr\Log\LoggerInterface; |
||||
use swentel\nostr\Key\Key; |
||||
use Symfony\Component\Console\Attribute\AsCommand; |
||||
use Symfony\Component\Console\Command\Command; |
||||
use Symfony\Component\Console\Input\InputInterface; |
||||
use Symfony\Component\Console\Input\InputOption; |
||||
use Symfony\Component\Console\Output\OutputInterface; |
||||
use Symfony\Component\Console\Style\SymfonyStyle; |
||||
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBagInterface; |
||||
|
||||
/** |
||||
* Prewarms magazine index cache, author metadata cache, and optional comment thread cache. |
||||
* Does not persist comments to MySQL; comments are cache-only in this app. |
||||
*/ |
||||
#[AsCommand( |
||||
name: 'app:prewarm', |
||||
description: 'Refresh magazine indices, NIP-09 deletions, profile metadata, and comment caches', |
||||
)] |
||||
final class PrewarmCommand extends Command |
||||
{ |
||||
public function __construct( |
||||
private readonly MagazineRefresher $magazineRefresher, |
||||
private readonly Nip09DeletionApplier $nip09DeletionApplier, |
||||
private readonly CacheService $cacheService, |
||||
private readonly NostrClient $nostrClient, |
||||
private readonly ArticleRepository $articleRepository, |
||||
private readonly ArticleCommentThreadLoader $commentThreadLoader, |
||||
private readonly ParameterBagInterface $params, |
||||
private readonly LoggerInterface $logger, |
||||
) { |
||||
parent::__construct(); |
||||
} |
||||
|
||||
protected function configure(): void |
||||
{ |
||||
$this |
||||
->addOption('no-magazine', null, InputOption::VALUE_NONE, 'Skip magazine 30040 index fetch') |
||||
->addOption('no-deletions', null, InputOption::VALUE_NONE, 'Skip NIP-09 kind 5 deletion sync (30023/30024 DB + 30040 magazine cache)') |
||||
->addOption('deletion-since', null, InputOption::VALUE_REQUIRED, 'strtotime() window start for kind 5 fetch', '-2 month') |
||||
->addOption('no-metadata', null, InputOption::VALUE_NONE, 'Skip Nostr profile metadata cache') |
||||
->addOption('no-comments', null, InputOption::VALUE_NONE, 'Skip comment thread cache') |
||||
->addOption('magazine-budget', null, InputOption::VALUE_REQUIRED, 'Seconds wall time for magazine relay refresh', '30') |
||||
->addOption('metadata-limit', null, InputOption::VALUE_REQUIRED, 'Max distinct author pubkeys to warm (0 = all)', '0') |
||||
->addOption('metadata-batch', null, InputOption::VALUE_REQUIRED, 'Kind-0 metadata: pubkeys per Nostr REQ (batched)', '50') |
||||
->addOption('comments-max', null, InputOption::VALUE_REQUIRED, 'Newest N articles to warm comment cache for (0 = all, order: createdAt DESC)', '20') |
||||
->addOption('comments-budget', null, InputOption::VALUE_REQUIRED, 'Max seconds for the whole comments phase', '120'); |
||||
} |
||||
|
||||
protected function execute(InputInterface $input, OutputInterface $output): int |
||||
{ |
||||
$this->disableCliExecutionTimeLimit(); |
||||
|
||||
$io = new SymfonyStyle($input, $output); |
||||
$keys = new Key(); |
||||
|
||||
if (!$input->getOption('no-magazine')) { |
||||
$budget = max(1, (int) $input->getOption('magazine-budget')); |
||||
$io->section('Magazine index (kinds 30040)'); |
||||
try { |
||||
$this->magazineRefresher->refreshFromRelays($budget, []); |
||||
$io->success('Magazine indices refreshed (within budget).'); |
||||
} catch (\Throwable $e) { |
||||
$this->logger->error('app:prewarm magazine failed', ['e' => $e]); |
||||
$io->warning('Magazine refresh failed: '.$e->getMessage()); |
||||
} |
||||
} else { |
||||
$io->note('Skipping magazine (--no-magazine).'); |
||||
} |
||||
|
||||
// MagazineRefresher sets max_execution_time (e.g. 60 for budget 30); restore before metadata. |
||||
$this->disableCliExecutionTimeLimit(); |
||||
|
||||
if (!$input->getOption('no-deletions')) { |
||||
$io->section('NIP-09 deletions (kind 5 → 30023/30024 / 30040)'); |
||||
$sinceStr = (string) $input->getOption('deletion-since'); |
||||
$since = strtotime($sinceStr); |
||||
if ($since === false) { |
||||
$since = strtotime('-2 month'); |
||||
} |
||||
$until = time(); |
||||
$deletionPubkeys = []; |
||||
foreach ($this->articleRepository->findDistinctAuthorPubkeys() as $pk) { |
||||
if (\is_string($pk) && 64 === \strlen($pk)) { |
||||
$deletionPubkeys[] = $pk; |
||||
} |
||||
} |
||||
$npubParam = (string) $this->params->get('npub'); |
||||
if (str_starts_with($npubParam, 'npub')) { |
||||
try { |
||||
$sitePk = $keys->convertToHex($npubParam); |
||||
if ($sitePk !== '' && 64 === \strlen($sitePk) && !\in_array($sitePk, $deletionPubkeys, true)) { |
||||
$deletionPubkeys[] = $sitePk; |
||||
} |
||||
} catch (\Throwable) { |
||||
} |
||||
} |
||||
if ($deletionPubkeys === []) { |
||||
$io->note('No author pubkeys; skipping kind 5 deletion fetch.'); |
||||
} else { |
||||
try { |
||||
$kind5 = $this->nostrClient->fetchKind5DeletionEventsForAuthors( |
||||
$deletionPubkeys, |
||||
$since, |
||||
$until, |
||||
40 |
||||
); |
||||
$st = $this->nip09DeletionApplier->apply($kind5); |
||||
$io->writeln(sprintf( |
||||
'Kind 5 events: <info>%d</info> (deduped). Articles removed: <info>%d</info>; magazine root/category cache entries removed: <info>%d</info> / <info>%d</info>.', |
||||
\count($kind5), |
||||
$st['articles_removed'], |
||||
$st['magazine_roots'], |
||||
$st['magazine_categories'] |
||||
)); |
||||
} catch (\Throwable $e) { |
||||
$this->logger->error('app:prewarm NIP-09 failed', ['exception' => $e]); |
||||
$io->warning('NIP-09 step failed: '.$e->getMessage()); |
||||
} |
||||
} |
||||
} else { |
||||
$io->note('Skipping NIP-09 deletions (--no-deletions).'); |
||||
} |
||||
|
||||
$this->disableCliExecutionTimeLimit(); |
||||
|
||||
if (!$input->getOption('no-metadata')) { |
||||
$io->section('Author metadata (cache)'); |
||||
$pubkeys = $this->articleRepository->findDistinctAuthorPubkeys(); |
||||
$npubParam = (string) $this->params->get('npub'); |
||||
if (str_starts_with($npubParam, 'npub')) { |
||||
try { |
||||
$sitePk = $keys->convertToHex($npubParam); |
||||
if ($sitePk !== '' && !\in_array($sitePk, $pubkeys, true)) { |
||||
$pubkeys[] = $sitePk; |
||||
} |
||||
} catch (\Throwable) { |
||||
// ignore bad npub |
||||
} |
||||
} |
||||
$limit = (int) $input->getOption('metadata-limit'); |
||||
if ($limit > 0) { |
||||
$pubkeys = \array_slice($pubkeys, 0, $limit); |
||||
} |
||||
$toWarm = []; |
||||
foreach ($pubkeys as $pubkey) { |
||||
if (strlen($pubkey) === 64) { |
||||
$toWarm[] = $pubkey; |
||||
} |
||||
} |
||||
$total = \count($toWarm); |
||||
$n = 0; |
||||
if ($total === 0) { |
||||
$io->note('No valid author pubkeys to warm.'); |
||||
} else { |
||||
$batchSize = max(1, min(200, (int) $input->getOption('metadata-batch'))); |
||||
$io->writeln(sprintf( |
||||
'Fetching kind-0 metadata: <info>%d</info> author(s) in Nostr requests of up to <info>%d</info> pubkeys each.', |
||||
$total, |
||||
$batchSize |
||||
)); |
||||
$bar = $io->createProgressBar($total); |
||||
$bar->start(); |
||||
try { |
||||
foreach (array_chunk($toWarm, $batchSize) as $chunk) { |
||||
$fetched = $this->nostrClient->fetchKind0MetadataForAuthors($chunk, $batchSize); |
||||
$n += $this->cacheService->putPrewarmMetadataBatch($chunk, $fetched, $keys); |
||||
$bar->advance(\count($chunk)); |
||||
} |
||||
} catch (\Throwable $e) { |
||||
$this->logger->error('app:prewarm metadata batch failed', ['exception' => $e]); |
||||
$io->error($e->getMessage()); |
||||
$bar->finish(); |
||||
$io->newLine(2); |
||||
|
||||
return Command::FAILURE; |
||||
} |
||||
$bar->finish(); |
||||
$io->newLine(2); |
||||
} |
||||
$io->success(sprintf('Warmed metadata for %d of %d author(s).', $n, $total)); |
||||
} else { |
||||
$io->note('Skipping metadata (--no-metadata).'); |
||||
} |
||||
|
||||
if ($input->getOption('no-comments')) { |
||||
$io->note('Skipping comments (--no-comments).'); |
||||
|
||||
return Command::SUCCESS; |
||||
} |
||||
|
||||
$maxArticles = (int) $input->getOption('comments-max'); |
||||
|
||||
$io->section('Comment / interaction cache'); |
||||
$deadline = microtime(true) + max(1, (int) $input->getOption('comments-budget')); |
||||
$qb = $this->articleRepository->createQueryBuilder('a') |
||||
->where('a.slug IS NOT NULL') |
||||
->andWhere("a.slug != ''") |
||||
->andWhere('a.pubkey IS NOT NULL') |
||||
->andWhere("a.pubkey != ''") |
||||
->orderBy('a.createdAt', 'DESC'); |
||||
if ($maxArticles > 0) { |
||||
$qb->setMaxResults($maxArticles); |
||||
} |
||||
$articles = $qb->getQuery()->getResult(); |
||||
$w = 0; |
||||
/** @var Article $article */ |
||||
foreach ($articles as $article) { |
||||
if (microtime(true) >= $deadline) { |
||||
$io->warning('Comment phase stopped: comments-budget reached.'); |
||||
break; |
||||
} |
||||
$slug = trim((string) $article->getSlug()); |
||||
$pubkey = (string) $article->getPubkey(); |
||||
if ($slug === '' || strlen($pubkey) !== 64) { |
||||
continue; |
||||
} |
||||
$kind = $article->getKind()?->value ?? 30023; |
||||
$coordinate = $kind.':'.$pubkey.':'.$slug; |
||||
$eventHex = (string) ($article->getEventId() ?? ''); |
||||
try { |
||||
$this->commentThreadLoader->load($coordinate, $eventHex !== '' ? $eventHex : null); |
||||
++$w; |
||||
} catch (\Throwable $e) { |
||||
$this->logger->warning('app:prewarm comment load', ['coord' => $coordinate, 'error' => $e->getMessage()]); |
||||
} |
||||
} |
||||
$io->success(sprintf('Warmed comment cache for %d of %d article(s).', $w, \count($articles))); |
||||
|
||||
return Command::SUCCESS; |
||||
} |
||||
|
||||
private function disableCliExecutionTimeLimit(): void |
||||
{ |
||||
@set_time_limit(0); |
||||
@ini_set('max_execution_time', '0'); |
||||
} |
||||
} |
||||
@ -0,0 +1,103 @@
@@ -0,0 +1,103 @@
|
||||
<?php |
||||
|
||||
declare(strict_types=1); |
||||
|
||||
namespace App\Controller; |
||||
|
||||
use App\Service\MagazineContentService; |
||||
use App\Service\MagazineRefresher; |
||||
use Psr\Log\LoggerInterface; |
||||
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBagInterface; |
||||
use Symfony\Component\HttpFoundation\JsonResponse; |
||||
use Symfony\Component\HttpFoundation\Request; |
||||
use Symfony\Component\HttpFoundation\Response; |
||||
use Symfony\Component\HttpKernel\Attribute\AsController; |
||||
use Symfony\Component\Routing\Attribute\Route; |
||||
use Twig\Environment; |
||||
|
||||
/** Stale-first: the main request only reads {@see \App\Service\MagazineIndexStore}; this refetches Nostr, updates that store, and returns HTML fragments for Stimulus to patch the document. */ |
||||
#[AsController] |
||||
final class MagazineSyncController |
||||
{ |
||||
public function __construct( |
||||
private readonly Environment $twig, |
||||
private readonly MagazineRefresher $refresher, |
||||
private readonly MagazineContentService $magazineContent, |
||||
private readonly ParameterBagInterface $params, |
||||
private readonly LoggerInterface $logger, |
||||
) { |
||||
} |
||||
|
||||
#[Route('/ux/magazine-sync', name: 'ux_magazine_sync', methods: ['GET'])] |
||||
public function __invoke(Request $request): JsonResponse |
||||
{ |
||||
try { |
||||
$page = (string) $request->query->get('page', 'article'); |
||||
if (!\in_array($page, ['home', 'category', 'article', 'articles'], true)) { |
||||
$page = 'article'; |
||||
} |
||||
$slug = (string) $request->query->get('slug', ''); |
||||
|
||||
$prefer = $slug !== '' ? [$slug] : []; |
||||
|
||||
try { |
||||
$this->refresher->refreshFromRelays(20, $prefer); |
||||
} catch (\Throwable $e) { |
||||
$this->logger->warning('MagazineSyncController: refresh failed', [ |
||||
'message' => $e->getMessage(), |
||||
'exception' => $e, |
||||
]); |
||||
|
||||
return new JsonResponse( |
||||
['ok' => false, 'error' => 'refresh_failed', 'message' => $e->getMessage()], |
||||
Response::HTTP_OK |
||||
); |
||||
} |
||||
|
||||
$community = (bool) $this->params->get('community_articles'); |
||||
$tags = $this->magazineContent->getHomeCategoryAIndexTagsFromStoreOnly(); |
||||
$globals = [ |
||||
'magazine_community_articles' => $community, |
||||
]; |
||||
|
||||
$header = $this->twig->render('ux/magazine/header_ul.html.twig', array_merge($globals, [ |
||||
'cats' => $tags, |
||||
])); |
||||
|
||||
$body = null; |
||||
if ($page === 'home') { |
||||
$body = $this->twig->render('ux/magazine/home_body.html.twig', array_merge($globals, [ |
||||
'indices' => $tags, |
||||
])); |
||||
} elseif ($page === 'category' && $slug !== '') { |
||||
$data = $this->magazineContent->getCategoryPageData($slug); |
||||
$body = $this->twig->render('ux/magazine/category_body.html.twig', array_merge($globals, [ |
||||
'list' => $data['list'], |
||||
'category' => $data['category'], |
||||
])); |
||||
} elseif ($page === 'articles') { |
||||
$body = null; |
||||
} |
||||
|
||||
return new JsonResponse([ |
||||
'ok' => true, |
||||
'header' => $header, |
||||
'body' => $body, |
||||
]); |
||||
} catch (\Throwable $e) { |
||||
$this->logger->error('MagazineSyncController: unexpected failure', [ |
||||
'message' => $e->getMessage(), |
||||
'exception' => $e, |
||||
]); |
||||
|
||||
return new JsonResponse( |
||||
[ |
||||
'ok' => false, |
||||
'error' => 'server_error', |
||||
'message' => 'Magazine UI sync could not be rendered.', |
||||
], |
||||
Response::HTTP_OK |
||||
); |
||||
} |
||||
} |
||||
} |
||||
@ -0,0 +1,177 @@
@@ -0,0 +1,177 @@
|
||||
<?php |
||||
|
||||
declare(strict_types=1); |
||||
|
||||
namespace App\Service; |
||||
|
||||
use App\Entity\Article; |
||||
use App\Entity\Event; |
||||
use App\Repository\ArticleRepository; |
||||
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBagInterface; |
||||
|
||||
/** |
||||
* Magazine index events for templates. Reads {@see MagazineIndexStore} first; on a cold cache or when |
||||
* the last successful relay sync is older than {@see self::ROOT_REVALIDATE_SECONDS}, the service |
||||
* calls {@see MagazineRefresher} so the root index (and nav) can pick up new categories. |
||||
*/ |
||||
final class MagazineContentService |
||||
{ |
||||
/** Re-fetch root from relays at most this often so new `a` tags appear in the header. */ |
||||
private const ROOT_REVALIDATE_SECONDS = 300; |
||||
|
||||
public function __construct( |
||||
private readonly MagazineIndexStore $store, |
||||
private readonly MagazineRefresher $refresher, |
||||
private readonly ParameterBagInterface $params, |
||||
private readonly ArticleRepository $articleRepository, |
||||
private readonly NostrClient $nostrClient, |
||||
) { |
||||
} |
||||
|
||||
/** |
||||
* "indices" for the home template: Nostr `a` tag rows for each category. |
||||
* |
||||
* @return list<array<int, string>> |
||||
*/ |
||||
public function getHomeCategoryIndexTags(): array |
||||
{ |
||||
$npub = (string) $this->params->get('npub'); |
||||
$dTag = (string) $this->params->get('d_tag'); |
||||
if ($this->store->getRoot($npub, $dTag) === null) { |
||||
$this->refresher->refreshFromRelays(20, []); |
||||
} elseif ($this->shouldRevalidateRootFromRelay()) { |
||||
$this->refresher->refreshFromRelays(20, []); |
||||
} |
||||
|
||||
return $this->getHomeCategoryAIndexTagsFromStoreOnly(); |
||||
} |
||||
|
||||
/** |
||||
* Category `a` tags from the persisted root only (no relay). Used after /ux/magazine-sync |
||||
* has already called {@see MagazineRefresher::refreshFromRelays}. |
||||
* |
||||
* @return list<array<int, string>> |
||||
*/ |
||||
public function getHomeCategoryAIndexTagsFromStoreOnly(): array |
||||
{ |
||||
return $this->categoryATagsFromStoredRoot(); |
||||
} |
||||
|
||||
/** |
||||
* @return list<array<int, string>> |
||||
*/ |
||||
private function categoryATagsFromStoredRoot(): array |
||||
{ |
||||
$npub = (string) $this->params->get('npub'); |
||||
$dTag = (string) $this->params->get('d_tag'); |
||||
$mag = $this->store->getRoot($npub, $dTag); |
||||
|
||||
return $this->categoryATagsFromMag($mag); |
||||
} |
||||
|
||||
/** |
||||
* @return list<array<int, string>> |
||||
*/ |
||||
private function categoryATagsFromMag(?Event $mag): array |
||||
{ |
||||
if ($mag === null) { |
||||
return []; |
||||
} |
||||
$tags = $mag->getTags(); |
||||
$cats = array_filter($tags, static function (mixed $tag): bool { |
||||
return \is_array($tag) && ($tag[0] ?? null) === 'a'; |
||||
}); |
||||
|
||||
return array_values($cats); |
||||
} |
||||
|
||||
private function shouldRevalidateRootFromRelay(): bool |
||||
{ |
||||
$age = $this->refresher->getSecondsSinceLastRelayRun(); |
||||
if ($age === null) { |
||||
return true; |
||||
} |
||||
|
||||
return $age > self::ROOT_REVALIDATE_SECONDS; |
||||
} |
||||
|
||||
/** |
||||
* @return array{list: list<Article>, category: array{title: string, summary: string}} |
||||
*/ |
||||
public function getCategoryPageData(string $slug): array |
||||
{ |
||||
$catIndex = $this->store->getCategory($slug); |
||||
if ($catIndex === null) { |
||||
$this->refresher->refreshFromRelays(20, [$slug]); |
||||
$catIndex = $this->store->getCategory($slug); |
||||
} |
||||
$list = []; |
||||
$coordinates = []; |
||||
$category = []; |
||||
if ($catIndex) { |
||||
foreach ($catIndex->getTags() as $tag) { |
||||
if ($tag[0] === 'title') { |
||||
$category['title'] = (string) $tag[1]; |
||||
} |
||||
if ($tag[0] === 'summary') { |
||||
$category['summary'] = (string) $tag[1]; |
||||
} |
||||
if ($tag[0] === 'a') { |
||||
$coordinates[] = $tag[1]; |
||||
} |
||||
} |
||||
} |
||||
|
||||
if (!empty($coordinates)) { |
||||
$pairs = []; |
||||
foreach ($coordinates as $coordinate) { |
||||
$parts = explode(':', (string) $coordinate, 3); |
||||
if (\count($parts) < 3) { |
||||
continue; |
||||
} |
||||
$slugPart = trim((string) $parts[2]); |
||||
if ($slugPart === '') { |
||||
continue; |
||||
} |
||||
$pairs[] = [ |
||||
'pubkey' => (string) $parts[1], |
||||
'slug' => $slugPart, |
||||
]; |
||||
} |
||||
$byAddress = $this->articleRepository->findByAuthorAndSlugIndexed($pairs); |
||||
$missing = []; |
||||
foreach ($coordinates as $coordinate) { |
||||
$parts = explode(':', (string) $coordinate, 3); |
||||
if (\count($parts) < 3) { |
||||
continue; |
||||
} |
||||
$k = (string) $parts[1]."\0".trim((string) $parts[2]); |
||||
if (!isset($byAddress[$k])) { |
||||
$missing[] = (string) $coordinate; |
||||
} |
||||
} |
||||
if ($missing !== []) { |
||||
$this->nostrClient->ingestMissingLongformForCategoryCoordinates($missing); |
||||
$byAddress = $this->articleRepository->findByAuthorAndSlugIndexed($pairs); |
||||
} |
||||
foreach ($coordinates as $coordinate) { |
||||
$parts = explode(':', (string) $coordinate, 3); |
||||
if (\count($parts) < 3) { |
||||
continue; |
||||
} |
||||
$k = (string) $parts[1]."\0".trim((string) $parts[2]); |
||||
if (isset($byAddress[$k])) { |
||||
$list[] = $byAddress[$k]; |
||||
} |
||||
} |
||||
} |
||||
|
||||
$category['title'] = $category['title'] ?? ''; |
||||
$category['summary'] = $category['summary'] ?? ''; |
||||
|
||||
return [ |
||||
'list' => $list, |
||||
'category' => $category, |
||||
]; |
||||
} |
||||
} |
||||
@ -0,0 +1,116 @@
@@ -0,0 +1,116 @@
|
||||
<?php |
||||
|
||||
declare(strict_types=1); |
||||
|
||||
namespace App\Service; |
||||
|
||||
use App\Entity\Event; |
||||
use Psr\Cache\CacheItemPoolInterface; |
||||
use Psr\Cache\InvalidArgumentException; |
||||
|
||||
/** |
||||
* Read/write persisted magazine Nostr index events (kinds 30040) without callback-based relay I/O |
||||
* on the request path. Updated by {@see MagazineRefresher} or the /ux/magazine-sync action. |
||||
*/ |
||||
final class MagazineIndexStore |
||||
{ |
||||
private const ROOT_PREFIX = 'mroot_v1_'; |
||||
private const CAT_PREFIX = 'mcat_v1_'; |
||||
|
||||
/** 30 days — we refresh on page load, TTL is a safety cap if sync stops working. */ |
||||
private const PERSIST_TTL = 2_592_000; |
||||
|
||||
public function __construct( |
||||
private readonly CacheItemPoolInterface $pool, |
||||
) { |
||||
} |
||||
|
||||
public function getRoot(string $npub, string $dTag): ?Event |
||||
{ |
||||
$item = $this->pool->getItem($this->rootKey($npub, $dTag)); |
||||
if (!$item->isHit()) { |
||||
return null; |
||||
} |
||||
|
||||
return $this->unwrap($item->get()); |
||||
} |
||||
|
||||
public function getCategory(string $slug): ?Event |
||||
{ |
||||
if ($slug === '') { |
||||
return null; |
||||
} |
||||
$item = $this->pool->getItem(self::CAT_PREFIX.$slug); |
||||
if (!$item->isHit()) { |
||||
return null; |
||||
} |
||||
|
||||
return $this->unwrap($item->get()); |
||||
} |
||||
|
||||
/** |
||||
* @throws InvalidArgumentException |
||||
*/ |
||||
public function putRoot(string $npub, string $dTag, Event $event): void |
||||
{ |
||||
$item = $this->pool->getItem($this->rootKey($npub, $dTag)); |
||||
$item->set(serialize($event)); |
||||
$item->expiresAfter(self::PERSIST_TTL); |
||||
$this->pool->save($item); |
||||
} |
||||
|
||||
/** |
||||
* @throws InvalidArgumentException |
||||
*/ |
||||
public function putCategory(string $slug, Event $event): void |
||||
{ |
||||
if ($slug === '') { |
||||
return; |
||||
} |
||||
$item = $this->pool->getItem(self::CAT_PREFIX.$slug); |
||||
$item->set(serialize($event)); |
||||
$item->expiresAfter(self::PERSIST_TTL); |
||||
$this->pool->save($item); |
||||
} |
||||
|
||||
/** |
||||
* Remove a cached category index (NIP-09 / local invalidation). |
||||
* |
||||
* @throws InvalidArgumentException |
||||
*/ |
||||
public function deleteCategory(string $slug): void |
||||
{ |
||||
if ($slug === '') { |
||||
return; |
||||
} |
||||
$this->pool->deleteItem(self::CAT_PREFIX.$slug); |
||||
} |
||||
|
||||
/** |
||||
* Remove the cached root magazine index for this npub + d_tag. |
||||
* |
||||
* @throws InvalidArgumentException |
||||
*/ |
||||
public function deleteRoot(string $npub, string $dTag): void |
||||
{ |
||||
$this->pool->deleteItem($this->rootKey($npub, $dTag)); |
||||
} |
||||
|
||||
private function rootKey(string $npub, string $dTag): string |
||||
{ |
||||
return self::ROOT_PREFIX.hash('sha256', $npub."\0".$dTag); |
||||
} |
||||
|
||||
private function unwrap(mixed $value): ?Event |
||||
{ |
||||
if (!\is_string($value) || $value === '') { |
||||
return null; |
||||
} |
||||
$e = unserialize($value, ['allowed_classes' => [Event::class]]); |
||||
if (!$e instanceof Event) { |
||||
return null; |
||||
} |
||||
|
||||
return $e; |
||||
} |
||||
} |
||||
@ -0,0 +1,177 @@
@@ -0,0 +1,177 @@
|
||||
<?php |
||||
|
||||
declare(strict_types=1); |
||||
|
||||
namespace App\Service; |
||||
|
||||
use App\Entity\Event; |
||||
use Psr\Cache\CacheItemPoolInterface; |
||||
use Psr\Cache\InvalidArgumentException; |
||||
use Psr\Log\LoggerInterface; |
||||
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBagInterface; |
||||
|
||||
/** |
||||
* Pulls magazine indices from relays within a wall-clock budget and persists them to {@see MagazineIndexStore}. |
||||
*/ |
||||
final class MagazineRefresher |
||||
{ |
||||
private const RELAY_STAMP_KEY = 'mag_relay_v1'; |
||||
|
||||
public function __construct( |
||||
private readonly NostrClient $nostrClient, |
||||
private readonly MagazineIndexStore $store, |
||||
private readonly ParameterBagInterface $params, |
||||
private readonly LoggerInterface $logger, |
||||
private readonly CacheItemPoolInterface $appCache, |
||||
) { |
||||
} |
||||
|
||||
/** |
||||
* Fetches the root index then each category index until $budgetSeconds elapses. $preferSlugs |
||||
* are requested first (e.g. current /cat route) so they are less likely to miss the budget. |
||||
*/ |
||||
public function refreshFromRelays(int $budgetSeconds = 8, array $preferSlugs = []): void |
||||
{ |
||||
$budgetSeconds = max(1, min(30, $budgetSeconds)); |
||||
$deadline = microtime(true) + $budgetSeconds; |
||||
$npub = (string) $this->params->get('npub'); |
||||
$dTag = (string) $this->params->get('d_tag'); |
||||
|
||||
// Do not set max_execution_time to the *remaining* soft budget: PHP resets the timer, so |
||||
// after a 6s root fetch, "2s left" would become a 2s hard cap for the *next* relay I/O |
||||
// (e.g. slow TLS) and can fatal. Cap once with headroom; the $deadline loop limits work. |
||||
$this->applyExecutionTimeCap($budgetSeconds); |
||||
|
||||
$defaultRelay = (string) $this->params->get('default_relay'); |
||||
$relayLabel = (string) (parse_url($defaultRelay, \PHP_URL_HOST) ?: $defaultRelay); |
||||
|
||||
$root = $this->nostrClient->getMagazineIndex($npub, $dTag); |
||||
if ($root === null) { |
||||
$this->logger->warning(sprintf( |
||||
'MagazineRefresher: root index not returned (tried from %s)', |
||||
$relayLabel |
||||
), [ |
||||
'd_tag' => $dTag, |
||||
'relay' => $defaultRelay, |
||||
]); |
||||
|
||||
return; |
||||
} |
||||
|
||||
$this->store->putRoot($npub, $dTag, $root); |
||||
|
||||
$slugs = $this->orderedCategorySlugs($this->categorySlugsFromRoot($root), $preferSlugs); |
||||
foreach ($slugs as $slug) { |
||||
if (microtime(true) >= $deadline) { |
||||
$this->logger->notice('MagazineRefresher: stopped at time budget; some categories not fetched', [ |
||||
'unprocessed_from' => $slug, |
||||
]); |
||||
break; |
||||
} |
||||
try { |
||||
$cat = $this->nostrClient->getMagazineIndex($npub, $slug); |
||||
if ($cat !== null) { |
||||
$this->store->putCategory($slug, $cat); |
||||
} |
||||
} catch (\Throwable $e) { |
||||
$this->logger->error(sprintf( |
||||
'MagazineRefresher: category fetch failed (relays from %s): %s', |
||||
$relayLabel, |
||||
$e->getMessage() |
||||
), [ |
||||
'slug' => $slug, |
||||
'message' => $e->getMessage(), |
||||
'relay' => $defaultRelay, |
||||
]); |
||||
} |
||||
} |
||||
|
||||
$this->touchLastRelayTime(); |
||||
} |
||||
|
||||
/** |
||||
* @throws InvalidArgumentException |
||||
*/ |
||||
public function getSecondsSinceLastRelayRun(): ?int |
||||
{ |
||||
try { |
||||
$item = $this->appCache->getItem(self::RELAY_STAMP_KEY); |
||||
} catch (InvalidArgumentException) { |
||||
return null; |
||||
} |
||||
if (!$item->isHit()) { |
||||
return null; |
||||
} |
||||
|
||||
return time() - (int) $item->get(); |
||||
} |
||||
|
||||
/** |
||||
* Child category indices are kind 30040; each root "a" tag is a NIP-33 address |
||||
* kind:hexpubkey:d-identifier. The third segment is the child #d (e.g. the long |
||||
* newsroom-…-category-… string), not a shortened title. |
||||
* |
||||
* @return list<string> |
||||
*/ |
||||
private function categorySlugsFromRoot(Event $root): array |
||||
{ |
||||
$slugs = []; |
||||
foreach ($root->getTags() as $tag) { |
||||
if (($tag[0] ?? null) !== 'a' || !isset($tag[1])) { |
||||
continue; |
||||
} |
||||
$parts = explode(':', (string) $tag[1], 3); |
||||
if (\count($parts) < 3) { |
||||
continue; |
||||
} |
||||
$s = trim((string) end($parts)); |
||||
if ($s !== '' && !\in_array($s, $slugs, true)) { |
||||
$slugs[] = $s; |
||||
} |
||||
} |
||||
|
||||
return $slugs; |
||||
} |
||||
|
||||
/** |
||||
* @param list<string> $allFromRoot |
||||
* @param list<string> $prefer |
||||
* @return list<string> |
||||
*/ |
||||
private function orderedCategorySlugs(array $allFromRoot, array $prefer): array |
||||
{ |
||||
$prefer = array_values(array_filter($prefer, static function (string $s): bool { |
||||
return $s !== ''; |
||||
})); |
||||
$out = $prefer; |
||||
foreach ($allFromRoot as $s) { |
||||
if (!\in_array($s, $out, true)) { |
||||
$out[] = $s; |
||||
} |
||||
} |
||||
|
||||
return $out; |
||||
} |
||||
|
||||
/** |
||||
* @throws InvalidArgumentException |
||||
*/ |
||||
private function touchLastRelayTime(): void |
||||
{ |
||||
$item = $this->appCache->getItem(self::RELAY_STAMP_KEY); |
||||
$item->set((string) time()); |
||||
$item->expiresAfter(86_400); |
||||
$this->appCache->save($item); |
||||
} |
||||
|
||||
/** |
||||
* One generous ceiling for PHP so relay/WebSocket I/O in one Nostr call can outlast the soft |
||||
* $deadline by seconds without a fatal, while the loop still stops *starting* new fetches in time. |
||||
*/ |
||||
private function applyExecutionTimeCap(int $budgetSeconds): void |
||||
{ |
||||
$sec = max(30, min(120, $budgetSeconds + 30)); |
||||
@set_time_limit($sec); |
||||
@ini_set('max_execution_time', (string) $sec); |
||||
} |
||||
} |
||||
@ -0,0 +1,359 @@
@@ -0,0 +1,359 @@
|
||||
<?php |
||||
|
||||
declare(strict_types=1); |
||||
|
||||
namespace App\Service; |
||||
|
||||
use App\Entity\Event as MagazineNostrEvent; |
||||
use App\Enum\KindsEnum; |
||||
use App\Repository\ArticleRepository; |
||||
use Doctrine\ORM\EntityManagerInterface; |
||||
use Psr\Log\LoggerInterface; |
||||
use swentel\nostr\Key\Key; |
||||
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBagInterface; |
||||
|
||||
/** |
||||
* Applies NIP-09 (kind 5) deletion requests to local MySQL articles and magazine 30040 cache. |
||||
* |
||||
* Relays are not authoritative; we only remove data we can validate (same pubkey as deletion request). |
||||
* For cached 30040 category indices (keyed by `d` only), we require the stored event’s author |
||||
* to match the deletion — not just an `a` tag whose own pubkey matches, so colliding `d` values |
||||
* across authors cannot wipe another author’s cache entry. |
||||
*/ |
||||
final class Nip09DeletionApplier |
||||
{ |
||||
public function __construct( |
||||
private readonly EntityManagerInterface $entityManager, |
||||
private readonly ArticleRepository $articleRepository, |
||||
private readonly MagazineIndexStore $magazineIndexStore, |
||||
private readonly ParameterBagInterface $params, |
||||
private readonly LoggerInterface $logger, |
||||
) { |
||||
} |
||||
|
||||
/** |
||||
* @param list<object> $deletionEvents Kind-5 events from relays (e.g. {@see NostrClient::fetchKind5DeletionEventsForAuthors}) |
||||
* |
||||
* @return array{articles_removed: int, magazine_roots: int, magazine_categories: int} |
||||
*/ |
||||
public function apply(array $deletionEvents): array |
||||
{ |
||||
$articlesRemoved = 0; |
||||
$articlesPendingFlush = 0; |
||||
$roots = 0; |
||||
$cats = 0; |
||||
$seenArticleIds = []; |
||||
|
||||
foreach ($deletionEvents as $ev) { |
||||
if (!\is_object($ev)) { |
||||
continue; |
||||
} |
||||
if ((int) ($ev->kind ?? 0) !== KindsEnum::DELETION_REQUEST->value) { |
||||
continue; |
||||
} |
||||
$deletionPubkey = (string) ($ev->pubkey ?? ''); |
||||
if (64 !== \strlen($deletionPubkey)) { |
||||
continue; |
||||
} |
||||
|
||||
[$eIds, $eKinds] = $this->parseETags($ev); |
||||
$aAddrs = $this->parseATags($ev); |
||||
|
||||
foreach ($eIds as $i => $eId) { |
||||
if (64 !== \strlen($eId)) { |
||||
continue; |
||||
} |
||||
$declared = $eKinds[$i] ?? null; |
||||
if ($declared !== null |
||||
&& !\in_array($declared, [30023, 30024, 30040, 1], true)) { |
||||
// Other kinds: we do not mirror in this app; skip. |
||||
continue; |
||||
} |
||||
if ($declared === 1) { |
||||
continue; |
||||
} |
||||
if ($this->removeArticleByEventIdIfValid($eId, $deletionPubkey, $declared, $seenArticleIds)) { |
||||
++$articlesRemoved; |
||||
++$articlesPendingFlush; |
||||
continue; |
||||
} |
||||
// No article row: 30040 index (or mis-tagged kind); only skip unrelated kinds. |
||||
if ($declared === null || \in_array($declared, [30023, 30024, 30040], true)) { |
||||
$mag = $this->tryRemoveMagazine30040ByEventId($eId, $deletionPubkey); |
||||
if ($mag === 1) { |
||||
++$roots; |
||||
} elseif ($mag === 2) { |
||||
++$cats; |
||||
} |
||||
} |
||||
} |
||||
|
||||
foreach ($aAddrs as $addr) { |
||||
$r = $this->removeByNip33Address($addr, $deletionPubkey, $seenArticleIds); |
||||
$articlesRemoved += $r['articles']; |
||||
$articlesPendingFlush += $r['articles']; |
||||
$roots += $r['roots']; |
||||
$cats += $r['cats']; |
||||
} |
||||
} |
||||
|
||||
if ($articlesPendingFlush > 0) { |
||||
try { |
||||
$this->entityManager->flush(); |
||||
} catch (\Throwable $e) { |
||||
$this->logger->error('Nip09DeletionApplier: flush failed', ['exception' => $e]); |
||||
} |
||||
} |
||||
|
||||
return [ |
||||
'articles_removed' => $articlesRemoved, |
||||
'magazine_roots' => $roots, |
||||
'magazine_categories' => $cats, |
||||
]; |
||||
} |
||||
|
||||
/** 0 = none, 1 = root cache, 2 = category cache */ |
||||
private function tryRemoveMagazine30040ByEventId(string $eventId, string $deletionPubkey): int |
||||
{ |
||||
$npub = (string) $this->params->get('npub'); |
||||
$dTag = (string) $this->params->get('d_tag'); |
||||
if ($npub === '' || $dTag === '') { |
||||
return 0; |
||||
} |
||||
$root = $this->magazineIndexStore->getRoot($npub, $dTag); |
||||
if ($root === null) { |
||||
return 0; |
||||
} |
||||
if ($this->eventIdMatches($root, $eventId) && $this->pubkeyEquals($root->getPubkey(), $deletionPubkey)) { |
||||
$this->magazineIndexStore->deleteRoot($npub, $dTag); |
||||
$this->logger->notice('NIP-09: removed cached magazine root index', [ |
||||
'event_id' => $eventId, |
||||
]); |
||||
|
||||
return 1; |
||||
} |
||||
foreach ($this->categorySlugsFromRoot($root) as $slug) { |
||||
$cat = $this->magazineIndexStore->getCategory($slug); |
||||
if ($cat === null) { |
||||
continue; |
||||
} |
||||
if ($this->eventIdMatches($cat, $eventId) && $this->pubkeyEquals($cat->getPubkey(), $deletionPubkey)) { |
||||
$this->magazineIndexStore->deleteCategory($slug); |
||||
$this->logger->notice('NIP-09: removed cached magazine category index', [ |
||||
'event_id' => $eventId, |
||||
'slug' => $slug, |
||||
]); |
||||
|
||||
return 2; |
||||
} |
||||
} |
||||
|
||||
return 0; |
||||
} |
||||
|
||||
private function eventIdMatches(MagazineNostrEvent $e, string $eventId): bool |
||||
{ |
||||
$a = strtolower($e->getId()); |
||||
$b = strtolower($eventId); |
||||
|
||||
return $a === $b; |
||||
} |
||||
|
||||
private function pubkeyEquals(string $a, string $b): bool |
||||
{ |
||||
if (64 !== \strlen($a) || 64 !== \strlen($b)) { |
||||
return $a === $b; |
||||
} |
||||
|
||||
return strtolower($a) === strtolower($b); |
||||
} |
||||
|
||||
/** |
||||
* @return list<string> |
||||
*/ |
||||
private function categorySlugsFromRoot(MagazineNostrEvent $root): array |
||||
{ |
||||
$slugs = []; |
||||
foreach ($root->getTags() as $tag) { |
||||
if (($tag[0] ?? null) !== 'a' || !isset($tag[1])) { |
||||
continue; |
||||
} |
||||
$parts = explode(':', (string) $tag[1], 3); |
||||
if (\count($parts) < 3) { |
||||
continue; |
||||
} |
||||
$s = trim((string) end($parts)); |
||||
if ($s !== '' && !\in_array($s, $slugs, true)) { |
||||
$slugs[] = $s; |
||||
} |
||||
} |
||||
|
||||
return $slugs; |
||||
} |
||||
|
||||
/** |
||||
* @param array<string, true> $seenArticleIds |
||||
*/ |
||||
private function removeArticleByEventIdIfValid( |
||||
string $eId, |
||||
string $deletionPubkey, |
||||
?int $declaredKind, |
||||
array &$seenArticleIds, |
||||
): bool { |
||||
if (isset($seenArticleIds[$eId])) { |
||||
return false; |
||||
} |
||||
$article = $this->articleRepository->findOneByEventId($eId); |
||||
if ($article === null) { |
||||
return false; |
||||
} |
||||
if (!$this->pubkeyEquals($article->getPubkey() ?? '', $deletionPubkey)) { |
||||
$this->logger->debug('NIP-09: ignore e tag (pubkey mismatch)', [ |
||||
'event_id' => $eId, |
||||
]); |
||||
|
||||
return false; |
||||
} |
||||
$k = $article->getKind()?->value; |
||||
if ($declaredKind !== null && $k !== null && $declaredKind !== $k) { |
||||
return false; |
||||
} |
||||
if ($k !== null && !\in_array($k, [KindsEnum::LONGFORM->value, KindsEnum::LONGFORM_DRAFT->value], true)) { |
||||
return false; |
||||
} |
||||
$this->entityManager->remove($article); |
||||
$seenArticleIds[$eId] = true; |
||||
$this->logger->notice('NIP-09: removed article from database', [ |
||||
'event_id' => $eId, |
||||
'kind' => $k, |
||||
]); |
||||
|
||||
return true; |
||||
} |
||||
|
||||
/** |
||||
* NIP-33: `kind:pubkeyhex:d-identifier` |
||||
* |
||||
* @param array<string, true> $seenArticleIds |
||||
* |
||||
* @return array{articles: int, roots: int, cats: int} |
||||
*/ |
||||
private function removeByNip33Address(string $addr, string $deletionPubkey, array &$seenArticleIds): array |
||||
{ |
||||
$out = ['articles' => 0, 'roots' => 0, 'cats' => 0]; |
||||
$parts = explode(':', $addr, 3); |
||||
if (\count($parts) < 3) { |
||||
return $out; |
||||
} |
||||
$kind = (int) $parts[0]; |
||||
$pk = (string) $parts[1]; |
||||
$d = trim((string) $parts[2]); |
||||
if ($d === '' || !$this->pubkeyEquals($pk, $deletionPubkey)) { |
||||
return $out; |
||||
} |
||||
|
||||
if ($kind === KindsEnum::LONGFORM->value || $kind === KindsEnum::LONGFORM_DRAFT->value) { |
||||
$article = $this->articleRepository->findOneBy(['pubkey' => $pk, 'slug' => $d]); |
||||
if ($article !== null) { |
||||
$eid = (string) ($article->getEventId() ?? ''); |
||||
$dedupeKey = $eid !== '' ? $eid : 'ps:'.$pk."\0".$d; |
||||
if (!isset($seenArticleIds[$dedupeKey])) { |
||||
$this->entityManager->remove($article); |
||||
$seenArticleIds[$dedupeKey] = true; |
||||
++$out['articles']; |
||||
$this->logger->notice('NIP-09: removed article (a tag)', [ |
||||
'address' => $addr, |
||||
]); |
||||
} |
||||
} |
||||
|
||||
return $out; |
||||
} |
||||
|
||||
if ($kind === KindsEnum::PUBLICATION_INDEX->value) { |
||||
$npub = (string) $this->params->get('npub'); |
||||
$siteD = (string) $this->params->get('d_tag'); |
||||
$siteHex = ''; |
||||
if (str_starts_with($npub, 'npub1')) { |
||||
try { |
||||
$h = (new Key())->convertToHex($npub); |
||||
if (64 === \strlen($h)) { |
||||
$siteHex = $h; |
||||
} |
||||
} catch (\Throwable) { |
||||
} |
||||
} |
||||
if ($npub !== '' && $siteD !== '' && $d === $siteD && $siteHex !== '' && $this->pubkeyEquals($pk, $siteHex)) { |
||||
$this->magazineIndexStore->deleteRoot($npub, $siteD); |
||||
++$out['roots']; |
||||
$this->logger->notice('NIP-09: removed magazine root (a tag)', ['address' => $addr]); |
||||
} else { |
||||
// Category cache is keyed by `d` only; the same d string can appear for different |
||||
// authors' 30040 events. Only remove if the cached event was authored by this deletion. |
||||
$cachedCat = $this->magazineIndexStore->getCategory($d); |
||||
if ($cachedCat === null) { |
||||
$this->logger->debug('NIP-09: skip category delete (nothing cached for d)', [ |
||||
'address' => $addr, |
||||
'd' => $d, |
||||
]); |
||||
} elseif (!$this->pubkeyEquals($cachedCat->getPubkey(), $deletionPubkey)) { |
||||
$this->logger->debug('NIP-09: skip category delete (cached index author != deletion author)', [ |
||||
'address' => $addr, |
||||
'd' => $d, |
||||
]); |
||||
} else { |
||||
$this->magazineIndexStore->deleteCategory($d); |
||||
++$out['cats']; |
||||
$this->logger->notice('NIP-09: removed magazine category (a tag)', [ |
||||
'address' => $addr, |
||||
'd' => $d, |
||||
]); |
||||
} |
||||
} |
||||
} |
||||
|
||||
return $out; |
||||
} |
||||
|
||||
/** |
||||
* @return array{0: list<string>, 1: list<?int>} e-ids and parallel k kinds (NIP-09 example order) |
||||
*/ |
||||
private function parseETags(object $ev): array |
||||
{ |
||||
$eIds = []; |
||||
$kinds = []; |
||||
foreach ($ev->tags ?? [] as $tag) { |
||||
if (!\is_array($tag) || !isset($tag[0], $tag[1])) { |
||||
continue; |
||||
} |
||||
if ($tag[0] === 'e') { |
||||
$eIds[] = (string) $tag[1]; |
||||
} |
||||
if ($tag[0] === 'k') { |
||||
$kinds[] = (int) $tag[1]; |
||||
} |
||||
} |
||||
$pairs = []; |
||||
for ($i = 0; $i < \count($eIds); ++$i) { |
||||
$pairs[] = $kinds[$i] ?? null; |
||||
} |
||||
|
||||
return [$eIds, $pairs]; |
||||
} |
||||
|
||||
/** |
||||
* @return list<string> NIP-33 addresses |
||||
*/ |
||||
private function parseATags(object $ev): array |
||||
{ |
||||
$a = []; |
||||
foreach ($ev->tags ?? [] as $tag) { |
||||
if (!\is_array($tag) || ($tag[0] ?? null) !== 'a' || !isset($tag[1])) { |
||||
continue; |
||||
} |
||||
$a[] = (string) $tag[1]; |
||||
} |
||||
|
||||
return $a; |
||||
} |
||||
} |
||||
@ -0,0 +1,3 @@
@@ -0,0 +1,3 @@
|
||||
<div class="category-body" data-magazine-sync-target="pageBody"> |
||||
<twig:Organisms:CardList :list="list" class="article-list" /> |
||||
</div> |
||||
@ -0,0 +1,10 @@
@@ -0,0 +1,10 @@
|
||||
<ul data-magazine-sync-target="headerNav"> |
||||
{% for category in cats %} |
||||
<li><twig:Molecules:CategoryLink :category="category" /></li> |
||||
{% endfor %} |
||||
{% if magazine_community_articles %} |
||||
<li> |
||||
<a href="{{ path('articles') }}">Latest Articles</a> |
||||
</li> |
||||
{% endif %} |
||||
</ul> |
||||
Loading…
Reference in new issue