Refactor markup generator for embedded events into Svelte snippets
- Eliminate a component that is no longer needed.
- Reduce duplicate code.
- Tidy up code along the way.
- Ran `deno fmt` to auto-format code (hence the large diff).
Alexandria is a reader and writer for curated publications, including e-books.
Alexandria is a reader and writer for curated publications, including e-books.
For a thorough introduction, please refer to our [project documention](https://next-alexandria.gitcitadel.eu/publication?d=gitcitadel-project-documentation-by-stella-v-1), viewable on Alexandria, or to the Alexandria [About page](https://next-alexandria.gitcitadel.eu/about).
It also contains a [universal event viewer](https://next-alexandria.gitcitadel.eu/events), with which you can search our relays, some aggregator relays, and your own relay list, to find and view event data.
It also contains a
[universal event viewer](https://next-alexandria.gitcitadel.eu/events), with
which you can search our relays, some aggregator relays, and your own relay
list, to find and view event data.
## Issues and Patches
## Issues and Patches
If you would like to suggest a feature or report a bug, please use the [Alexandria Contact page](https://next-alexandria.gitcitadel.eu/contact).
If you would like to suggest a feature or report a bug, please use the
You can also contact us [on Nostr](https://next-alexandria.gitcitadel.eu/events?id=nprofile1qqsggm4l0xs23qfjwnkfwf6fqcs66s3lz637gaxhl4nwd2vtle8rnfqprfmhxue69uhhg6r9vehhyetnwshxummnw3erztnrdaks5zhueg), directly.
Make sure that you have [Node.js](https://nodejs.org/en/download/package-manager) (v22 or above) or [Deno](https://docs.deno.com/runtime/getting_started/installation/) (v2) installed.
Make sure that you have
[Node.js](https://nodejs.org/en/download/package-manager) (v22 or above) or
Once you've cloned this repo, install dependencies with NPM:
Once you've cloned this repo, install dependencies with NPM:
@ -43,7 +55,8 @@ deno task dev
## Building
## Building
Alexandria is configured to run on a Node server. The [Node adapter](https://svelte.dev/docs/kit/adapter-node) works on Deno as well.
Alexandria is configured to run on a Node server. The
[Node adapter](https://svelte.dev/docs/kit/adapter-node) works on Deno as well.
To build a production version of your app with Node, use:
To build a production version of your app with Node, use:
@ -71,7 +84,8 @@ deno task preview
## Docker + Deno
## Docker + Deno
This application is configured to use the Deno runtime. A Docker container is provided to handle builds and deployments.
This application is configured to use the Deno runtime. A Docker container is
provided to handle builds and deployments.
To build the app for local development:
To build the app for local development:
@ -87,9 +101,11 @@ docker run -d -p 3000:3000 local-alexandria
## Testing
## Testing
_These tests are under development, but will run. They will later be added to the container._
_These tests are under development, but will run. They will later be added to
the container._
To run the Vitest suite we've built, install the program locally and run the tests.
To run the Vitest suite we've built, install the program locally and run the
tests.
```bash
```bash
npm run test
npm run test
@ -103,4 +119,8 @@ npx playwright test
## Markup Support
## Markup Support
Alexandria supports both Markdown and AsciiDoc markup for different content types. For a detailed list of supported tags and features in the basic and advanced markdown parsers, as well as information about AsciiDoc usage for publications and wikis, see [MarkupInfo.md](./src/lib/utils/markup/MarkupInfo.md).
Alexandria supports both Markdown and AsciiDoc markup for different content
types. For a detailed list of supported tags and features in the basic and
advanced markdown parsers, as well as information about AsciiDoc usage for
The relay selector will be a singleton that tracks, rates, and ranks Nostr relays to help the application determine which relay should be used to handle each request. It will weight relays based on observed characteristics, then use these weights to implement a weighted round robin algorithm for selecting relays, with some additional modifications to account for domain-specific features of Nostr.
The relay selector will be a singleton that tracks, rates, and ranks Nostr
relays to help the application determine which relay should be used to handle
each request. It will weight relays based on observed characteristics, then use
these weights to implement a weighted round robin algorithm for selecting
relays, with some additional modifications to account for domain-specific
features of Nostr.
## Relay Weights
## Relay Weights
@ -9,63 +14,92 @@ The relay selector will be a singleton that tracks, rates, and ranks Nostr relay
Relays are broadly divided into three categories:
Relays are broadly divided into three categories:
1. **Public**: no authorization is required
1. **Public**: no authorization is required
2. **Private Write**: authorization is required to write to this relay, but not to read
2. **Private Write**: authorization is required to write to this relay, but not
3. **Private Read and Write**: authorization is required to use any features of this relay
to read
3. **Private Read and Write**: authorization is required to use any features of
this relay
The broadest level of relay selection is based on these categories.
The broadest level of relay selection is based on these categories.
- For users that are not logged in, public relays are used exclusively.
- For users that are not logged in, public relays are used exclusively.
- For logged-in users, public and private read relays are initially rated equally for read operations.
- For logged-in users, public and private read relays are initially rated
- For logged-in users, private write relays are preferred above public relays for write operations.
equally for read operations.
- For logged-in users, private write relays are preferred above public relays
for write operations.
### User Preferences
### User Preferences
The relay selector will respect user relay preferences while still attempting to optimize for responsiveness and success rate.
The relay selector will respect user relay preferences while still attempting to
optimize for responsiveness and success rate.
- User inbox relays will be stored in a separate list from general-purpose relays, and weighted and sorted separately using the same algorithm as the general-purpose relay list.
- Local relays (beginning with `wss://localhost` or `ws://localhost`) will be stored _unranked_ in a separate list, and used when the relay selector is operating on a web browser (as opposed to a server).
- User inbox relays will be stored in a separate list from general-purpose
- When a caller requests relays from the relay selector, the selector will return:
relays, and weighted and sorted separately using the same algorithm as the
general-purpose relay list.
- Local relays (beginning with `wss://localhost` or `ws://localhost`) will be
stored _unranked_ in a separate list, and used when the relay selector is
operating on a web browser (as opposed to a server).
- When a caller requests relays from the relay selector, the selector will
return:
- The highest-ranked general-purpose relay
- The highest-ranked general-purpose relay
- The highest-ranked user inbox relay
- The highest-ranked user inbox relay
- (If on browser) any local relays
- (If on browser) any local relays
### Weighted Metrics
### Weighted Metrics
Several weighted metrics are used to compute a relay's score. The score is used to rank relays to determine which to prefer when fetching events.
Several weighted metrics are used to compute a relay's score. The score is used
to rank relays to determine which to prefer when fetching events.
#### Response Time
#### Response Time
The response time weight of each relay is computed according to the logarithmic function $`r(t) = -log(t) + 1`$, where $`t`$ is the median response time in seconds. This function has a few features which make it useful:
The response time weight of each relay is computed according to the logarithmic
function $`r(t) = -log(t) + 1`$, where $`t`$ is the median response time in
seconds. This function has a few features which make it useful:
- $`r(1) = 1`$, making a response time of 1s the netural point. This causes the algorithm to prefer relays that respond in under 1s.
- $`r(1) = 1`$, making a response time of 1s the netural point. This causes the
- $`r(0.3) \approx 1.5`$ and $`r(3) \approx 0.5`$. This clusters the 0.5 to 1.5 weight range in the 300ms to 3s response time range, which is a sufficiently rapid response time to keep user's from switching context.
algorithm to prefer relays that respond in under 1s.
- The function has a long tail, so it doesn't discount slower response times too heavily, too quickly.
- $`r(0.3) \approx 1.5`$ and $`r(3) \approx 0.5`$. This clusters the 0.5 to 1.5
weight range in the 300ms to 3s response time range, which is a sufficiently
rapid response time to keep user's from switching context.
- The function has a long tail, so it doesn't discount slower response times too
heavily, too quickly.
#### Success Rate
#### Success Rate
The success rate $`s(x)`$ is computed as the fraction of total requests sent to the relay that returned at least one event in response. The optimal score is 1, meaning the relay successfully responds to 100% of requests.
The success rate $`s(x)`$ is computed as the fraction of total requests sent to
the relay that returned at least one event in response. The optimal score is 1,
meaning the relay successfully responds to 100% of requests.
#### Trust Level
#### Trust Level
Certain relays may be assigned a constant "trust level" score $`T`$. This modifier is a number in the range $`[-0.5, 0.5]`$ that indicates how much a relay is trusted by the GitCitadel organization.
Certain relays may be assigned a constant "trust level" score $`T`$. This
modifier is a number in the range $`[-0.5, 0.5]`$ that indicates how much a
relay is trusted by the GitCitadel organization.
A few factors contribute to a higher trust rating:
A few factors contribute to a higher trust rating:
- Effective filtering of spam and abusive content.
- Effective filtering of spam and abusive content.
- Good data transparency, including such policies as honoring deletion requests.
- Good data transparency, including such policies as honoring deletion requests.
- Event aggregation policies that aim at synchronization with the broader relay network.
- Event aggregation policies that aim at synchronization with the broader relay
network.
#### Preferred Vendors
#### Preferred Vendors
Certain relays may be assigned a constant "preferred vendor" score $`V`$. This modifier is a number in the range $`[0, 0.5]`$. It is used to increase the priority of GitCitadel's preferred relay vendors.
Certain relays may be assigned a constant "preferred vendor" score $`V`$. This
modifier is a number in the range $`[0, 0.5]`$. It is used to increase the
priority of GitCitadel's preferred relay vendors.
### Overall Weight
### Overall Weight
The overall weight of a relay is calculated as $`w(t, x) = r(t) \times s(x) + T + V`$. The `RelaySelector` class maintains a list of relays sorted by their overall weights. The weights may be updated at runtime when $`t`$ or $`x`$ change. On update, the relay list is re-sorted to account for the new weights.
The overall weight of a relay is calculated as
$`w(t, x) = r(t) \times s(x) + T + V`$. The `RelaySelector` class maintains a
list of relays sorted by their overall weights. The weights may be updated at
runtime when $`t`$ or $`x`$ change. On update, the relay list is re-sorted to
account for the new weights.
## Algorithm
## Algorithm
The relay weights contribute to a weighted round robin (WRR) algorithm for relay selection. Pseudocode for the algorithm is given below:
The relay weights contribute to a weighted round robin (WRR) algorithm for relay
selection. Pseudocode for the algorithm is given below:
```pseudocode
```pseudocode
Constants and Variables:
Constants and Variables:
@ -86,11 +120,13 @@ Function getRelay:
## Class Methods
## Class Methods
The `RelaySelector` class should expose the following methods to support updates to relay weights. Pseudocode for each method is given below.
The `RelaySelector` class should expose the following methods to support updates
to relay weights. Pseudocode for each method is given below.
### Add Response Time Datum
### Add Response Time Datum
This function updates the class state by side effect. Locking should be used in concurrent use cases.
This function updates the class state by side effect. Locking should be used in
concurrent use cases.
```pseudocode
```pseudocode
Constants and Variables:
Constants and Variables:
@ -123,7 +159,8 @@ Function addResponseTimeDatum:
### Add Success Rate Datum
### Add Success Rate Datum
This function updates the class state by side effect. Locking should be used in concurrent use cases.
This function updates the class state by side effect. Locking should be used in
@ -142,7 +173,7 @@ export function deduplicateAndCombineEvents(
constfinalEventMap=newMap<string,NDKEvent>();
constfinalEventMap=newMap<string,NDKEvent>();
constseenCoordinates=newSet<string>();
constseenCoordinates=newSet<string>();
allEventsToProcess.forEach(event=>{
allEventsToProcess.forEach((event)=>{
if(!event.id)return;
if(!event.id)return;
// For replaceable events, only add if it's the chosen version
// For replaceable events, only add if it's the chosen version
@ -174,11 +205,20 @@ export function deduplicateAndCombineEvents(
// Log deduplication results if any duplicates were found
// Log deduplication results if any duplicates were found
if(duplicateCoordinatesFound>0){
if(duplicateCoordinatesFound>0){
console.log(`[eventDeduplication] deduplicateAndCombineEvents: Found ${duplicateCoordinatesFound} duplicate coordinates out of ${replaceableEventsProcessed} replaceable events`);
console.log(
console.log(`[eventDeduplication] deduplicateAndCombineEvents: Reduced from ${initialCount} to ${finalCount} events (${reduction} removed)`);
`[eventDeduplication] deduplicateAndCombineEvents: Found ${duplicateCoordinatesFound} duplicate coordinates out of ${replaceableEventsProcessed} replaceable events`,
Alexandria supports multiple markup formats for different use cases. Below is a summary of the supported tags and features for each parser, as well as the formats used for publications and wikis.
Alexandria supports multiple markup formats for different use cases. Below is a
summary of the supported tags and features for each parser, as well as the
formats used for publications and wikis.
## Basic Markup Parser
## Basic Markup Parser
The **basic markup parser** follows the [Nostr best-practice guidelines](https://github.com/nostrability/nostrability/issues/146) and supports:
@ -18,7 +22,8 @@ The **basic markup parser** follows the [Nostr best-practice guidelines](https:/
- **Links:**`[text](url)`
- **Links:**`[text](url)`
- **Images:**``
- **Images:**``
- **Hashtags:**`#hashtag`
- **Hashtags:**`#hashtag`
- **Nostr identifiers:** npub, nprofile, nevent, naddr, note, with or without `nostr:` prefix (note is deprecated)
- **Nostr identifiers:** npub, nprofile, nevent, naddr, note, with or without
`nostr:` prefix (note is deprecated)
- **Emoji shortcodes:**`:smile:` will render as 😄
- **Emoji shortcodes:**`:smile:` will render as 😄
## Advanced Markup Parser
## Advanced Markup Parser
@ -26,17 +31,25 @@ The **basic markup parser** follows the [Nostr best-practice guidelines](https:/
The **advanced markup parser** includes all features of the basic parser, plus:
The **advanced markup parser** includes all features of the basic parser, plus:
- **Inline code:** `` `code` ``
- **Inline code:** `` `code` ``
- **Syntax highlighting:** for code blocks in many programming languages (from [highlight.js](https://highlightjs.org/))
- **Syntax highlighting:** for code blocks in many programming languages (from
[highlight.js](https://highlightjs.org/))
- **Tables:** Pipe-delimited tables with or without headers
- **Tables:** Pipe-delimited tables with or without headers
- **Footnotes:**`[^1]` or `[^Smith]`, which should appear where the footnote shall be placed, and will be displayed as unique, consecutive numbers
- **Footnotes:**`[^1]` or `[^Smith]`, which should appear where the footnote
- **Footnote References:**`[^1]: footnote text` or `[^Smith]: Smith, Adam. 1984 "The Wiggle Mysteries`, which will be listed in order, at the bottom of the event, with back-reference links to the footnote, and text footnote labels appended
shall be placed, and will be displayed as unique, consecutive numbers
- **Wikilinks:**`[[NIP-54]]` will render as a hyperlink and goes to [NIP-54](./events?d=nip-54)
- **Footnote References:**`[^1]: footnote text` or
`[^Smith]: Smith, Adam. 1984 "The Wiggle Mysteries`, which will be listed in
order, at the bottom of the event, with back-reference links to the footnote,
and text footnote labels appended
- **Wikilinks:**`[[NIP-54]]` will render as a hyperlink and goes to
[NIP-54](./events?d=nip-54)
## Publications and Wikis
## Publications and Wikis
**Publications** and **wikis** in Alexandria use **AsciiDoc** as their primary markup language, not Markdown.
**Publications** and **wikis** in Alexandria use **AsciiDoc** as their primary
markup language, not Markdown.
AsciiDoc supports a much broader set of formatting, semantic, and structural features, including:
AsciiDoc supports a much broader set of formatting, semantic, and structural
features, including:
- Section and document structure
- Section and document structure
- Advanced tables, callouts, admonitions
- Advanced tables, callouts, admonitions
@ -48,7 +61,8 @@ AsciiDoc supports a much broader set of formatting, semantic, and structural fea
### Advanced Content Types
### Advanced Content Types
Alexandria supports rendering of advanced content types commonly used in academic, technical, and business documents:
Alexandria supports rendering of advanced content types commonly used in
academic, technical, and business documents:
#### Math Rendering
#### Math Rendering
@ -113,18 +127,26 @@ TikZ diagrams for mathematical illustrations:
### Rendering Features
### Rendering Features
- **Automatic Detection**: Content types are automatically detected based on syntax
- **Automatic Detection**: Content types are automatically detected based on
- **Fallback Display**: If rendering fails, the original source code is displayed
syntax
- **Fallback Display**: If rendering fails, the original source code is
displayed
- **Source Code**: Click "Show source" to view the original code
- **Source Code**: Click "Show source" to view the original code
- **Responsive Design**: All rendered content is responsive and works on mobile devices
- **Responsive Design**: All rendered content is responsive and works on mobile
devices
For more information on AsciiDoc, see the [AsciiDoc documentation](https://asciidoc.org/).
For more information on AsciiDoc, see the
[AsciiDoc documentation](https://asciidoc.org/).
---
---
**Note:**
**Note:**
- The markdown parsers are primarily used for comments, issues, and other user-generated content.
- The markdown parsers are primarily used for comments, issues, and other
- Publications and wikis are rendered using AsciiDoc for maximum expressiveness and compatibility.
user-generated content.
- All URLs are sanitized to remove tracking parameters, and YouTube links are presented in a clean, privacy-friendly format.
- Publications and wikis are rendered using AsciiDoc for maximum expressiveness
- [Here is a test markup file](/tests/integration/markupTestfile.md) that you can use to test out the parser and see how things should be formatted.
and compatibility.
- All URLs are sanitized to remove tracking parameters, and YouTube links are
presented in a clean, privacy-friendly format.
- [Here is a test markup file](/tests/integration/markupTestfile.md) that you
can use to test out the parser and see how things should be formatted.
# This is a testfile for writing mathematic formulas in NostrMarkup
# This is a testfile for writing mathematic formulas in NostrMarkup
This document covers the rendering of formulas in TeX/LaTeX and AsciiMath notation, or some combination of those within the same page. It is meant to be rendered by clients utilizing MathJax.
This document covers the rendering of formulas in TeX/LaTeX and AsciiMath
notation, or some combination of those within the same page. It is meant to be
If you want the entire document to be rendered as mathematics, place the entire thing in a backtick-codeblock, but know that this makes the document slower to load, it is harder to format the prose, and the result is less legible. It also doesn't increase portability, as it's easy to export markup as LaTeX files, or as PDFs, with the formulas rendered.
rendered by clients utilizing MathJax.
The general idea, is that anything placed within `single backticks` is inline code, and inline-code will all be scanned for typical mathematics statements and rendered with best-effort. (For more precise rendering, use Asciidoc.) We will not render text that is not marked as inline code, as mathematical formulas, as that is prose.
If you want the entire document to be rendered as mathematics, place the entire
thing in a backtick-codeblock, but know that this makes the document slower to
If you want the TeX to be blended into the surrounding text, wrap the text within single `$`. Otherwise, use double `$$` symbols, for display math, and it will appear on its own line.
load, it is harder to format the prose, and the result is less legible. It also
doesn't increase portability, as it's easy to export markup as LaTeX files, or
as PDFs, with the formulas rendered.
The general idea, is that anything placed within `single backticks` is inline
code, and inline-code will all be scanned for typical mathematics statements and
rendered with best-effort. (For more precise rendering, use Asciidoc.) We will
not render text that is not marked as inline code, as mathematical formulas, as
that is prose.
If you want the TeX to be blended into the surrounding text, wrap the text
within single `$`. Otherwise, use double `$$` symbols, for display math, and it
will appear on its own line.
## TeX Examples
## TeX Examples
@ -16,36 +28,25 @@ Same equation, in the display mode: `$$\sqrt{x}$$`
Something more complex, inline: `$\mathbb{N} = \{ a \in \mathbb{Z} : a > 0 \}$`
Something more complex, inline: `$\mathbb{N} = \{ a \in \mathbb{Z} : a > 0 \}$`
LaTeX ypesetting won't be rendered. Use NostrMarkup delimeter tables for this sort of thing.
LaTeX ypesetting won't be rendered. Use NostrMarkup delimeter tables for this
sort of thing.
`\\begin{tabular}{|c|c|c|l|r|}
`\\begin{tabular}{|c|c|c|l|r|}
\\hline
\\hline
@ -69,13 +70,17 @@ We also recognize common LaTeX statements:
Greek letters are a snap: `$\Psi$`, `$\psi$`, `$\Phi$`, `$\phi$`.
Greek letters are a snap: `$\Psi$`, `$\psi$`, `$\Phi$`, `$\phi$`.
Equations within text are easy--- A well known Maxwell thermodynamic relation is `$\left.{\partial T \over \partial P}\right|_{s} = \left.{\partial v \over \partial s}\right|_{P}$`.
Equations within text are easy--- A well known Maxwell thermodynamic relation is
`$\left.{\partial T \over \partial P}\right|_{s} = \left.{\partial v \over \partial s}\right|_{P}$`.
You can also set aside equations like so: `\begin{eqnarray} du &=& T\ ds -P\ dv, \qquad \mbox{first law.}\label{fl}\\ ds &\ge& {\delta q \over T}.\qquad \qquad \mbox{second law.} \label{sl} \end {eqnarray}`
Asciimath doesn't use `$` or `$$` delimiters, but we are using it to make mathy stuff easier to find. If you want it inline, include it inline. If you want it on a separate line, put a hard-return before and after.
Asciimath doesn't use `$` or `$$` delimiters, but we are using it to make mathy
stuff easier to find. If you want it inline, include it inline. If you want it
on a separate line, put a hard-return before and after.
Inline text example here `$E=mc^2$` and another `$1/(x+1)$`; very simple.
Inline text example here `$E=mc^2$` and another `$1/(x+1)$`; very simple.
@ -109,19 +114,23 @@ Using the quadratic formula, the roots of `$x^2-6x+4=0$` are
Advanced alignment and matrices looks like this:
Advanced alignment and matrices looks like this:
A `$3xx3$` matrix, `$$((1,2,3),(4,5,6),(7,8,9))$$` and a `$2xx1$` matrix, or vector, `$$((1),(0))$$`.
A `$3xx3$` matrix, `$$((1,2,3),(4,5,6),(7,8,9))$$` and a `$2xx1$` matrix, or
vector, `$$((1),(0))$$`.
The outer brackets determine the delimiters e.g. `$|(a,b),(c,d)|=ad-bc$`.
The outer brackets determine the delimiters e.g. `$|(a,b),(c,d)|=ad-bc$`.
A general `$m xx n$` matrix `$$((a_(11), cdots , a_(1n)),(vdots, ddots, vdots),(a_(m1), cdots , a_(mn)))$$`
expect(indexEvent.tags).toContainEqual(["title","Document with Special Characters: Test & More!"]);
"d",
"document-with-special-characters-test-more",
]);
expect(indexEvent.tags).toContainEqual([
"title",
"Document with Special Characters: Test & More!",
]);
expect(sectionEvents).toHaveLength(1);
expect(sectionEvents).toHaveLength(1);
});
});
it("should handle document with very long title",()=>{
it("should handle document with very long title",()=>{
constcontent=`= This is a very long document title that should be handled properly by the system and should not cause any issues with the d-tag generation or any other functionality
constcontent=
`= This is a very long document title that should be handled properly by the system and should not cause any issues with the d-tag generation or any other functionality
expect(indexEvent.tags).toContainEqual(["title","This is a very long document title that should be handled properly by the system and should not cause any issues with the d-tag generation or any other functionality"]);
expect(indexEvent.tags).toContainEqual([
"title",
"This is a very long document title that should be handled properly by the system and should not cause any issues with the d-tag generation or any other functionality",
buttercat1791
7 months ago