checkpoint - zetteleditor and compose tree

doc/compose_tree.md

@@ -0,0 +1,173 @@
+# NKBIP-01 Hierarchical Parsing Technical Plan
+
+## Overview
+
+This document outlines the complete restart plan for implementing NKBIP-01 compliant hierarchical AsciiDoc parsing using proper Asciidoctor tree processor extensions.
+
+## Current State Analysis
+
+### Problems Identified
+1. **Dual Architecture Conflict**: Two competing parsing implementations exist:
+   - `publication_tree_factory.ts` - AST-first approach (currently used)
+   - `publication_tree_extension.ts` - Extension approach (incomplete)
+
+2. **Missing Proper Extension Registration**: Current code doesn't follow the official Asciidoctor extension pattern you provided
+
+3. **Incomplete NKBIP-01 Compliance**: Testing with `deep_hierarchy_test.adoc` may not produce the exact structures shown in `docreference.md`
+
+## NKBIP-01 Specification Summary
+
+From `test_data/AsciidocFiles/docreference.md`:
+
+### Event Types
+- **30040**: Index events (collections/hierarchical containers)
+- **30041**: Content events (actual article sections)
+
+### Parse Level Behaviors
+- **Level 2**: Only `==` sections → 30041 events (subsections included in content)
+- **Level 3**: `==` → 30040 indices, `===` → 30041 content events  
+- **Level 4+**: Full hierarchy with each level becoming separate events
+
+### Key Rules
+1. If a section has subsections at target level → becomes 30040 index
+2. If no subsections at target level → becomes 30041 content event
+3. Content inclusion: 30041 events include all content below parse level
+4. Hierarchical references: Parent indices use `a` tags to reference children
+
+## Proposed Architecture
+
+### Core Pattern: Asciidoctor Tree Processor Extension
+
+Following the pattern you provided:
+
+```javascript
+// Extension registration pattern
+module.exports = function (registry) {
+  registry.treeProcessor(function () {
+    var self = this
+    self.process(function (doc) {
+      // Process document and build PublicationTree
+      return doc
+    })
+  })
+}
+```
+
+### Implementation Components
+
+1. **PublicationTreeProcessor** (`src/lib/utils/publication_tree_processor.ts`)
+   - Implements the tree processor extension
+   - Registers with Asciidoctor during document processing
+   - Builds PublicationTree with NDK events during AST traversal
+   - Returns result via closure to avoid Ruby compatibility issues
+
+2. **Unified Parser Interface** (`src/lib/utils/asciidoc_publication_parser.ts`)
+   - Single entry point for all parsing operations
+   - Manages extension registration and cleanup
+   - Provides clean API for ZettelEditor integration
+
+3. **Enhanced ZettelEditor Integration**
+   - Replace `publication_tree_factory.ts` usage
+   - Use proper extension-based parsing
+   - Maintain current preview and publishing workflow
+
+## Technical Implementation Plan
+
+### Phase 1: Core Tree Processor (`publication_tree_processor.ts`)
+
+```typescript
+export function registerPublicationTreeProcessor(
+  registry: Registry,
+  ndk: NDK,
+  parseLevel: number,
+  options?: ProcessorOptions
+): { getResult: () => ProcessorResult | null }
+```
+
+**Key Features:**
+- Follows Asciidoctor extension pattern exactly
+- Builds events during AST traversal (not after)
+- Preserves original AsciiDoc content in events
+- Handles all parse levels (2-7) with proper NKBIP-01 compliance
+- Uses closure pattern to return results safely
+
+### Phase 2: Unified Parser Interface (`asciidoc_publication_parser.ts`)
+
+```typescript
+export async function parseAsciiDocWithTree(
+  content: string,
+  ndk: NDK,
+  parseLevel: number = 2
+): Promise<PublicationTreeResult>
+```
+
+**Responsibilities:**
+- Create Asciidoctor instance
+- Register tree processor extension
+- Execute parsing with extension
+- Return PublicationTree and events
+- Clean up resources
+
+### Phase 3: ZettelEditor Integration
+
+**Changes to `ZettelEditor.svelte`:**
+- Replace `createPublicationTreeFromContent()` calls
+- Use new `parseAsciiDocWithTree()` function
+- Maintain existing preview/publishing interface
+- No changes to component props or UI
+
+### Phase 4: Validation Testing
+
+**Test Suite:**
+1. Parse `deep_hierarchy_test.adoc` at levels 2-7
+2. Verify event structures match `docreference.md` examples
+3. Validate content preservation and tag inheritance
+4. Test publish workflow end-to-end
+
+## File Organization
+
+### Files to Create
+1. `src/lib/utils/publication_tree_processor.ts` - Core tree processor extension
+2. `src/lib/utils/asciidoc_publication_parser.ts` - Unified parser interface
+3. `tests/unit/publication_tree_processor.test.ts` - Comprehensive test suite
+
+### Files to Modify
+1. `src/lib/components/ZettelEditor.svelte` - Update parsing calls
+2. `src/routes/new/compose/+page.svelte` - Verify integration works
+
+### Files to Remove (After Validation)
+1. `src/lib/utils/publication_tree_factory.ts` - Replace with processor
+2. `src/lib/utils/publication_tree_extension.ts` - Merge concepts into processor
+
+## Success Criteria
+
+1. **NKBIP-01 Compliance**: All parse levels produce structures exactly matching `docreference.md`
+2. **Content Preservation**: Original AsciiDoc content preserved in events (not converted to HTML)
+3. **Proper Extension Pattern**: Uses official Asciidoctor tree processor registration
+4. **Zero Regression**: Current ZettelEditor functionality unchanged
+5. **Performance**: No degradation in parsing or preview speed
+6. **Test Coverage**: Comprehensive validation with `deep_hierarchy_test.adoc`
+
+## Development Sequence
+
+1. **Study & Plan** ✓ (Current phase)
+2. **Implement Core Processor** - Create `publication_tree_processor.ts`
+3. **Build Unified Interface** - Create `asciidoc_publication_parser.ts`  
+4. **Integrate with ZettelEditor** - Update parsing calls
+5. **Validate with Test Documents** - Verify NKBIP-01 compliance
+6. **Clean Up Legacy Code** - Remove old implementations
+7. **Documentation & Testing** - Comprehensive test suite
+
+## Risk Mitigation
+
+- **Incremental Integration**: Keep old code until new implementation validated
+- **Extensive Testing**: Use both test documents for validation
+- **Performance Monitoring**: Ensure no degradation in user experience
+- **Rollback Plan**: Can revert to `publication_tree_factory.ts` if needed
+
+## References
+
+- NKBIP-01 Specification: `test_data/AsciidocFiles/docreference.md`
+- Test Document: `test_data/AsciidocFiles/deep_hierarchy_test.adoc`
+- Asciidoctor Extensions: [Official Documentation](https://docs.asciidoctor.org/asciidoctor.js/latest/extend/extensions/)
+- Current Implementation: `src/lib/components/ZettelEditor.svelte:64`

src/lib/components/ZettelEditor.svelte

@@ -48,6 +48,10 @@ import Asciidoctor from "asciidoctor";
   let contentType = $state<'article' | 'scattered-notes' | 'none'>('none');
 
   // Effect to create PublicationTree when content changes
+  // Uses tree processor extension as Michael envisioned:
+  // "register a tree processor extension in our Asciidoctor instance"
+  // "use the AST that Asciidoctor generates during parsing"
+  // "publication tree side-loads into memory as AsciiDoc is parsed"
   $effect(() => {
     if (!content.trim() || !ndk) {
       publicationResult = null;
@@ -56,27 +60,28 @@ import Asciidoctor from "asciidoctor";
       return;
     }
     
-    // Create PublicationTree asynchronously
+    // Use tree factory with corrected AST parser
     createPublicationTreeFromContent(content, ndk, parseLevel)
       .then(result => {
+        console.log("Tree factory result:", result);
         publicationResult = result;
         contentType = result.metadata.contentType;
         
-        // Export events for compatibility
+        // Export events for publishing workflow
         return exportEventsFromTree(result);
       })
       .then(events => {
         generatedEvents = events;
-        console.log("AST-based events generated:", {
+        
+        console.log("Tree factory result:", {
           contentType,
           indexEvent: !!events.indexEvent,
           contentEvents: events.contentEvents.length,
           parseLevel: parseLevel
         });
-        console.log("Updated generatedEvents state:", generatedEvents);
       })
       .catch(error => {
-        console.error("PublicationTree creation error:", error);
+        console.error("Tree factory error:", error);
         publicationResult = null;
         generatedEvents = null;
         contentType = 'none';
@@ -119,6 +124,12 @@ import Asciidoctor from "asciidoctor";
   let parsedSections = $derived.by(() => {
     if (!publicationResult) return [];
     
+    console.log("Preview: publicationResult structure:", {
+      hasContentEvents: !!publicationResult.contentEvents,
+      contentEventsLength: publicationResult.contentEvents?.length,
+      keys: Object.keys(publicationResult)
+    });
+    
     // Convert PublicationTree events to preview format
     return publicationResult.contentEvents.map((event: any) => {
       const title = event.tags.find((t: string[]) => t[0] === 'title')?.[1] || 'Untitled';