Lean Blueprint and Sphinxdocrs Roadmap

Status as of 2026-11-11. This plan extends dsport’s existing Sphinxdocrs H14 work; it does not restart the Rust port of Sphinx.

Current Baseline

  • Signals uses Lean v4.35.0-rc3 and pins Mathlib to the matching release. Its blueprint target runs LeanArchitect’s package-wide facet for all four Signals libraries (:blueprint and :blueprintJson, completing at 3,628 core / 3,811 doc-gen4 Lake jobs across 35 Signals modules), sanitizes generated .artifacts/*.tex files via build_api_docs.py --sanitize-blueprint-tex, and renders the HTML with plasTeX. See src/signals/Makefile:45-105, src/signals/build_api_docs.py:92-265, src/signals/lakefile.toml:5-15, src/signals/blueprint/src/content.tex, and plan-index.md.

  • The Sphinx HTML build (docs/conf.py:_build_signals_blueprint) checks a content-addressed SHA-256 stamp (src/signals/.lake/build/sphinx-blueprint-inputs.sha256) over all src/signals Lean sources, blueprint/src inputs (excluding *.paux), Makefile, and build_api_docs.py, skipping subprocess invocation entirely when inputs are unchanged and streaming build output to .tmp/signals-blueprint.log when a rebuild is needed. See docs/conf.py:208-276, docs/_toc.yml:43, docs/signals-blueprint.md:1-13, and src/signals/Makefile:45-105.

Root-Cause Diagnosis and Remediation of signals-api / plasTeX Bottlenecks (2026-11-11)

1. Why plasTeX Repeatedly Printed [ sect0002.html ] and WARNING: Using default renderer for active::&

With split-level = 0 in src/signals/blueprint/src/plastex.cfg, the single \chapter{Signals Lean library} in src/signals/blueprint/src/content.tex should emit only index.html, sect0001.html (containing Sections 1.1–1.9 and all 1,540 formal nodes), and dep_graph_document.html.

Investigation revealed four TeX tokenization hazards in the raw @[blueprint (title := "...")] and (statement := /-- ... -/) strings emitted by LeanArchitect into .lake/build/blueprint/module/**/*.artifacts/*.tex:

  1. Unbalanced square brackets inside \begin{definition}[title] (specifically half-open intervals such as e ∈ [0, 1) in def:signals-ddf-planetary-orbit): plasTeX’s TeXFragment.readArgument balances [ and ] tokens even inside {...} groups. Because [0, 1) has an opening [ and a closing ), the bracket depth counter incremented from 1 to 2 at [0, 1) and only decremented back to 1 at the end of \begin{definition}[...]. As a result, plasTeX consumed the closing ] as part of the title and swallowed ~2,000 lines of subsequent \section and theorem nodes inside <span class="definition_thmtitle"> until encountering another ].

  2. Unescaped % signs in titles and statements (e.g., 50%, 98%, 37.5%): treated by LaTeX/plasTeX as comment characters (catcode 14), commenting out the closing ] on line 1 of .artifacts/*.tex.

  3. Unescaped & outside matrix environments (e.g., Al/Si & Al/GaN in titles or prose): parsed as TeX alignment tab tokens (active::&, catcode 4), triggering WARNING: Using default renderer for active::&, corrupting the DOM section stack, and causing plasTeX to spawn a phantom empty sect0002.html page ([ sect0002.html ] [ sect0002.html ] ...).

  4. Unescaped currency $ in titles (e.g., $876k/yr): opened an unclosed math environment inside the optional [title] argument.

Fix Implemented (src/signals/build_api_docs.py:sanitize_blueprint_tex): Added an idempotent pre-plasTeX sanitizer (python3 build_api_docs.py --sanitize-blueprint-tex) that rewrites generated .artifacts/*.tex files in place:

  • Escapes unescaped % \(\to\) \%,

  • Escapes unescaped & \(\to\) \& outside \begin{*matrix}...\end{*matrix},

  • Escapes unescaped currency $ before digits (\$(?=\d)) when line 1 has an odd number of $ delimiters, and

  • Replaces nested [ and ] inside line-1 \begin{...}[title] arguments with $\lbrack$ and $\rbrack$.

With this sanitizer, sect0002.html and WARNING: Using default renderer for active::& are completely eliminated, and sect0001.html renders all 9 sections (1.1–1.9) and 1,540 theorem/definition/lemma nodes cleanly.

2. Why signals-api and blueprint Rebuilt Every Time (and How Incremental Caching Works)

Four separate issues caused every make docs or make -C src/signals blueprint run to rebuild from scratch:

  1. Self-Invalidating web.paux Hash in src/signals/Makefile: The blueprint target previously computed current via find blueprint/src .lake/build/blueprint -type f before running plastex, which included blueprint/src/web.paux (2.5 MB). Because plastex overwrites blueprint/src/web.paux during rendering, saving the pre-plastex hash into .lake/build/blueprint-web.sha256 guaranteed that the next run hashed the post-plastex web.paux and always saw a mismatch. Excluding ! -name '*.paux' restores deterministic cache hits.

  2. Missing Input Stamps on lean-api and :blueprint: lean-api previously had no input stamp and unconditionally invoked lake build Signals @doc-gen4/coreDocs signalsApiDocs and build_api_docs.py, while blueprint unconditionally ran lake build :blueprint. We added .lake/build/lean-api-inputs.sha256 and .lake/build/blueprint-lean-inputs.sha256 in src/signals/Makefile, plus .lake/build/sphinx-blueprint-inputs.sha256 in docs/conf.py.

  3. Unconditional SQLite Database Deletion in build_api_docs.py: build_api_docs.py previously unlinked .lake/build/api-docs-signals.db and deleted .lake/build/doc/ on every invocation. Inspection of doc-gen4 (DocGen4/DB.lean:updateModuleDb and DocGen4/DB/Schema.lean) confirmed that doc-gen4 natively supports incremental per-module SQLite updates: updateModuleDb runs DELETE FROM modules WHERE name = ?, and all 18 child tables declare FOREIGN KEY (module_name) REFERENCES modules(name) ON DELETE CASCADE. We updated build_api_docs.py to persist per-module (source_sha256, olean_stat) hashes in .lake/build/api-docs-cache.json:

    • 0 modules changed: build_api_docs.py exits in < 0.02s.

    • \(k < N\) modules changed (e.g., editing Signals/DFT.lean alone): build_api_docs.py retains api-docs-signals.db and passes only the \(k\) changed modules to signalsApiDocs, updating their rows in place before running doc-gen4 fromDb and rewriting only modified HTML files.

    • Global config/schema change or SIGNALS_API_FORCE_REBUILD=1: performs a clean full rebuild.

  4. Excessive Terminal Output During Agent Builds: lake build :blueprint emits >25 KB of ( /workspaces/.../.artifacts/...tex ).. progress lines, and plastex emits per-file token streams. Redirecting these to .lake/build/blueprint-lake.log, .lake/build/plastex.log, and .tmp/signals-blueprint.log (dumping tail -n 60 only on non-zero exit) reduces normal terminal output to 3–5 summary lines.

  • leoliu0/ratex is cloned at v0.5.2 under src/ratex; it is a complete, self-contained TeX engine. The local checkout occupies about 1.2 GB. Building it from source requires Rust 1.88. See src/ratex/Cargo.toml:17 and src/ratex/README.md:50-52,399-406.

  • dsport is cloned at af216e1 under src/dsport. Its H14 work already includes a versioned source-analysis model, Arborium-based Lean syntax analysis, a Lean domain/xref path, and source-neutral autodoc. H14 explicitly says Lean analysis is syntax-level, not elaboration or typeclass information. See src/dsport/README.md:273-285, src/dsport/docs/sphinxdocrs-port-plan.md:2196-2212,2349-2380,2436-2475, and src/dsport/src/sphinxdocrs/src/source_analysis/lean.rs:1-4.

  • dsport has a separate source-documentation LSP integration plan, but it is marked proposed, not implemented. It recommends static analysis by default with opt-in lsp and hybrid modes. Its planned LSP mapping covers symbols, source ranges, detail/signature text, hover, locations, and diagnostics; it leaves visibility, aliases, noindex, and stable IDs to static analysis. See src/dsport/docs/source-docs-lsp-integration-plan.md:3,45,368-399.

  • The dsport Sphinx submodule is an exception: its gitlink requests 4ef6748f097b4315a9989d039f29408d5bde0e7d, which was unavailable from both declared upstream and the westurner fork. The worktree is restored at the available upstream master b04a210; dsport reports this submodule as modified. Resolve and record a valid upstream pin before using that checkout for version-sensitive parity tests. The URL is in src/dsport/.gitmodules:25-27.

  • dsport also has a src/RaTeX submodule from erweixin/RaTeX, a math renderer. It is distinct from the full TeX engine leoliu0/ratex cloned at src/ratex. See src/dsport/.gitmodules:4-6.

Design Boundaries

  • Lean’s elaborated environment remains authoritative for declaration identity, proof status, and inferred dependencies. Do not infer leanok, sorry status, uses, or proofUses from Arborium syntax.

  • LSP is a useful route to elaborated symbol details, but generic LSP responses are not automatically blueprint evidence. The proposed dsport adapter maps document symbols, hover, definitions, references, and diagnostics; it does not specify theorem proof-completion or uses/proofUses extraction. Keep LeanArchitect’s Lake export authoritative for those fields unless a Lean-specific RPC or equivalent semantic export is implemented and parity- tested. See src/dsport/docs/source-docs-lsp-integration-plan.md:61,368,389.

  • Keep blueprint records separate from generic SourceDeclaration. The latter is suitable for names, signatures, documentation, visibility, and source spans; blueprint readiness and proof/dependency semantics need their own schema. See src/dsport/src/sphinxdocrs/src/source_analysis.rs:15,151,270-362.

  • Keep the current LeanArchitect/leanblueprint path available until Rust output passes parity checks. Ratex replaces only the print/PDF engine; it does not replace Lean extraction or the HTML renderer.

  • Prefer an incremental Sphinxdocrs extension and its existing Python/Rust compatibility boundary over replacing the complete Sphinx application in one step. Existing H14 domain, source-analysis, and autodoc behavior should be reused.

Proposed Phases

Phase 0: Pin and Capture the Baseline

  • Resolve dsport’s missing Sphinx gitlink. Either recover the exact object or intentionally update the parent gitlink to an available commit and record the new parity baseline. Do not silently treat the current master fallback as the recorded version.

  • Record the Signals toolchain, Mathlib/Physlib revisions, LeanArchitect tag, dsport revision, and Ratex tag used for each comparison.

  • Save reproducible outputs from the current workflow: package-wide LeanArchitect JSON/TeX, blueprint HTML, dependency graph, and print source.

  • Acceptance: a clean build can regenerate all four library indexes and the current HTML graph from pinned inputs.

Phase 1: Ratex PDF Compatibility Spike

  • Build or install Ratex v0.5.2; for source builds, use the documented Rust 1.88+ toolchain. Test the existing blueprint/src/print.tex with explicit XeLaTeX mode and the Ratex multipass driver.

  • Check unicode-math, expl3, mathtools, hyperref, geometry, local macro files, links, theorem text, and generated Lean inputs. Compare PDF text, page count, and link destinations against a TeX Live reference.

  • If compatible, add a configurable RATEX backend to blueprint-pdf, keeping TeX Live as a fallback until CI passes. If it is not compatible, retain the current PDF target while recording the unsupported package or command.

  • The prior Ubuntu 24.04 package estimate for a minimal XeLaTeX/latexmk setup was roughly 0.4 GB installed; reserve about 0.5 GB because apt dependencies and recommends vary. latexmk itself is under 1 MB installed. Package references: latexmk, XeTeX, and LaTeX extras.

  • Acceptance: the current print document builds reproducibly with Ratex in CI and has no unexplained differences from the reference PDF.

Phase 2: Define a Versioned Blueprint Export

  • Use LeanArchitect’s Lake JSON facet as the producer. Inspect its exact current JSON output before fixing a Rust schema; normalize it into a versioned BlueprintSnapshot containing nodes, labels, node kinds, statements/proofs, Lean declarations, module/source locations, uses, proofUses, readiness, and diagnostics.

  • Keep source declarations linked by stable IDs, but store blueprint metadata independently. Preserve the distinction between declaration existence and proof readiness.

  • Capture fixtures for all four Signals libraries, including custom labels, multiple Lean declarations per node, missing dependencies, sorry, and theorem-to-theorem edges.

  • Acceptance: deterministic JSON with explicit schema/toolchain/backend versions; invalid labels or references fail validation with actionable diagnostics.

Phase 2.5: Optional Lean LSP Semantic Enrichment

  • Reuse dsport’s planned SourceLspClient and source-provider boundary rather than creating a blueprint-specific LSP process manager. Start with its static, lsp, hybrid, and auto policy; static remains the default for reproducible CI. The current dsport document explicitly marks this LSP path proposed, so first land and verify that provider independently.

  • Configure the Lean server per Signals workspace, initially through the Lake-selected toolchain and a command such as lake env lean --server. Probe the exact server version, capabilities, workspace root, initialize/open lifecycle, request limits, and shutdown behavior. Do not assume that an editor’s already-running server has loaded the same Lake environment.

  • Use standard document-symbol, hover, definition/reference, and diagnostic responses to enrich declaration names, spans, signatures, source links, and diagnostics. Reconcile to static declarations by canonical file URI, source range, then qualified name. Preserve static IDs and policy metadata; report conflicts instead of silently choosing a value.

  • Separately probe whether the pinned Lean server exposes a supported Lean-specific RPC for declaration constants, theorem proof completion, and proof dependencies. textDocument/hover, references, and diagnostics alone are not sufficient to set leanok or derive uses/proofUses.

  • Until such an RPC is verified, merge LSP-enriched source records with the authoritative LeanArchitect JSON nodes. If a future RPC produces blueprint semantics, compare it field-for-field with lake build :blueprintJson before allowing it to replace those fields.

  • Keep LSP opt-in, timeout-bounded, shell-free, and explicit about fallback. Default documentation builds must not start a language server. Protected LSP mode requires process/stdio lifecycle and platform sandbox guarantees; do not label an unsandboxed child as protected. See src/dsport/docs/source-docs-lsp-integration-plan.md:399,536.

  • Acceptance: the static path works without an installed server; lsp mode fails clearly when the configured server is unavailable; hybrid mode is deterministic; and the fixture matrix records exactly which fields came from static analysis, LSP, or LeanArchitect semantic output.

Phase 3: Rust Importer and Validator

  • Add a focused Rust module in sphinxdocrs (or a separate crate only if the interface warrants it) to read the export, validate node identity and edges, and map declaration IDs into dsport’s existing source-domain records.

  • Preserve snapshots and cache identity using the source hashes and request identity already present in dsport’s source-analysis model.

  • Do not make the Rust importer parse Lean source or interpret generic LSP responses to reconstruct proof status or dependency edges. The Arborium adapter remains useful for syntax; LSP may enrich declaration details; the LeanArchitect export remains the semantic source until a tested Lean RPC replacement exists.

  • Acceptance: Rust round-trips the fixture schema deterministically and reports unsupported schema versions instead of dropping fields.

Phase 4: Sphinxdocrs Blueprint & API Extension (with Incremental SQLite + JSON Caching)

  • Add a Rust crate/module (sphinxdocrs::blueprint and PyO3 wrapper sphinxcontrib_lean_blueprint_rs) that consumes both LeanArchitect’s .lake/build/blueprint/module/**/*.json (:blueprintJson) and doc-gen4’s SQLite database .lake/build/api-docs-signals.db directly via serde_json and rusqlite:

    1. Elimination of plasTeX and LaTeX Macro Intermediaries: By reading :blueprintJson + content.tex section structure directly in Rust rather than expanding \newleannode / \inputleanmodule through plasTeX, the extension completely bypasses Python plasTeX tokenization (eliminating web.paux serialization overhead and TeX catcode failures on [0, 1), %, &, and $).

    2. Content-Addressed Per-Module & Per-Node Incremental Cache: Maintain a fine-grained Rust cache manifest (.lake/build/sphinxdocrs-blueprint-cache.bin keyed by blake3 hashes of each module’s .olean, :blueprintJson file, and api-docs-signals.db row digest):

      • Module API Pages (signals-api/Signals/*.html): Render HTML directly from api-docs-signals.db in Rust using parallel rayon workers, applying the HOSTED_DOCS (Mathlib, Physlib, Init, Std, Lake, Lean) and Architect source-link rewrites in a single streaming pass at generation time (replacing the post-hoc Python regex pass in build_api_docs.py:rewrite_html_links) and re-emitting only modules whose SQLite rows changed.

      • Blueprint Nodes & Dependency Graph (sect0001.html, dep_graph_document.html): Parse content.tex section prose and \inputleanmodule{...} directives into a typed AST; join each node label (def:*, lem:*, thm:*) with its :blueprintJson metadata (title, statement, proof, uses, proofUses, leanok, notready, discussion, leandecls) and resolved signals-api declaration URLs; incrementally re-render only modified sections and dirty connected components of the dependency DAG.

    3. Dual Execution Mode (Python Sphinx Extension + Native sphinxdocrs):

      • Mode A (Drop-in Python Sphinx Extension): Register a PyO3 extension module in docs/conf.py (extensions.append("sphinxcontrib_lean_blueprint_rs")) that replaces _build_signals_blueprint and _copy_signals_blueprint, emitting incremental blueprint and API pages directly into app.outdir and registering theorem labels (thm:signals-*, def:signals-*, lem:signals-*) in Sphinx’s standard std:label and lean:decl domains so MyST documents can cross-reference blueprint theorems via {ref}.

      • Mode B (Native sphinxdocrs H14 Builder): Register the same Rust BlueprintSnapshot and DocGen4SqliteReader providers directly in src/dsport/src/sphinxdocrs/src/source_analysis/lean.rs.

  • Render both the interactive SVG/Canvas dependency graph and an accessible textual dependency list. Reuse existing Lean domain anchors; do not duplicate declarations in a parallel object index.

  • Acceptance: the Sphinx page has stable URLs, working theorem links, searchable labels, zero sect0002.html phantom splits, sub-100 ms incremental rebuilds when a single Lean module or MyST doc changes, and graph edges exactly matching LeanArchitect’s export.

Phase 5: Rust leanblueprint-Compatible CLI

  • Add a Rust command layer for extract, validate, web, and pdf. Initially extract orchestrates the existing Lake facet; web uses the Sphinxdocrs extension; pdf delegates to the proven Ratex target.

  • Keep leanblueprint new and its existing Python path until a separate non-interactive project-init contract is defined. Do not port setup prompts merely for nominal API parity.

  • Acceptance: a single documented command builds the JSON, site, and optional PDF while preserving Python fallback and clear error reporting.

Phase 6: Parity, CI, and Cutover

  • Compare Rust and current outputs for every fixture: labels, declarations, statements, proof-ready status, dependency edges, source links, rendered graph, and PDF artifacts. For LSP-enabled fixtures, also compare each enriched field, provenance, fallback decision, and cache invalidation against the static-plus-LeanArchitect baseline.

  • Run both paths in CI until the Rust route is stable. Then make the Rust route the default while keeping an opt-in Python fallback for at least one release cycle.

  • Remove leanblueprint Python dependencies only after output parity, failure diagnostics, docs deployment, and the rollback path are verified.

Immediate Next Steps

  1. Decide whether to repair dsport’s Sphinx pin to a known reachable commit.

  2. Inspect lake build :blueprintJson output and save a small baseline fixture.

  3. Read and reconcile dsport’s proposed source-docs-lsp-integration-plan.md with this blueprint-specific semantic contract.

  4. Prototype the Lean LSP provider on one propagation declaration, recording symbol/type/location responses separately from blueprint proof/dependency metadata.

  5. Build Ratex and compile the existing print.tex before changing the PDF target.

  6. Prototype one Rust importer for the two existing propagation nodes and their single dependency edge; do not begin with a whole-library rewrite.