Lean Blueprint and Sphinxdocrs Roadmap¶
Status as of 2026-11-11. This plan extends dsport’s existing Sphinxdocrs H14 work; it does not restart the Rust port of Sphinx.
Current Baseline¶
Signals uses Lean
v4.35.0-rc3and pins Mathlib to the matching release. Itsblueprinttarget runs LeanArchitect’s package-wide facet for all four Signals libraries (:blueprintand:blueprintJson, completing at 3,628 core / 3,811 doc-gen4 Lake jobs across 35Signalsmodules), sanitizes generated.artifacts/*.texfiles viabuild_api_docs.py --sanitize-blueprint-tex, and renders the HTML with plasTeX. Seesrc/signals/Makefile:45-105,src/signals/build_api_docs.py:92-265,src/signals/lakefile.toml:5-15,src/signals/blueprint/src/content.tex, andplan-index.md.The Sphinx HTML build (
docs/conf.py:_build_signals_blueprint) checks a content-addressed SHA-256 stamp (src/signals/.lake/build/sphinx-blueprint-inputs.sha256) over allsrc/signalsLean sources,blueprint/srcinputs (excluding*.paux),Makefile, andbuild_api_docs.py, skipping subprocess invocation entirely when inputs are unchanged and streaming build output to.tmp/signals-blueprint.logwhen a rebuild is needed. Seedocs/conf.py:208-276,docs/_toc.yml:43,docs/signals-blueprint.md:1-13, andsrc/signals/Makefile:45-105.
Root-Cause Diagnosis and Remediation of signals-api / plasTeX Bottlenecks (2026-11-11)¶
1. Why plasTeX Repeatedly Printed [ sect0002.html ] and WARNING: Using default renderer for active::&¶
With split-level = 0 in src/signals/blueprint/src/plastex.cfg, the single
\chapter{Signals Lean library} in src/signals/blueprint/src/content.tex
should emit only index.html, sect0001.html (containing Sections 1.1–1.9
and all 1,540 formal nodes), and dep_graph_document.html.
Investigation revealed four TeX tokenization hazards in the raw
@[blueprint (title := "...")] and (statement := /-- ... -/) strings emitted
by LeanArchitect into .lake/build/blueprint/module/**/*.artifacts/*.tex:
Unbalanced square brackets inside
\begin{definition}[title](specifically half-open intervals such ase ∈ [0, 1)indef:signals-ddf-planetary-orbit):plasTeX’sTeXFragment.readArgumentbalances[and]tokens even inside{...}groups. Because[0, 1)has an opening[and a closing), the bracket depth counter incremented from1to2at[0, 1)and only decremented back to1at the end of\begin{definition}[...]. As a result,plasTeXconsumed the closing]as part of the title and swallowed ~2,000 lines of subsequent\sectionand theorem nodes inside<span class="definition_thmtitle">until encountering another].Unescaped
%signs in titles and statements (e.g.,50%,98%,37.5%): treated by LaTeX/plasTeX as comment characters (catcode 14), commenting out the closing]on line 1 of.artifacts/*.tex.Unescaped
&outside matrix environments (e.g.,Al/Si & Al/GaNin titles or prose): parsed as TeX alignment tab tokens (active::&,catcode 4), triggeringWARNING: Using default renderer for active::&, corrupting the DOM section stack, and causingplasTeXto spawn a phantom emptysect0002.htmlpage ([ sect0002.html ] [ sect0002.html ] ...).Unescaped currency
$in titles (e.g.,$876k/yr): opened an unclosed math environment inside the optional[title]argument.
Fix Implemented (src/signals/build_api_docs.py:sanitize_blueprint_tex):
Added an idempotent pre-plasTeX sanitizer (python3 build_api_docs.py --sanitize-blueprint-tex)
that rewrites generated .artifacts/*.tex files in place:
Escapes unescaped
%\(\to\)\%,Escapes unescaped
&\(\to\)\&outside\begin{*matrix}...\end{*matrix},Escapes unescaped currency
$before digits (\$(?=\d)) when line 1 has an odd number of$delimiters, andReplaces nested
[and]inside line-1\begin{...}[title]arguments with$\lbrack$and$\rbrack$.
With this sanitizer, sect0002.html and WARNING: Using default renderer for active::&
are completely eliminated, and sect0001.html renders all 9 sections (1.1–1.9)
and 1,540 theorem/definition/lemma nodes cleanly.
2. Why signals-api and blueprint Rebuilt Every Time (and How Incremental Caching Works)¶
Four separate issues caused every make docs or make -C src/signals blueprint
run to rebuild from scratch:
Self-Invalidating
web.pauxHash insrc/signals/Makefile: Theblueprinttarget previously computedcurrentviafind blueprint/src .lake/build/blueprint -type fbefore runningplastex, which includedblueprint/src/web.paux(2.5 MB). Becauseplastexoverwritesblueprint/src/web.pauxduring rendering, saving the pre-plastexhash into.lake/build/blueprint-web.sha256guaranteed that the next run hashed the post-plastexweb.pauxand always saw a mismatch. Excluding! -name '*.paux'restores deterministic cache hits.Missing Input Stamps on
lean-apiand:blueprint:lean-apipreviously had no input stamp and unconditionally invokedlake build Signals @doc-gen4/coreDocs signalsApiDocsandbuild_api_docs.py, whileblueprintunconditionally ranlake build :blueprint. We added.lake/build/lean-api-inputs.sha256and.lake/build/blueprint-lean-inputs.sha256insrc/signals/Makefile, plus.lake/build/sphinx-blueprint-inputs.sha256indocs/conf.py.Unconditional SQLite Database Deletion in
build_api_docs.py:build_api_docs.pypreviously unlinked.lake/build/api-docs-signals.dband deleted.lake/build/doc/on every invocation. Inspection ofdoc-gen4(DocGen4/DB.lean:updateModuleDbandDocGen4/DB/Schema.lean) confirmed thatdoc-gen4natively supports incremental per-module SQLite updates:updateModuleDbrunsDELETE FROM modules WHERE name = ?, and all 18 child tables declareFOREIGN KEY (module_name) REFERENCES modules(name) ON DELETE CASCADE. We updatedbuild_api_docs.pyto persist per-module(source_sha256, olean_stat)hashes in.lake/build/api-docs-cache.json:0 modules changed:
build_api_docs.pyexits in< 0.02s.\(k < N\) modules changed (e.g., editing
Signals/DFT.leanalone):build_api_docs.pyretainsapi-docs-signals.dband passes only the \(k\) changed modules tosignalsApiDocs, updating their rows in place before runningdoc-gen4 fromDband rewriting only modified HTML files.Global config/schema change or
SIGNALS_API_FORCE_REBUILD=1: performs a clean full rebuild.
Excessive Terminal Output During Agent Builds:
lake build :blueprintemits >25 KB of( /workspaces/.../.artifacts/...tex )..progress lines, andplastexemits per-file token streams. Redirecting these to.lake/build/blueprint-lake.log,.lake/build/plastex.log, and.tmp/signals-blueprint.log(dumpingtail -n 60only on non-zero exit) reduces normal terminal output to 3–5 summary lines.
leoliu0/ratexis cloned atv0.5.2undersrc/ratex; it is a complete, self-contained TeX engine. The local checkout occupies about 1.2 GB. Building it from source requires Rust 1.88. Seesrc/ratex/Cargo.toml:17andsrc/ratex/README.md:50-52,399-406.dsport is cloned at
af216e1undersrc/dsport. Its H14 work already includes a versioned source-analysis model, Arborium-based Lean syntax analysis, a Lean domain/xref path, and source-neutral autodoc. H14 explicitly says Lean analysis is syntax-level, not elaboration or typeclass information. Seesrc/dsport/README.md:273-285,src/dsport/docs/sphinxdocrs-port-plan.md:2196-2212,2349-2380,2436-2475, andsrc/dsport/src/sphinxdocrs/src/source_analysis/lean.rs:1-4.dsport has a separate source-documentation LSP integration plan, but it is marked proposed, not implemented. It recommends static analysis by default with opt-in
lspandhybridmodes. Its planned LSP mapping covers symbols, source ranges, detail/signature text, hover, locations, and diagnostics; it leaves visibility, aliases,noindex, and stable IDs to static analysis. Seesrc/dsport/docs/source-docs-lsp-integration-plan.md:3,45,368-399.The dsport Sphinx submodule is an exception: its gitlink requests
4ef6748f097b4315a9989d039f29408d5bde0e7d, which was unavailable from both declared upstream and the westurner fork. The worktree is restored at the available upstream masterb04a210; dsport reports this submodule as modified. Resolve and record a valid upstream pin before using that checkout for version-sensitive parity tests. The URL is insrc/dsport/.gitmodules:25-27.dsport also has a
src/RaTeXsubmodule fromerweixin/RaTeX, a math renderer. It is distinct from the full TeX engineleoliu0/ratexcloned atsrc/ratex. Seesrc/dsport/.gitmodules:4-6.
Design Boundaries¶
Lean’s elaborated environment remains authoritative for declaration identity, proof status, and inferred dependencies. Do not infer
leanok,sorrystatus,uses, orproofUsesfrom Arborium syntax.LSP is a useful route to elaborated symbol details, but generic LSP responses are not automatically blueprint evidence. The proposed dsport adapter maps document symbols, hover, definitions, references, and diagnostics; it does not specify theorem proof-completion or
uses/proofUsesextraction. Keep LeanArchitect’s Lake export authoritative for those fields unless a Lean-specific RPC or equivalent semantic export is implemented and parity- tested. Seesrc/dsport/docs/source-docs-lsp-integration-plan.md:61,368,389.Keep blueprint records separate from generic
SourceDeclaration. The latter is suitable for names, signatures, documentation, visibility, and source spans; blueprint readiness and proof/dependency semantics need their own schema. Seesrc/dsport/src/sphinxdocrs/src/source_analysis.rs:15,151,270-362.Keep the current LeanArchitect/leanblueprint path available until Rust output passes parity checks. Ratex replaces only the print/PDF engine; it does not replace Lean extraction or the HTML renderer.
Prefer an incremental Sphinxdocrs extension and its existing Python/Rust compatibility boundary over replacing the complete Sphinx application in one step. Existing H14 domain, source-analysis, and autodoc behavior should be reused.
Proposed Phases¶
Phase 0: Pin and Capture the Baseline¶
Resolve dsport’s missing Sphinx gitlink. Either recover the exact object or intentionally update the parent gitlink to an available commit and record the new parity baseline. Do not silently treat the current master fallback as the recorded version.
Record the Signals toolchain, Mathlib/Physlib revisions, LeanArchitect tag, dsport revision, and Ratex tag used for each comparison.
Save reproducible outputs from the current workflow: package-wide LeanArchitect JSON/TeX, blueprint HTML, dependency graph, and print source.
Acceptance: a clean build can regenerate all four library indexes and the current HTML graph from pinned inputs.
Phase 1: Ratex PDF Compatibility Spike¶
Build or install Ratex
v0.5.2; for source builds, use the documented Rust 1.88+ toolchain. Test the existingblueprint/src/print.texwith explicit XeLaTeX mode and the Ratex multipass driver.Check
unicode-math,expl3,mathtools,hyperref,geometry, local macro files, links, theorem text, and generated Lean inputs. Compare PDF text, page count, and link destinations against a TeX Live reference.If compatible, add a configurable
RATEXbackend toblueprint-pdf, keeping TeX Live as a fallback until CI passes. If it is not compatible, retain the current PDF target while recording the unsupported package or command.The prior Ubuntu 24.04 package estimate for a minimal XeLaTeX/latexmk setup was roughly 0.4 GB installed; reserve about 0.5 GB because apt dependencies and recommends vary.
latexmkitself is under 1 MB installed. Package references: latexmk, XeTeX, and LaTeX extras.Acceptance: the current print document builds reproducibly with Ratex in CI and has no unexplained differences from the reference PDF.
Phase 2: Define a Versioned Blueprint Export¶
Use LeanArchitect’s Lake JSON facet as the producer. Inspect its exact current JSON output before fixing a Rust schema; normalize it into a versioned
BlueprintSnapshotcontaining nodes, labels, node kinds, statements/proofs, Lean declarations, module/source locations,uses,proofUses, readiness, and diagnostics.Keep source declarations linked by stable IDs, but store blueprint metadata independently. Preserve the distinction between declaration existence and proof readiness.
Capture fixtures for all four Signals libraries, including custom labels, multiple Lean declarations per node, missing dependencies,
sorry, and theorem-to-theorem edges.Acceptance: deterministic JSON with explicit schema/toolchain/backend versions; invalid labels or references fail validation with actionable diagnostics.
Phase 2.5: Optional Lean LSP Semantic Enrichment¶
Reuse dsport’s planned
SourceLspClientand source-provider boundary rather than creating a blueprint-specific LSP process manager. Start with itsstatic,lsp,hybrid, andautopolicy; static remains the default for reproducible CI. The current dsport document explicitly marks this LSP path proposed, so first land and verify that provider independently.Configure the Lean server per Signals workspace, initially through the Lake-selected toolchain and a command such as
lake env lean --server. Probe the exact server version, capabilities, workspace root, initialize/open lifecycle, request limits, and shutdown behavior. Do not assume that an editor’s already-running server has loaded the same Lake environment.Use standard document-symbol, hover, definition/reference, and diagnostic responses to enrich declaration names, spans, signatures, source links, and diagnostics. Reconcile to static declarations by canonical file URI, source range, then qualified name. Preserve static IDs and policy metadata; report conflicts instead of silently choosing a value.
Separately probe whether the pinned Lean server exposes a supported Lean-specific RPC for declaration constants, theorem proof completion, and proof dependencies.
textDocument/hover, references, and diagnostics alone are not sufficient to setleanokor deriveuses/proofUses.Until such an RPC is verified, merge LSP-enriched source records with the authoritative LeanArchitect JSON nodes. If a future RPC produces blueprint semantics, compare it field-for-field with
lake build :blueprintJsonbefore allowing it to replace those fields.Keep LSP opt-in, timeout-bounded, shell-free, and explicit about fallback. Default documentation builds must not start a language server. Protected LSP mode requires process/stdio lifecycle and platform sandbox guarantees; do not label an unsandboxed child as protected. See
src/dsport/docs/source-docs-lsp-integration-plan.md:399,536.Acceptance: the static path works without an installed server;
lspmode fails clearly when the configured server is unavailable;hybridmode is deterministic; and the fixture matrix records exactly which fields came from static analysis, LSP, or LeanArchitect semantic output.
Phase 3: Rust Importer and Validator¶
Add a focused Rust module in
sphinxdocrs(or a separate crate only if the interface warrants it) to read the export, validate node identity and edges, and map declaration IDs into dsport’s existing source-domain records.Preserve snapshots and cache identity using the source hashes and request identity already present in dsport’s source-analysis model.
Do not make the Rust importer parse Lean source or interpret generic LSP responses to reconstruct proof status or dependency edges. The Arborium adapter remains useful for syntax; LSP may enrich declaration details; the LeanArchitect export remains the semantic source until a tested Lean RPC replacement exists.
Acceptance: Rust round-trips the fixture schema deterministically and reports unsupported schema versions instead of dropping fields.
Phase 4: Sphinxdocrs Blueprint & API Extension (with Incremental SQLite + JSON Caching)¶
Add a Rust crate/module (
sphinxdocrs::blueprintand PyO3 wrappersphinxcontrib_lean_blueprint_rs) that consumes both LeanArchitect’s.lake/build/blueprint/module/**/*.json(:blueprintJson) anddoc-gen4’s SQLite database.lake/build/api-docs-signals.dbdirectly viaserde_jsonandrusqlite:Elimination of
plasTeXand LaTeX Macro Intermediaries: By reading:blueprintJson+content.texsection structure directly in Rust rather than expanding\newleannode/\inputleanmodulethroughplasTeX, the extension completely bypasses PythonplasTeXtokenization (eliminatingweb.pauxserialization overhead and TeX catcode failures on[0, 1),%,&, and$).Content-Addressed Per-Module & Per-Node Incremental Cache: Maintain a fine-grained Rust cache manifest (
.lake/build/sphinxdocrs-blueprint-cache.binkeyed byblake3hashes of each module’s.olean,:blueprintJsonfile, andapi-docs-signals.dbrow digest):Module API Pages (
signals-api/Signals/*.html): Render HTML directly fromapi-docs-signals.dbin Rust using parallelrayonworkers, applying theHOSTED_DOCS(Mathlib,Physlib,Init,Std,Lake,Lean) andArchitectsource-link rewrites in a single streaming pass at generation time (replacing the post-hoc Python regex pass inbuild_api_docs.py:rewrite_html_links) and re-emitting only modules whose SQLite rows changed.Blueprint Nodes & Dependency Graph (
sect0001.html,dep_graph_document.html): Parsecontent.texsection prose and\inputleanmodule{...}directives into a typed AST; join each node label (def:*,lem:*,thm:*) with its:blueprintJsonmetadata (title,statement,proof,uses,proofUses,leanok,notready,discussion,leandecls) and resolvedsignals-apideclaration URLs; incrementally re-render only modified sections and dirty connected components of the dependency DAG.
Dual Execution Mode (Python Sphinx Extension + Native
sphinxdocrs):Mode A (Drop-in Python Sphinx Extension): Register a PyO3 extension module in
docs/conf.py(extensions.append("sphinxcontrib_lean_blueprint_rs")) that replaces_build_signals_blueprintand_copy_signals_blueprint, emitting incremental blueprint and API pages directly intoapp.outdirand registering theorem labels (thm:signals-*,def:signals-*,lem:signals-*) in Sphinx’s standardstd:labelandlean:decldomains so MyST documents can cross-reference blueprint theorems via{ref}.Mode B (Native
sphinxdocrsH14 Builder): Register the same RustBlueprintSnapshotandDocGen4SqliteReaderproviders directly insrc/dsport/src/sphinxdocrs/src/source_analysis/lean.rs.
Render both the interactive SVG/Canvas dependency graph and an accessible textual dependency list. Reuse existing Lean domain anchors; do not duplicate declarations in a parallel object index.
Acceptance: the Sphinx page has stable URLs, working theorem links, searchable labels, zero
sect0002.htmlphantom splits, sub-100 ms incremental rebuilds when a single Lean module or MyST doc changes, and graph edges exactly matching LeanArchitect’s export.
Phase 5: Rust leanblueprint-Compatible CLI¶
Add a Rust command layer for
extract,validate,web, andpdf. Initiallyextractorchestrates the existing Lake facet;webuses the Sphinxdocrs extension;pdfdelegates to the proven Ratex target.Keep
leanblueprint newand its existing Python path until a separate non-interactive project-init contract is defined. Do not port setup prompts merely for nominal API parity.Acceptance: a single documented command builds the JSON, site, and optional PDF while preserving Python fallback and clear error reporting.
Phase 6: Parity, CI, and Cutover¶
Compare Rust and current outputs for every fixture: labels, declarations, statements, proof-ready status, dependency edges, source links, rendered graph, and PDF artifacts. For LSP-enabled fixtures, also compare each enriched field, provenance, fallback decision, and cache invalidation against the static-plus-LeanArchitect baseline.
Run both paths in CI until the Rust route is stable. Then make the Rust route the default while keeping an opt-in Python fallback for at least one release cycle.
Remove leanblueprint Python dependencies only after output parity, failure diagnostics, docs deployment, and the rollback path are verified.
Immediate Next Steps¶
Decide whether to repair dsport’s Sphinx pin to a known reachable commit.
Inspect
lake build :blueprintJsonoutput and save a small baseline fixture.Read and reconcile dsport’s proposed
source-docs-lsp-integration-plan.mdwith this blueprint-specific semantic contract.Prototype the Lean LSP provider on one propagation declaration, recording symbol/type/location responses separately from blueprint proof/dependency metadata.
Build Ratex and compile the existing
print.texbefore changing the PDF target.Prototype one Rust importer for the two existing propagation nodes and their single dependency edge; do not begin with a whole-library rewrite.