Workflow Simplification and Performance Plan¶
This plan covers the path from source chats to searchable Sphinx documentation:
data/chats + docs/_toc.yml
|
+--> tagged symlink map (chat overlays)
|
+--> transform-md --> docs/chats/*.myst.md [canonical searchable source]
\--> docs/chats/*.ipynb [optional notebook artifact]
|
+--> sphinx-build --> docs/_build/html
+--> search.html + _static/searchtools.js
|
+--> docindex --> OxiRS or Meilisearch
Current baseline¶
The current workspace contains approximately:
325 top-level source Markdown/JSON documents
172 generated MyST documents
172 generated notebooks
206 generated HTML files
46 MB of selected source input
24 MB of generated MyST input
The main avoidable costs are repeated file discovery, transforming each input again for each output format, rebuilding or parsing unchanged documents, and parsing rendered HTML after Sphinx has already resolved the document tree.
The first implementation is available through make chat_overlays, which
writes .tmp/workflow/chat-manifest.json, and
tools/workflow_transform.py, which transforms only records whose content hash
or requested outputs are stale. The manifest also fingerprints output formats
and transform settings, removes stale generated outputs, and writes atomically.
make transform_md_all now uses this incremental runner.
Design principles¶
docs/_toc.ymlis the only project workflow configuration file. Its Jupyter Book keys remain standard (format,root,chapters,parts), while project-specific settings live undersustainablefactory.chat_sources.ReST and MyST are the canonical searchable/documentation formats. Notebook output is a secondary artifact and should not be required for a documentation-only or search-only build.
Every stage consumes a manifest produced by the preceding stage instead of independently rescanning directories.
Incremental work is keyed by content hash plus relevant configuration, not only modification time.
Each stage writes atomically and reports enough counters to explain a slow build or an unexpected index size.
Phase 0: establish one manifest¶
The initial implementation adds a small generated manifest under
.tmp/workflow/ containing, for each
selected source document:
source path and stable document key
content hash and byte size
extracted tags and normalized tag paths
selected output paths
transform configuration/version
last successful transform and indexing status
create_symlinks.py produces the selection portion of this manifest and
use it to remove stale tag symlinks. It should not follow generated tag-folder
symlinks while selecting source files. The manifest should be reproducible from
docs/_toc.yml and should make the selected file list inspectable in CI.
The current runner provides the initial per-file incremental behavior. The remaining work is to move parsing/output fan-out into transform-md itself, so the batch process can parse each input once and write all requested formats atomically. Notebook generation remains requested by the current project target; a MyST-only target should be added before disabling notebook output.
Phase 1: make transform-md incremental¶
The initial manifest-aware batch mode transforms only new or changed source files.
Parse and normalize each source once, then fan out the in-memory result to MyST and notebook serializers. The current per-output loop can otherwise repeat input parsing and transformation work.
Keep MyST as the default output for
transform_md_all; move notebook generation behind an explicit output flag or separate target when notebooks are not needed.The manifest is written atomically after successful transforms. The runner must also write outputs to a temporary sibling and rename them into place. Do not update the manifest until every requested output succeeds.
Remove stale generated files when their source disappears from the manifest or when an output format is removed.
Add bounded worker parallelism only after measuring parser memory. Prefer process workers for CPU-heavy parsing and avoid creating one worker per file.
Acceptance criteria:
A no-op transform performs zero content transformations.
A one-file source edit rewrites only that source’s outputs.
MyST-only mode does not create or read notebook outputs.
A failed transform leaves the previous valid output and manifest intact.
The manifest producer and incremental runner implement the selection, hashing,
configuration fingerprinting, stale-output cleanup, and atomic promotion pieces
of this phase. make transform_md_all and the default manage.py transform
path use them. Direct custom indir/outdir calls remain available for
one-off conversions.
Phase 2: make Sphinx consume the manifest and TOC¶
Keep
toctreeyamlresponsible for the navigational tree and native Sphinx toctree resolution. Do not duplicate document discovery in a second TOC implementation.Use normal incremental Sphinx builds by default. Reserve
-Efor clean rebuilds and make-j autoan opt-in after checking extension safety.Register generated chat pages only through the TOC or an explicit generated subtree; do not make Sphinx discover every artifact in
docs/chats/.Keep large aggregate documents such as
tables_and_figuresas an explicit project choice. If their build cost dominates, split them into stable topic documents and include those in_toc.yml.Ensure extensions invalidate only the documents they actually affect. The YAML TOC extension should record the TOC file as a dependency so changing
_toc.ymlinvalidates the root navigation without forcing unrelated source rewrites.Keep the Sphinx theme optional in development, but test the configured theme in CI so fallback behavior does not conceal a packaging failure.
Treat Sphinx’s generated
search.html,_static/searchtools.js, andsearchindex.jsas the static search UI contract. Do not copy Sphinx’s versioned search assets into the repository; verify their presence in the built output instead.Keep the optional enhanced UI behind the boolean
docindex_searchtools_enhanced. Itsdocindex_searchtoolsconfiguration keeps native Sphinx, OxiRS, and Meilisearch results visibly separate.Treat the WebMCP
webmcp.jsonmanifest as a public, derived interface. Keep its page/navigation/doctree summary stable and do not expose raw Sphinx pickle files or private backend credentials.
Acceptance criteria:
A source-free documentation edit does not rerun transform-md.
A TOC-only edit rebuilds navigation and affected pages only.
A no-op Sphinx build does not rewrite HTML files.
The build log reports changed, copied, and skipped document counts.
Phase 3: reduce docindex work¶
Stop treating the entire HTML output directory as the change set. Feed docindex the manifest’s changed HTML pages, or consume Sphinx’s build environment/inventory to identify changed documents.
Preserve the current project exclusion in
docs/conf.py; generic docindex defaults should remain project-neutral.Add an incremental index mode that upserts changed documents and deletes documents whose source key disappeared. Keep the existing atomic full rebuild as a recovery command.
Prefer indexing canonical MyST/chat records directly when their schema is sufficient. Use HTML section parsing for rendered pages that need public URL and heading extraction, rather than reparsing both representations.
For OxiRS and Meilisearch, select one backend for normal local builds. Use the multi-backend path only for an intentional replication run.
Keep batch submission and task polling asynchronous, but measure batch size, queue latency, retry count, and bytes submitted. Tune these against the selected backend rather than using one universal batch size.
Avoid calling optimize after every small incremental update. Schedule optimization after a threshold of updates or as a separate maintenance target.
Acceptance criteria:
An unchanged Sphinx build submits zero index batches.
A one-page change submits only that page and its stale replacements.
A deleted page is removed from all relevant indices.
Full rebuild remains available and produces the same document keys as the incremental path.
Phase 4: observability and regression protection¶
Record per-stage metrics in .tmp/workflow/last-run.json and print a concise
summary:
selected, changed, skipped, failed, and deleted files
input/output bytes
transform, Sphinx, parse, submit, and wait durations
index document counts and batch counts
cache hit rate and retry count
Add tests for manifest determinism, stale symlink cleanup, incremental output selection, TOC dependency invalidation, and incremental index deletion. Add a small fixture benchmark to CI; do not benchmark the full chat corpus on every pull request.
Recommended implementation order¶
Finish the merged
_toc.ymlschema and manifest-producing overlay command.Add transform-md MyST-only and manifest-aware incremental mode.
Make the Makefile and
manage.pycall the same staged commands.Add Sphinx changed-document instrumentation and TOC dependency tracking.
Add docindex incremental upsert/delete with full rebuild fallback.
Tune parallelism, batch sizes, and optimization thresholds from recorded measurements.
This order keeps the workflow behavior stable while making each performance improvement measurable and reversible.
The manifest and incremental transform runner are implemented first. The incremental Sphinx and docindex phases remain planned work; full docindex rebuilds continue to be the supported indexing operation until their delete and upsert semantics are implemented and tested.