Sphinx-Meilisearch Integration Architecture¶
Overview¶
This document outlines the architecture for integrating Meilisearch full-text search with Sphinx documentation, enabling efficient indexing of chat exports, processed markdown, and HTML documentation.
1. System Architecture¶
1.1 Components¶
┌─────────────────────────────────────────────────────────────┐
│ Data Processing Pipeline │
└─────────────────────────────────────────────────────────────┘
↓
data/chats/ ──→ transform-md ──→ docs/chats/
(.md, .json) (.myst.md, .ipynb)
↓ ↓
sphinx/ ──→ docs/_build/html/ │
.rst, .md ↓
↓ ┌──────────────────┐
└──────────────────────→ Meilisearch ←─┤ Indexer Module │
(Port 7700) └──────────────────┘
↓
┌────────────────────┐
│ Search UI / API │
│ (sphinx-pagefind │
│ or custom) │
└────────────────────┘
1.2 Key Integration Points¶
Document Sources: Chat exports (JSON/Markdown), Sphinx-built HTML, MyST markdown
Indexing Strategy:
Index raw chats by segments
Index HTML build output by sections
Index source markdown for development
Index Management:
Track file modifications with inode/hash
Rebuild on
make htmlvia hooksBatch operations for performance
Search UI:
Sphinx-pagefind for static site search
JSON API for programmatic access
Snippet highlighting and relevance ranking
Native Sphinx search and optional DocIndex search are separate modes; configure the enhanced UI with
docindex_searchtools_enhancedanddocindex_searchtoolsindocs/conf.py.sphinxcontrib.webmcpexposes the same public search modes, navigation, page context, and compact doctree metadata throughdocument.modelContext.
2. File Structure¶
sustainablefactory/
├── Dockerfile.meilisearch # Fedora toolbox container
├── requirements-meilisearch.txt # Python dependencies
├── src/
│ └── meilisearch_integration/ # New module
│ ├── __init__.py
│ ├── config.py # Configuration and schemas
│ ├── indexer.py # Core indexing logic
│ ├── chat_parser.py # Chat extraction module
│ ├── html_parser.py # HTML section parser
│ ├── hooks.py # Sphinx/Make hooks
│ ├── api.py # Meilisearch API wrapper
│ └── cli.py # CLI commands
├── docs/
│ └── search-config.json # Search UI configuration
├── Makefile # Updated with meilisearch targets
├── pyproject.toml # Updated with search dependencies
└── .env.meilisearch # Configuration (local only)
3. Data Model & Indexing Strategy¶
3.1 Document Structure in Meilisearch¶
Each indexed document represents a searchable unit (can vary by source):
// Chat document (from docs/chats/)
{
id: string // Unique identifier: "chat_<filename>_<segment>"
type: "chat"
title: string // Chat title from metadata
filename: string // Original filename
segment_index: number // Segment number within chat
content: string // Text content (max 100KB per document)
summary: string // Optional AI-generated summary
metadata: {
source_file: string
chat_type: "gemini" | "copilot" | "custom"
tags: string[]
date_indexed: ISO8601
concepts: string[] // Extracted entities/concepts
}
relevance_score: number // Set during search
}
// Sphinx HTML document (from docs/_build/html/)
{
id: string // "html_<page_path>_<section_id>"
type: "sphinx"
title: string // Page/section title
url: string // Relative URL path
path: string // File path in _build/html/
section_title: string // Heading text
content: string // HTML converted to text
code_snippets: CodeSnippet[] // Extracted code blocks
metadata: {
source_file: string
heading_level: number // 1-6
breadcrumb: string[]
last_built: ISO8601
sphinx_role: string // e.g., "doc", "ref", "py:class"
}
}
// MyST Markdown document (from docs/chats/)
{
id: string // "myst_<filename>"
type: "myst"
title: string
filename: string
content: string
frontmatter: Record<string, any>
metadata: {
source_file: string
last_modified: ISO8601
word_count: number
}
}
3.2 Indexing Strategy¶
Source 1: Chat Exports (JSON/Markdown)¶
Input: data/chats/*.json or docs/chats/*.myst.md
Strategy: Split by conversation turns or sections
Frequency: On-demand or via make meilisearch_index_chats
# Pseudocode: chat_parser.py
for chat_file in docs/chats:
if file_extension == '.json':
turns = parse_json_turns(file)
documents = [
Document(
id=f"chat_{stem}_{i}",
type="chat",
content=turn["content"],
metadata={"turn": i, "role": turn["role"]}
)
for i, turn in enumerate(turns)
]
elif file_extension == '.myst.md':
sections = split_by_markdown_headings(file)
documents = [
Document(
id=f"chat_{stem}_{i}",
type="chat",
content=section,
title=extract_heading(section)
)
for i, section in enumerate(sections)
]
meilisearch.add_documents(documents)
Source 2: Sphinx HTML Output¶
Input: docs/_build/html/
Strategy: Parse HTML, extract sections by heading levels
Frequency: Automatic after make html (via hook)
# Pseudocode: html_parser.py
for html_file in docs/_build/html:
if html_file.endswith('.html'):
sections = extract_sections_from_html(html_file)
documents = [
Document(
id=f"html_{page_id}_{section_id}",
type="sphinx",
title=section["title"],
content=section["text"],
url=section["url"],
metadata={
"heading_level": section["level"],
"breadcrumb": section["breadcrumb"]
}
)
for section_id, section in enumerate(sections)
]
meilisearch.add_documents(documents)
Source 3: MyST Markdown Source Files¶
Input: docs/*.myst.md, docs/chats/*.myst.md
Strategy: Index entire file + extract sections
Frequency: On-demand via make meilisearch_index_myst
4. Hook System (Django-Haystack Style)¶
4.1 Sphinx Build Hooks¶
Integrate with Sphinx’s event system to auto-index on document changes:
# sustainablefactory/src/meilisearch_integration/hooks.py
# Hook into Sphinx config
def setup(app):
"""Register Sphinx event handlers for Meilisearch indexing."""
app.connect("config-inited", on_config_inited)
app.connect("build-finished", on_build_finished)
app.connect("env-updated", on_env_updated)
def on_config_inited(app, config):
"""Initialize Meilisearch client on Sphinx startup."""
config.meilisearch_enabled = True
config.meilisearch_host = os.getenv("MEILISEARCH_HOST", "localhost")
config.meilisearch_port = int(os.getenv("MEILISEARCH_PORT", 7700))
config.meilisearch_api_key = os.getenv("MEILISEARCH_API_KEY")
def on_build_finished(app, exception):
"""Index HTML documents after build completion."""
if exception or not app.config.meilisearch_enabled:
return
logger.info("Indexing Sphinx HTML output to Meilisearch...")
indexer = SphinxHTMLIndexer(
host=app.config.meilisearch_host,
port=app.config.meilisearch_port
)
indexer.index_html_build(app.outdir)
def on_env_updated(app, env):
"""Track document changes for selective re-indexing."""
# Get list of updated files from Sphinx environment
changed_files = env.get_updated_docs()
logger.debug(f"Updated docs: {changed_files}")
4.2 Make Targets with Hooks¶
# Makefile additions
.PHONY: meilisearch_start
meilisearch_start:
@echo "meilisearch_start # Start Meilisearch server"
podman run -d -p 7700:7700 -e MEILI_MASTER_KEY=dev-key \
--name meilisearch-dev \
sustainablefactory-meilisearch:latest
.PHONY: meilisearch_stop
meilisearch_stop:
@echo "meilisearch_stop # Stop Meilisearch server"
podman stop meilisearch-dev || true
podman rm meilisearch-dev || true
.PHONY: meilisearch_index_chats
meilisearch_index_chats:
@echo "meilisearch_index_chats # Index chat exports to Meilisearch"
python3 -m sustainablefactory.meilisearch_integration.cli index-chats --source docs/chats
.PHONY: meilisearch_index_html
meilisearch_index_html:
@echo "meilisearch_index_html # Index Sphinx HTML to Meilisearch"
python3 -m sustainablefactory.meilisearch_integration.cli index-html --source docs/_build/html
.PHONY: meilisearch_index_all
meilisearch_index_all: meilisearch_index_chats meilisearch_index_html
@echo "meilisearch_index_all # Index all sources"
.PHONY: build_with_search
build_with_search: build meilisearch_index_html
@echo "build_with_search # Build docs and index to Meilisearch"
4.3 File Change Tracking¶
Use file hashing to avoid re-indexing unchanged documents:
# sustainablefactory/src/meilisearch_integration/cache.py
import json
import hashlib
from pathlib import Path
from typing import Dict
class IndexCache:
"""Track file hashes to detect changes."""
def __init__(self, cache_file: Path = Path(".meilisearch_cache")):
self.cache_file = cache_file
self.hashes = self._load_cache()
def _load_cache(self) -> Dict[str, str]:
if self.cache_file.exists():
return json.loads(self.cache_file.read_text())
return {}
def save_cache(self):
self.cache_file.write_text(json.dumps(self.hashes, indent=2))
def file_changed(self, filepath: Path) -> bool:
"""Check if file has been modified since last index."""
new_hash = hashlib.md5(filepath.read_bytes()).hexdigest()
old_hash = self.hashes.get(str(filepath))
if new_hash != old_hash:
self.hashes[str(filepath)] = new_hash
return True
return False
def get_changed_files(self, directory: Path) -> list[Path]:
"""Get list of changed files in directory."""
return [
f for f in directory.rglob("*")
if f.is_file() and self.file_changed(f)
]
5. CLI Interface¶
# Index chat exports
$ python3 -m sustainablefactory.meilisearch_integration.cli index-chats \
--source docs/chats \
--batch-size 100 \
--skip-unchanged
# Index Sphinx HTML
$ python3 -m sustainablefactory.meilisearch_integration.cli index-html \
--source docs/_build/html \
--headings 1-3 \
--exclude-pages "genindex,search,modindex"
# Manage indices
$ python3 -m sustainablefactory.meilisearch_integration.cli list-indices
# Clear index
$ python3 -m sustainablefactory.meilisearch_integration.cli clear-index \
--index chats --confirm
# Search (testing)
$ python3 -m sustainablefactory.meilisearch_integration.cli search \
--query "lignin vitrimer" \
--index all \
--limit 10
# Status
$ python3 -m sustainablefactory.meilisearch_integration.cli status
6. Search UI Integration¶
6.1 Sphinx-Pagefind¶
Sphinx-Pagefind provides static site search without requiring a server.
Configuration (docs/conf.py):
extensions = [
'sphinx_pagefind',
]
sphinx_pagefind_excluded_extensions = ['.pdf', '.zip']
sphinx_pagefind_root_selector = 'div.main-content'
6.2 Custom JSON API Search UI¶
Lightweight HTML/JS interface querying Meilisearch directly:
<!-- docs/_static/meilisearch-search.html -->
<div id="meilisearch-search">
<input type="text" id="search-input" placeholder="Search docs and chats...">
<div id="search-results"></div>
</div>
<script>
const MEILISEARCH_HOST = 'http://localhost:7700';
const searchInput = document.getElementById('search-input');
const resultsDiv = document.getElementById('search-results');
searchInput.addEventListener('input', async (e) => {
const query = e.target.value;
if (query.length < 2) return;
try {
const response = await fetch(`${MEILISEARCH_HOST}/indexes/all/search`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ q: query, limit: 20 })
});
const data = await response.json();
displayResults(data.hits);
} catch (error) {
console.error('Search error:', error);
}
});
function displayResults(hits) {
resultsDiv.innerHTML = hits.map(hit => `
<div class="search-result">
<h3><a href="${hit.url || '#'}">${hit.title}</a></h3>
<p>${hit.content.substring(0, 150)}...</p>
</div>
`).join('');
}
</script>
7. Performance Considerations¶
7.1 Batch Indexing¶
# Batch documents to avoid hitting Meilisearch limits
BATCH_SIZE = 1000
def index_documents_batched(documents: List[Document]):
for i in range(0, len(documents), BATCH_SIZE):
batch = documents[i:i+BATCH_SIZE]
client.index('all').add_documents(batch)
logger.info(f"Indexed batch {i//BATCH_SIZE + 1}")
7.2 Selective Re-indexing¶
Track file hashes to skip unchanged files
Use Meilisearch’s
updateDocumentsfor modifications onlyImplement incremental indexing for large builds
7.3 Index Management¶
# Meilisearch settings for optimal search
settings = {
"searchableAttributes": [
"title",
"content",
"summary",
"concepts"
],
"filterableAttributes": [
"type",
"date_indexed",
"filename"
],
"sortableAttributes": [
"date_indexed"
],
"synonyms": {
"lignin": ["kraft lignin", "kraft-lignin", "lig"],
"vitrimer": ["vitrimer polymer", "vitrimeric"],
}
}
8. Integration with Build Pipeline¶
8.1 Extended Makefile¶
.PHONY: build
build: aggregate_data transform_md_all docs meilisearch_index_html
@echo "build # Full build pipeline with search indexing"
.PHONY: rebuild_index
rebuild_index:
@echo "rebuild_index # Force full re-index of all sources"
python3 -m sustainablefactory.meilisearch_integration.cli \
clear-index --index all --confirm
$(MAKE) meilisearch_index_all
8.2 CI/CD Considerations¶
Index builds in CI to pre-populate production Meilisearch
Store index data in persistent volume for containerized deployments
Export/import indices for multi-environment consistency
9. Deployment Scenarios¶
9.1 Local Development¶
# Start Meilisearch
make meilisearch_start
# Build docs and index
make build_with_search
# Serve with search
make serve
9.2 Container (Podman)¶
# Build container
podman build -f Dockerfile.meilisearch -t sustainablefactory-search:latest
# Run with volumes
podman run -d \
-p 7700:7700 -p 8000:8000 \
-v $(pwd):/workspace \
-e MEILISEARCH_MASTER_KEY=prod-key \
sustainablefactory-search:latest
9.3 Production¶
Run Meilisearch as a separate service
Use persistent storage for indices
Implement authentication/rate limiting
Cache search results at CDN layer
10. Future Enhancements¶
GraphRAG Integration: Use Meilisearch as vector store for graph-based retrieval
Semantic Search: Add embedding-based search using sentence-transformers
Analytics: Track popular searches, query patterns
Auto-suggestions: Implement query completion with entity extraction
Faceted Search: Enable filtering by document type, date, concepts
Multi-language Support: Lemmatization and language-specific analyzers