Measured on real code, reproducibly.

How we experimented

SynaFS ships a built-in syna bench, but its corpus is synthetic — recall is 1.0 by construction, which proves the pipeline runs but says nothing about quality. The harness here is different: it indexes real source trees and grades retrieval against a gold set whose answers are fixed by inspection, swapping only the embedder so the comparison is clean.

1. Build the release binary with the local semantic embedder: cargo build --release --features coderank.
2. Assemble corpora — copy code files only (excluding node_modules, target, build output) from real repositories into a clean tree.
3. Write 18 natural-language queries, each a paraphrase that avoids the codebase's own identifiers; fix the gold answer to the file that primarily implements that concept.
4. Index the SynaFS tree twice — once with the offline lexical hash embedder, once with CodeRankEmbed — and run the identical hybrid pipeline (RRF over vector + BM25 + trigram).
5. Grade at file level: a hit counts if its path ends with the gold path. recall@k = share of queries with a gold hit in the top k; MRR = mean of 1/rank.
6. Measure engine latency in-process via syna bench; measure indexing throughput by wall-clock over the real trees.

Gold-set examples (the queries contain none of the target's identifiers)

Reference machine: x86-64 Linux, CPU-only inference, the pure-Rust engine (brute-force vector search over an in-memory snapshot). Your numbers will vary by corpus, embedder, and hardware. Raw output lives in bench/results.json.

Retrieval quality — semantic vs lexical

bench/gold.json is 18 natural-language queries over the SynaFS source. Each is a paraphrase that deliberately avoids the codebase's own identifiers; the gold answer is the file that primarily implements that concept (e.g. "compress response header fields for an http2 stream" → syna-grpc/src/hpack.rs). Relevance is graded at file level. We index the same tree twice and run the identical hybrid pipeline, changing only the embedder.

Semantic embeddings roughly double MRR (0.216 → 0.434) and recall@1 (0.111 → 0.333). The biggest per-query wins are exactly where lexical overlap is weakest: "get notified of every write across a whole mounted filesystem" → fanotify.rs climbs from rank 10 → 1, and "demand and check the caller's certificate during the secure handshake" → tls.rs from 6 → 1. Several hard queries (hpack.rs, wal.rs, ws.rs) are missed by both in the top-10 — the set is small and untuned, not rigged toward a win.

Indexing throughput — real repositories

End-to-end syna index (tree-walk → tree-sitter chunk → embed → persist) over real source trees, code files only. The offline hash embedder isolates engine/chunking throughput; semantic indexing is bound by CPU model inference and is far slower (it is the quality path, not the throughput path).

Index size is the current M0 JSON snapshot (~7–9 KB/chunk, full uncompressed vectors); the planned RocksDB + usearch + PQ layout compresses this substantially.

Engine latency — in-process

From syna bench at 1,000 files / 5,000 chunks. These are in-process (no per-call snapshot reload), so they reflect engine latency, not CLI start-up. Corpus is synthetic; the latency is real.

• Embed-cache hit rate on a 1-line edit: 80% — editing 1 of 5 functions re-embeds only that function's chunks; the other 4/5 are served from cache. This is the incremental-on-write invariant, measured directly.
• The ~20 ms reindex cost is dominated by the M0 executor rebuilding the index O(n) per commit; usearch incremental add/remove will cut it. It grows with corpus size — it is not yet a fixed per-edit cost.

Filesystem performance — the POSIX floor

SynaFS is a real FUSE filesystem, so we measured ordinary file ops through the mount against the raw backing disk, plus the cost of a semantic search done by listing a magic directory. The honest picture: metadata is essentially free, every latency stays under ~100 µs, and a semantic query runs in a fraction of a millisecond — but raw byte streaming carries real overhead, because the current pure-Rust FUSE copies bytes through userspace (kernel FUSE_PASSTHROUGH would close that gap).

Scaling — overcoming brute-force with ANN

The biggest M0 limitation was exact brute-force vector search — O(N) per query. So we implemented a pure-Rust HNSW approximate-nearest-neighbour index (no external crates) behind the same VectorIndex trait, selectable with SYNA_ANN=hnsw, and measured it against exact search over clustered 768-dim vectors. Search becomes sublinear: 26× faster at 100k while keeping 98.7% recall@10.

Memory — 32× smaller with product quantization

M0 kept every vector as full f32 — ~3 KB each, so a few million chunks no longer fit in RAM. SynaFS now ships a pure-Rust PQ-compressed index (SYNA_ANN=pq): each vector becomes a 96-byte code, the raw f32s move to an on-disk tier, and search re-scores the top candidates exactly from disk — so RAM drops 32× while recall@10 stays at brute-force. Reproduce with syna pq-bench.

Standard benchmark — CodeSearchNet-style

The 18-query gold set above is hand-written; this is its held-out complement. Following the CodeSearchNet method, a harness auto-harvests 186 docstring→function pairs from the real source and strips the doc comments from the indexed code, so the query text is never in the index (no leakage). Semantic embeddings win decisively on this larger, bias-free set.

Honesty notes

• The vector store is no longer RAM-bound: vectors are PQ-compressed to 96 bytes (32× smaller than full f32), with the raw vectors on an on-disk tier and exact rerank, so recall@10 holds at 1.0 even at 100k while RAM drops 307 MB → 9.6 MB (SYNA_ANN=pq; reproduce with syna pq-bench). And the scan is no longer O(N): an IVF coarse quantizer (≈√N cells, probe a few) touches only ~8% of the codes at 100k — recall@10 stays ~0.99 and that fraction keeps shrinking as the corpus grows. §E adds pure-Rust HNSW for 26× faster exact search on top.
• Beyond the 18-query hand set, a CodeSearchNet-style harness auto-harvests 186 held-out docstring→function pairs with the doc comments stripped from the indexed code (no leakage): semantic CodeRankEmbed scores R@1 0.75 / R@10 0.96 / MRR 0.82 vs the lexical baseline's 0.42 / 0.78 / 0.53 (bench/csn_bench.py). And it's no longer only our own source: the same harness now runs across 7 external third-party repos (TS/JS + Python, 320+ held-out pairs, bench/csn_multi.py). Out-of-domain retrieval is harder, but semantic still wins — e.g. on the third-party arcflo project, recall@10 0.45 vs 0.20 and MRR 0.29 vs 0.17 over lexical. Honest remainder: absolute external numbers trail the Rust target SynaFS was tuned on, and a public CoIR leaderboard is still future work.
• Everything here is reproducible from a clean checkout with the two commands below.

# build with the local semantic embedder, then run the harness
cargo build --release --features coderank
python3 bench/run_bench.py            # → bench/results.json
python3 bench/run_bench.py --skip-coderank   # lexical only, no model download

Full write-up: docs/benchmarks.md.