Measured on real code, reproducibly.

How we experimented

SynaFS ships a built-in syna bench, but its corpus is synthetic — recall is 1.0 by construction, which proves the pipeline runs but says nothing about quality. The harness here is different: it indexes real source trees and grades retrieval against a gold set whose answers are fixed by inspection, swapping only the embedder so the comparison is clean.

1. Build the release binary with the local semantic embedder: cargo build --release --features coderank.
2. Assemble corpora — copy code files only (excluding node_modules, target, build output) from real repositories into a clean tree.
3. Write 18 natural-language queries, each a paraphrase that avoids the codebase's own identifiers; fix the gold answer to the file that primarily implements that concept.
4. Index the SynaFS tree twice — once with the offline lexical hash embedder, once with CodeRankEmbed — and run the identical hybrid pipeline (RRF over vector + BM25 + trigram).
5. Grade at file level: a hit counts if its path ends with the gold path. recall@k = share of queries with a gold hit in the top k; MRR = mean of 1/rank.
6. Measure engine latency in-process via syna bench; measure indexing throughput by wall-clock over the real trees.

Gold-set examples (the queries contain none of the target's identifiers)

Reference machine: x86-64 Linux, CPU-only inference, the pure-Rust engine (brute-force vector search over an in-memory snapshot). Your numbers will vary by corpus, embedder, and hardware. Raw output lives in bench/results.json.

Retrieval quality — semantic vs lexical

bench/gold.json is 18 natural-language queries over the SynaFS source. Each is a paraphrase that deliberately avoids the codebase's own identifiers; the gold answer is the file that primarily implements that concept (e.g. "compress response header fields for an http2 stream" → syna-grpc/src/hpack.rs). Relevance is graded at file level. We index the same tree twice and run the identical hybrid pipeline, changing only the embedder.

Semantic embeddings roughly double MRR (0.210 → 0.434) and recall@1 (0.111 → 0.333). The biggest per-query wins are exactly where lexical overlap is weakest: "get notified of every write across a whole mounted filesystem" → fanotify.rs climbs from rank 10 → 1, and "demand and check the caller's certificate during the secure handshake" → tls.rs from 6 → 1. Several hard queries (hpack.rs, wal.rs, ws.rs) are missed by both in the top-10 — the set is small and untuned, not rigged toward a win.

Indexing throughput — real repositories

End-to-end syna index (tree-walk → tree-sitter chunk → embed → persist) over real source trees, code files only. The offline hash embedder isolates engine/chunking throughput; semantic indexing is bound by CPU model inference and is far slower (it is the quality path, not the throughput path).

Index size is the current split snapshot (manifest.json + docs.bin + units.ndjson): full vectors still dominate disk. PQ is shipped as an opt-in scale path (SYNA_ANN=pq); making it automatic at large N remains future work.

Engine latency — in-process

From syna bench at 1,000 files / 5,000 chunks. These are in-process (no per-call snapshot reload), so they reflect engine latency, not CLI start-up. Corpus is synthetic; the latency is real.

• Embed-cache hit rate on a 1-line edit: 80% — editing 1 of 5 functions re-embeds only that function's chunks; the other 4/5 are served from cache. This is the incremental-on-write invariant, measured directly.
• The ~20 ms reindex cost reflects this benchmark configuration and hardware. Newly-created docs are folded into the live index incrementally; larger corpora still need separate measurement because symbol-graph refresh, persistence, and ANN maintenance can dominate different workloads.

Filesystem performance — the POSIX floor

SynaFS is a real FUSE filesystem, so we measured ordinary file ops through the mount against the raw backing disk, plus the cost of a semantic search done by listing a magic directory. The honest picture: metadata is essentially free, every latency stays under ~100 µs, and a semantic query runs in a fraction of a millisecond — but raw byte streaming carries real overhead, because the current pure-Rust FUSE copies bytes through userspace (kernel FUSE_PASSTHROUGH would close that gap).

Scaling — overcoming brute-force with ANN

Exact brute-force vector search is O(N) per query, so SynaFS includes a pure-Rust HNSW approximate-nearest-neighbour index behind the same VectorIndex trait and selectable with SYNA_ANN=hnsw. The current checked-in public artifact is a 10k-vector smoke run: 2.6× faster p50 search with recall 1.000. Full 100k headlines remain pending until regenerated and committed.

Memory — 32× smaller with product quantization

M0 kept every vector as full f32 — ~3 KB each, so a few million chunks no longer fit in RAM. SynaFS now ships a pure-Rust PQ-compressed index (SYNA_ANN=pq): each vector becomes a 96-byte code, the raw f32s move to an on-disk tier, and search re-scores the top candidates exactly from disk — so RAM drops 32× while recall@10 stays at brute-force. Reproduce with syna pq-bench.

SIMD — vectorizing the hot dot-product kernel

Every vector backend — the brute-force scan, HNSW's distance, PQ's exact rerank — bottlenecks on the same dot product over f32 embeddings. We replaced the scalar loop with an AVX2+FMA kernel (runtime-detected via std::arch, scalar fallback), keeping it pure-Rust and dependency-free. Isolated, the kernel is 5.5–8.7× faster; end-to-end search shows ~2.8× because the dot is one stage of many.

Incremental write path — reindex independent of history

A one-line edit used to rebuild the whole index and re-serialize every vector — O(N) per edit, growing as a file accrues versions. Now an edit folds only its new chunks into the live index and appends them to docs.ndjson; only the small vector-free manifest is rewritten. Per-edit cost is O(changed) and independent of edit history.

Standard benchmark — CodeSearchNet-style

The 18-query gold set above is hand-written; this is its held-out complement. Following the CodeSearchNet method, a harness auto-harvests 186 docstring→function pairs from the real source and strips the doc comments from the indexed code, so the query text is never in the index (no leakage). Semantic embeddings win decisively on this larger, bias-free set.

Warm-model daemon — query latency

With the real CodeRankEmbed embedder, every cold syna query reloads the 137M-parameter model (~140 ms) before a ~36 ms search. A per-repo daemon holds the engine and its model resident; query/edit/commit/index route to it over gRPC-Web, paying the load once per session instead of once per call.

Ranking priors — when the test file outranks the code it tests

Test files repeat the implementation's vocabulary almost verbatim, and RRF scores cluster within a few percent, so a well-named test regularly edged out the very code it exercises. Two priors fix this. Stage 2.5 applies a mild demotion to test-suite paths (tests/, test_*.py, *.test.js, *_test.go, conftest.py, …) — about 7 ranks at the top, enough to flip a near-tie toward the implementation, and uniform so genuinely test-seeking queries (whose competitors are also tests) keep their order. Stage 3.5 caps any one file at two chunks in the top-N, with displaced hits backfilling the tail; the failure that motivated it was a single well-named test file flooding an entire top-3 while the answer sat one rank below.

GPU embedding — 3.8× off the cold index, verified on Metal

Embedding dominates indexing wall time — on a real corpus it is the overwhelming majority of it, which makes the forward pass the only thing worth accelerating. The embedder now runs on Apple Metal behind an opt-in build feature (--features coderank-metal, SYNA_DEVICE=metal), and the win is close to the arithmetic limit of moving that pass onto the GPU. Peak memory drops too, because model tensors live in GPU-side allocations instead of the malloc heap.

Where the time actually goes — and what we did not build because of it

Every benchmark above this point uses the synthetic hash embedder, which is near-free by design — that is what isolates the index, store and search machinery. Profiling the real CodeRankEmbed model tells a much blunter story about production, and it is the single most useful measurement in this document.

Deferred on purpose

These were all on the list and are all recorded as declined, so the decision is explicit rather than forgotten. Each sits below the noise floor of a real embedder — building them would add risk and complexity for a gain no deployment could observe.

What a session actually costs — cold start and payload

Two clusters of work, both aimed at the parts of a session that are pure overhead. Cold start is everything paid before the first useful answer — loading a 137M-parameter model, rebuilding an index structure, re-reading a tree to check for edits. Payload is what each reply carries: the same answer can arrive as pretty-printed JSON full of fields no tool reads, or as compact text.

Honesty notes

• The vector store is no longer RAM-bound: vectors are PQ-compressed to 96 bytes (32× smaller than full f32), with the raw vectors on an on-disk tier and exact rerank, so recall@10 holds at 1.0 even at 100k while RAM drops 307 MB → 9.6 MB (SYNA_ANN=pq; reproduce with syna pq-bench). And the scan is no longer O(N): an IVF coarse quantizer (≈√N cells, probe a few) touches only ~8% of the codes at 100k — recall@10 stays ~0.99 and that fraction keeps shrinking as the corpus grows. §E adds pure-Rust HNSW for 2.6× faster exact search on top.
• Beyond the 18-query hand set, a CodeSearchNet-style harness auto-harvests 186 held-out docstring→function pairs with the doc comments stripped from the indexed code (no leakage): semantic CodeRankEmbed scores R@1 0.75 / R@10 0.96 / MRR 0.82 vs the lexical baseline's 0.42 / 0.78 / 0.53 (bench/csn_bench.py). And it's no longer only our own source: the same harness now runs across 7 external third-party repos (TS/JS + Python, private-generated held-out pairs, bench/csn_multi.py). Out-of-domain retrieval is harder; the current arcflo aggregate is recall@10 0.343 and MRR 0.162, without a public semantic-vs-lexical split, so it is not yet a publishable external win claim. Honest remainder: absolute external numbers trail the Rust target SynaFS was tuned on, and a public CoIR leaderboard is still future work.
• Everything here is reproducible from a clean checkout with the two commands below.

# build with the local semantic embedder, then run the harness
cargo build --release --features coderank
python3 bench/run_bench.py            # → bench/results.json
python3 bench/run_bench.py --skip-coderank   # lexical only, no model download

Benchmark write-up ships with the source repo as docs/benchmarks.md.