Why Vector Search Alone Isn't Enough for Personal AI: RecalAI's Hybrid Retrieval Design

Why this isn’t another generic hybrid-search tutorial

There are already credible public implementations of hybrid local retrieval. Vannevar documents FTS5, vector search and weighted RRF for Markdown; SearchKit shows vector/BM25/RRF modes in a Swift on-device context. Simply repeating “combine keyword and semantic search” would not establish new value. This article instead examines the actual RecalAI iOS source code: anchor/strict/broad lexical routes, weighted fusion, deterministic (not neural) reranking, explicit evidence budgets and validated citation IDs, while identifying experiments that have not yet been completed.

The short answer

A vector index can retrieve passages that sound related to a question. A personal knowledge base must do more: find the correct source, exact model identifier, date, version, amount, or quoted clause, and let the reader reopen it. Semantic similarity is not evidence of factual identity.

RecalAI’s documented local-first design separates source preservation, parsing, immediate full-text search, background embeddings, hybrid retrieval, ranking, evidence selection, and grounded generation. This describes a design pipeline; it does not prove that hybrid search is always faster or more accurate than a single retrieval method.

For the broader system boundaries, see Personal AI Agent architecture and Agent Memory systems. This article addresses a narrower engineering question: how does an imported document become reliably searchable and citable?

An imported file is not automatically answer-ready

At least four readiness states matter:

1. Source preserved. The original file can be recovered or previewed. Its text may not yet be searchable.
2. Parsed and indexed for lexical search. Text extraction and an FTS index are ready; scanned pages may need OCR, and failures require explicit reporting.
3. Semantic enrichment complete. Chunk embeddings, summaries, labels and optional image understanding have finished. Incomplete embeddings must not be reported as completed semantic retrieval.
4. Grounded question answering ready. The selected knowledge scope, retrieved evidence, generation budget and citation references are valid for this question.

Separating these states improves perceived responsiveness without pretending background work has finished. A document can be searchable by exact terms while its semantic index is still pending. The tradeoff is more state management: the UI and retrieval pipeline must agree about readiness, errors and retries.

Evidence still required: verify the current RecalAI source code on each supported platform, including import-state persistence, crash recovery, retry behavior, and library visibility after relaunch. This draft does not claim cross-platform equivalence has been measured.

Lexical retrieval and vector retrieval solve different problems

Consider a document containing the exact identifier LFM2.5-2.6B-Q4_K_M. Searching for this identifier is not the same task as asking, “Could that smaller on-device model run on an older phone?”

An FTS engine can locate literal identifiers, paths, error strings, exact wording and names. Semantic retrieval can help with paraphrases and conceptually related passages. Both fail in different ways: tokenization and multilingual handling affect lexical search, while embedding quality, chunk boundaries and similarity thresholds affect vector search. Numbers, negation, access-control scope and neighboring version identifiers remain dangerous ambiguity points.

SQLite’s FTS5 documentation specifies bm25(), snippet() and highlight(). In FTS5’s BM25 convention, lower returned scores mean better matches. The actual tokenizer, scoring and index implementation in a given RecalAI build must be confirmed from its code rather than inferred from the use of the term “FTS.”

Query	Lexical strength	Semantic opportunity	Failure risk
Exact error, model name, API identifier	Literal localization	Related context	Incorrect identifier
“Which note discussed device/cloud tradeoffs?”	Keyword anchors	Paraphrases	Generic discussion outranks source
Date, amount or version difference	Exact spans	Supporting context	Numeric or negation confusion
Query in a different language	Depends on tokenizer and vocabulary	Potential cross-language recall	Model coverage varies

Fuse ranks, deduplicate chunks, then rerank

Raw lexical scores and vector similarities live on different scales. Adding them directly without calibration can produce arbitrary rankings. Reciprocal Rank Fusion (RRF) combines the positions from two or more ranked lists:

RRF(d) = Σ 1 / (k + rank_i(d))

Here rank_i(d) is a document’s rank in retrieval list i, and k is a smoothing constant. Cormack, Clarke and Büttcher described the method in SIGIR 2009. That research does not establish that a particular RecalAI configuration outperforms alternatives on its user’s data.

Deduplication matters because overlapping chunks from a single source can otherwise dominate the limited context window. Reranking has a separate job: among plausible candidates, which passages actually help answer this question? A snippet that contains “permission” and “cloud” may still fail to identify the specific permission or condition the user asked about. Rerank latency, memory use and language performance also require real-device validation.

Evidence selection is the real boundary before generation

Retrieval is candidate discovery, not an answer certificate. Before passing passages to a generator, restrict them to the user’s selected and authorized knowledge scope, preserve source_id, chunk_id and page or location references, reduce near-duplicates, and allocate an explicit token budget. Where sources disagree, do not silently omit the inconvenient contradictory source.

Citation validation must at least ensure that cited IDs refer to retrieved passages the user can still access. A correctly formatted citation proves traceability, not that the model interpreted the passage correctly. Unsupported questions should result in a clearly communicated lack of evidence rather than invented file references.

Readers who want to understand actual product scope can continue to the RecalAI product page, which distinguishes the local-first direction from currently unavailable public app-store downloads.

Why device limits change the design

On an older iPhone, document parsing, FTS queries, embedding, reranking, model inference, previews and UI rendering all compete for memory, compute time and battery. Lower-memory devices may need tighter concurrency, smaller candidate sets and shorter generation context. Larger devices may support more candidates or a larger model, but this is not a guarantee of better factual accuracy.

A robust design separates candidate retrieval, evidence selection and the generation budget, and adapts each to device capabilities and current pressure. During memory warnings, preserve source and index consistency and user-visible progress before expensive background enrichment. This article does not provide fabricated latency, recall, battery or device-speed comparisons.

Verified against the current RecalAI iOS source code

This section reflects a static source inspection on October 23, 2026, not an on-device benchmark. The examined files are Sources/Core/Search/PMSearchService.swift, Sources/Core/AI/PMContextBuilder.swift, Sources/Core/AI/PMAssistantModels.swift, and Sources/Core/ModelDelivery/PMModelTechnicalSpecification.swift. A final release should pin the reviewed Git commit. Only non-sensitive implementation details are summarized here.

The chunk search path is more specific than “FTS plus vectors.” It combines exact substring matches, anchor-term matches, strict lexical matches, broad lexical matches, and dense-vector candidates. The dense branch requires semantic search readiness and query embeddings; the current code filters cosine similarity at 0.30 and keeps up to 50 dense candidates. These are observed code constants, not empirically optimized thresholds.

Weighted RRF uses k = 60. The current five list weights are 3.0 for exact, 2.8 for anchors, 1.7 for strict lexical, 0.9 for broad lexical, and 1.0 for dense retrieval. Query-signal boosts are subsequently applied. There is no standalone neural reranker model in the first mobile release. The implementation’s rerank() is deterministic, rewarding query-term coverage and exact numeric/date/model-token signals. Describing this as a deployed cross-encoder would be inaccurate.

The PMContextBuilder.build() path defaults to 2048 estimated tokens, up to 8 evidence chunks, and no more than 3 chunks per knowledge source. It removes volatile UI metadata and shares the remaining context budget across passages. The budget approximation is character-based for mixed scripts; it is not an exact model-tokenizer count.

Citation handling is also checked in code: PMGroundedResponseParser.parse() receives the set of allowed chunk_id values. An ANSWER or PARTIAL result with missing or unauthorized source IDs falls back to NOT_FOUND. That validates citation identity, not the complete semantic correctness of the model’s interpretation.

These observations do not establish equivalence across Android, macOS and Windows, or relative retrieval accuracy and latency on devices. The article is a source-backed engineering analysis, not a performance benchmark.

The repository also contains Tests/PMRetrievalCoreTests.swift for normalization, similarity, fallback thresholds and evaluation math, plus Tests/Fixtures/rag_acceptance.json with 11 defined test cases spanning Chinese/English, cross-language, OCR errors, long PDFs, identifiers, dates, versions and no-answer scenarios. These are test definitions—not evidence that the tests passed in this session or that measured retrieval quality generalizes to users.

What an honest retrieval experiment must measure

Before labeling this an empirical product benchmark, compare four configurations on the same dataset and hardware: FTS-only, vector-only, fusion-only, and fusion plus rerank.

The evaluation set should include exact identifiers, paraphrases, multilingual queries, negation, conflicting documents and unanswerable questions. Fix chunking, corpus, model versions and query definitions. Record retrieval metrics such as Recall@k, MRR or nDCG alongside citation correctness, appropriate abstention, P50/P95 latency, peak memory and time to first searchable result. Retain commands, device specifications and raw results. AI-generated explanatory diagrams cannot replace logs or screenshots.

When simpler search is enough

If a small collection is queried mostly by exact field or identifier, a reliable FTS index may be the most maintainable answer. If users ask highly paraphrased topical questions about short text, semantic retrieval may be sufficient for candidate discovery. Hybrid retrieval becomes compelling when exact strings, diverse formats, languages and grounded citations coexist—but its extra complexity must be justified empirically.

The engineering takeaway is not “install a vector database.” It is preserve the source → parse and expose accurate readiness → retrieve lexical and semantic candidates → fuse and deduplicate → select evidence → generate within constraints → validate citations. Architecture is a hypothesis; reproducible user-task evidence is the acceptance test.

Sources and publication gaps

• SQLite FTS5 reference for lexical ranking and snippets.
• Cormack, Clarke and Buettcher, RRF, SIGIR 2009 for rank fusion methodology.
• XBSTACK Personal AI Agent architecture for the documented first-party design.
• XBSTACK Agent Memory systems for the distinction between retrieval, memory and task state.

Before publication: inspect current retrieval source code and versions; conduct controlled device tests; confirm differentiated search intent; generate the article-specific GPT cover and diagrams; finalize bilingual metadata and reciprocal links; pass publishing and live verification gates.

Topic path / AI Agents

Continue from one agent pattern to the complete production system

The AI Agent hub organizes architecture, memory, tool use, evaluation, security, deployment and multi-agent coordination into a single learning path.

Open AI Agent hub → Review the architecture guide Subscribe