← All guides AI Release
RU Subscribe to @ai_release1

How to Set Up Local Semantic File Search with Rummage and EmbeddingGemma 2

Updated: 24.10.2026 · AI Release · @ai_release1
How to Set Up Local Semantic File Search with Rummage and EmbeddingGemma 2

TL;DR

- Python 3.10 or newer with venv and pip (confirm the exact minimum in the project's pyproject.toml — it changes between releases)

- Rummage installed and available on your PATH

- An EmbeddingGemma 2 checkpoint you can reach: a local directory or a Hugging Face repo ID (take the exact ID from the model card — it differs between releases)

- About 2 GB of free disk for the weights plus the index (the model card lists the real download size)

- A Hugging Face read token if the model repository is gated

Requirements and setup

Illustration: Requirements and setup
Illustration: Requirements and setup

Semantic search means you compare *meaning*, not keywords: every chunk of text becomes a vector (a list of numbers), and a query becomes a vector too. The closest vectors win. This is the same idea used in a local RAG setup on a mini PC, only here the index is a single file on disk instead of a vector database.

The pipeline you are building:

./corpus/                 raw files (.md, .txt, ...)
   |  Rummage: walk + text extraction
   v
./corpus_txt/             plain-text mirror of the corpus
   |  build_index.py: chunk -> embed
   v
./index/vectors.npy       float32 matrix, one row per chunk
./index/chunks.jsonl      one record per row: path, offset, text
   |  search.py: embed the query, cosine similarity
   v
top-k results: score, path@offset, snippet

Example: a 40-page runbook.md becomes roughly 60 chunks of 800 characters; asking "how do I rotate the API key" returns the chunk containing the rotation procedure even though the word "rotate" never appears in your query phrasing.

Before touching any code, verify the base layer:

python3 --version
rummage --help | head -n 20
df -h .

Example output (versions will differ on your machine):

Python 3.11.9
Usage: rummage [OPTIONS] COMMAND [ARGS]...

If rummage --help returns command not found, install Rummage first and re-open the shell so PATH is refreshed — that is the single most common failure at this stage.

Step 1. Create an isolated environment and stage the corpus

What to do: create a virtual environment (an isolated Python folder so this project's packages cannot break your system Python), install the two libraries the indexer needs, and put the files you want to search into ./corpus.

mkdir -p ~/semantic-search && cd ~/semantic-search
python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install sentence-transformers numpy

sentence-transformers loads the embedding model and produces the vectors; numpy stores and compares them. Installing inside .venv is why python -m pip is used instead of a bare pip — it guarantees the packages land in the active environment.

Now stage the files and let Rummage extract text:

mkdir -p corpus corpus_txt
cp ~/Documents/notes/*.md corpus/

# Template: Rummage's subcommand and flag names change between releases.
# Confirm them with `rummage --help` before running this.
rummage <extract-subcommand> --input ./corpus --output ./corpus_txt

Example: if rummage --help lists extract, the call becomes rummage extract --input ./corpus --output ./corpus_txt. If your build only walks directories and prints paths, skip the extraction step and point the indexer at ./corpus directly — the loader below only needs plain-text files.

Why: every later step reads ./corpus_txt. Getting a clean text mirror first means the chunker never has to guess about binary formats.

How to check:

ls corpus_txt | head
find corpus_txt -name '*.txt' | wc -l
python -c "import sentence_transformers, numpy; print(sentence_transformers.__version__, numpy.__version__)"

You should see your file names, a count greater than zero, and two version numbers. A count of 0 means the extraction wrote nothing — re-read rummage --help before continuing.

Step 2. Build the vector index

Illustration: Step 2. Build the vector index
Illustration: Step 2. Build the vector index

What to do: save this as build_index.py. It walks the text mirror, cuts each file into overlapping chunks, embeds every chunk, and writes two files that stay row-aligned: vectors.npy (the numbers) and chunks.jsonl (the text plus its source).

import json, os, pathlib
import numpy as np
from sentence_transformers import SentenceTransformer

ROOT = pathlib.Path(os.environ.get("CORPUS_DIR", "./corpus_txt"))
OUT = pathlib.Path("./index"); OUT.mkdir(exist_ok=True)
MODEL_ID = os.environ["EMBED_MODEL"]          # path or repo ID from the model card
CHUNK_CHARS, OVERLAP = 800, 100               # overlap keeps sentences from being cut in half

model = SentenceTransformer(MODEL_ID)

def split(text):
    step = CHUNK_CHARS - OVERLAP
    for start in range(0, max(len(text) - OVERLAP, 1), step):
        piece = text[start:start + CHUNK_CHARS].strip()
        if piece:
            yield start, piece

records, payload = [], []
for path in sorted(ROOT.rglob("*.txt")) + sorted(ROOT.rglob("*.md")):
    text = path.read_text(encoding="utf-8", errors="ignore")
    for offset, piece in split(text):
        records.append({"path": str(path), "offset": offset, "text": piece})
        payload.append(piece)

if not records:
    raise SystemExit(f"no .txt/.md files under {ROOT.resolve()}")

vectors = model.encode(payload, batch_size=32,
                       normalize_embeddings=True, show_progress_bar=True)
np.save(OUT / "vectors.npy", vectors.astype("float32"))
with open(OUT / "chunks.jsonl", "w", encoding="utf-8") as f:
    for r in records:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")
print(f"indexed {len(records)} chunks, matrix shape {vectors.shape}")

Run it:

export EMBED_MODEL="<path-or-repo-id-from-the-model-card>"
python build_index.py

Why it works: normalize_embeddings=True scales every vector to length 1, so a plain dot product between two vectors equals cosine similarity — the cheapest correct comparison you can do. The 100-character overlap means a sentence split across a boundary still appears intact in one of the two neighbouring chunks.

Two things to confirm against the EmbeddingGemma model card before you trust the numbers: whether the model expects separate prompt prefixes for documents and for queries, and whether it recommends a reduced output dimension. If prefixes are required, add them in both scripts at the same time.

Example: 120 markdown files of average 3 KB each produce roughly 500–600 chunks; the printed shape reads (560, <dim>), where <dim> is whatever the model card specifies.

How to check:

wc -l index/chunks.jsonl
python -c "import numpy as np; v=np.load('index/vectors.npy'); print(v.shape, v.dtype, np.linalg.norm(v[0]))"

Line count must equal the number of rows in the matrix, dtype must be float32, and the norm must print 1.0 (or 0.99999994). If the norm is not 1, normalization did not run and your scores will be meaningless.

Step 3. Query the index and verify relevance

Illustration: Step 3. Query the index and verify relevance
Illustration: Step 3. Query the index and verify relevance

What to do: save this as search.py and run a query you already know the answer to.

import json, os, sys
import numpy as np
from sentence_transformers import SentenceTransformer

MODEL_ID = os.environ["EMBED_MODEL"]          # must be the SAME checkpoint as the index
q = " ".join(sys.argv[1:])
if not q:
    raise SystemExit('usage: python search.py "your question"')

model = SentenceTransformer(MODEL_ID)
vectors = np.load("index/vectors.npy")
rows = [json.loads(line) for line in open("index/chunks.jsonl", encoding="utf-8")]

query_vector = model.encode([q], normalize_embeddings=True)[0]
scores = vectors @ query_vector

for i in np.argsort(scores)[::-1][:3]:
    hit = rows[i]
    print(f"{scores[i]:.3f}  {hit['path']}@{hit['offset']}")
    print("    " + hit["text"][:160].replace("\n", " ") + "\n")
python search.py "how do I rotate the API key"

Example output (scores depend on your model and corpus — what matters is the ordering):

0.7xx  corpus_txt/runbook.txt@1200
    Rotate the API key every 90 days via the admin console ...

0.6xx  corpus_txt/onboarding.txt@0
    New engineers receive a read-only key on day one ...

Why it works: the query goes through exactly the same model as the chunks, so both live in one vector space; the matrix multiply scores every chunk in a single call, which is why a few thousand chunks still answer in well under a second.

How to check: pick a question whose answer sits in a file you named yourself, for example "which port does the staging API listen on". The top hit must be that file. Then run a deliberately unrelated query such as "sourdough hydration" — the top score should drop noticeably and the snippets should look irrelevant. That contrast is your proof the index is semantic and not just returning whatever it read first.

Troubleshooting

Problem. The indexer stops immediately after the imports.

source .venv/bin/activate
python -m pip install sentence-transformers numpy
python -c "import sys; print(sys.executable)"

The printed path must contain /.venv/.

Problem. The model download fails on a gated repository.

huggingface-cli login          # paste a READ token
export EMBED_MODEL="<exact-id-from-the-model-card>"
python build_index.py

Problem. Search crashes with a shape error after you swap the model.

rm -rf index
python build_index.py
python search.py "how do I rotate the API key"

If the corpus keeps growing, move the same vectors into a dedicated store — see Local RAG: running Qdrant or Chroma on your own mini PC.

Summary and next steps

You now have a local index that turns a plain-language question into a ranked list of file locations and snippets, with no network access needed after the model is cached. The natural next step is to feed the top three chunks into a chat model as context, and if you want OS-level search instead of a script, compare with AI-powered search in Windows 11.

FAQ

Do my files leave the machine?

No. Only the model weights are downloaded once; chunking, embedding and scoring all run in your Python process. To prove it, disconnect the network and rerun python search.py "..." — it still works because the checkpoint is in the local cache.

Can I search PDF and DOCX files directly?

Only if Rummage extracts their text into ./corpus_txt. The loader in Step 2 reads .txt and .md only, so convert other formats first or extend the glob patterns — check the Rummage documentation for which formats its extractor supports.

How large a corpus can this handle?

A brute-force NumPy scan is comfortable for tens of thousands of chunks, because it is one matrix multiply. Past that, rebuild the index in a vector database rather than growing vectors.npy.

Why do my scores never reach 1.0?

Normalized cosine similarity of two different texts practically never equals 1.0. Judge results by the gap between the first and the third hit, not by the absolute number; if all three scores cluster tightly together, your chunks are too small or the model needs the query prefix recommended on its model card.

🤖 AI summary

Related guides

Claude Code for Teams: Setting Up Subagents and a Shared AI Superagent

Пошаговый гид по настройке Claude Code для нескольких разработчиков: субагенты, MCP-серверы, общие правила и единый супе

Local RAG: running Qdrant or Chroma on your own mini PC

Собираем локальную RAG-систему без облака: векторная база Qdrant/Chroma, эмбеддинги и подключение к локальной LLM.

How to Use AI-Powered Search in Windows 11: Commands and Setup

Рассказываем, как находить файлы, запускать программы и управлять ПК через новый поиск Windows 11 с поддержкой Copilot.