# How to Set Up Local Semantic File Search with Rummage and EmbeddingGemma 2

> AI Release · @ai_release1 · https://ai-release.org/guides/kak-nastroit-lokalnyy-semanticheskiy-poisk-po-faylam-s_en.html

## TL;DR

- **Time to complete:** 30–45 minutes on a laptop that already has a working Python toolchain.
- **Difficulty:** Intermediate
- **You will need:**
  - Python 3.10 or newer with `venv` and `pip` (confirm the exact minimum in the project's `pyproject.toml` — it changes between releases)
  - Rummage installed and available on your `PATH`
  - An EmbeddingGemma 2 checkpoint you can reach: a local directory or a Hugging Face repo ID (take the exact ID from the model card — it differs between releases)
  - About 2 GB of free disk for the weights plus the index (the model card lists the real download size)
  - A Hugging Face **read** token if the model repository is gated

## Requirements and setup
![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-nastroit-lokalnyy-semanticheskiy-poisk-po-faylam-s-en-1.jpg?v=1792832271)


Semantic search means you compare *meaning*, not keywords: every chunk of text becomes a vector (a list of numbers), and a query becomes a vector too. The closest vectors win. This is the same idea used in a [local RAG setup on a mini PC](https://ai-release.org/guides/lokalnyy-rag-na-mini-pk.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide), only here the index is a single file on disk instead of a vector database.

The pipeline you are building:

```
./corpus/                 raw files (.md, .txt, ...)
   |  Rummage: walk + text extraction
   v
./corpus_txt/             plain-text mirror of the corpus
   |  build_index.py: chunk -> embed
   v
./index/vectors.npy       float32 matrix, one row per chunk
./index/chunks.jsonl      one record per row: path, offset, text
   |  search.py: embed the query, cosine similarity
   v
top-k results: score, path@offset, snippet
```

**Example:** a 40-page `runbook.md` becomes roughly 60 chunks of 800 characters; asking "how do I rotate the API key" returns the chunk containing the rotation procedure even though the word "rotate" never appears in your query phrasing.

Before touching any code, verify the base layer:

- [ ] `python3 --version` prints `3.10` or newer
- [ ] `python3 -m venv --help` prints usage (the `venv` module is present)
- [ ] `rummage --help` prints a subcommand list
- [ ] `echo $EMBED_MODEL` prints the checkpoint path or repo ID you intend to use
- [ ] `df -h .` shows at least 2 GB free
- [ ] `huggingface-cli whoami` prints your username — only needed if the repo is gated

```bash
python3 --version
rummage --help | head -n 20
df -h .
```

**Example output** (versions will differ on your machine):

```
Python 3.11.9
Usage: rummage [OPTIONS] COMMAND [ARGS]...
```

If `rummage --help` returns `command not found`, install Rummage first and re-open the shell so `PATH` is refreshed — that is the single most common failure at this stage.

## Step 1. Create an isolated environment and stage the corpus

**What to do:** create a virtual environment (an isolated Python folder so this project's packages cannot break your system Python), install the two libraries the indexer needs, and put the files you want to search into `./corpus`.

```bash
mkdir -p ~/semantic-search && cd ~/semantic-search
python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install sentence-transformers numpy
```

`sentence-transformers` loads the embedding model and produces the vectors; `numpy` stores and compares them. Installing inside `.venv` is why `python -m pip` is used instead of a bare `pip` — it guarantees the packages land in the active environment.

Now stage the files and let Rummage extract text:

```bash
mkdir -p corpus corpus_txt
cp ~/Documents/notes/*.md corpus/

# Template: Rummage's subcommand and flag names change between releases.
# Confirm them with `rummage --help` before running this.
rummage <extract-subcommand> --input ./corpus --output ./corpus_txt
```

**Example:** if `rummage --help` lists `extract`, the call becomes `rummage extract --input ./corpus --output ./corpus_txt`. If your build only walks directories and prints paths, skip the extraction step and point the indexer at `./corpus` directly — the loader below only needs plain-text files.

**Why:** every later step reads `./corpus_txt`. Getting a clean text mirror first means the chunker never has to guess about binary formats.

**How to check:**

```bash
ls corpus_txt | head
find corpus_txt -name '*.txt' | wc -l
python -c "import sentence_transformers, numpy; print(sentence_transformers.__version__, numpy.__version__)"
```

You should see your file names, a count greater than zero, and two version numbers. A count of `0` means the extraction wrote nothing — re-read `rummage --help` before continuing.

## Step 2. Build the vector index
![Illustration: Step 2. Build the vector index](https://ai-release.org/guides/img/kak-nastroit-lokalnyy-semanticheskiy-poisk-po-faylam-s-en-2.jpg?v=1792832277)


**What to do:** save this as `build_index.py`. It walks the text mirror, cuts each file into overlapping chunks, embeds every chunk, and writes two files that stay row-aligned: `vectors.npy` (the numbers) and `chunks.jsonl` (the text plus its source).

```python
import json, os, pathlib
import numpy as np
from sentence_transformers import SentenceTransformer

ROOT = pathlib.Path(os.environ.get("CORPUS_DIR", "./corpus_txt"))
OUT = pathlib.Path("./index"); OUT.mkdir(exist_ok=True)
MODEL_ID = os.environ["EMBED_MODEL"]          # path or repo ID from the model card
CHUNK_CHARS, OVERLAP = 800, 100               # overlap keeps sentences from being cut in half

model = SentenceTransformer(MODEL_ID)

def split(text):
    step = CHUNK_CHARS - OVERLAP
    for start in range(0, max(len(text) - OVERLAP, 1), step):
        piece = text[start:start + CHUNK_CHARS].strip()
        if piece:
            yield start, piece

records, payload = [], []
for path in sorted(ROOT.rglob("*.txt")) + sorted(ROOT.rglob("*.md")):
    text = path.read_text(encoding="utf-8", errors="ignore")
    for offset, piece in split(text):
        records.append({"path": str(path), "offset": offset, "text": piece})
        payload.append(piece)

if not records:
    raise SystemExit(f"no .txt/.md files under {ROOT.resolve()}")

vectors = model.encode(payload, batch_size=32,
                       normalize_embeddings=True, show_progress_bar=True)
np.save(OUT / "vectors.npy", vectors.astype("float32"))
with open(OUT / "chunks.jsonl", "w", encoding="utf-8") as f:
    for r in records:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")
print(f"indexed {len(records)} chunks, matrix shape {vectors.shape}")
```

Run it:

```bash
export EMBED_MODEL="<path-or-repo-id-from-the-model-card>"
python build_index.py
```

**Why it works:** `normalize_embeddings=True` scales every vector to length 1, so a plain dot product between two vectors equals cosine similarity — the cheapest correct comparison you can do. The 100-character overlap means a sentence split across a boundary still appears intact in one of the two neighbouring chunks.

Two things to confirm against the EmbeddingGemma model card before you trust the numbers: whether the model expects separate prompt prefixes for documents and for queries, and whether it recommends a reduced output dimension. If prefixes are required, add them in both scripts at the same time.

**Example:** 120 markdown files of average 3 KB each produce roughly 500–600 chunks; the printed shape reads `(560, <dim>)`, where `<dim>` is whatever the model card specifies.

**How to check:**

```bash
wc -l index/chunks.jsonl
python -c "import numpy as np; v=np.load('index/vectors.npy'); print(v.shape, v.dtype, np.linalg.norm(v[0]))"
```

Line count must equal the number of rows in the matrix, dtype must be `float32`, and the norm must print `1.0` (or `0.99999994`). If the norm is not 1, normalization did not run and your scores will be meaningless.

## Step 3. Query the index and verify relevance
![Illustration: Step 3. Query the index and verify relevance](https://ai-release.org/guides/img/kak-nastroit-lokalnyy-semanticheskiy-poisk-po-faylam-s-en-3.jpg?v=1792832284)


**What to do:** save this as `search.py` and run a query you already know the answer to.

```python
import json, os, sys
import numpy as np
from sentence_transformers import SentenceTransformer

MODEL_ID = os.environ["EMBED_MODEL"]          # must be the SAME checkpoint as the index
q = " ".join(sys.argv[1:])
if not q:
    raise SystemExit('usage: python search.py "your question"')

model = SentenceTransformer(MODEL_ID)
vectors = np.load("index/vectors.npy")
rows = [json.loads(line) for line in open("index/chunks.jsonl", encoding="utf-8")]

query_vector = model.encode([q], normalize_embeddings=True)[0]
scores = vectors @ query_vector

for i in np.argsort(scores)[::-1][:3]:
    hit = rows[i]
    print(f"{scores[i]:.3f}  {hit['path']}@{hit['offset']}")
    print("    " + hit["text"][:160].replace("\n", " ") + "\n")
```

```bash
python search.py "how do I rotate the API key"
```

**Example output** (scores depend on your model and corpus — what matters is the ordering):

```
0.7xx  corpus_txt/runbook.txt@1200
    Rotate the API key every 90 days via the admin console ...

0.6xx  corpus_txt/onboarding.txt@0
    New engineers receive a read-only key on day one ...
```

**Why it works:** the query goes through exactly the same model as the chunks, so both live in one vector space; the matrix multiply scores every chunk in a single call, which is why a few thousand chunks still answer in well under a second.

**How to check:** pick a question whose answer sits in a file you named yourself, for example "which port does the staging API listen on". The top hit must be that file. Then run a deliberately unrelated query such as "sourdough hydration" — the top score should drop noticeably and the snippets should look irrelevant. That contrast is your proof the index is semantic and not just returning whatever it read first.

## Troubleshooting

**Problem. The indexer stops immediately after the imports.**

- Error: `ModuleNotFoundError: No module named 'sentence_transformers'`
- Cause: the virtual environment is not active, so `python` resolves to the system interpreter where nothing was installed.
- Solution: activate and reinstall, then confirm which interpreter you are using:

```bash
source .venv/bin/activate
python -m pip install sentence-transformers numpy
python -c "import sys; print(sys.executable)"
```

The printed path must contain `/.venv/`.

**Problem. The model download fails on a gated repository.**

- Error: `OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like <model-id> is not the path to a directory containing a file named config.json.`
- Cause: the repo ID is misspelled, the machine is offline with an empty cache, or the repo is gated and no token was supplied.
- Solution: copy the ID character-for-character from the model card, then authenticate once and retry:

```bash
huggingface-cli login          # paste a READ token
export EMBED_MODEL="<exact-id-from-the-model-card>"
python build_index.py
```

**Problem. Search crashes with a shape error after you swap the model.**

- Error: `ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 768 is different from 1024)`
- Cause: `vectors.npy` was built with one checkpoint and the query is embedded with another, so the dimensions no longer match.
- Solution: always rebuild after changing `EMBED_MODEL`; the two files must be regenerated together:

```bash
rm -rf index
python build_index.py
python search.py "how do I rotate the API key"
```

If the corpus keeps growing, move the same vectors into a dedicated store — see [Local RAG: running Qdrant or Chroma on your own mini PC](https://ai-release.org/guides/lokalnyy-rag-na-mini-pk.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## Summary and next steps

You now have a local index that turns a plain-language question into a ranked list of file locations and snippets, with no network access needed after the model is cached. The natural next step is to feed the top three chunks into a chat model as context, and if you want OS-level search instead of a script, compare with [AI-powered search in Windows 11](https://ai-release.org/guides/kak-ispolzovat-ii-poisk-v-windows-11-komandy-i-nastroyk.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## FAQ

**Do my files leave the machine?**
No. Only the model weights are downloaded once; chunking, embedding and scoring all run in your Python process. To prove it, disconnect the network and rerun `python search.py "..."` — it still works because the checkpoint is in the local cache.

**Can I search PDF and DOCX files directly?**
Only if Rummage extracts their text into `./corpus_txt`. The loader in Step 2 reads `.txt` and `.md` only, so convert other formats first or extend the glob patterns — check the Rummage documentation for which formats its extractor supports.

**How large a corpus can this handle?**
A brute-force NumPy scan is comfortable for tens of thousands of chunks, because it is one matrix multiply. Past that, rebuild the index in a vector database rather than growing `vectors.npy`.

**Why do my scores never reach 1.0?**
Normalized cosine similarity of two different texts practically never equals 1.0. Judge results by the gap between the first and the third hit, not by the absolute number; if all three scores cluster tightly together, your chunks are too small or the model needs the query prefix recommended on its model card.