# How to Run an OSINT Self-Audit: Check What Personal Data Is Publicly Available Using Watson and Free Tools

> AI Release · @ai_release1 · https://ai-release.org/guides/kak-proverit-kakie-dannye-o-vas-est-v-otkrytom-dostupe_en.html

## TL;DR
- Time to complete: 45–60 minutes
- Difficulty: Intermediate
- You need: Linux (Ubuntu 22.04+), macOS 13+, or Windows 11 with WSL2; Python 3.10+; `pipx`; ExifTool; `jq`; `curl`; a browser; a Watson workspace (Watson is the OSINT aggregation tool where you collect and de-duplicate findings — its menu names and export formats change between releases, so **check the documentation** for your build before clicking anything).

## Requirements and setup
![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-proverit-kakie-dannye-o-vas-est-v-otkrytom-dostupe-en-1.jpg?v=1793176778)


- OS: Ubuntu 22.04+ / macOS 13+ / Windows 11 + WSL2. On Windows, run every command below inside the WSL terminal, not PowerShell.
- Python 3.10 or newer and `pipx` (an installer that puts CLI tools in isolated environments, so they don't break system packages).
- ExifTool — reads metadata (author name, GPS coordinates, software) hidden inside photos and PDFs.
- `jq` — a command-line JSON/URL encoder used below to build search URLs safely.
- At least 2 GB free disk space for downloaded copies of your own files.
- Access to the mailbox you are auditing: breach checks and account-recovery steps send confirmation emails there.
- A Watson workspace created and reachable in your browser. Record the exact menu path you used in `report/watson-notes.txt`.

Install the toolchain:

```bash
# Debian/Ubuntu/WSL
sudo apt update && sudo apt install -y exiftool jq curl pipx
pipx ensurepath

# macOS
brew install exiftool jq pipx
```

**Example:** on Ubuntu 22.04, `exiftool -ver` prints a version string such as `12.40`, and `jq --version` prints `jq-1.6`.

**What to do:** verify all four tools respond.
**Why:** a missing binary fails silently later, when you are already 20 minutes into the audit.
**How to check:**

```bash
python3 --version && exiftool -ver && jq --version && pipx --version
```

Expected: four lines of version output, no `command not found`. Exact numbers differ by distribution — that is fine.

Create the working folders and the breach log:

```bash
mkdir -p ~/osint-audit/{raw,report} && cd ~/osint-audit
printf 'email,breach,date,data_classes\n' > raw/breaches.csv
```

**How to check:** `ls raw` prints `breaches.csv`.

## Step 1. Map what is already indexed about you

**What to do:** build a fixed list of search queries (called "dorks" — search operators such as `site:` and `filetype:` that narrow results) and open each one in a browser. Store the queries in a file so the audit is repeatable in 30 days.

```bash
cd ~/osint-audit
cat > dorks.txt <<'EOF'
"<your-full-name>"
"<your-full-name>" site:linkedin.com
"<your-full-name>" filetype:pdf
"<your-email>"
"<your-email>" filetype:xlsx
site:pastebin.com "<your-email>"
EOF
```

Replace `<your-full-name>` with your name exactly as it appears on your passport or contract, and `<your-email>` with the address you are auditing.

Convert the queries into clickable, URL-encoded links:

```bash
while read -r q; do
  printf 'https://duckduckgo.com/?q=%s\n' "$(printf '%s' "$q" | jq -sRr @uri)"
done < dorks.txt > raw/dork-urls.txt
```

**Why:** `jq -sRr @uri` percent-encodes quotes and spaces, so the query survives the trip through the browser address bar. Hand-typed dorks break on the first `"`.

**How to check:** `wc -l raw/dork-urls.txt` prints `6`, and the first line of `head -1 raw/dork-urls.txt` looks like `https://duckduckgo.com/?q=%22John%20Doe%22`.

Open each URL, and for every hit that is genuinely you, append one row to the findings log:

```bash
cat >> raw/breaches.csv <<'EOF'
<your-email>,<breach-name>,<breach-date>,<data-classes>
EOF
```

**Example:** `<your-email>,ExampleForum2019,2019-04-02,email addresses; hashed passwords`.

Next, check known breaches: open `https://haveibeenpwned.com`, enter the same address, and copy each listed breach into `raw/breaches.csv` using the exact fields the site shows. The site's API requires a separate key — **check the documentation** if you want to automate this step.

**Why this works:** breach lists are public and indexed, so they surface your old addresses faster than any other source.

**How to check:** `column -s, -t raw/breaches.csv` shows a readable table with one header row plus one row per breach.

## Step 2. Enumerate usernames and strip metadata from your own files
![Illustration: Step 2. Enumerate usernames and strip metadata from your own files](https://ai-release.org/guides/img/kak-proverit-kakie-dannye-o-vas-est-v-otkrytom-dostupe-en-2.jpg?v=1793176786)


**What to do:** find which sites still hold an account under your handle, then read the metadata out of the files you publish.

```bash
pipx install sherlock-project
sherlock <your-username> --timeout 10 --print-found | tee raw/sherlock.txt
```

`--timeout 10` is the per-site request limit in seconds; `--print-found` hides the "[*] checking…" noise and keeps only positive matches.

**Example:** a run that finds 14 accounts prints 14 lines starting with `[+]`, e.g. `[+] GitHub: https://github.com/<your-username>`.

**Why:** an old handle tied to a forum you forgot about is a direct path from your name to your posts, and often to a reusable password.

**How to check:** `grep -c '^\[+\]' raw/sherlock.txt` returns the number of live accounts. Anything you no longer use goes on the deletion list.

Now scan the files you actually publish — CVs, photos, contracts — for hidden author and GPS fields:

```bash
exiftool -r -csv -Author -Creator -Artist -GPSLatitude -GPSLongitude -CreateDate \
  ~/Downloads > raw/metadata.csv
```

**Why:** word processors and phones write your real name and shooting location into the file, and those fields survive re-upload to job boards.

**How to check:** `awk -F, 'NR>1 && ($2!="" || $3!="" || $4!="")' raw/metadata.csv` lists only files that still carry an author name. An empty result means the files are clean.

**Example:** a hit looks like `photo.jpg,John Doe,,,55.7558,37.6173` — author plus coordinates. Remove them before the next upload:

```bash
exiftool -all= -overwrite_original ~/Downloads/photo.jpg
```

**How to check:** re-run the `awk` command; the file disappears from the output.

If you later automate these loops with an agent that has access to your mailbox or cloud drive, read [AI agent security: protecting your data during automation](https://ai-release.org/guides/bezopasnost-ii-agentov-kak-zaschitit-dannye-i-izbezhat.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) first — an audit bot with write access can leak exactly what you are trying to contain.

## Step 3. Consolidate the findings in Watson and verify the result
![Illustration: Step 3. Consolidate the findings in Watson and verify the result](https://ai-release.org/guides/img/kak-proverit-kakie-dannye-o-vas-est-v-otkrytom-dostupe-en-3.jpg?v=1793176793)


**What to do:** merge the two raw sources into one normalised CSV, import it into Watson, and tag each row with a severity.

Create `normalize.py` in `~/osint-audit`:

```python
import csv, pathlib

fields = ["finding", "type", "source", "url_or_handle", "exposure", "action"]
rows = []

for r in csv.DictReader(open("raw/breaches.csv")):
    if not r.get("email"):
        continue
    rows.append({"finding": r["breach"], "type": "breach", "source": "HIBP",
                 "url_or_handle": r["email"], "exposure": r["data_classes"],
                 "action": "rotate password + enable 2FA"})

for line in open("raw/sherlock.txt"):
    line = line.strip()
    if not line.startswith("[+]"):
        continue
    rows.append({"finding": "account exists", "type": "username",
                 "source": "sherlock", "url_or_handle": line.split()[-1],
                 "exposure": "public profile", "action": "review or delete"})

out = pathlib.Path("report/findings.csv")
with out.open("w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=fields)
    w.writeheader()
    w.writerows(rows)
print(f"{len(rows)} findings -> {out}")
```

```bash
python3 normalize.py
```

**Example:** with 3 breaches and 14 live accounts, the script prints `17 findings -> report/findings.csv`.

Import `report/findings.csv` into your Watson workspace, then de-duplicate on `url_or_handle` and set severity: `breach` with password data = high; `username` on a site you still use = medium; abandoned account = low. The import button, column-mapping screen and export format differ per build — **check the documentation** for the exact path, then write it into `report/watson-notes.txt`.

**Why:** Watson gives you one deduplicated view instead of three files, so the deletion list is generated once and re-checked later.

**How to check:** in Watson, the row count equals the number the script printed, and the severity column has no empty cells. Then export the report and confirm the export reopens:

```bash
head -3 report/findings.csv
```

Expected: a header line plus two data lines containing a real handle or email, not `,,,`.

Finally, close the loop on the two highest-severity items and log the date you did it. Re-run `python3 normalize.py` after 14 days — the same script on the same inputs must return the same or a smaller count.

Before you store the report on a laptop, note that it now contains your addresses, handles and breach history: encrypt the volume, and if you use Windows, keep the recovery key escrowed — see [BitLocker: how to recover your key and avoid losing data](https://ai-release.org/guides/bitlocker-vosstanovlenie-klyucha.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

Audit checklist:

- [ ] `raw/dork-urls.txt` has 6 queries and every hit is logged in `raw/breaches.csv`
- [ ] `raw/sherlock.txt` reviewed; abandoned accounts marked for deletion
- [ ] `raw/metadata.csv` returns no author or GPS fields for published files
- [ ] `report/findings.csv` imported into Watson with severity on every row
- [ ] Passwords rotated and 2FA enabled on every high-severity breach
- [ ] Re-scan scheduled in 14 days

## Troubleshooting

**Problem. `pip install sherlock-project` refuses to install.**
- Ошибка: `error: externally-managed-environment`
- Причина: Debian 12, Ubuntu 23.04+ and recent Fedora mark the system Python as externally managed (PEP 668), so `pip` cannot write into it.
- Решение: do not use `sudo pip`. Install through `pipx` instead, then reload the shell:

```bash
pipx install sherlock-project
pipx ensurepath
exec $SHELL -l
sherlock --version
```

**Problem. ExifTool reports nothing for a folder full of photos.**
- Ошибка: `    0 image files read`
- Причина: the path points at a directory without `-r`, or the shell passed an unexpanded glob such as `*.jpg` to a folder with no JPEGs.
- Решение: pass the directory itself with recursion, and confirm the folder exists:

```bash
ls -1 ~/Downloads | head
exiftool -r -csv -Author -Artist ~/Downloads | head -5
```

**Problem. The breach API returns an error instead of JSON.**
- Ошибка: `curl: (22) The requested URL returned error: 401`
- Причина: the HIBP v3 API requires an `hibp-api-key` header, and keyless requests are rejected — **check the documentation** for current key tiers.
- Решение: run the check manually in the browser at `https://haveibeenpwned.com` and paste the results into `raw/breaches.csv`; the audit does not depend on the API.

## Summary and next steps
You now have a reproducible four-file pipeline (`dorks.txt`, `breaches.csv`, `sherlock.txt`, `findings.csv`) plus a severity-tagged Watson report, and every high-severity item has a dated remediation. Re-run `python3 normalize.py` in 14 days and the row count should fall or stay flat — if it grows, a new account or breach has appeared. If you plan to widen the audit to exported message archives and need to clean them before analysis, use [How to prepare data for a neural network: dataset cleaning and labeling](https://ai-release.org/guides/kak-podgotovit-dannye-dlya-neyroseti-chistka-i-razmetka.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## FAQ

**Is it legal to run this audit on myself?**
Yes — every step above reads public sources or data tied to your own accounts. Do not point `sherlock` or the dorks at other people's handles without their consent.

**How long does data take to disappear from search results?**
It varies by engine and by page; there is no fixed number. Removing the underlying page or account is what actually makes the result drop out, so log the removal date in `findings.csv` and re-check after 14 days.

**Does deleting an account remove the data a breach exposed?**
No. A breach copy exists outside the service's control, so the only real fix is rotating the password everywhere it was reused and enabling 2FA.

**Do I have to pay for any of these tools?**
The commands in this guide use free packages (`exiftool`, `jq`, `pipx`, `sherlock-project`). Watson's plan limits and the HIBP API key are separate — check the documentation for both before you scale the audit up.