# How to Reverse-Engineer a Competitor's Marketing Funnel from a Single Link with AI

> AI Release · @ai_release1 · https://ai-release.org/guides/kak-razobrat-marketingovuyu-voronku-konkurenta-po-ssylk_en.html

## TL;DR
- **Time to complete:** 60–90 minutes for one competitor domain.
- **Difficulty:** Intermediate.
- **You will need:** macOS, Linux, or Windows 11 with WSL2 (all commands below are bash); `curl` and `jq`; Python 3 with `beautifulsoup4`; an LLM API key that exposes an OpenAI-compatible `/v1/chat/completions` endpoint; about 50 MB of free disk space; a text editor.
- **Output:** a `funnel.jsonl` dataset plus a `funnel-report.md` table that maps each funnel layer (ads → social → SEO) to evidence you can open in a browser.

## Requirements and setup
![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-razobrat-marketingovuyu-voronku-konkurenta-po-ssylk-en-1.jpg)


- **OS and shell.** macOS, Ubuntu, or Windows 11 + WSL2. Run `bash --version` — you should see a version line, not `command not found`.
- **curl.** Check with `curl --version`. The scripts use `-sS`, `-o`, `-w '%{http_code}'`, and `-A`; all are present in any current build.
- **jq.** `jq --version` must print a version. Install with `brew install jq` (macOS) or `sudo apt install jq` (Debian/Ubuntu).
- **Python and parser.** `python3 --version`, then `python3 -m pip install beautifulsoup4`. BeautifulSoup is an HTML parser — it turns raw markup into a tree you can query by tag.
- **LLM endpoint.** Export `LLM_API_KEY`, `LLM_BASE_URL`, and `LLM_MODEL`. The exact model ID differs per provider — **check your provider's documentation** for the current identifier; do not guess it.
- **Legal check before anything else.** Read `https://<competitor-domain>/robots.txt` and the terms of every platform you touch. Public ad libraries (Google Ads Transparency Center, Meta Ad Library) have their own access rules that change over time — **verify the current URL and limits on the platform's help page** before automating anything.
- **Disk.** Raw HTML for 200 pages is typically a few MB; 50 MB is comfortable.

## Step 1. Map the funnel and capture the raw inputs

**What to do:** create a workspace and pull three things per competitor: the landing page HTML, `robots.txt`, and `sitemap.xml`. Then record the ad-library and social results manually into a notes file.

**Why:** an LLM can only reason about text you actually give it. The funnel has three layers, and each lives in a different place:

| Funnel layer | Where to look | What to capture | What the AI does with it |
|---|---|---|---|
| Ads (top) | Public ad libraries; landing pages with UTM parameters | Ad copy, CTA, destination URL | Groups ads into offers and angles |
| Social (middle) | Public profile bios, pinned posts, link-in-bio | Bio text, recurring hooks, posting cadence | Detects the message that repeats |
| SEO (bottom) | `sitemap.xml`, `<title>`, meta description, H1/H2, `og:` tags | Page inventory and on-page fields | Clusters pages into topic silos |

```bash
mkdir -p ~/funnel/<competitor> && cd ~/funnel/<competitor>
UA="Mozilla/5.0 (funnel-research; contact: you@example.com)"

curl -sS -A "$UA" -o home.html     -w "%{http_code}\n" "https://<competitor-domain>/"
curl -sS -A "$UA" -o robots.txt    -w "%{http_code}\n" "https://<competitor-domain>/robots.txt"
curl -sS -A "$UA" -o sitemap.xml   -w "%{http_code}\n" "https://<competitor-domain>/sitemap.xml"

grep -c "<loc>" sitemap.xml
```

**Example:** the three `curl` calls print `200`, `200`, `200`, and `grep` prints a URL count such as `184`. Replace `<competitor-domain>` with the bare host, e.g. `example-shop.com` — no `https://`, no trailing slash.

**How to check:** `ls -la` shows three non-empty files; `grep -c "<loc>" sitemap.xml` returns a number greater than 0. If `sitemap.xml` is empty, look for a `Sitemap:` line inside `robots.txt` and fetch that path instead.

## Step 2. Normalize pages to JSONL and run the LLM analysis
![Illustration: Step 2. Normalize pages to JSONL and run the LLM analysis](https://ai-release.org/guides/img/kak-razobrat-marketingovuyu-voronku-konkurenta-po-ssylk-en-2.jpg)


**What to do:** extract a fixed set of fields from every saved page into one JSON object per line (JSONL — a text file where each line is a complete JSON record), then send that file to the model with a strict output schema.

**Why:** models drift when fed raw HTML. A fixed field list makes the output comparable across pages and lets you verify each claim against the source file later.

```python
# extract.py — run inside ~/funnel/<competitor>
import json, glob
from bs4 import BeautifulSoup

def meta(soup, key):
    tag = soup.find("meta", attrs={"name": key}) or soup.find("meta", attrs={"property": key})
    return (tag.get("content") or "").strip() if tag else ""

rows = []
for path in sorted(glob.glob("pages/*.html")):
    soup = BeautifulSoup(open(path, encoding="utf-8", errors="ignore").read(), "html.parser")
    rows.append({
        "source_file": path,
        "title": soup.title.get_text(strip=True) if soup.title else "",
        "description": meta(soup, "description"),
        "og_title": meta(soup, "og:title"),
        "h1": [h.get_text(strip=True) for h in soup.find_all("h1")],
        "h2": [h.get_text(strip=True) for h in soup.find_all("h2")][:20],
        "links": len(soup.find_all("a")),
    })

with open("funnel.jsonl", "w", encoding="utf-8") as f:
    for r in rows:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")
print(f"wrote {len(rows)} records")
```

**Example:** `python3 extract.py` prints `wrote 184 records`, and `head -n 1 funnel.jsonl | jq .` shows one object with `title`, `description`, `h1`, `h2`, `links`.

Then call the model with a schema that forbids free-form prose:

```bash
export LLM_API_KEY="<your-key>"
export LLM_BASE_URL="https://<provider-endpoint>/v1/chat/completions"
export LLM_MODEL="<model-id-from-provider-docs>"

jq -n --rawfile data funnel.jsonl --arg m "$LLM_MODEL" '{
  model: $m,
  temperature: 0,
  messages: [
    {role: "system", content: "You are a funnel analyst. Return only JSON: an array of objects with keys layer, claim, evidence_quote, source_file. layer is one of ads|social|seo. evidence_quote must be copied verbatim from the input. If you cannot quote it, omit the row."},
    {role: "user", content: $data}
  ]
}' > payload.json

curl -sS "$LLM_BASE_URL" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d @payload.json | jq -r '.choices[0].message.content' > funnel-ai.json
```

The `evidence_quote` field is what makes this usable: it forces every AI claim to be a substring you can grep. Temperature `0` keeps the same input from producing different clusters on each run.

If you would rather keep the funnel map as a spreadsheet than as JSON, the same JSONL loads directly into a sheet — the workflow in [ChatGPT Office: How to Create Documents and Spreadsheets with AI](https://ai-release.org/guides/chatgpt-office-kak-sozdavat-dokumenty-i-tablitsy-s-pomo.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) covers the table structure and column formatting.

**How to check:** `jq 'length' funnel-ai.json` returns a row count greater than 0, and `jq -r '.[].layer' funnel-ai.json | sort | uniq -c` shows how many rows landed in `ads`, `social`, and `seo`. A layer with zero rows means you did not feed that layer's text yet.

## Step 3. Verify every AI claim and assemble the report
![Illustration: Step 3. Verify every AI claim and assemble the report](https://ai-release.org/guides/img/kak-razobrat-marketingovuyu-voronku-konkurenta-po-ssylk-en-3.jpg)


**What to do:** run three checks — quote validity, layer coverage, and a manual spot check — then write `funnel-report.md`.

**Why:** an LLM will occasionally paraphrase instead of quoting, or invent a plausible-sounding keyword that appears nowhere on the site. The grep check catches both.

```bash
# 1. Every evidence_quote must exist verbatim in the extracted dataset
jq -r '.[].evidence_quote' funnel-ai.json | while read -r q; do
  [ -z "$q" ] && continue
  grep -qF -- "$q" funnel.jsonl || echo "NOT FOUND: $q"
done

# 2. Layer coverage
jq -r '.[].layer' funnel-ai.json | sort | uniq -c

# 3. Manual spot check: open 3 source_file paths and confirm the claim on screen
jq -r '.[0:3][] | "\(.layer)\t\(.source_file)\t\(.claim)"' funnel-ai.json
```

**Example of a clean run:** step 1 prints nothing at all (no `NOT FOUND` lines), step 2 prints `31 ads`, `18 social`, `57 seo`, and step 3 prints three tab-separated rows whose `source_file` values you can open locally.

Checklist before you publish the report:

- [ ] Zero `NOT FOUND` lines from the quote check.
- [ ] All three layers (`ads`, `social`, `seo`) have at least one row.
- [ ] Each ad row's destination URL opens and matches the ad copy you recorded.
- [ ] `wc -l funnel.jsonl` equals the number of HTML files in `pages/`.
- [ ] The report states the capture date, because ad libraries rotate creatives.

For a shareable deck instead of a Markdown file, the layout rules in [How to Create Presentations with AI: Best Tools and Tips](https://ai-release.org/guides/kak-sozdavat-prezentatsii-s-pomoschyu-ii-luchshie-servi.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) apply directly — one slide per funnel layer, evidence quotes in the speaker notes.

**How to check:** `grep -c "^| " funnel-report.md` returns the number of table rows, and opening the file in a viewer shows three sections titled Ads, Social, SEO with a source column in each.

## Troubleshooting

**Problem. curl cannot reach the domain at all.**
- Error: `curl: (6) Could not resolve host: competitor-domian.com`
- Cause: typo in the placeholder, or the host is behind a private DNS you are not on.
- Solution: run `dig +short <competitor-domain>` (or `nslookup`). No output means DNS, not the site. Fix the spelling and retry.

**Problem. The page returns a bot-protection block instead of HTML.**
- Error: `HTTP/2 403` printed by `-w "%{http_code}\n"`, and `home.html` contains a challenge page rather than your expected `<title>`.
- Cause: a CDN or WAF (web application firewall) rejected the request because of the user agent or request rate.
- Solution: set an honest, contactable user agent (`-A "Mozilla/5.0 (funnel-research; contact: you@example.com)"`), add `--compressed`, slow down to one request per second, and if it still fails, fetch the page manually in a browser and save it into `pages/` — do not attempt to bypass the protection.

**Problem. The model call returns prose instead of JSON.**
- Error: `json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)` when you parse `funnel-ai.json`.
- Cause: the endpoint returned an error envelope or a refusal, or the response was truncated by the token limit on a large `funnel.jsonl`.
- Solution: inspect the raw body first with `curl -sS -i ... | head -n 20`; if the input is too large, split `funnel.jsonl` into batches of about 40 records with `split -l 40 funnel.jsonl batch-` and run the same prompt per batch, then merge the arrays.

## Summary and next steps
You now have a reproducible pipeline: capture the raw ad, social, and SEO inputs, normalize them to JSONL, and let the model cluster them under a verbatim-quote constraint you can grep. Re-run the same three steps weekly against the same workspace so the diff between `funnel-ai.json` snapshots shows which offers and keywords the competitor added or dropped.

## FAQ

**Can I run this without an API key?**
Yes — paste `funnel.jsonl` into any chat interface manually and keep the same schema and `evidence_quote` rule. You lose batching and automation, not accuracy.

**Is scraping a competitor's site allowed?**
It depends on `robots.txt`, the site's terms, and your jurisdiction. Read both first; public ad libraries are usually the safer source for ad creatives.

**How do I know the AI did not invent a keyword?**
The `evidence_quote` field must be a verbatim substring of `funnel.jsonl`. If the grep check prints `NOT FOUND`, delete that row.

**How large can the input be?**
It is limited by your model's context window — check your provider's documentation for the current limit and batch accordingly.