How to Reverse-Engineer a Competitor's Marketing Funnel from a Single Link with AI

TL;DR

• Time to complete: 60–90 minutes for one competitor domain.
• Difficulty: Intermediate.
• You will need: macOS, Linux, or Windows 11 with WSL2 (all commands below are bash); curl and jq; Python 3 with beautifulsoup4; an LLM API key that exposes an OpenAI-compatible /v1/chat/completions endpoint; about 50 MB of free disk space; a text editor.
• Output: a funnel.jsonl dataset plus a funnel-report.md table that maps each funnel layer (ads → social → SEO) to evidence you can open in a browser.

Requirements and setup

• OS and shell. macOS, Ubuntu, or Windows 11 + WSL2. Run bash --version — you should see a version line, not command not found.
• curl. Check with curl --version. The scripts use -sS, -o, -w '%{http_code}', and -A; all are present in any current build.
• jq. jq --version must print a version. Install with brew install jq (macOS) or sudo apt install jq (Debian/Ubuntu).
• Python and parser. python3 --version, then python3 -m pip install beautifulsoup4. BeautifulSoup is an HTML parser — it turns raw markup into a tree you can query by tag.
• LLM endpoint. Export LLM_API_KEY, LLM_BASE_URL, and LLM_MODEL. The exact model ID differs per provider — check your provider's documentation for the current identifier; do not guess it.
• Legal check before anything else. Read https://<competitor-domain>/robots.txt and the terms of every platform you touch. Public ad libraries (Google Ads Transparency Center, Meta Ad Library) have their own access rules that change over time — verify the current URL and limits on the platform's help page before automating anything.
• Disk. Raw HTML for 200 pages is typically a few MB; 50 MB is comfortable.

Step 1. Map the funnel and capture the raw inputs

What to do: create a workspace and pull three things per competitor: the landing page HTML, robots.txt, and sitemap.xml. Then record the ad-library and social results manually into a notes file.

Why: an LLM can only reason about text you actually give it. The funnel has three layers, and each lives in a different place:

mkdir -p ~/funnel/<competitor> && cd ~/funnel/<competitor>
UA="Mozilla/5.0 (funnel-research; contact: you@example.com)"

curl -sS -A "$UA" -o home.html     -w "%{http_code}\n" "https://<competitor-domain>/"
curl -sS -A "$UA" -o robots.txt    -w "%{http_code}\n" "https://<competitor-domain>/robots.txt"
curl -sS -A "$UA" -o sitemap.xml   -w "%{http_code}\n" "https://<competitor-domain>/sitemap.xml"

grep -c "<loc>" sitemap.xml

Example: the three curl calls print 200, 200, 200, and grep prints a URL count such as 184. Replace <competitor-domain> with the bare host, e.g. example-shop.com — no https://, no trailing slash.

How to check: ls -la shows three non-empty files; grep -c "<loc>" sitemap.xml returns a number greater than 0. If sitemap.xml is empty, look for a Sitemap: line inside robots.txt and fetch that path instead.

Step 2. Normalize pages to JSONL and run the LLM analysis

What to do: extract a fixed set of fields from every saved page into one JSON object per line (JSONL — a text file where each line is a complete JSON record), then send that file to the model with a strict output schema.

Why: models drift when fed raw HTML. A fixed field list makes the output comparable across pages and lets you verify each claim against the source file later.

# extract.py — run inside ~/funnel/<competitor>
import json, glob
from bs4 import BeautifulSoup

def meta(soup, key):
    tag = soup.find("meta", attrs={"name": key}) or soup.find("meta", attrs={"property": key})
    return (tag.get("content") or "").strip() if tag else ""

rows = []
for path in sorted(glob.glob("pages/*.html")):
    soup = BeautifulSoup(open(path, encoding="utf-8", errors="ignore").read(), "html.parser")
    rows.append({
        "source_file": path,
        "title": soup.title.get_text(strip=True) if soup.title else "",
        "description": meta(soup, "description"),
        "og_title": meta(soup, "og:title"),
        "h1": [h.get_text(strip=True) for h in soup.find_all("h1")],
        "h2": [h.get_text(strip=True) for h in soup.find_all("h2")][:20],
        "links": len(soup.find_all("a")),
    })

with open("funnel.jsonl", "w", encoding="utf-8") as f:
    for r in rows:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")
print(f"wrote {len(rows)} records")

Example: python3 extract.py prints wrote 184 records, and head -n 1 funnel.jsonl | jq . shows one object with title, description, h1, h2, links.

Then call the model with a schema that forbids free-form prose:

export LLM_API_KEY="<your-key>"
export LLM_BASE_URL="https://<provider-endpoint>/v1/chat/completions"
export LLM_MODEL="<model-id-from-provider-docs>"

jq -n --rawfile data funnel.jsonl --arg m "$LLM_MODEL" '{
  model: $m,
  temperature: 0,
  messages: [
    {role: "system", content: "You are a funnel analyst. Return only JSON: an array of objects with keys layer, claim, evidence_quote, source_file. layer is one of ads|social|seo. evidence_quote must be copied verbatim from the input. If you cannot quote it, omit the row."},
    {role: "user", content: $data}
  ]
}' > payload.json

curl -sS "$LLM_BASE_URL" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d @payload.json | jq -r '.choices[0].message.content' > funnel-ai.json

The evidence_quote field is what makes this usable: it forces every AI claim to be a substring you can grep. Temperature 0 keeps the same input from producing different clusters on each run.

If you would rather keep the funnel map as a spreadsheet than as JSON, the same JSONL loads directly into a sheet — the workflow in ChatGPT Office: How to Create Documents and Spreadsheets with AI covers the table structure and column formatting.

How to check: jq 'length' funnel-ai.json returns a row count greater than 0, and jq -r '.[].layer' funnel-ai.json | sort | uniq -c shows how many rows landed in ads, social, and seo. A layer with zero rows means you did not feed that layer's text yet.

Step 3. Verify every AI claim and assemble the report

What to do: run three checks — quote validity, layer coverage, and a manual spot check — then write funnel-report.md.

Why: an LLM will occasionally paraphrase instead of quoting, or invent a plausible-sounding keyword that appears nowhere on the site. The grep check catches both.

# 1. Every evidence_quote must exist verbatim in the extracted dataset
jq -r '.[].evidence_quote' funnel-ai.json | while read -r q; do
  [ -z "$q" ] && continue
  grep -qF -- "$q" funnel.jsonl || echo "NOT FOUND: $q"
done

# 2. Layer coverage
jq -r '.[].layer' funnel-ai.json | sort | uniq -c

# 3. Manual spot check: open 3 source_file paths and confirm the claim on screen
jq -r '.[0:3][] | "\(.layer)\t\(.source_file)\t\(.claim)"' funnel-ai.json

Example of a clean run: step 1 prints nothing at all (no NOT FOUND lines), step 2 prints 31 ads, 18 social, 57 seo, and step 3 prints three tab-separated rows whose source_file values you can open locally.

Checklist before you publish the report:

• [ ] Zero NOT FOUND lines from the quote check.
• [ ] All three layers (ads, social, seo) have at least one row.
• [ ] Each ad row's destination URL opens and matches the ad copy you recorded.
• [ ] wc -l funnel.jsonl equals the number of HTML files in pages/.
• [ ] The report states the capture date, because ad libraries rotate creatives.

For a shareable deck instead of a Markdown file, the layout rules in How to Create Presentations with AI: Best Tools and Tips apply directly — one slide per funnel layer, evidence quotes in the speaker notes.

How to check: grep -c "^| " funnel-report.md returns the number of table rows, and opening the file in a viewer shows three sections titled Ads, Social, SEO with a source column in each.

Troubleshooting

Problem. curl cannot reach the domain at all.

• Error: curl: (6) Could not resolve host: competitor-domian.com
• Cause: typo in the placeholder, or the host is behind a private DNS you are not on.
• Solution: run dig +short <competitor-domain> (or nslookup). No output means DNS, not the site. Fix the spelling and retry.

Problem. The page returns a bot-protection block instead of HTML.

• Error: HTTP/2 403 printed by -w "%{http_code}\n", and home.html contains a challenge page rather than your expected <title>.
• Cause: a CDN or WAF (web application firewall) rejected the request because of the user agent or request rate.
• Solution: set an honest, contactable user agent (-A "Mozilla/5.0 (funnel-research; contact: you@example.com)"), add --compressed, slow down to one request per second, and if it still fails, fetch the page manually in a browser and save it into pages/ — do not attempt to bypass the protection.

Problem. The model call returns prose instead of JSON.

• Error: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) when you parse funnel-ai.json.
• Cause: the endpoint returned an error envelope or a refusal, or the response was truncated by the token limit on a large funnel.jsonl.
• Solution: inspect the raw body first with curl -sS -i ... | head -n 20; if the input is too large, split funnel.jsonl into batches of about 40 records with split -l 40 funnel.jsonl batch- and run the same prompt per batch, then merge the arrays.

Summary and next steps

You now have a reproducible pipeline: capture the raw ad, social, and SEO inputs, normalize them to JSONL, and let the model cluster them under a verbatim-quote constraint you can grep. Re-run the same three steps weekly against the same workspace so the diff between funnel-ai.json snapshots shows which offers and keywords the competitor added or dropped.

FAQ

Can I run this without an API key?

Yes — paste funnel.jsonl into any chat interface manually and keep the same schema and evidence_quote rule. You lose batching and automation, not accuracy.

Is scraping a competitor's site allowed?

It depends on robots.txt, the site's terms, and your jurisdiction. Read both first; public ad libraries are usually the safer source for ad creatives.

How do I know the AI did not invent a keyword?

The evidence_quote field must be a verbatim substring of funnel.jsonl. If the grep check prints NOT FOUND, delete that row.

How large can the input be?

It is limited by your model's context window — check your provider's documentation for the current limit and batch accordingly.

🤖 AI summary

• Updated: 2026-10-19
• Topics: анализ рекламы конкурента по ссылке, разбор маркетинговой воронки ии, какие объявления крутит конкурент проверить
• Summary: Практический гайд: какие ИИ-инструменты собрать в цепочку, чтобы по одному URL увидеть платные объявления, органику, контент-план и структуру лендингов конкурента.
• Key facts:
  • TL;DR Time to complete: 60–90 minutes for one competitor domain.
  • Difficulty: Intermediate.
  • You will need: macOS, Linux, or Windows 11 with WSL2 (all commands below are bash); curl and jq ; Python 3 with beautifulsoup4 ; an LLM API key that exposes an OpenAI-compatible /v1/chat/completions endpoint; about 50 MB of free disk space; a text editor.
  • Output: a funnel.jsonl dataset plus a funnel-report.md table that maps each funnel layer (ads → social → SEO) to evidence you can open in a browser.
  • Requirements and setup Illustration: Requirements and setup OS and shell.
  • macOS, Ubuntu, or Windows 11 + WSL2.

Related guides

ChatGPT Office: How to Create Documents and Spreadsheets with AI

Пошаговый гайд по работе с ChatGPT Office: создание документов, таблиц и презентаций с помощью ИИ.

How to Create Videos with AI and Remotion: Automation Guide

Учимся использовать Claude Opus 5.5 и Remotion для автоматического создания программируемых видео: от сценария до готово

How to Automate App Publishing to App Store and Google Play with AI Agents

Настраиваем MCP-сервер shipstores, чтобы ИИ-агент сам собирал, загружал и публиковал ваши iOS и Android приложения в сто