# How we killed duplicate posts in a Telegram channel: idempotent sends and rate limits

> AI Release · @ai_release1 · https://ai-release.org/guides/kak-my-ubili-dubli-v-telegram-kanale_en.html

## TL;DR

- **Time to complete:** 45 minutes
- **Difficulty:** Intermediate
- **You will need:** Python 3.9+ with `requests` installed, the `sqlite3` module (ships with Python), a bot token from @BotFather, a test channel where that bot is an administrator with the "Post messages" permission, `curl`, and the Telegram Bot API documentation open in a browser tab (Bot API limits change — check the current numbers there before you set production values).

## Requirements and setup
![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-my-ubili-dubli-v-telegram-kanale-en-1.jpg?v=1793343667)


- Operating system: Linux or macOS with `bash`, or Windows 10/11 with WSL2. All commands below are `bash`.
- Python 3.9 or newer. Check with `python3 --version`.
- A virtual environment in the project folder: `python3 -m venv .venv && source .venv/bin/activate`.
- The `requests` package: `pip install requests`.
- The `sqlite3` CLI for manual inspection (optional but used in verification below). On Debian/Ubuntu: `sudo apt install sqlite3`.
- A bot token stored in an environment variable, never in code.
- A channel: create a private test channel, add your bot as administrator, and note both its numeric id (`-100…`) and its public username if it has one.

The three protection layers you are about to build look like this:

```
publisher ──► [1] idempotency key ──► [2] rate limiter ──► [3] Bot API sendMessage
                     │                       │                       │
                SQLite outbox           token bucket            message_id
                (UNIQUE key)          (per-minute cap)              │
                     └──────────── skip on duplicate ◄─────────────┘
                                                                  │
                                                       https://t.me/<username>/<id>
```

| Layer | What it stops | Where it lives |
|---|---|---|
| Idempotency key | The same text being posted twice after a retry, restart, or double-clicked admin button | SQLite table `outbox`, `UNIQUE` primary key |
| Rate limiter | `429 Too Many Requests` during broadcast loops | In-process token bucket, one worker |
| `t.me` permalink check | Silent misconfiguration: wrong channel, deleted post, wrong username | `https://t.me/<username>/<message_id>` |

**Example:** a bot with token `<token-from-BotFather>` posting to `-1001234567890` with username `my_test_channel` produces the permalink `https://t.me/my_test_channel/42`.

For the bot-creation and deployment part itself, follow [An AI-powered Telegram bot in Python: from scratch to deploy](https://ai-release.org/guides/telegram-bot-s-ii-na-python.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) — this guide starts where that one ends.

**What to do:** create the project skeleton and export the environment variables.

```bash
mkdir -p ~/tg-dedup && cd ~/tg-dedup
python3 -m venv .venv && source .venv/bin/activate
pip install requests

export TG_BOT_TOKEN="<token-from-BotFather>"
export TG_CHANNEL="-1001234567890"
export TG_CHANNEL_USER="my_test_channel"
```

**Why:** the key must be derived from the message content and the target channel, so the script needs both values at runtime; keeping the token in the environment prevents it from leaking into git.

**How to check:** `echo "${TG_BOT_TOKEN:0:10}"` prints the first ten characters of the token, and `python3 -c "import requests; print(requests.__version__)"` prints a version number.

## Step 1. Build the idempotency key and the dedup store

An idempotency key is a fingerprint of an operation: run the same operation twice, get the same fingerprint, and the second run is discarded instead of executed. Here the fingerprint is a SHA-256 hash of the channel id plus the normalized text.

**What to do:** create `dedup.py` with the key function and the SQLite schema.

```python
# dedup.py
import hashlib, sqlite3

SCHEMA = """
CREATE TABLE IF NOT EXISTS outbox (
    idem_key   TEXT PRIMARY KEY,
    channel_id TEXT NOT NULL,
    text       TEXT NOT NULL,
    status     TEXT NOT NULL DEFAULT 'pending',  -- pending | sent | failed
    message_id INTEGER,
    attempts   INTEGER NOT NULL DEFAULT 0,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    sent_at    TEXT
);
"""

def idempotency_key(channel_id: str, text: str) -> str:
    normalized = " ".join(text.split()).strip().lower()
    return hashlib.sha256(f"{channel_id}:{normalized}".encode()).hexdigest()[:32]

def init_db(path="outbox.db"):
    con = sqlite3.connect(path)
    con.executescript(SCHEMA)
    con.commit()
    con.close()

if __name__ == "__main__":
    init_db()
```

`TEXT PRIMARY KEY` on `idem_key` makes the database itself the arbiter: the second insert of the same key fails or is ignored, so two processes cannot both send.

**Why:** normalization (collapsing whitespace, lowercasing) means `"Hello  World"` and `"hello world"` map to one key — this is what kills the classic duplicate caused by a re-rendered template.

**How to check:**

```bash
python3 dedup.py
python3 -c "
from dedup import idempotency_key
print(idempotency_key('-1001234567890', 'Hello  World'))
print(idempotency_key('-1001234567890', 'hello world'))
"
```

Expected output: two identical 32-character hex lines. If they differ, your normalization is broken — fix it before continuing. `ls -l outbox.db` should show the database file.

**Example:** `sqlite3 outbox.db ".tables"` prints `outbox`.

## Step 2. Send through the Bot API with a rate limiter and a permalink check
![Illustration: Step 2. Send through the Bot API with a rate limiter and a permalink check](https://ai-release.org/guides/img/kak-my-ubili-dubli-v-telegram-kanale-en-2.jpg?v=1793343674)


The Bot API has no idempotency parameter for `sendMessage` — the deduplication must happen on your side (confirm the current parameter list in the official docs). The rate limiter is a token bucket: it records timestamps of recent sends and sleeps when the per-minute cap is reached. The Bot API documentation states a global cap of roughly 30 messages per second and about 20 messages per minute to one group or channel; re-check those numbers in the docs, because Telegram adjusts them.

**What to do:** create `publisher.py`.

```python
# publisher.py
import os, sqlite3, time, sys, requests
from dedup import idempotency_key, init_db

BOT_TOKEN = os.environ["TG_BOT_TOKEN"]
CHANNEL   = os.environ["TG_CHANNEL"]
USERNAME  = os.environ["TG_CHANNEL_USER"]
API       = f"https://api.telegram.org/bot{BOT_TOKEN}"
DB        = "outbox.db"

class RateLimiter:
    def __init__(self, per_minute=20):
        self.per_minute, self.stamps = per_minute, []

    def wait(self):
        now = time.monotonic()
        self.stamps = [t for t in self.stamps if now - t < 60]
        if len(self.stamps) >= self.per_minute:
            time.sleep(60 - (now - self.stamps[0]) + 0.5)
        self.stamps.append(time.monotonic())

def claim(key, text):
    con = sqlite3.connect(DB)
    cur = con.execute(
        "INSERT INTO outbox (idem_key, channel_id, text) VALUES (?,?,?) "
        "ON CONFLICT(idem_key) DO NOTHING", (key, CHANNEL, text))
    con.commit()
    claimed = cur.rowcount == 1
    con.close()
    return claimed

def publish(text, limiter):
    key = idempotency_key(CHANNEL, text)
    if not claim(key, text):
        print(f"skip: duplicate idem_key={key}")
        return
    for _ in range(3):
        limiter.wait()
        r = requests.post(f"{API}/sendMessage",
                          json={"chat_id": CHANNEL, "text": text,
                                "disable_web_page_preview": True}, timeout=15)
        if r.status_code != 429:
            break
        wait = r.json().get("parameters", {}).get("retry_after", 30)
        print(f"429: sleeping {wait}s")
        time.sleep(wait)
    body = r.json()
    if not body.get("ok"):
        print(f"failed: {body.get('description')}")
        return
    message_id = body["result"]["message_id"]
    con = sqlite3.connect(DB)
    con.execute("UPDATE outbox SET status='sent', message_id=?, sent_at=datetime('now') "
                "WHERE idem_key=?", (message_id, key))
    con.commit(); con.close()
    print(f"sent: message_id={message_id} https://t.me/{USERNAME}/{message_id}")

if __name__ == "__main__":
    init_db()
    publish(" ".join(sys.argv[1:]), RateLimiter(per_minute=20))
```

**Why:** the row is inserted *before* the network call. If the process crashes between insert and send, the row stays `pending` and is visible in the database — you never get an invisible duplicate, only an explicit retry decision. Honoring `retry_after` instead of a fixed sleep is what prevents a second `429` immediately after the first.

**How to check:** run the script once. Expected output:

```
sent: message_id=42 https://t.me/my_test_channel/42
```

Open the permalink — the post must be there. Then `sqlite3 outbox.db "SELECT status, message_id FROM outbox;"` shows `sent|42`.

**Example:** for a broadcast of 50 texts with `per_minute=20`, the limiter inserts three sleep windows and the script finishes in roughly 2 minutes instead of being throttled by the API.

## Step 3. Verify that a duplicate run changes nothing
![Illustration: Step 3. Verify that a duplicate run changes nothing](https://ai-release.org/guides/img/kak-my-ubili-dubli-v-telegram-kanale-en-3.jpg?v=1793343682)


**What to do:** run the same publish command twice and inspect all three layers.

```bash
python3 publisher.py "Idempotency smoke test"
python3 publisher.py "Idempotency smoke test"
```

Expected output — one send, one skip:

```
sent: message_id=42 https://t.me/my_test_channel/42
skip: duplicate idem_key=<32-hex-string>
```

Then verify the store and the public permalink:

```bash
sqlite3 outbox.db "SELECT status, COUNT(*) FROM outbox GROUP BY status;"
curl -s -o /dev/null -w "%{http_code}\n" https://t.me/my_test_channel/42
```

Expected: `sent|1` from the database and `200` from `curl`. A `404` means the username or the message id is wrong, not that the post is missing from the channel.

**Why:** this proves the three layers agree — the database has exactly one accepted row, the channel has exactly one message, and the message is reachable at its canonical URL.

**How to check (pre-flight list before switching to production):**

- [ ] Running the same text twice produces one `sent:` and one `skip:` line
- [ ] `SELECT COUNT(*) FROM outbox GROUP BY idem_key HAVING COUNT(*)>1;` returns no rows
- [ ] The permalink returns HTTP `200`
- [ ] `429` appears at most once in the log and is followed by the correct sleep
- [ ] Only one publisher process runs (no cron overlap, no duplicated worker)

**Example:** `ps aux | grep publisher.py | grep -v grep` must show exactly one line. Two processes bypass the in-process limiter and double the send rate.

## Troubleshooting

**Problem. The broadcast dies partway with a rate-limit error**

- Ошибка: `{"ok":false,"error_code":429,"description":"Too Many Requests: retry after 35","parameters":{"retry_after":35}}`
- Причина: the loop ignores the per-minute cap, or several workers share the same bot token.
- Решение: set `per_minute` below the documented cap, honor `retry_after` from the response body, and keep a single publisher process. In the code above `time.sleep(wait)` does this; verify with a log line `429: sleeping 35s`. Wrap the worker in `flock -n /tmp/tg.lock python3 publisher.py "text"` so cron cannot overlap.

**Problem. Two identical posts appear in the channel**

- Ошибка: `sqlite3.IntegrityError: UNIQUE constraint failed: outbox.idem_key`
- Причина: the text was sent by a path that does not use `claim()`, or someone ran a manual `INSERT` and then a `sendMessage` anyway. The exception is the database correctly refusing the duplicate — the bug is that the sender ignored it and posted regardless.
- Решение: route every send through `claim()` and use `ON CONFLICT(idem_key) DO NOTHING` so the conflict becomes a `skip:` line, not a crash. Detect historical damage with `sqlite3 outbox.db "SELECT idem_key, COUNT(*) c FROM outbox GROUP BY idem_key HAVING c>1;"` — this must return no rows.

**Problem. Editing a post repeatedly returns an error**

- Ошибка: `400 Bad Request: message is not modified`
- Причина: `editMessageText` was called with text identical to the current post — typical for a polling loop that re-renders a counter or a timestamp.
- Решение: hash the text before editing and skip when unchanged. `python3 -c "import hashlib;print(hashlib.sha256('same'.encode()).hexdigest()[:8])"` gives you the comparison value; store it next to `message_id` and compare before calling the API.

## Summary and next steps

You now have a publisher where the database, not the network, decides whether a message is new, and where every send leaves a permalink you can verify in one `curl`. To extend it into a scheduled content pipeline with retries, queues and monitoring, continue with [AI agents for task automation: a guide](https://ai-release.org/guides/ii-agenty-dlya-avtomatizacii.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## FAQ

**Does the Telegram Bot API have a built-in idempotency key for sendMessage?**
No idempotency parameter is documented for `sendMessage`; deduplication must be implemented on the client side, as shown in Step 1. Check the current parameter list in the official Bot API docs before relying on any field.

**Why check `t.me` if the database already says `sent`?**
The database proves what you intended to send, while the permalink proves the message exists in the channel you think it does. A wrong `TG_CHANNEL` id produces a clean database and an empty channel.

**Can I use the `t.me` check for a private channel?**
Public previews exist only for channels with a public username. For a private channel, verify with your own `message_id` record, or mirror the post into a private test channel that has a username.

**What rate limit should I set in production?**
Start at `per_minute=20` for a single channel and always honor `retry_after` from a `429` response. The exact global and per-chat limits are published in the Bot API documentation — confirm them there, as they change.