← All guides AI Release
RU Subscribe to @ai_release1

How we killed duplicate posts in a Telegram channel: idempotent sends and rate limits

Updated: 30.10.2026 · AI Release · @ai_release1
How we killed duplicate posts in a Telegram channel: idempotent sends and rate limits

TL;DR

Requirements and setup

Illustration: Requirements and setup
Illustration: Requirements and setup

The three protection layers you are about to build look like this:

publisher ──► [1] idempotency key ──► [2] rate limiter ──► [3] Bot API sendMessage
                     │                       │                       │
                SQLite outbox           token bucket            message_id
                (UNIQUE key)          (per-minute cap)              │
                     └──────────── skip on duplicate ◄─────────────┘
                                                                  │
                                                       https://t.me/<username>/<id>
LayerWhat it stopsWhere it lives
Idempotency keyThe same text being posted twice after a retry, restart, or double-clicked admin buttonSQLite table `outbox`, `UNIQUE` primary key
Rate limiter`429 Too Many Requests` during broadcast loopsIn-process token bucket, one worker
`t.me` permalink checkSilent misconfiguration: wrong channel, deleted post, wrong username`https://t.me/<username>/<message_id>`

Example: a bot with token <token-from-BotFather> posting to -1001234567890 with username my_test_channel produces the permalink https://t.me/my_test_channel/42.

For the bot-creation and deployment part itself, follow An AI-powered Telegram bot in Python: from scratch to deploy — this guide starts where that one ends.

What to do: create the project skeleton and export the environment variables.

mkdir -p ~/tg-dedup && cd ~/tg-dedup
python3 -m venv .venv && source .venv/bin/activate
pip install requests

export TG_BOT_TOKEN="<token-from-BotFather>"
export TG_CHANNEL="-1001234567890"
export TG_CHANNEL_USER="my_test_channel"

Why: the key must be derived from the message content and the target channel, so the script needs both values at runtime; keeping the token in the environment prevents it from leaking into git.

How to check: echo "${TG_BOT_TOKEN:0:10}" prints the first ten characters of the token, and python3 -c "import requests; print(requests.__version__)" prints a version number.

Step 1. Build the idempotency key and the dedup store

An idempotency key is a fingerprint of an operation: run the same operation twice, get the same fingerprint, and the second run is discarded instead of executed. Here the fingerprint is a SHA-256 hash of the channel id plus the normalized text.

What to do: create dedup.py with the key function and the SQLite schema.

# dedup.py
import hashlib, sqlite3

SCHEMA = """
CREATE TABLE IF NOT EXISTS outbox (
    idem_key   TEXT PRIMARY KEY,
    channel_id TEXT NOT NULL,
    text       TEXT NOT NULL,
    status     TEXT NOT NULL DEFAULT 'pending',  -- pending | sent | failed
    message_id INTEGER,
    attempts   INTEGER NOT NULL DEFAULT 0,
    created_at TEXT NOT NULL DEFAULT (datetime('now')),
    sent_at    TEXT
);
"""

def idempotency_key(channel_id: str, text: str) -> str:
    normalized = " ".join(text.split()).strip().lower()
    return hashlib.sha256(f"{channel_id}:{normalized}".encode()).hexdigest()[:32]

def init_db(path="outbox.db"):
    con = sqlite3.connect(path)
    con.executescript(SCHEMA)
    con.commit()
    con.close()

if __name__ == "__main__":
    init_db()

TEXT PRIMARY KEY on idem_key makes the database itself the arbiter: the second insert of the same key fails or is ignored, so two processes cannot both send.

Why: normalization (collapsing whitespace, lowercasing) means "Hello World" and "hello world" map to one key — this is what kills the classic duplicate caused by a re-rendered template.

How to check:

python3 dedup.py
python3 -c "
from dedup import idempotency_key
print(idempotency_key('-1001234567890', 'Hello  World'))
print(idempotency_key('-1001234567890', 'hello world'))
"

Expected output: two identical 32-character hex lines. If they differ, your normalization is broken — fix it before continuing. ls -l outbox.db should show the database file.

Example: sqlite3 outbox.db ".tables" prints outbox.

Step 2. Send through the Bot API with a rate limiter and a permalink check

Illustration: Step 2. Send through the Bot API with a rate limiter and a permalink check
Illustration: Step 2. Send through the Bot API with a rate limiter and a permalink check

The Bot API has no idempotency parameter for sendMessage — the deduplication must happen on your side (confirm the current parameter list in the official docs). The rate limiter is a token bucket: it records timestamps of recent sends and sleeps when the per-minute cap is reached. The Bot API documentation states a global cap of roughly 30 messages per second and about 20 messages per minute to one group or channel; re-check those numbers in the docs, because Telegram adjusts them.

What to do: create publisher.py.

# publisher.py
import os, sqlite3, time, sys, requests
from dedup import idempotency_key, init_db

BOT_TOKEN = os.environ["TG_BOT_TOKEN"]
CHANNEL   = os.environ["TG_CHANNEL"]
USERNAME  = os.environ["TG_CHANNEL_USER"]
API       = f"https://api.telegram.org/bot{BOT_TOKEN}"
DB        = "outbox.db"

class RateLimiter:
    def __init__(self, per_minute=20):
        self.per_minute, self.stamps = per_minute, []

    def wait(self):
        now = time.monotonic()
        self.stamps = [t for t in self.stamps if now - t < 60]
        if len(self.stamps) >= self.per_minute:
            time.sleep(60 - (now - self.stamps[0]) + 0.5)
        self.stamps.append(time.monotonic())

def claim(key, text):
    con = sqlite3.connect(DB)
    cur = con.execute(
        "INSERT INTO outbox (idem_key, channel_id, text) VALUES (?,?,?) "
        "ON CONFLICT(idem_key) DO NOTHING", (key, CHANNEL, text))
    con.commit()
    claimed = cur.rowcount == 1
    con.close()
    return claimed

def publish(text, limiter):
    key = idempotency_key(CHANNEL, text)
    if not claim(key, text):
        print(f"skip: duplicate idem_key={key}")
        return
    for _ in range(3):
        limiter.wait()
        r = requests.post(f"{API}/sendMessage",
                          json={"chat_id": CHANNEL, "text": text,
                                "disable_web_page_preview": True}, timeout=15)
        if r.status_code != 429:
            break
        wait = r.json().get("parameters", {}).get("retry_after", 30)
        print(f"429: sleeping {wait}s")
        time.sleep(wait)
    body = r.json()
    if not body.get("ok"):
        print(f"failed: {body.get('description')}")
        return
    message_id = body["result"]["message_id"]
    con = sqlite3.connect(DB)
    con.execute("UPDATE outbox SET status='sent', message_id=?, sent_at=datetime('now') "
                "WHERE idem_key=?", (message_id, key))
    con.commit(); con.close()
    print(f"sent: message_id={message_id} https://t.me/{USERNAME}/{message_id}")

if __name__ == "__main__":
    init_db()
    publish(" ".join(sys.argv[1:]), RateLimiter(per_minute=20))

Why: the row is inserted *before* the network call. If the process crashes between insert and send, the row stays pending and is visible in the database — you never get an invisible duplicate, only an explicit retry decision. Honoring retry_after instead of a fixed sleep is what prevents a second 429 immediately after the first.

How to check: run the script once. Expected output:

sent: message_id=42 https://t.me/my_test_channel/42

Open the permalink — the post must be there. Then sqlite3 outbox.db "SELECT status, message_id FROM outbox;" shows sent|42.

Example: for a broadcast of 50 texts with per_minute=20, the limiter inserts three sleep windows and the script finishes in roughly 2 minutes instead of being throttled by the API.

Step 3. Verify that a duplicate run changes nothing

Illustration: Step 3. Verify that a duplicate run changes nothing
Illustration: Step 3. Verify that a duplicate run changes nothing

What to do: run the same publish command twice and inspect all three layers.

python3 publisher.py "Idempotency smoke test"
python3 publisher.py "Idempotency smoke test"

Expected output — one send, one skip:

sent: message_id=42 https://t.me/my_test_channel/42
skip: duplicate idem_key=<32-hex-string>

Then verify the store and the public permalink:

sqlite3 outbox.db "SELECT status, COUNT(*) FROM outbox GROUP BY status;"
curl -s -o /dev/null -w "%{http_code}\n" https://t.me/my_test_channel/42

Expected: sent|1 from the database and 200 from curl. A 404 means the username or the message id is wrong, not that the post is missing from the channel.

Why: this proves the three layers agree — the database has exactly one accepted row, the channel has exactly one message, and the message is reachable at its canonical URL.

How to check (pre-flight list before switching to production):

Example: ps aux | grep publisher.py | grep -v grep must show exactly one line. Two processes bypass the in-process limiter and double the send rate.

Troubleshooting

Problem. The broadcast dies partway with a rate-limit error

Problem. Two identical posts appear in the channel

Problem. Editing a post repeatedly returns an error

Summary and next steps

You now have a publisher where the database, not the network, decides whether a message is new, and where every send leaves a permalink you can verify in one curl. To extend it into a scheduled content pipeline with retries, queues and monitoring, continue with AI agents for task automation: a guide.

FAQ

Does the Telegram Bot API have a built-in idempotency key for sendMessage?

No idempotency parameter is documented for sendMessage; deduplication must be implemented on the client side, as shown in Step 1. Check the current parameter list in the official Bot API docs before relying on any field.

Why check t.me if the database already says sent?

The database proves what you intended to send, while the permalink proves the message exists in the channel you think it does. A wrong TG_CHANNEL id produces a clean database and an empty channel.

Can I use the t.me check for a private channel?

Public previews exist only for channels with a public username. For a private channel, verify with your own message_id record, or mirror the post into a private test channel that has a username.

What rate limit should I set in production?

Start at per_minute=20 for a single channel and always honor retry_after from a 429 response. The exact global and per-chat limits are published in the Bot API documentation — confirm them there, as they change.

🤖 AI summary

Related guides

AI Assistants for Learning English: Best Bots and Tools in Telegram

Обзор эффективных способов учить английский с помощью ИИ прямо в Telegram: боты, диалоговые тренажёры, генерация упражне

An AI-powered Telegram bot in Python: from scratch to deploy

Пишем Telegram-бота с подключением LLM на Python: получение токена, обработка сообщений, память диалога и запуск на серв

Best AI models and neural networks in 2026

Обзор главных ИИ-моделей года — GPT, Claude, Gemini, DeepSeek и другие: возможности, тесты и ссылки на наши обзоры.