Inference in Large Language Models: How Text Becomes Numbers

⚡ The Gist in 5 Seconds - Inference in LLMs is the conversion of a prompt into numbers, processing through transformer layers, and autoregressive generation of the response token by token. - Where available: a translation of an article by Arpit Bhayani was published in the Piter blog on Habr on September 28, 2024; reading time is 14 minutes. - Limitation: the more tokens in the text, the more computations, API overhead, and the risk of hitting the context window limit. ### 🔍 What Was Found The translated article by Arpit Bhayani for the Piter blog (Habr, September 28, 2024) explains that models like GPT-4, Claude, and Llama are decoder-only transformers. This means they are autoregressive: at each pass, they generate one new token based on all previously generated ones. A model's size is determined by its number of parameters: for example, a model with 7 billion parameters contains 7 billion floating-point numbers. The author provides a detailed breakdown of tokenization using the Byte Pair Encoding (BPE) method. Text is converted into UTF-8 bytes, then frequent pairs of characters are merged into tokens. For the word "unhappiness," the process looks like this: from individual characters to the tokens ['un', 'happi', 'ness']. The example with the phrase "The AI model generates text" yields the identifiers [464, 15592, 2746, 18616, 2420]. Vector representations of tokens are built via an embedding matrix: with a vocabulary of 50,000 tokens and a dimensionality of 4096, the matrix has the shape [50000, 4096]. Positional encoding, such as RoPE, is added to the vectors. ### 💡 Why It Matters Understanding inference matters for anyone working with LLM APIs. Tokenization directly affects performance and costs: the more tokens required to process a text, the more computations and the higher the cost of requests. If a tokenizer was trained primarily on English, non-English text often requires more tokens, which increases expenses and can bring you closer to the context window limit. Knowing these mechanics helps better estimate costs and limitations when working with models. ### 🧩 Context The transformer architecture replaced models that processed text sequentially. Its key advantage is parallel analysis of entire sequences, which speeds up training and deployment. The basic unit is a transformer layer, consisting of a self-attention mechanism and a feed-forward network; a large model contains dozens of such layers. The self-attention mechanism computes queries (Q), keys (K), and values (V) for each token by multiplying the input vector representations by learned weight matrices. These matrices are initially random and are adjusted during backpropagation to extract useful patterns from the data.

🔗 Read on habr.com