AGI as a Cognitive OS: What If We Search for Programs Instead of Weights

⚡ The gist in 5 seconds - Researchers propose looking for learned programs — cognitive algorithms — in neural network weights, rather than individual parameters. - The hypothesis paper was published on Habr by author Minebot. - It's still a hypothesis: there is no universal "neural network decompiler," and the discovered structures require further interpretation. ### 🔍 What was found In Minebot's article, published on Habr on September 25, an analogy is drawn between an LLM and a binary without source code: the model brilliantly writes code, translates, and jokes, but there is no documentation or code — only hundreds of gigabytes of weights. The authors treat behavior as a function with memory: (input, state) → (action, new state). Mathematically, a neural network and a program are two representations of the same computation: in one case the structure is smeared across the weights, in the other it is written explicitly. Key examples: the Rule 110 cellular automaton — 8 bits of rules and a loop yield a Turing-complete system (M. Cook, 2004). In the work of Nanda et al. (2023), a one-layer transformer was trained on modular addition mod 113 — after a memorization phase, it began to generalize via an algorithm using sines and cosines. In doing so, hundreds of thousands of the model's parameters are described by a pair of formulas. The authors also recall the limitations of the universal approximation theorem: a finite feed-forward network cannot replace a program with infinite memory, but with bounded input and time, the behavior can be reproduced. ### 💡 Why it matters If the hypothesis is true, then strong AI should not only be trained but also searched for — as a program with readable code, explicit memory, and understandable modules. Today, understanding modern models works like an experimental science: observation, hypothesis, intervention, measuring the effect. This is slower and less reliable than reverse engineering. Extracting programs could shift AI debugging and improvement from guessing based on results to explicit algorithms — from distillation and pruning to program synthesis and circuit analysis, which are already used for compression and interpretation. ### 🧩 Context The article is based on the observation that an LLM is "compiled intelligence," and we don't know how to read its source code. A neural network and a program are not opposites, but two ways of representing a computation. Complexity, meanwhile, is determined not only by the number of parameters but also by recurrence and memory: parameters ≠ complexity. A small program applied repeatedly can generate arbitrarily complex behavior. The theoretical limit of compression is set by the Kolmogorov complexity of the function: if a pattern is structural, the network may be orders of magnitude larger than its minimal description. The question of whether huge networks use billions of parameters merely as a search space, while the discovered pattern is in fact compact, remains open. The authors list existing tools for compression and mechanistic interpretability, but admit that a universal neural network decompiler does not yet exist.

🔗 Read on habr.com