Have open-weight LLMs caught up with closed ones? A production check

⚡ The gist in 5 seconds - The best open-weight models trail closed ones by an average of 4 months on the ECI index, which Epoch AI computes across many benchmarks for the period from January 1 to May 28, 2026. - From April to August 2026, open releases included DeepSeek V4, Kimi K3, and GLM-5.3, while Meta returned to open weights with Muse Glimmer. - Limitation: the aggregate gap says little about your specific use case, and the authors themselves warn the real gap could be up to six months. ### 🔍 What was found Epoch AI computed a composite capability index, ECI, across many benchmarks. For the period from January 1 to May 28, 2026, the best open models trail the closed frontier by roughly 4 months — about 8 index points, roughly the difference between GPT-5 and GPT-5.5. The gap hasn't shrunk but has slightly widened: according to data from 2023 through October 2025, it was about three months. The authors warn that four months may understate the real gap, since open models are tuned to public benchmarks while closed labs keep their best systems in-house. Under a stricter criterion, the gap reaches about six months. Expert Nathan Lambert, after Kimi K3's release, estimated the gap at 3–5 months, but that's an interpretation, not an independent measurement using Epoch's methodology. Specific 2026 releases: DeepSeek V4 — a preview with MIT-licensed weights came out April 24 (Pro at 1.6 trillion parameters, 49 billion active; Flash at 284 billion, 13 billion active; 1M-token context), and on August 13 Pro left preview. Moonshot AI introduced Kimi K3 on July 16 (2.8 trillion parameters, 104 billion active, 1M-token context), with weights published July 27. Meta released Muse Glimmer, a 30-billion-parameter model under Apache 2.0, on August 10; it runs on a single consumer GPU. Zhipu released GLM-5.3 via API on August 14 and published the weights on August 28 after a cybersecurity review. According to OpenRouter, by May Chinese open-weight models accounted for about 61% of tokens consumed (that's one router's statistics, not the whole market). ### 💡 Why it matters For an engineer, this is a signal: on some tasks open models are already indistinguishable from closed ones, and you can deploy weights inside your perimeter, saving on API costs. The author gives an example: when classifying disputed-payment tickets on an anonymized set of a few hundred examples, the difference from an expensive closed model stayed within the variance between repeated runs. But the same model, when generating explanations for antifraud analysts, confidently cited internal rules that weren't in the context. The average ECI gap says nothing about your task: a model can be great at field extraction and noticeably worse on long agentic chains. The final decision depends on your data and latency requirements. ### 🧩 Context In the spring, Meta's flagship Muse Spark became available only in Meta products and in a closed API preview for select partners, without open weights, and for the past two years the pace in open models has been set by Chinese labs

🔗 Read on habr.com