RU

GPT-6.1-Sol and GPT-6-Astra Are Obsessed With Appendices: AI Reviewers Are Ruining Peer Review

Published: 2026-11-21 · Author: AI Release · @ai_releaseorg
GPT-6.1-Sol and GPT-6-Astra Are Obsessed With Appendices: AI Reviewers Are Ruining Peer Review

⚡ The gist

🔍 What happened

A team of researchers ran 2,000 papers from four popular conferences through OpenAI's new models and compared their comments with what human reviewers actually write. The gap turned out to be huge. For GPT-5, less than 30% of the criticism targeted appendices; for GPT-6.1-Sol and GPT-6-Astra, it's roughly 50–70% — and the effect holds across all four conferences, so it's not venue-specific.

The paper's authors note: "The difference between generations is large. For GPT-5, only a small share of weaknesses cited relate to the appendix. Both new models, GPT-6.1-Sol and GPT-6-Astra, lean on it far more heavily." Human reviews show the opposite picture — 8.4% of comments about appendices versus 62% for the models. The AI was more than seven times as likely to flag problems in supplementary materials, and three times as likely to criticize the appendix alone without a single complaint about the main text.

One more detail: how exactly the model reads the PDF. The API, Codex, and the web interface parse documents differently — Codex doesn't even read the middle of a medium-sized PDF, retrieving it in "chunks" later if needed. But all access methods loved the appendix.

🧠 Our take

The cause, as we see it, is simple and unpleasant: context windows have grown, and models now genuinely chew through the entire PDF. But "reading everything" ≠ "understanding what matters." As the paper notes, the model barely criticizes the first pages, where authors frame the problem — while the appendix gets dissected line by line. Appendices are often raw, unpolished, full of tables and logs: it's easy to find a formal error there and nearly impossible to find a scientific one.

What this changes depends on whose problem it is. If AI reviewers are used as a draft for a human, the damage is minor. If they're used as a replacement, the conference gets a stream of nitpicks about a hyperparameter table instead of a conversation about the method. And the most alarming part of the paper: the share of appendix comments in live human reviews rose to 27.6% after these models launched. The authors flag this as a correlation, not a proven cause — and we wouldn't rush to conclusions either.

The winners, in our assessment, aren't OpenAI but the people selling "review checking" on top of these models: the demand for a filter is obvious. The losers are paper authors and program committees: time gets burned on formal complaints. And there's another question: why did models with stronger reading skills get stuck on footnotes? It doesn't explain any quality improvement, and the paper says it outright: "The ability to read a paper in detail did not teach them which parts matter more."

⚠️ What we don't know

We're working from a summary of the research, not the paper itself: we don't have a link to the primary source, so there's no way to verify the sample, the counting methodology, or how the criticism was normalized. The 27.6% figure is a correlation with the models' release, not proof that humans are copying AI habits. There's also no estimate of how many of these comments were actually valid — maybe some appendices really are that bad.

💬 Worth debating

Should AI reviews be allowed into program committees at all without section-level restrictions, or should a model's review be limited to the main text?
If human reviewers are already picking up the models' habits, what needs fixing — the tool or the review process?
Should 50–70% of criticism landing in appendices be treated as a model bug, or as an honest signal that authors are hiding weaknesses in supplementary materials?

Forecast: within two quarters, at least one public benchmark will appear that evaluates AI review quality separately for the main text and separately for appendices.

Disclaimer: single source, summary of a study — not officially confirmed.

Source: The New York Times

🤖 AI summary
  • Date: 2026-11-21
  • Source: www.nytimes.com
  • Tags: OpenAI, Pir-revyu, GPT-6, Nauchnye, Benchmarks, AI, LLM, neural networks
  • Key facts:
    • OpenAI's new models — GPT-6.1-Sol and GPT-6-Astra — pull 50–70% of their review comments from paper appendices rather than the main text.
    • For humans, appendices account for 8.4% of complaints; for AI, it's 62% — a gap of more than seven times.
    • Side effect: the share of appendix nitpicks in live human reviews has jumped from under 11% to 27.6%.
    • A team of researchers ran 2,000 papers from four popular conferences through OpenAI's new models and compared their comments with what human reviewers actually write.
    • The gap turned out to be huge.
    • For GPT-5, less than 30% of the criticism targeted appendices; for GPT-6.1-Sol and GPT-6-Astra, it's roughly 50–70% — and the effect holds across all four conferences, so it's not venue-specific.
    • The paper's authors note: "The difference between generations is large.
    • For GPT-5, only a small share of weaknesses cited relate to the appendix.
OpenAIPir-revyuGPT-6NauchnyeBenchmarksAILLMneural networks
← PreviousMotorola expands its Android 17 phone list: Razr 50 and Edge 50 join

Source: www.nytimes.com · post in Telegram