QA Engineer Ran 6500 E2E Tests Through an AI Agent in OpenCode: What Came of It

⚡ The Gist in 5 Seconds

• A manual QA engineer from TravelLine hooked up an AI agent in OpenCode and audited roughly 6500 E2E tests ahead of a migration from Java to.NET.
• The agent read local repositories of the product and automated tests, pulled data from TestIT via an MCP server; the model was GPT-5.5.
• The entry bar is strict: without full context and a polished instruction, the agent confuses suites and sections and returns an incomplete list of tests.

🔍 What Happened

TravelLine tester Ilsur Mingazov published a case study on Habr: a team with a product history of over 15 years had accumulated about 6500 E2E tests. The trigger for the audit was a planned migration of automated tests from Java to.NET. Simply rewriting everything "as is" means dragging the accumulated tech debt onto the new stack as well. So before the migration, the tests were sorted into four buckets: migrate, rewrite, push down to the integration/API level, delete as duplicates.

Here's the mechanics. The product repository and the automated test project were cloned locally, and OpenCode navigated the folders itself, matching method calls to classes. For TestIT, they first tried direct API requests, then moved everything to an MCP server — the agent called ready-made tools, and the server hit the API under the hood. Lightweight Flash models failed at mapping the architecture, so the choice fell on GPT-5.5: it held a long system prompt with test pyramid rules and a strict markdown report format. The very first section exposed a problem: the agent was navigating by suites rather than sections and returned an incomplete list. They had to send it a screenshot of the tree, explain the section-subsection division, and lock the workflow into the instructions.

🧠 Our Take

It's curious who the real protagonist of this story is. Not the model, not MCP, but the manual QA engineer who suddenly became the agent's context provider. Our assessment: test auditing is exactly the class of tasks where AI doesn't write code but reads someone else's and sorts it into piles. Copy-pasting methods into a chat is pointless, which is why local repositories and MCP were needed. It's precisely the combination of "access to source code + access to TMS via a proper protocol" that turns a chatbot into an auditor.

What this changes in practice. Usually a stack migration means rewriting blindly: you port everything, because no one will give you the money or time to manually review 6500 scenarios. Here a third option emerged — a cheap analysis first, then the port. For the wallet, that means a shift in costs: fewer man-hours on reading, more on setting up the agent and verifying its output. From experience, nobody counts for the second item, yet it's what eats the savings.

Who wins — teams with a live API on their TMS and repositories that can be cloned. Who loses — those whose tests live in spreadsheets and wikis, as well as proponents of the "feed it a list of scenario names and wait for a miracle" approach. We believe the article's value lies not in the model but in the honest account of debugging the agent: half the text is about how it made mistakes and how it was corrected.

⚠️ What We Don't Know

The text contains not a single effectiveness figure: how many tests actually ended up in the "delete" bucket, how many hours were saved, what the tokens and subscription cost. The author rounded the size of the test base, anonymized internal entities, and we take the model name GPT-5.5 as given from the article without verifying it separately. The weak point is "one team, one case": reproducibility depends on how similar your TestIT sections and repository structure are.

Prediction: within a quarter, several more public cases will appear where an MCP server for a TMS (TestIT, Allure, Zephyr) is part of the standard audit pipeline. If they don't appear, it means the barrier isn't in the models but in context setup.

Disclaimer: a single source — one team's case study; the author rounded the figures and anonymized internal entities. Not officially confirmed.

Source: Habr