← All guides AI Release
RU Subscribe to @ai_release1

How to Set Up an AI E2E Tester for Your Project: Automated Tests for Web and Mobile Apps

Updated: 10.11.2026 · AI Release · @ai_release1
How to Set Up an AI E2E Tester for Your Project: Automated Tests for Web and Mobile Apps

TL;DR

Requirements and setup

Illustration: Requirements and setup
Illustration: Requirements and setup

Pre-flight checklist:

Step 1. Freeze a green baseline before the AI touches anything

Illustration: Step 1. Freeze a green baseline before the AI touches anything
Illustration: Step 1. Freeze a green baseline before the AI touches anything

What to do: create an e2e project, install the web runner and one mobile driver, write one smoke test by hand, and run it.

mkdir -p e2e && cd e2e
npm init -y
npm i -D @playwright/test appium
npx playwright install --with-deps
npx appium driver install uiautomator2
npx playwright --version

Create playwright.config.ts:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  timeout: 30_000,
  retries: 1,
  use: {
    baseURL: process.env.BASE_URL ?? 'https://staging.<your-domain>',
    trace: 'on-first-retry',
    video: 'retain-on-failure',
  },
  reporter: [['html', { open: 'never' }], ['list']],
});

And tests/smoke.spec.ts:

import { test, expect } from '@playwright/test';

test('login page renders', async ({ page }) => {
  await page.goto('/login');
  await expect(page.getByRole('button', { name: /sign in/i })).toBeVisible();
});

For Android, create mobile/caps/android.json. Field names below are Appium's standard capability keys; replace the values:

{
  "platformName": "Android",
  "appium:automationName": "UiAutomator2",
  "appium:deviceName": "<device-name-printed-by-adb-devices>",
  "appium:app": "/absolute/path/to/app-debug.apk",
  "appium:noReset": false
}

Why: an AI generator inherits the quality of its starting point. If the baseline suite is red, every failure after generation is ambiguous — you cannot tell whether the model wrote a bad selector or the build is broken.

How to check:

BASE_URL=https://staging.<your-domain> npx playwright test tests/smoke.spec.ts

Expected on screen: 1 passed and an exit code of 0 (echo $?). For mobile, npx appium driver list --installed must list uiautomator2, and adb devices must show your emulator as device, not offline.

Step 2. Connect the AI tester: credentials, config, first generated spec

Illustration: Step 2. Connect the AI tester: credentials, config, first generated spec
Illustration: Step 2. Connect the AI tester: credentials, config, first generated spec

What to do: store the model credentials in .env, declare the target project in a config file, and add a small generator script that converts a plain-language scenario into a spec file.

.env (never commit it):

BASE_URL=https://staging.<your-domain>
AI_BASE_URL=https://<your-llm-endpoint>/v1
AI_API_KEY=<your-api-key>
AI_MODEL=<model-id-from-provider-docs>
VariableWhat to put thereExample format
`BASE_URL`URL of the app under test`https://staging.<your-domain>`
`AI_BASE_URL`OpenAI-compatible base URL, usually ending in `/v1``https://<your-llm-endpoint>/v1`
`AI_API_KEY`Secret key from the provider console`<your-api-key>`
`AI_MODEL`Exact model identifier — copy it from the provider docs, do not guess`<model-id-from-provider-docs>`

.ai-qe/config.yaml describes what the tester may touch. The key names below are a starting schema for your own wrapper — if your tool ships its own config format, map these values onto it and verify against its documentation.

target:
  web:
    base_url: https://staging.<your-domain>
    framework: playwright
  mobile:
    platform: android
    capabilities: mobile/caps/android.json
model:
  base_url_env: AI_BASE_URL
  api_key_env: AI_API_KEY
  model_env: AI_MODEL
generation:
  output_dir: tests/generated
  language: typescript
  max_retries: 2

scripts/gen-spec.mjs:

import { writeFileSync, mkdirSync } from 'node:fs';

const prompt = process.argv.slice(2).join(' ');
if (!prompt) throw new Error('Usage: node scripts/gen-spec.mjs "<scenario>"');

const res = await fetch(`${process.env.AI_BASE_URL}/chat/completions`, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    Authorization: `Bearer ${process.env.AI_API_KEY}`,
  },
  body: JSON.stringify({
    model: process.env.AI_MODEL,
    messages: [
      { role: 'system', content: 'You write Playwright TypeScript specs for the given baseURL. Use getByRole and getByTestId only. Output code only, no prose.' },
      { role: 'user', content: prompt },
    ],
  }),
});

if (!res.ok) throw new Error(`LLM HTTP ${res.status}: ${await res.text()}`);
const data = await res.json();
const code = data.choices[0].message.content.replace(/^```[a-z]*\n|```$/g, '');
mkdirSync('tests/generated', { recursive: true });
writeFileSync('tests/generated/checkout.spec.ts', code);
console.log('written tests/generated/checkout.spec.ts');

Run it:

set -a; source .env; set +a
node scripts/gen-spec.mjs "Log in as a standard user, add the first product to the cart, pay with the saved card, and assert the order confirmation heading is visible."

Expected output: written tests/generated/checkout.spec.ts.

Why: the tester only needs three things — a reachable endpoint, a key, and a constraint on output format. The system message pins the model to Playwright TypeScript with role/test-id locators, so the file lands directly in tests/ and runs without editing. set -a exports every assignment in .env, which is why the script sees the variables.

How to check:

head -20 tests/generated/checkout.spec.ts
node -e "fetch(process.env.AI_BASE_URL + '/models',{headers:{Authorization:'Bearer '+process.env.AI_API_KEY}}).then(r=>console.log(r.status))"

You should see an import { test, expect } from '@playwright/test'; line and a printed 200. If the file is empty or contains prose instead of code, the model answered outside the requested format — tighten the system message before debugging anything else. For browser-level automation details (contexts, headless flags, remote debugging), see How to Connect an AI Agent to Chrome.

Step 3. Run the generated suite locally and gate pull requests in CI

What to do: execute the generated spec, inspect the report, then let CI run the same command on every pull request.

npx playwright test tests/generated --reporter=list
npx playwright show-report

Expected on screen: 1 passed, and show-report opens http://localhost:9323 with a tree of tests; failed steps carry a screenshot, trace and video because of the trace/video settings from Step 1.

Add .github/workflows/e2e.yml:

name: e2e
on: [pull_request]
jobs:
  web:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test
        env:
          BASE_URL: ${{ secrets.BASE_URL }}
          AI_BASE_URL: ${{ secrets.AI_BASE_URL }}
          AI_API_KEY: ${{ secrets.AI_API_KEY }}
          AI_MODEL: ${{ secrets.AI_MODEL }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/

Why: the local run proves the generated code is syntactically and semantically valid against a live environment; the CI run proves it stays valid when someone else's branch changes the DOM. if: always() keeps the HTML report even when the job fails, which is where the failing selector is.

How to check: open the pull request, wait for the e2e / web job, and confirm a green check plus a downloadable playwright-report artifact. A red build with an artifact attached still counts as a working pipeline — download the report and read the failing step.

Result: generated specs live in tests/generated, hand-written baselines in tests/, and every pull request is gated by both.

Troubleshooting

Problem. The baseline cannot start a browser after a fresh clone

Problem. The generator script dies on the first call

echo "${AI_API_KEY:0:6}... (length ${#AI_API_KEY})"
set -a; source .env; set +a
node scripts/gen-spec.mjs "Open the pricing page and assert three plan cards are visible."

If the length is 0, the variable never loaded; if it is non-zero and the error persists, rotate the key in the provider console.

Problem. The generated spec times out on a valid page

Summary and next steps

You now have a Playwright baseline, an AI generator wired to an OpenAI-compatible endpoint through .env, and a CI gate that uploads the HTML report on every pull request. Next, extend the same pattern to mobile by pointing the generator at your Appium capabilities file, and read How to connect n8n to a local LLM with Ollama if the tests must run without sending code to an external provider.

FAQ

Do I need a paid LLM subscription to run an AI e2e tester?

No — you need any OpenAI-compatible endpoint. A hosted API key or a self-hosted model behind the same /chat/completions shape both work; only AI_BASE_URL, AI_API_KEY and AI_MODEL change.

Will AI-generated tests replace hand-written ones?

Keep the hand-written smoke tests as the trusted baseline and treat tests/generated as an expanding layer. Generated specs are cheapest to re-create, not to maintain by hand.

Does the same setup work for mobile apps?

Yes, with a different runner: Appium plus a platform driver (uiautomator2 for Android) and a capabilities JSON instead of a browser config. The generator and CI pattern stay identical.

How do I keep the API key out of the repository?

Store it in .env locally and in CI secrets, add .env to .gitignore, and pass only the secret names — never the values — into workflow YAML.

🤖 AI summary

Related guides

How to Connect an AI Agent to Chrome: Browser Automation Setup

Пошаговый гайд по подключению ИИ-агентов к Chrome через расширения и API для автоматизации действий в фоновых вкладках.

How to Access OpenAI Models via Google Cloud: Step-by-Step Setup

Пошаговая инструкция: как включить модели OpenAI в Google Cloud, настроить проект, получить ключи и вызвать их через Ver

How to connect n8n to a local LLM: step-by-step private automation setup with Ollama

Пошаговая инструкция: поднимаем n8n в Docker, разворачиваем локальную модель через Ollama и настраиваем OpenAI-совместим