# How to Set Up an AI E2E Tester for Your Project: Automated Tests for Web and Mobile Apps > AI Release · @ai_release1 · https://ai-release.org/guides/kak-podklyuchit-ii-testirovschik-e2e-k-proektu-nastroyk_en.html ## TL;DR - **Time to complete:** 45–60 minutes - **Difficulty:** Intermediate - **You will need:** Node.js LTS, `@playwright/test` (web E2E runner), Appium + a platform driver (mobile E2E runner), `adb` for Android devices, an OpenAI-compatible LLM endpoint with an API key, a git repository, and a CI runner (GitHub Actions example below) - **Result:** an AI tester that turns a plain-language scenario into a runnable Playwright spec, executes it locally and in CI, and stores the trace/video/report as artifacts ## Requirements and setup ![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-podklyuchit-ii-testirovschik-e2e-k-proektu-nastroyk-en-1.jpg?v=1794300614) - **OS:** macOS, Linux or Windows with WSL2. Windows native shells work, but every command below is written for `bash`. - **Node.js:** LTS installed. Check with `node -v`. The exact major version is not fixed by this guide — if your project pins one, use `.nvmrc`. - **Disk space:** at least 2 GB free. Playwright browser binaries and an Android emulator image are the two heavy items. - **Credentials:** an LLM API key plus its base URL from your provider. The endpoint must be OpenAI-compatible (`POST {base_url}/chat/completions`). Two setup walkthroughs for getting such a key: [How to Get a GPT-6 Astra API Key and Connect the Model](https://ai-release.org/guides/kak-poluchit-api-klyuch-gpt-6-astra-i-podklyuchit-model.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) and [How to Connect Kimi K3.1 Pro via API](https://ai-release.org/guides/kak-podklyuchit-kimi-k3-1-pro-cherez-api-klyuchi-tseny.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide). - **Test target:** a staging URL (`https://staging.`) and, for mobile, a debug `.apk` (Android) or `.app`/`.ipa` build. Pre-flight checklist: - [ ] `node -v` prints a version and `npm -v` works - [ ] `npx playwright --version` prints a version - [ ] `npx appium --version` prints a version - [ ] `adb devices` lists at least one device or emulator - [ ] `curl -s -o /dev/null -w "%{http_code}" $AI_BASE_URL/models -H "Authorization: Bearer $AI_API_KEY"` returns `200` - [ ] `.env` is listed in `.gitignore` ## Step 1. Freeze a green baseline before the AI touches anything ![Illustration: Step 1. Freeze a green baseline before the AI touches anything](https://ai-release.org/guides/img/kak-podklyuchit-ii-testirovschik-e2e-k-proektu-nastroyk-en-2.jpg?v=1794300620) **What to do:** create an `e2e` project, install the web runner and one mobile driver, write one smoke test by hand, and run it. ```bash mkdir -p e2e && cd e2e npm init -y npm i -D @playwright/test appium npx playwright install --with-deps npx appium driver install uiautomator2 npx playwright --version ``` Create `playwright.config.ts`: ```ts import { defineConfig } from '@playwright/test'; export default defineConfig({ testDir: './tests', timeout: 30_000, retries: 1, use: { baseURL: process.env.BASE_URL ?? 'https://staging.', trace: 'on-first-retry', video: 'retain-on-failure', }, reporter: [['html', { open: 'never' }], ['list']], }); ``` And `tests/smoke.spec.ts`: ```ts import { test, expect } from '@playwright/test'; test('login page renders', async ({ page }) => { await page.goto('/login'); await expect(page.getByRole('button', { name: /sign in/i })).toBeVisible(); }); ``` For Android, create `mobile/caps/android.json`. Field names below are Appium's standard capability keys; replace the values: ```json { "platformName": "Android", "appium:automationName": "UiAutomator2", "appium:deviceName": "", "appium:app": "/absolute/path/to/app-debug.apk", "appium:noReset": false } ``` **Why:** an AI generator inherits the quality of its starting point. If the baseline suite is red, every failure after generation is ambiguous — you cannot tell whether the model wrote a bad selector or the build is broken. **How to check:** ```bash BASE_URL=https://staging. npx playwright test tests/smoke.spec.ts ``` Expected on screen: `1 passed` and an exit code of `0` (`echo $?`). For mobile, `npx appium driver list --installed` must list `uiautomator2`, and `adb devices` must show your emulator as `device`, not `offline`. ## Step 2. Connect the AI tester: credentials, config, first generated spec ![Illustration: Step 2. Connect the AI tester: credentials, config, first generated spec](https://ai-release.org/guides/img/kak-podklyuchit-ii-testirovschik-e2e-k-proektu-nastroyk-en-3.jpg?v=1794300628) **What to do:** store the model credentials in `.env`, declare the target project in a config file, and add a small generator script that converts a plain-language scenario into a spec file. `.env` (never commit it): ```bash BASE_URL=https://staging. AI_BASE_URL=https:///v1 AI_API_KEY= AI_MODEL= ``` | Variable | What to put there | Example format | |---|---|---| | `BASE_URL` | URL of the app under test | `https://staging.` | | `AI_BASE_URL` | OpenAI-compatible base URL, usually ending in `/v1` | `https:///v1` | | `AI_API_KEY` | Secret key from the provider console | `` | | `AI_MODEL` | Exact model identifier — copy it from the provider docs, do not guess | `` | `.ai-qe/config.yaml` describes what the tester may touch. The key names below are a starting schema for your own wrapper — if your tool ships its own config format, map these values onto it and verify against its documentation. ```yaml target: web: base_url: https://staging. framework: playwright mobile: platform: android capabilities: mobile/caps/android.json model: base_url_env: AI_BASE_URL api_key_env: AI_API_KEY model_env: AI_MODEL generation: output_dir: tests/generated language: typescript max_retries: 2 ``` `scripts/gen-spec.mjs`: ```js import { writeFileSync, mkdirSync } from 'node:fs'; const prompt = process.argv.slice(2).join(' '); if (!prompt) throw new Error('Usage: node scripts/gen-spec.mjs ""'); const res = await fetch(`${process.env.AI_BASE_URL}/chat/completions`, { method: 'POST', headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${process.env.AI_API_KEY}`, }, body: JSON.stringify({ model: process.env.AI_MODEL, messages: [ { role: 'system', content: 'You write Playwright TypeScript specs for the given baseURL. Use getByRole and getByTestId only. Output code only, no prose.' }, { role: 'user', content: prompt }, ], }), }); if (!res.ok) throw new Error(`LLM HTTP ${res.status}: ${await res.text()}`); const data = await res.json(); const code = data.choices[0].message.content.replace(/^```[a-z]*\n|```$/g, ''); mkdirSync('tests/generated', { recursive: true }); writeFileSync('tests/generated/checkout.spec.ts', code); console.log('written tests/generated/checkout.spec.ts'); ``` Run it: ```bash set -a; source .env; set +a node scripts/gen-spec.mjs "Log in as a standard user, add the first product to the cart, pay with the saved card, and assert the order confirmation heading is visible." ``` Expected output: `written tests/generated/checkout.spec.ts`. **Why:** the tester only needs three things — a reachable endpoint, a key, and a constraint on output format. The system message pins the model to Playwright TypeScript with role/test-id locators, so the file lands directly in `tests/` and runs without editing. `set -a` exports every assignment in `.env`, which is why the script sees the variables. **How to check:** ```bash head -20 tests/generated/checkout.spec.ts node -e "fetch(process.env.AI_BASE_URL + '/models',{headers:{Authorization:'Bearer '+process.env.AI_API_KEY}}).then(r=>console.log(r.status))" ``` You should see an `import { test, expect } from '@playwright/test';` line and a printed `200`. If the file is empty or contains prose instead of code, the model answered outside the requested format — tighten the system message before debugging anything else. For browser-level automation details (contexts, headless flags, remote debugging), see [How to Connect an AI Agent to Chrome](https://ai-release.org/guides/kak-podklyuchit-ii-agenta-k-chrome-nastroyka-avtomatiza.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide). ## Step 3. Run the generated suite locally and gate pull requests in CI **What to do:** execute the generated spec, inspect the report, then let CI run the same command on every pull request. ```bash npx playwright test tests/generated --reporter=list npx playwright show-report ``` Expected on screen: `1 passed`, and `show-report` opens `http://localhost:9323` with a tree of tests; failed steps carry a screenshot, trace and video because of the `trace`/`video` settings from Step 1. Add `.github/workflows/e2e.yml`: ```yaml name: e2e on: [pull_request] jobs: web: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: lts/* - run: npm ci - run: npx playwright install --with-deps - run: npx playwright test env: BASE_URL: ${{ secrets.BASE_URL }} AI_BASE_URL: ${{ secrets.AI_BASE_URL }} AI_API_KEY: ${{ secrets.AI_API_KEY }} AI_MODEL: ${{ secrets.AI_MODEL }} - uses: actions/upload-artifact@v4 if: always() with: name: playwright-report path: playwright-report/ ``` **Why:** the local run proves the generated code is syntactically and semantically valid against a live environment; the CI run proves it stays valid when someone else's branch changes the DOM. `if: always()` keeps the HTML report even when the job fails, which is where the failing selector is. **How to check:** open the pull request, wait for the `e2e / web` job, and confirm a green check plus a downloadable `playwright-report` artifact. A red build with an artifact attached still counts as a working pipeline — download the report and read the failing step. **Result:** generated specs live in `tests/generated`, hand-written baselines in `tests/`, and every pull request is gated by both. ## Troubleshooting **Problem. The baseline cannot start a browser after a fresh clone** - Error: `browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium-/chrome-linux/chrome` - Cause: the `@playwright/test` package is installed, but the browser binaries for that build are not. - Solution: run `npx playwright install --with-deps` in the same shell/user context, and make sure the CI step above runs before `npx playwright test`. Verify with `npx playwright install --list`. **Problem. The generator script dies on the first call** - Error: `Error: LLM HTTP 401: {"error":{"message":"Incorrect API key provided","type":"invalid_request_error","code":"invalid_api_key"}}` - Cause: `AI_API_KEY` is not exported into the current shell — editing `.env` without `set -a; source .env; set +a` leaves the variable empty, so the request goes out with `Bearer undefined`. A second common cause is a whitespace or quote character copied with the key. - Solution: check the value is visible and looks right, then re-run: ```bash echo "${AI_API_KEY:0:6}... (length ${#AI_API_KEY})" set -a; source .env; set +a node scripts/gen-spec.mjs "Open the pricing page and assert three plan cards are visible." ``` If the length is `0`, the variable never loaded; if it is non-zero and the error persists, rotate the key in the provider console. **Problem. The generated spec times out on a valid page** - Error: `TimeoutError: locator.click: Timeout 30000ms exceeded.` - Cause: the model guessed a locator from marketing copy (for example a button label) that differs in the staging build, or the element sits inside an iframe. - Solution: regenerate with an explicit constraint instead of hand-patching the file — add `use only data-testid attributes` to the system message, ask the frontend team to add `data-testid="checkout-submit"` to the button, then re-run `node scripts/gen-spec.mjs ""`. For mobile sessions, `Error: Could not find a connected Android device` means `adb devices` returned an empty list — start the emulator and confirm it shows `device` before rerunning. ## Summary and next steps You now have a Playwright baseline, an AI generator wired to an OpenAI-compatible endpoint through `.env`, and a CI gate that uploads the HTML report on every pull request. Next, extend the same pattern to mobile by pointing the generator at your Appium capabilities file, and read [How to connect n8n to a local LLM with Ollama](https://ai-release.org/guides/kak-podklyuchit-n8n-k-lokalnoy-yazykovoy-modeli-poshago.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide) if the tests must run without sending code to an external provider. ## FAQ **Do I need a paid LLM subscription to run an AI e2e tester?** No — you need any OpenAI-compatible endpoint. A hosted API key or a self-hosted model behind the same `/chat/completions` shape both work; only `AI_BASE_URL`, `AI_API_KEY` and `AI_MODEL` change. **Will AI-generated tests replace hand-written ones?** Keep the hand-written smoke tests as the trusted baseline and treat `tests/generated` as an expanding layer. Generated specs are cheapest to re-create, not to maintain by hand. **Does the same setup work for mobile apps?** Yes, with a different runner: Appium plus a platform driver (`uiautomator2` for Android) and a capabilities JSON instead of a browser config. The generator and CI pattern stay identical. **How do I keep the API key out of the repository?** Store it in `.env` locally and in CI secrets, add `.env` to `.gitignore`, and pass only the secret names — never the values — into workflow YAML.