How to Build an AI Agent for iPhone with ESP32: Control Your Phone via Neural Network
An ESP32 is a cheap Wi-Fi/Bluetooth microcontroller — a small chip board you can program over USB. An iPhone cannot be opened up and scripted from the inside, so the practical trick is the opposite of what most people expect: the ESP32 does not "hack" the phone. It becomes a tiny always-on agent that (a) talks to a neural network over Wi-Fi, and (b) hands the result to the iPhone through Apple's built-in **Shortcuts** app — Apple's visual automation tool that can run actions like sending a message, starting a timer or opening an app. This guide is for beginners with an ESP32 dev board (ESP32-WROOM-32, ESP32-S3) and an iPhone on iOS 16 or newer, and it walks you from a bare board to a working "say it → phone does it" loop. If you have never wired an LLM to real tools before, read How to turn a regular LLM chat into a full-fledged AI agent first — the "agent" idea is exactly what you are building in hardware.
Requirements and preparation
- ESP32 board with Wi-Fi and Bluetooth: ESP32-WROOM-32 or ESP32-S3-DevKitC. ESP32-S3 is the safer pick because it has more RAM.
- Arduino IDE 2.x with the **esp32** board package installed, or Espressif's ESP-IDF if you prefer C toolchains.
- USB data cable plus the right driver — CP210x or CH340, depending on your board's USB chip.
- An LLM API key from a provider that supports **tool calling** (also called function calling — the model answers with a structured "run this function" object instead of prose).
- An iPhone with iOS 16+, the built-in **Shortcuts** app, and the phone on the same 2.4 GHz Wi-Fi network as the ESP32. ESP32 radios do not see 5 GHz networks at all.
- A plain text editor for JSON. VS Code, Notepad++ or the Arduino IDE itself is fine.
Step-by-step instructions
1. Install the board support and pick the port. In Arduino IDE go to **Tools → Board → esp32 → ESP32 Dev Module**, then **Tools → Port** and select your device, for example `/dev/cu.usbserial-0001` on macOS or `COM5` on Windows.
**Example:** open **File → Examples → 01.Basics → Blink**, press Upload, and watch the on-board LED blink.
Expected result: the console prints `Leaving... Hard resetting via RTS pin...` and the LED pulses. Why this works: it proves the USB driver, the cable and the board selection are all correct before you add networking complexity.
2. Connect the board to Wi-Fi and print its IP address. Flash this sketch:
#include <WiFi.h>
#include <WebServer.h>
const char* SSID = "Home_2G"; // must be a 2.4 GHz network
const char* PASS = "your-password";
WebServer server(80);
void handleCommand() {
if (!server.hasArg("plain")) { server.send(400, "text/plain", "no body"); return; }
String body = server.arg("plain"); // {"tool":"Send_Message","args":{"text":"on my way"}}
Serial.println("LLM asked for: " + body);
server.send(200, "application/json", "{\"status\":\"ok\"}");
}
void setup() {
Serial.begin(115200);
WiFi.begin(SSID, PASS);
while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); }
Serial.println("\nIP: " + WiFi.localIP().toString());
server.on("/command", HTTP_POST, handleCommand);
server.begin();
}
void loop() { server.handleClient(); }Expected result: the Serial Monitor at **115200 baud** shows dots, then `IP: 192.168.1.42`.
Why this works: the board now has a stable address on your local network, so the phone can find it without any cloud service.
3. Test the endpoint from your computer before touching the phone.
curl -X POST http://192.168.1.42/command \
-H "Content-Type: application/json" \
-d '{"tool":"Send_Message","args":{"text":"on my way"}}'Expected result: `{"status":"ok"}` and the same JSON echoed in the Serial Monitor.
Why this works: you isolate the network layer from the AI layer. If this fails, the problem is Wi-Fi, not the model.
4. Describe your phone actions as a tool schema. This JSON is the contract between the model and the phone:
[
{"type":"function","function":{"name":"Send_Message","description":"Send a text message to a contact",
"parameters":{"type":"object","properties":{"to":{"type":"string"},"text":{"type":"string"}},
"required":["to","text"]}}},
{"type":"function","function":{"name":"Start_Timer","description":"Start a countdown timer",
"parameters":{"type":"object","properties":{"minutes":{"type":"integer"}},"required":["minutes"]}}}
]Expected result: the model can now answer with `{"name":"Start_Timer","arguments":"{\"minutes\":10}"}` instead of a sentence.
Why this works: tool calling turns free text into a machine-readable instruction, which is the core of any agent — the same pattern is described in the LLM-to-agent guide.
5. Create matching Shortcuts on the iPhone. Open **Shortcuts → +**, name the shortcut exactly `Send_Message` (no spaces, case-sensitive), add the action **Send Message**, and set the recipient and message text as variables.
**Example:** add a second shortcut called `Start_Timer` with the **Start Timer** action set to 10 minutes.
Expected result: tapping `Send_Message` in the Shortcuts app sends a real message. Why this works: the shortcut name is the only thing the ESP32 and the model need to reference — names act as function identifiers.
6. Let the phone pull commands from the ESP32. In a new shortcut, add **Get Contents of URL**, set the URL to `http://192.168.1.42/command`, method **POST**, and attach a **Get Current Weather**-style dummy or your own trigger, then add **Run Shortcut** with the name taken from the response. Wrap it in a **Repeat** loop with a short delay, or trigger it from **Automation → Personal Automation**.
Expected result: the phone polls the board and executes the returned tool name.
Why this works: iOS does not let external hardware execute Shortcuts directly, so the phone becomes the "hands" and the ESP32 becomes the "brain". If you want the same pull model on a desktop instead, the setup in How to Connect an AI Agent to Chrome shows how browser automation is wired the same way.
7. Add the model call to the ESP32. Use an HTTPS POST to your provider's chat completions endpoint, sending the user text plus the `tools` array from step 4. Parse the JSON response, extract the tool name and arguments, and forward them to the phone.
**Example:** the Serial Monitor should print `tool=Start_Timer args={"minutes":10}`.
Expected result: an end-to-end run where you type "start a 10 minute timer" into your test client and the iPhone timer actually starts.
Why this works: the board never needs to understand language — it only relays a decision the model already made.
8. Run this checklist before you call the build finished:
- [ ] Board shows a stable IP in the Serial Monitor.
- [ ] `curl` returns `{"status":"ok"}`.
- [ ] Every tool name in the JSON has an identically named Shortcut.
- [ ] The polling shortcut runs at least once per few seconds.
- [ ] The API key is stored in a constant, not pasted into a public repo.
Common problems and fixes
Symptom: the Serial Monitor prints dots forever and never shows an IP.The board is not joining the network. Fix: confirm the SSID is a 2.4 GHz network (ESP32 cannot see 5 GHz), re-type the password, and move the board closer to the router. Then re-check that `WiFi.status()` returns `WL_CONNECTED` before the server starts.
Symptom: the model replies with a normal sentence and no `tool_calls` field.Tool calling was not triggered. Fix: verify the `tools` array is actually in the request body, check that your chosen model supports function calling, and set `"tool_choice": "auto"`. Lowering the temperature also helps, because the model becomes less chatty.
Symptom: `curl` works, but nothing happens on the iPhone.The shortcut name does not match. Fix: compare names character by character — `Send_Message` is not the same as `send_message` or `Send Message`. Also check that the automation is enabled in **Shortcuts → Automation** and that the phone is unlocked, since iOS blocks some actions on a locked screen.
Symptom: upload fails with "Failed to connect to ESP32: Timed out waiting for packet header".Fix: close the Serial Monitor (it holds the port), then hold the **BOOT** button while the upload starts. If it still fails, reinstall the CP210x or CH340 driver.
FAQ
Can the ESP32 run the neural network by itself?
No. The board has far too little memory for a real language model. It sends the request to an API over Wi-Fi and only handles the small JSON result. Think of it as a remote control, not a brain.
Do I need to jailbreak the iPhone?
No. Everything here uses the built-in Shortcuts app and its Automation tab. Nothing in this setup requires unofficial system access.
Which ESP32 board should I buy?
Any board with Wi-Fi and Bluetooth works. ESP32-S3 is the better choice for beginners because it has more RAM and headroom for larger JSON buffers; ESP32-WROOM-32 is fine for a minimal build.
How do I add voice input instead of typing?
Add a voice step on the phone side rather than on the board. The general pipeline — speech to text, then text to the model, then an action — is described in How to Create a Voice AI Assistant on Your Computer, and the same stages apply when the final action happens on an iPhone.
Is it safe to keep the API key on the board?
Treat it as exposed. Anyone with physical access to the ESP32 can read flash memory, so use a key with a spending limit, rotate it regularly, and keep the whole setup on your local network instead of exposing port 80 to the internet.
🤖 AI summary
- Updated: 2026-10-12
- Topics: ESP32 iPhone AI agent, Control iPhone via AI, DIY AI robot
- Summary: Learn how to control a real iPhone using an inexpensive ESP32 module and an AI model: from AirPlay interfaces to touch emulation via USB HID.
- Key facts:
- An ESP32 is a cheap Wi-Fi/Bluetooth microcontroller — a small chip board you can program over USB.
- An iPhone cannot be opened up and scripted from the inside, so the practical trick is the opposite of what most people expect: the ESP32 does not "hack" the phone.
- It becomes a tiny always-on agent that (a) talks to a neural network over Wi-Fi, and (b) hands the result to the iPhone through Apple& x27;s built-in Shortcuts app — Apple& x27;s visual automation tool that can run actions like sending a message, starting a timer or opening an app.
- This guide is for beginners with an ESP32 dev board (ESP32-WROOM-32, ESP32-S3) and an iPhone on iOS 16 or newer, and it walks you from a bare board to a working "say it → phone does it" loop.
- If you have never wired an LLM to real tools before, read How to turn a regular LLM chat into a full-fledged AI agent first — the "agent" idea is exactly what you are building in hardware.
- Requirements and preparation Illustration: Requirements and preparation ESP32 board with Wi-Fi and Bluetooth: ESP32-WROOM-32 or ESP32-S3-DevKitC.