EliotAndres/iphone-use-app: iPhone app where an LLM controls the device

EliotAndres/iphone-use-app is an open-source iOS application that allows a vision-enabled large language model to directly control an iPhone from the device itself without requiring a Mac at runtime. The setup requires a one-time USB pairing file created via a Mac, after which the app utilizes its own loopback VPN and CoreDevice capabilities to capture screenshots and execute touches, swipes, and typing actions. Evaluation benchmarks indicate model performance and costs vary depending on the chosen endpoint, with default configurations using Claude Haiku achieving high accuracy. Open article →