01
Vision-first grounding
The planner reasons from the current screenshot instead of requiring an accessibility tree, enabling interaction with text, icons, and visually defined controls.
Open source · v0.1.0 Alpha
An Android agent runtime that can see what it does.
PhoneAgent combines vision-language planning, deterministic Android capabilities, safe structured actions, post-action evidence, and bounded recovery into one auditable loop for real devices.
10:24
82%
PhoneAgentRuntime active
LIVE
USER TASK
Open Settings and find Battery.
01Observe
02Plan
03Execute
04Verify
05Recover / Replan
Open-source lineage
Inspired by Open-AutoGLM and recommended with Zhipu BigModel's autoglm-phone API.
Core capabilities
A small runtime with explicit trust boundaries.
PhoneAgent keeps each layer inspectable: observation, model planning, action parsing, Android execution, evidence verification, recovery, and trajectory persistence.
Safe action protocol
Model output is parsed through AST/JSON and validated against an action allow-list. Generated text is never executed as Python code.
do(action="Tap", element=[512, 438])
Deterministic app routing
Discover Launcher activities, resolve aliases, and prefer package/activity launch before falling back to visual navigation.
Evidence-aware verification
Separate command success, observable UI change, and deterministic semantic evidence instead of collapsing them into one success flag.
Bounded recovery
Failure episodes have explicit budgets. Non-idempotent actions such as Tap, Type, Swipe, and Back are never blindly replayed.
Auditable trajectories
Persist observations, model responses, actions, execution evidence, verification, recovery decisions, state transitions, and final results.
Architecture
Read the architecture document
Deterministic where possible. Visual where necessary.
PhoneAgent does not force every operation through a vision model. It combines Android system capabilities with a visual planning loop, then records evidence at each boundary.
01
Fast pathResolve and launch installed applications through package/activity state.
02
General pathUse the screenshot and task context to plan navigation inside an application.
03
Evidence pathVerify observable or deterministic effects before continuing or recovering.
User TaskNatural language goal
Agent RuntimeState machine · context · policy
App ResolverPackage / activity
↓
Deterministic launch
VLM PlannerScreenshot grounding
↓
Structured action
VerifierEvidence / recovery
↓
Bounded decision
ADB ExecutorTap · Type · Swipe · Launch
▯
Android DeviceObservation returns to runtime
↺
Quick Start
From clone to first run.
No frontend build step, no workflow framework, and no device-specific accessibility tree required.
Prepare the environment
Linux, Python 3.12+, Android Platform Tools, and an Android device with USB debugging enabled.
Configure the model
Use a vision-language model service compatible with the OpenAI Chat Completions API.
Run a task
Start with deterministic application launch, then move to multi-step visual tasks.
# Clone and install
$ git clone https://github.com/AuroraEchos/PhoneAgent.git
$ cd PhoneAgent
$ uv sync --extra dev
# Configure the VLM endpoint
$ cp .env.example .env
# Check the device and run
$ adb devices
$ uv run phoneagent "打开设置"
Runtime started · device connected · task initialized
Python 3.12+
Android + ADB
Vision-language model
Linux / Ubuntu
{
"type": "verification",
"step": 4,
"action": {
"name": "Launch",
"app": "设置"
},
"result": {
"command_success": true,
"observable_effect_verified": true,
"semantic_effect_verified": true,
"policy": "foreground_package_match"
}
}
Traceability
Every decision leaves evidence.
A run is not just terminal output. PhoneAgent records the state machine, model interaction, structured action, execution result, verification evidence, recovery policy, and final outcome.
- Reconstruct failed runs without relying on screenshots alone.
- Measure model latency, token usage, action frequency, and recovery behavior.
- Build reproducible evaluation and future experience-learning pipelines.
Roadmap
The runtime is small by design. The direction is not.
The first release freezes a reliable baseline. Future work can improve semantic correctness without obscuring the core loop.
Releasedv0.1.0
✓
Vision agent loopObserve, plan, execute, verify, recover.
✓
Android app resolverDiscovery, aliases, package launch.
✓
Safe action runtimeValidation, confirmation, bounded recovery.
Nextv0.x
→
Semantic verificationSubgoal-aware evidence and independent review.
→
Evaluation harnessTask suites, replay, metrics, and regression.
→
Provider abstractionMore local and hosted VLM backends.
Longer termResearch
◇
On-device runtimeAccessibility, MediaProjection, privacy controls.
◇
Experience learningReusable app knowledge from trajectories.
◇
Hybrid perceptionVision plus system UI signals where available.
Current boundaries
Honest about what v0.1.0 can prove.
PhoneAgent is an Alpha research and engineering runtime. It exposes uncertainty instead of converting weak evidence into confident claims.
Visual change is not semantic correctness
For coordinate actions, a changed screen does not prove that the intended UI target was selected.
Task completion is model-reported
The planner currently reports full completion through finish(...); there is no independent task judge yet.
Protected screens require takeover
DRM, FLAG_SECURE, passwords, verification codes, and payment flows may be unobservable or sensitive.
ADB is the current deployment boundary
A true on-device product requires an Android application, services, permissions, and privacy controls.
Build with PhoneAgent
A transparent baseline for Android agents.
Read the code, reproduce the loop, inspect the trajectory, and help improve reliable mobile interaction.