Madhumitha Venkatesan, Shicheng Wen, Jiajing Guo +3 more
Agents that drive software by looking at the screen and clicking, the way a person would, tend to wander off track when a surprise dialog pops up or the layout shifts. Part of why they are hard to trust is that their plan lives entirely in their head. You can watch them act but not see what they intend, let alone correct it before it goes wrong.
Plover pulls that plan into the open. Instead of a black box, you get a written, standing to-do list that the system follows and that you can read, edit, and steer with plain language or by pointing at the screenshot. A planner proposes and revises steps while an executor carries them out, and when you fix something, the work already done is kept rather than thrown away. The authors report that many agent failures turn out to be repairable this way, once the plan is visible and corrections stay local.
This is based on the abstract, so the paper holds the study design and how well the repairs worked.
Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We…
Evaluating covariate balance for long time horizon Markov decision processes
arXiv (cs.AI) · July 16, 2026BrainPilot: Automating Brain Discovery with Agentic Research
arXiv (cs.LG) · July 16, 2026An Introduction to Sparse Identification of Nonlinear Dynamics for Engineering Applications
arXiv (cs.CV) · July 16, 2026ESAR: Event-Based Synthetic Aperture Reconstruction