An Environment for User Interface Softbots

A user interface softbot is a software agent that controls an interactive system through the graphical user interface, relying on visual information from the system rather than an application programming interface or access to source code. Interface softbots have acted as autonomous agents in applications such as drawing and data recording, and the core vision processing algorithms have been incorporated into cognitive models for simple problem-solving tasks. Building interface softbots is still a time-consuming task, unfortunately, requiring experience with complex program components as well as the details of the visual interface. We have developed a prototype development environment that facilitates the development of interface softbots, streamlining the programming process and making it more accessible to new developers