Abstract: An adaptive vision based controller for controlling a robot arm comprises a camera, a segmenter for analyzing images from the camera, a tracker, sketcher and ranger responsive to information from the segmenter for creating a three dimensional segmented data list, a recognizer for receiving the data list and comparing data in the list against a database of plausible objects, and a planner interactive with the recognizer and responsive to task definitions for developing control outputs. The recognizer uses scenic information such as feature maps produced by the segmenter in conjunction with a knowledge base to construct a world model. The planner uses the world model and the task definitions to construct a plan in the form of a set of actions for accomplishing the defined task. By way of the control system, information about how the robot arm is actually performing a task can be compared with the desired task and the task can be updated if necessary.