Abstract: The technology is directed to training a system to generate pick instructions. A teleoperator system may receive data corresponding to a robot attempting a picking task including picking an item of an identified product type from a container. The data may include imagery of an end effector of the robot after the attempted picking task. The teleoperator system may display the imagery on a display and an input indicating whether the picking task was successfully or unsuccessfully performed by the robot may be received. The data may be labeled based on the input and transmitted to a processor for training a learning algorithm for use in generating future pick instructions.